Retry Amplification Calculator

When several layers each retry failed calls, the worst-case load on the bottom dependency is the product of (1 + retries) across layers. A gateway with 2 retries, a service with 3 and a database client with 3 can turn one user request into 3 × 4 × 4 = 48 calls when the dependency is down, so 1,000 RPS of users becomes 48,000 RPS of attempts. The calculator below shows the worst case, the expected load at a given failure rate, and how much a retry budget caps it, layer by layer.

The amplification formulas

Number the layers from the top (closest to the user) to the bottom (the one calling the dependency that fails). Layer i makes up to 1 + ri attempts for each call it receives. Every attempt that reaches a lower layer can itself be retried there, so attempts multiply:

worst_case      = Π (1 + r_i)                       over all layers
bottom_RPS      = user_RPS × worst_case

expected, with independent failures (p = failure rate per attempt at the bottom):
  bottom layer:  E_n = (1 − p^(r_n + 1)) / (1 − p)      calls per invocation
                 f_n = p^(r_n + 1)                       chance the layer gives up
  layer above:   E_i = (1 − f_(i+1)^(r_i + 1)) / (1 − f_(i+1)),  f_i = f_(i+1)^(r_i + 1)
  amplification  = Π E_i

with a retry budget b per layer (retries ≤ b × requests):
  worst_case_b   = Π min(1 + r_i, 1 + b)
  expected_b     ≈ Π min(E_i, 1 + b)

The expected-value formula is a geometric series: a layer stops at the first success, so with failure probability q per call it makes 1 + q + q² + … + q^r calls on average. Multiplying the layers is valid because each layer's attempts are independent repetitions of the layer below (Wald's identity). The budget version is an approximation, since a budget also changes how often each layer gives up.

Worked example

The calculator's defaults: gateway with 2 retries, order service with 3, database client with 3, 1,000 user RPS.

1. Worst case: (1 + 2) × (1 + 3) × (1 + 3) = 3 × 4 × 4 = 48. If the database is refusing every call, it receives 48,000 attempts per second while users still get errors.
2. Expected at 50% failures: the DB client makes (1 − 0.5⁴) ÷ 0.5 = 1.875 calls on average and gives up 0.5⁴ = 6.25% of the time. The service above therefore sees 6.25% failures and makes (1 − 0.0625⁴) ÷ 0.9375 ≈ 1.067 calls. The gateway sees almost no failures. Total ≈ 1.875 × 1.067 × 1.000 = 2.0×.
3. With a 10% budget: each layer can add at most 10% extra calls, so the worst case is 1.1³ = 1.33× instead of 48×.

The gap between 2× and 48× is the lesson. When failures are random and independent, retries at the bottom layer absorb them and the upper layers' retries rarely fire. When the dependency is hard down or overloaded, every attempt fails, and every layer's retries fire on every request.

Why real incidents look like the worst case

The independent-failure model is optimistic for one reason: overload makes failures correlated. A database at 100% CPU times out on most calls, not on a random few. Retries then add load to the component that is already saturated, which raises its failure rate further, which triggers more retries. That feedback loop is a retry storm, and it can keep a system down after the original trigger has gone. The retry storm troubleshooting guide covers how it shows up in metrics.

Use the sensitivity table in the calculator to see where your configuration turns dangerous. With the defaults, amplification stays near 2× until the bottom failure rate passes about 50%, then climbs steeply toward 48× as it approaches 100%.

Retry budgets and other controls

Backoff deserves a specific warning, because it is often treated as the fix. Backoff changes when retries happen, not how many happen. Three retries with jittered backoff still multiply load by up to four during a long outage; they just arrive more smoothly.

Timeouts interact with retries too. If an upper layer times out and retries while the lower layer is still working on the first attempt, the system does the work twice. Keep timeouts shorter as you go down the stack, so the inner layer gives up before the outer layer retries.

Load testing retry behavior

A normal load test against a healthy system will never show retry amplification, because nothing fails. To see it:

1. Measure attempts, not just requests. Export per-layer request counts, including retries, from each service's metrics. The ratio of downstream to upstream RPS is your observed amplification.
2. Inject a failure during a steady load. Hold a constant arrival rate, then degrade the dependency: reduce database capacity, add latency, or make a stub return 503s. Watch the dependency's incoming RPS. In LoadTester you can add a deploy-style annotation at that moment so it lines up on the charts.
3. Push past the knee. A stress test that runs above capacity shows whether the system recovers when load drops back, or whether retries keep it saturated.
4. Hold the load constant. Use an arrival-rate (open) model. A closed VU model slows down as errors and latency rise, hiding exactly the amplification you want to see; see open vs closed workload models.

Retries also inflate the error budget burn when the dependency is down; the error budget calculator shows how quickly that adds up.

Checklist

1. List every layer that retries, including SDKs, service meshes, gateways and client libraries with default retries.
2. Enter them here and read the worst-case amplification.
3. If it is above about 3×, keep retries at one layer and fail fast elsewhere.
4. Add a retry budget or circuit breaker at each remaining retrying layer.
5. Make timeouts shorter at inner layers than at outer ones.
6. Run a constant-rate test with an injected dependency failure and measure downstream RPS.