Retry Storms Under Load

A retry storm happens when a slowdown causes timeouts, every layer retries, and the retries multiply the load on the component that was already struggling, so it fails harder and triggers more retries. With 3 attempts per call at 3 layers, one user request can become up to 3 × 3 × 3 = 27 calls at the bottom of the stack. Stop it by retrying at one layer only, using exponential backoff with jitter, capping retries with a budget (for example 10-20% of normal traffic), adding circuit breakers, and making timeouts shrink as you go deeper. A load test with injected failure shows the amplification directly: compare RPS at the database with RPS at the front door.

What the symptom looks like

From outside, a retry storm looks like an outage that is bigger and lasts longer than its cause. Inside, the clues are a mismatch between traffic layers and a cluster of timeout messages:

# Edge RPS flat, downstream RPS jumps
ingress: 1,000 RPS   orders-svc: 1,000 -> 2,900 RPS   postgres: 3,100 -> 8,700 queries/s

# Client libraries logging retries
WARN  Retrying request to orders-svc (attempt 2/3): java.net.SocketTimeoutException: Read timed out
urllib3.connectionpool WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(...)'
grpc: code = Unavailable desc = upstream connect error or disconnect/reset before headers

# Envoy / Istio
upstream_rq_retry: climbing
upstream_rq_retry_overflow: 0      (no budget in place, so nothing is capping retries)

# And after the trigger is gone, errors persist

The signature that matters most is in the last line: the database recovered (or the slow deploy rolled back), yet error rates stay high because the retry load alone exceeds capacity. That self-sustaining overload is what distinguishes a storm from an ordinary failure.

The amplification math

If each layer makes up to A attempts per call (1 original + retries) and there are L layers that all retry, the worst-case calls at the bottom per user request are AL. In practice not every attempt fails, but during an outage most do, so the worst case is what you get exactly when you can least afford it.

Defaults stack up quietly. Many SDKs and HTTP client libraries retry by default (AWS SDKs in standard mode make up to 3 attempts), service meshes can add their own retry policies, and application code often wraps calls in yet another retry loop. Nobody chose 27x; it emerged. The retry amplification calculator works this out for your own layer counts and failure rates.

Most likely causes, ranked

1. Retries at several layers. The multiplication above. The most common and the most damaging.
2. Timeouts that do not shrink with depth. If the gateway times out at 5 s and the service behind it at 10 s, the gateway retries while the first attempt is still running. The service now does the work twice and the first result is thrown away. Inner timeouts must be shorter than outer ones (the 504 timeout ladder covers this ordering).
3. No backoff or no jitter. Immediate retries arrive while the dependency is still overloaded. Exponential backoff without jitter keeps clients that failed together retrying together, in synchronized waves.
4. Retrying non-retryable errors. Retrying 400s, 401s or 429s without honoring Retry-After, or retrying non-idempotent POSTs, adds load without any chance of success.
5. Thundering herd after recovery. When a dependency comes back, every client with a queued retry, every reconnecting websocket and every expired cache entry hits it in the same second.

How to confirm a retry storm

• Amplification ratio: divide RPS at each layer by RPS at the edge. In normal operation it reflects your fan-out (say 1.0 or 2.0). During the incident, a ratio that jumps while edge traffic is flat is retries.
• Attempt counters: Envoy exposes upstream_rq_retry, upstream_rq_retry_success and upstream_rq_retry_overflow; Resilience4j publishes a resilience4j.retry.calls metric tagged by kind; for clients without metrics, count retry log lines per minute.
• Duplicate work: the same request ID or idempotency key appearing several times in the downstream log.
• Persistence: errors stay high after the trigger (deploy rolled back, failover finished) and drop only when traffic is reduced or retries are disabled.

Fixes, with config

Retry at one layer, and only what is safe

Pick the layer closest to the user that can still make a sensible decision, usually the edge or the calling service. Disable automatic retries in the layers below it. Retry only idempotent operations, or POSTs carrying an idempotency key, and only on connection failures, 502/503/504 and explicit retryable gRPC codes. Honor Retry-After on 429 and 503.

Exponential backoff with full jitter

import random, time

def call_with_retry(fn, attempts=3, base=0.1, cap=2.0):
    for attempt in range(attempts):
        try:
            return fn()
        except RetryableError:
            if attempt == attempts - 1:
                raise
            # full jitter: random wait between 0 and the exponential cap
            time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))

Full jitter (random(0, min(cap, base × 2attempt))) spreads retries from many clients across the whole backoff window instead of synchronizing them.

Retry budgets

A retry budget allows retries only while they are a small fraction of normal requests, so they help with scattered failures but cannot multiply load during an outage. Envoy supports this per cluster; when retry_budget is set, it replaces max_retries as the limit on concurrent retries:

circuit_breakers:
  thresholds:
  - priority: DEFAULT
    retry_budget:
      budget_percent: { value: 20.0 }   # Envoy's default when the budget is enabled
      min_retry_concurrency: 3          # default

In application code, a token bucket that earns 0.1 tokens per successful request and spends one per retry implements a 10% budget in a few lines.

Circuit breakers

# Resilience4j (Spring Boot)
resilience4j.circuitbreaker.instances.orders.failure-rate-threshold=50
resilience4j.circuitbreaker.instances.orders.sliding-window-size=50
resilience4j.circuitbreaker.instances.orders.wait-duration-in-open-state=10s
resilience4j.circuitbreaker.instances.orders.permitted-number-of-calls-in-half-open-state=5

When the breaker is open, callers fail fast or serve a fallback, so the failing dependency gets room to recover. The half-open state sends a few probe calls before reopening the floodgates, which also limits the thundering herd.

Deadlines and load shedding

Propagate a deadline (gRPC does this natively; for HTTP pass a header with the remaining budget) so deeper layers stop work the caller has already given up on. On the server side, reject early with 503 when queues are full, so retries hit a fast, cheap rejection instead of consuming worker time. See rate limit testing for validating that path.

How to surface a retry storm in a load test

A normal load test against a healthy system will never show amplification, because nothing fails. You have to inject the failure:

1. Run a steady arrival-rate test at normal peak, for example 300 RPS, so the edge rate is fixed and any change downstream is amplification, not more users.
2. After a few minutes, degrade one dependency: add latency with tc qdisc add dev eth0 root netem delay 800ms on the database host or a Toxiproxy toxic, make a downstream return 503 for a fraction of calls, or scale it down.
3. Watch per-layer RPS, retry counters and connection pool usage. Compute the amplification ratio.
4. Remove the fault and time how long until error rate and downstream RPS return to baseline. Seconds is healthy; minutes, or never without intervention, is the storm.

Keep the load generator itself from retrying, so the edge rate stays constant and failures are counted honestly. In LoadTester, an error-rate threshold with auto-stop (for example errors < 5%) keeps a failure-injection test from running a broken system for the full duration; mark the fault injection and removal on the timeline with the annotations endpoint so the recovery time is easy to read. For multi-service setups, microservices load testing covers where to place the measurement points.

Diagnosis table

How to re-test and confirm

Repeat the failure-injection test with the same edge RPS and the same fault. Three numbers should change: the peak amplification ratio at the bottom layer should drop to roughly your normal fan-out plus the budget (for example 1.1-1.3x instead of 9x+), the error rate during the fault should be bounded and steady instead of growing, and recovery after removing the fault should take seconds. Also run the plain steady-load test again to check that the new timeouts and breaker thresholds do not trip under normal traffic. If you track reliability against an SLO, the error budget calculator shows how much a storm of a given length costs.