Coordinated Omission in Load Testing

Coordinated omission is a measurement error in which a load generator stops sending requests while the server is slow, so the requests that would have suffered the worst delays are never sent and never recorded. The result is latency percentiles that look healthy during a stall: a 3-second freeze can leave p99 at 1 ms. The term comes from Gil Tene, and the fix is to measure against a schedule (open model, constant arrival rate) or correct the histogram for the intended send times.

What coordinated omission is

Most load generators run a closed loop: send a request, wait for the response, record its latency, send the next one. That loop "coordinates" with the system under test. When the server stalls, the generator stalls with it, and it omits all the requests it would have sent during the stall. Those omitted requests are exactly the ones that real users would have experienced as slow.

Gil Tene (Azul Systems, author of HdrHistogram and wrk2) named and popularized the problem, most widely through his talk "How NOT to Measure Latency". His point is that the bias is not a small rounding error. It systematically removes the tail, which is the part of the distribution that p99 and p99.9 are meant to describe.

Two things have to be true for it to bite:

• The generator waits for a response before sending the next request (closed model, or a fixed number of connections).
• The system has periods where it is much slower than usual: GC pauses, lock contention, failovers, cold caches, a slow dependency, autoscaling lag.

A worked example with numbers

Illustrative setup: one connection, a test plan of one request every 10 ms (100 RPS) for 60 seconds. The server normally answers in 1 ms. At t = 30 s it freezes completely for 3 seconds, then recovers.

What the closed-loop generator records

Good periods: 57 s × 100 req/s = 5,700 samples at 1 ms
Stall:        1 sample at 3,000 ms
Total:        5,701 samples

p50  = 1 ms
p99  = 1 ms
p99.9 = 1 ms
max  = 3,000 ms
mean ≈ (5,700 × 1 + 3,000) / 5,701 ≈ 1.5 ms

One slow sample out of 5,701 is 0.018% of the data. It doesn't move any percentile you would put on a dashboard. The report says p99 is 1 ms during a test where the service was completely unavailable for 5% of the time.

What a user population would have experienced

Requests scheduled every 10 ms during the stall would have waited 3,000 ms, 2,990 ms, 2,980 ms, and so on down to 10 ms. Measured from their intended send time:

Good periods: 5,700 samples at 1 ms
Stall:        300 samples, evenly spread from 10 ms to 3,000 ms
Total:        6,000 samples

p50  = 1 ms
p99  ≈ 2,400 ms   (60 worst samples: 3,000 down to 2,410 ms)
p99.9 ≈ 2,950 ms
max  = 3,000 ms
mean ≈ (5,700 × 1 + 451,500) / 6,000 ≈ 76 ms

Same server, same stall, a p99 that differs by more than three orders of magnitude. Only the max is right in both columns, which is why Gil Tene recommends always looking at the max and the full percentile distribution, not just a couple of percentiles.

Why it happens in real load tests

The single-connection example is extreme, but the same mechanism applies with 100 or 1,000 VUs. Each VU is a closed loop. During a stall, every VU is stuck on one request, and the test sends at most one request per VU into the stall. With 100 VUs and a 3 s stall, the generator records 100 slow samples where the target rate would have produced thousands. Pauses that are short relative to response time cause smaller errors; pauses that are long relative to the request interval cause large ones.

Common sources of stalls that trigger it:

• Stop-the-world GC pauses in the JVM, .NET or Go (Go's are short but not zero).
• Database lock waits, vacuum or checkpoint spikes, connection pool exhaustion.
• Node.js event loop blocking on a synchronous call or a large JSON parse.
• Autoscaling, pod rescheduling, or a load balancer draining a target.
• The load generator itself pausing (its own GC, CPU starvation). This is coordinated omission on the client side and it hides problems just as well.

How to detect it in your results

• RPS dips while VUs are flat. In a VU-mode test, a dip in throughput is the generator backing off. Every dip is a window where samples went missing. See VUs vs RPS for why.
• Max is far above p99.9. If p99.9 is 40 ms and max is 3,000 ms, check how many samples you would expect during a 3 s window at your target rate. If it's hundreds, the percentiles are hiding them.
• Little's Law doesn't balance over the stall. Concurrency stays at VUs, response time spikes, and throughput collapses. Little's Law is satisfied, but by lowering λ, which is the omission.
• Server-side metrics disagree. The server's queue depth or in-flight request gauge doesn't grow during the stall, because nothing new arrived. Production traffic would have grown it.

How to avoid or correct it

1. Use an open model with a constant arrival rate

Schedule requests on a clock, independent of responses, and measure latency from the scheduled send time. If the server stalls, the generator keeps launching requests on schedule, each one on its own connection or VU, and each records its full wait. This is the open workload model.

The historical example is wrk2, Gil Tene's fork of wrk. It requires a rate (-R) and records latency from when each request should have been sent according to the constant-throughput schedule, so stalls in either the server or the client's own loop show up in the histogram:

# wrk2 builds a binary named wrk; -R (requests/sec) is required
wrk -t4 -c200 -d60s -R2000 --latency https://api.example.com/health

Vegeta (-rate) also generates load at a fixed rate. In k6, use the constant-arrival-rate or ramping-arrival-rate executor with enough maxVUs. Note the caveat: k6 measures http_req_duration from the actual send, so the arrival-rate model only protects you while VUs are available. When k6 can't start an iteration on time it counts it in dropped_iterations instead of recording it as slow. Treat any dropped iterations as a signal that the measurement is no longer complete.

2. Correct the histogram

If you must use a closed loop, HdrHistogram can back-fill the missing samples. In Java:

// expected interval between requests: 10 ms (in the histogram's units, here microseconds)
histogram.recordValueWithExpectedInterval(latencyMicros, 10_000);

// or correct an already-recorded histogram
Histogram corrected = raw.copyCorrectedForCoordinatedOmission(10_000);

When a recorded value exceeds the expected interval, HdrHistogram adds synthetic samples at value − interval, value − 2×interval, and so on, approximating the requests that should have been sent. It's a useful correction, but it relies on you knowing the intended interval, and it can't recreate the extra queueing those requests would have caused on the server. Avoiding the omission is better than correcting it.

3. Keep the generator healthy

A saturated or pausing generator causes the same effect from the client side. Keep generator CPU well below saturation, avoid GC-heavy scripting in the hot path, and run distributed generators when one machine isn't enough; see distributed load testing.

What it means when you read your results

• A closed-model test can tell you average throughput and latency under a fixed concurrency. Its p99 and p99.9 are optimistic whenever the system had pauses, and the error is largest exactly when the tail matters.
• Compare tools and runs only if they use the same model. A wrk run and a wrk2 run at "the same load" can disagree by orders of magnitude at p99 for a system with pauses, and both are reporting correctly what they measured.
• Before signing off on an SLO like "p99 under 500 ms", confirm the test was rate-based and that no iterations were dropped. The difference between percentiles is explained in p95 vs p99 latency.
• Look at latency over time, not just the summary. A one-line spike to 3 s in the time series is far easier to see than its effect on a percentile computed over the whole run.

LoadTester's exact RPS mode schedules requests at the configured arrival rate rather than waiting for each VU's previous response, and the live chart shows p50/p95/p99 over time alongside active VUs, so a stall shows up as both a latency spike and a jump in VUs.

Checklist

1. Run latency-critical tests in an arrival-rate (open) mode, not a fixed-VU loop.
2. Set the VU or connection ceiling to target rate × worst-case latency.
3. Check for dropped or delayed iterations before reading percentiles.
4. Always report max next to p99 and p99.9, and look at latency over time.
5. If you must use a closed loop, correct with HdrHistogram's expected interval and say so in the report.
6. Watch generator CPU and GC; a pausing client omits samples too.