Why Latency Increases Under Load

Latency increases under load because requests start waiting in queues for a busy resource: CPU, worker threads, database connections, locks or the garbage collector. Waiting time grows non-linearly with utilization. In the simplest queueing model, response time is the service time divided by (1 − utilization), so a resource at 50% busy doubles latency, at 80% multiplies it by 5 and at 90% by 10. That is why latency looks flat until roughly 70-80% utilization of the bottleneck and then rises steeply, and why the way to find your limit is a step-ramp test that locates that knee.

Why latency rises even when nothing is "slow"

Every request's latency is service time (the work itself) plus wait time (time spent queued for something). At low load, wait time is close to zero: when a request arrives, a worker is free. As load rises, the chance that the resource is already busy goes up, and requests start waiting behind each other.

The key point is that arrivals are not evenly spaced. Even at an average of 50% utilization, requests arrive in clumps: three in the same millisecond, then nothing for a while. The clump queues. The higher the average utilization, the less idle time there is to drain those clumps before the next one arrives, so queues get longer and last longer. Nothing about the code changed, yet p95 doubled.

This also explains a confusing pattern in test results: CPU at 70% "looks fine", yet latency is already noticeably worse than at 30%. The resource is not at capacity, but it is busy often enough that requests regularly wait.

The math: utilization vs. response time

The M/M/1 queue (random arrivals, one server, exponentially distributed service times) is the simplest model, and while real systems differ in detail, its shape is the one you will see in practice:

R = S / (1 − ρ)

where R is mean response time, S is mean service time and ρ (rho) is utilization: arrival rate divided by maximum service rate. Worked example with a 20 ms service time:

The p99 column comes from a property of this model: response time is exponentially distributed, so p99 = ln(100) × mean ≈ 4.6 × mean. Real services usually have less variable service times, so their tails are less extreme than this, but the trend holds: tail latency degrades at the same rate as the mean, from a higher starting point. More on reading tails in p95 vs p99 latency.

Two refinements matter in practice:

• Multiple servers. With many workers sharing one queue (8 cores, 50 threads, a pool of 20 DB connections), you can run at higher utilization before waiting becomes significant, but the curve then turns up more abruptly. The knee is later and sharper, not gone.
• Variability. Kingman's approximation says wait time scales with the variability of both arrivals and service times. A mix of 5 ms and 2 s requests on the same pool queues much worse than uniform 100 ms requests at the same utilization. Isolating slow endpoints onto their own pool is often a bigger win than adding capacity.

The knee of the latency curve

The knee is the point where adding a little more throughput costs a lot of latency. Below it, the system absorbs extra load cheaply. Above it, every additional percent of throughput buys a large latency increase, and small traffic fluctuations produce large latency swings. Capacity planning should put normal peak traffic below the knee, not at the theoretical maximum. The relationship between the two axes is covered in throughput vs latency.

Past 100% of capacity the queue grows without bound. Throughput flattens (the system cannot serve faster), latency climbs for as long as the overload lasts, and eventually timeouts turn latency into errors, typically 504s.

What actually saturates

The "server" in the model is whichever resource reaches high utilization first. In web services, the usual suspects:

• CPU. Run-queue length above the core count means threads are waiting for a CPU. Check vmstat 1 (the r column) or load average relative to cores. In containers, CPU throttling from a limit counts too: nr_throttled in the cgroup's cpu.stat.
• Connection pools. 20 DB connections each held for 50 ms give a ceiling of 400 queries per second. Many pools expose "pending" or "waiting" counts (HikariCP's hikaricp_connections_pending metric, for example).
• Worker or thread pools. A fixed number of request threads, Gunicorn workers, PHP-FPM children. When all are busy, requests wait in the accept queue, which is invisible in app metrics.
• Garbage collection. More requests in flight means more allocation and more frequent or longer GC pauses. Pauses stop all requests at once, which shows up in p99 before the mean.
• Lock contention. A mutex, a hot database row, or a single-threaded component (a Node event loop, Redis) behaves like a single-server queue: ρ near 1 arrives quickly.
• Downstream services. Your service may be idle while it waits on a dependency that is past its own knee.

Finding the bottleneck with the USE method

Brendan Gregg's USE method gives a checklist: for every resource, check Utilization, Saturation (queued work) and Errors. Saturation is the metric that maps directly onto queueing delay, and it is the one people forget to collect.

The resource whose saturation metric starts rising at the same load step as latency is your bottleneck. Fix it, and the knee moves to the next resource.

How to find the knee with a step-ramp test

A smooth ramp from 0 to max blurs the knee, because the system never settles at any load level. Use steps instead:

1. Use a fixed arrival rate (RPS), not a fixed number of VUs. In a closed model, VUs wait for slow responses before sending the next request, so as latency rises the offered load drops, and the knee is hidden. See open vs closed workload models and coordinated omission.
2. Choose steps from an estimate. If you believe capacity is around 1,000 RPS, step 200, 400, 600, 700, 800, 900, 1,000, 1,100, with finer steps near the expected knee.
3. Hold each step 2-5 minutes so caches, pools and autoscalers settle, and so you have enough samples for a stable p99 (at 500 RPS, 3 minutes is 90,000 requests).
4. Record per step: achieved RPS, p50, p95, p99, error rate, and the USE metrics above.

Example of what the result looks like (illustrative numbers):

Here the knee is between 600 and 700 RPS: p95 starts climbing faster than load. 800 RPS is past it, and at 900 RPS achieved throughput falls short of target, which means the system is saturated and the queue is growing. A sensible capacity number for this service is around 650 RPS with a p95 objective of 150 ms. The latency percentile calculator helps when you want to compare steps by percentile.

In LoadTester, run this in RPS mode with a p95 threshold and auto-stop, so the run ends at the first step that breaches your objective instead of pushing far past the knee.

Diagnosis table

Re-test and confirm the fix

Re-run the identical step-ramp (same steps, same hold times, same payloads) and compare runs step by step. A fix to the bottleneck should produce one of two outcomes: the knee moves to a higher RPS, or latency below the knee drops because service time dropped. If p95 at the old knee step improves but the knee itself stays put, you shortened service time without raising capacity. If nothing changes, the resource you fixed was not the bottleneck; go back to the USE table and look for the saturation metric that rose first. To catch the knee drifting back in later releases, keep this test as a baseline and compare against it on every release, as described in performance regression testing.