How to Read Load Test Results

Read load test results in a fixed order: errors first (a failing test can show great latency), then whether achieved throughput reached the target, then latency percentiles over time rather than as one summary number, and finally saturation signals on the servers and the load generator. Most results fall into a handful of recognizable shapes, such as the knee where latency bends upward, a plateau where RPS stops growing, or latency that climbs steadily over a long run, and each points at a different cause.

Read in this order

1. Errors. Error rate and, more importantly, which errors and when they started.
2. Throughput vs target. Did the test deliver the load it was supposed to?
3. Latency percentiles over time. p50, p95, p99 as lines across the run, not a single summary.
4. Saturation signals. CPU, memory, connection pools, queue depths on the system under test, and the same on the load generator.

The order matters because each step can invalidate the ones after it. Excellent p95 means nothing if 8% of requests returned 503 in 2 ms.

Step 1: errors

Look at three things:

• Rate: overall and per endpoint. A 0.5% overall error rate can be 20% on checkout.
• Type: HTTP status codes from the server (500, 502, 503, 504, 429) mean different things than client-side errors (connection reset by peer, timeout, EADDRNOTAVAIL, too many open files). Client-side errors can come from the generator itself.
• Onset: did errors start at a specific load level, at a specific time, or are they spread evenly? Evenly spread low-level errors often mean a broken test script or test data; errors that start above a certain RPS mean you found a limit.

The fast-failure trap: rejected requests are usually much faster than successful ones, so a rising error rate pulls latency percentiles down. Filter latency to successful responses, or at least read it next to the error chart. Specific errors have their own guides, for example 502 Bad Gateway under load.

Step 2: throughput vs target

Compare achieved RPS with the target at every stage of the run. In an arrival-rate (fixed RPS) test, achieved RPS should match the target; if it falls short, either the generator could not keep up or it ran out of VUs because responses got slow. In a VU (closed) test, RPS is an output: if latency doubles, RPS halves at the same VU count, and the test silently delivers less load than you think. The relationship is Little's Law: concurrency = throughput x latency.

Example: a test planned at 800 RPS that achieved 610 RPS with p95 of 300 ms did not pass at 800 RPS. It passed at 610. Report the achieved number.

Step 3: latency percentiles over time

Summary tables hide when things happened. Always look at p50, p95 and p99 plotted against time and against load. Rules of thumb:

• p50 is the typical request. If p50 moves, everything moved.
• p95 and p99 are the tail. A widening gap between p50 and p99 means queueing or contention is starting. See p95 vs p99 for which to put in an SLO.
• max is one request. It is useful for spotting timeouts (a max of exactly 30.0 s is a timeout setting) but not as a target.
• Ignore the first 30–60 seconds of warmup (JIT, connection pools, caches) when comparing, or report it separately.

Step 4: saturation signals

Latency tells you something is queueing; saturation metrics tell you where. On the system under test, check CPU (including steal time on VMs), memory and GC, connection pool usage (database, HTTP clients), thread or worker pool usage, and queue depths. On the generator, check CPU, network throughput, and open connections. A generator above about 80% CPU is a red flag: its own scheduling delay ends up in your latency numbers.

Recognizing the common patterns

The knee

Flat latency followed by a sharp rise is a queue forming in front of a saturated resource. Find which resource hit 100% (or its pool limit) at that load. Plan capacity at roughly 70–80% of the knee RPS; throughput vs latency explains why the curve is shaped this way.

Flat RPS plateau

When more load produces no more throughput, there is a hard cap: a connection pool size, a worker count, a single hot database row, a rate limiter, or a single-threaded process. If server CPU is low at the plateau, look for a pool or lock. If the generator is busy and the server is not, the cap is the generator.

Sawtooth: garbage collection or periodic jobs

Periodic spikes in p99 with a steady p50 are pauses that hit every in-flight request. Line them up against GC logs (JVM, .NET, Go, Node) or scheduled jobs. Fixes range from heap sizing and GC tuning to staggering cache TTLs so they do not all expire at once.

Latency climbing over time

If load is constant and latency drifts upward, the system is accumulating state. A short test will not show this; it is the reason soak tests exist. Check memory growth, open file descriptors, connection counts, and table sizes between the start and end of the run.

Errors only above a certain RPS

A clean run up to, say, 1,200 RPS and then 503s or connection resets beyond it usually means a configured limit, not a performance problem: a rate limiter, max_connections, a load balancer surge queue, or a file descriptor limit (the Linux default soft limit of 1,024 open files is a classic). Look for a round number in your configuration that matches the failure point.

Generator-side saturation

Signs the problem is the tool, not the system: generator CPU near 100%, achieved RPS below target while server metrics are calm, client errors like cannot assign requested address (ephemeral port exhaustion), and latency that improves when you split the same load across more generators. Fix the generator first or the run tells you nothing. Closed-loop tools can also hide latency through coordinated omission.

What a good result looks like

Comparing runs against a baseline

• Same test, same data, same environment. Change one thing at a time or the comparison is meaningless.
• Know your noise. Run the baseline two or three times. If p95 varies by 8% between identical runs, a 5% difference is not a regression.
• Compare at equal achieved RPS. A faster p95 at lower achieved throughput is not an improvement.
• Compare per endpoint. An aggregate can stay flat while one endpoint doubles.

LoadTester's run comparison puts two runs side by side and flags regressions against a baseline, which handles the mechanical part; the judgment about noise is still yours. The performance regression testing guide covers setting baselines and thresholds.

Reporting to stakeholders

Non-engineers need the answer, the limit, and the decision. One paragraph is enough:

We tested the checkout API at the forecast peak of 900 RPS plus 50% headroom (1,350 RPS) for 30 minutes on [date]. It met the target up to 1,150 RPS (p95 380 ms, errors 0.1%). Above 1,150 RPS, database connections ran out and p95 rose above 2 seconds. Risk: if the peak exceeds forecast by more than about 25%, checkout slows sharply. Recommendation: raise the connection pool and add a read replica by [date], then re-test; owner [name].

Put percentile tables, charts and configuration in an appendix. If you need to translate percentiles into "how many users were affected", the latency percentile calculator helps.