Inconsistent Load Test Results Between Runs

Load test results differ between runs because something other than your code changed between them: how warm the system was (JIT, caches, connection pools, autoscaled replicas), what else was running on shared hardware, whether the load generator saturated, the network path, the test data, or simply too few samples in the tail percentile you compare. Make runs comparable by fixing those inputs, excluding warm-up, using a fixed arrival rate, running each variant at least three times and comparing the median of their p95 values. Then set regression thresholds wider than the spread you measured between identical runs, or the check will fail at random.

What the symptom looks like

You run the same scenario against the same build twice and get p95 of 180 ms and 260 ms. CI flags a regression on a commit that only touched a README. A fix appears to improve latency by 20%, and the next run says it made things worse. The common lines in team chat: "results differ between runs", "flaky performance test", "the numbers are not reproducible".

Before chasing causes, measure the noise. Run the unchanged build five times, back to back, with the same settings:

Read it the way you would read any experiment. p50 is stable within a few milliseconds. p95 varies about 12% across runs 1, 2, 3 and 5. p99 swings more than 2x. Run 4 is different in kind, not degree: achieved RPS fell short of the target, which means the generator or the network had a problem in that run, and its numbers should not be compared at all.

Most likely causes, ranked

1. System state at start: warm-up, caches, JIT, pools

The first run after a deploy hits cold caches, an unJITted JVM, empty connection pools, and a database whose buffer cache does not hold the working set yet. The second run starts warm. If you compare a first run against a third run, you measure warmth. Signs: the first minutes of each run are slow, and runs right after a deploy are consistently worse.

2. Autoscaling state

If the previous run left 12 replicas behind and the scale-down stabilization window has not expired, the next run starts with three times the capacity. Run order then decides the result. Check replica count at the start of every run; the autoscaling lag guide covers how long scale-down takes.

3. Load generator saturation

A generator that runs out of CPU, sockets or bandwidth sends late and counts its own queueing as server latency. Whether it saturates depends on what else runs on the machine and on response sizes, so it varies between runs. Signs: achieved RPS below target, generator CPU above about 70-80%, latency rising on the client but not in server-side metrics. Closed-loop VU tests also hide stalls through coordinated omission, which makes some runs look better than they were.

4. Noisy neighbors and shared infrastructure

Shared CI runners, burstable cloud instances that run out of CPU credits, a staging database shared with other teams, a batch job or backup that happens to overlap. Signs: CPU steal (st in top), different results by time of day, and outliers that cluster rather than scatter.

5. Test data differences

Tests that create data grow the tables they read, so run 5 queries a bigger table than run 1. Randomized IDs hit cached rows in one run and cold rows in another. A test user's cart fills up across runs. Reset the data set or use a fixed seed between runs; load test data covers the patterns.

6. Network path

Different source regions, a CDN serving from cache in one run and from origin in another, DNS returning different load balancer nodes, or rate limiting kicking in. Signs: differences in connection time or TLS time while server processing time is stable.

7. Too few samples for the percentile

A p99 from 3,000 requests is decided by about 30 samples; one GC pause moves it. This is noise you cannot remove by controlling the environment, only by measuring more requests per window. The latency percentile calculator shows how much data each percentile rests on.

How to find which cause applies

• Check achieved vs target RPS first. Any run that did not hit the target rate is invalid for latency comparison. Look at generator CPU and network for that run.
• Plot p95 per minute within each run. A slow first few minutes that converge means warm-up. A spike in the middle of one run means an external event; line it up with cron jobs, backups and deploys.
• Compare server-side and client-side latency. If server processing time (from access logs or APM) is stable but client latency varies, the difference is in the network or the generator.
• Record environment state per run: replica count, instance types, database size, build version, time, and generator location. Differences in that list usually explain differences in results.
• Run the same build five times. The spread of those runs is your noise floor. Nothing smaller than it is a real change.

A protocol that makes runs comparable

1. Pin the environment. Same instance types (non-burstable), same replica count with autoscaling disabled or pinned to fixed min/max, same database snapshot, nothing else scheduled during the window. Use a dedicated environment if you can; production vs staging covers the trade-offs.
2. Reset state. Restore test data, flush or pre-warm caches deliberately (pick one and always do the same), and restart or warm the service the same way each time.
3. Use a fixed arrival rate. RPS-mode tests keep offered load identical between runs. VU-mode tests let throughput drift with latency, so two runs at "100 VUs" can be two different workloads.
4. Warm up, then measure. A 2-3 minute warm-up at the test rate, excluded from the compared numbers.
5. Measure enough requests. For p95, aim for thousands of requests in the measured window; for p99, tens of thousands. At 300 RPS, 5 minutes gives 90,000 requests, enough for a stable p99.
6. Run each variant three times or more, interleaved. A-B-A-B-A-B is better than A-A-A-B-B-B, because slow drift in the environment affects both variants equally.
7. Compare medians of p95. The median of three runs ignores one outlier. Report the spread with it.

This is the same discipline described in benchmarks methodology, applied to day-to-day regression checks.

Setting regression thresholds that do not flap

A threshold has to sit above the noise floor, or it fails on unchanged code. A simple, defensible rule:

relative noise = (max p95 − min p95) / median p95 across baseline runs of an unchanged build

With the four valid runs above: (238 − 212) / 223.5 ≈ 12%. A relative threshold of 15-20% on the median p95 then catches real regressions of that size without random failures. Add an absolute ceiling tied to your SLO (for example p95 < 400 ms) so slow creep below the relative threshold still gets caught eventually. If you need to detect changes smaller than your noise, reduce the noise first (longer windows, dedicated environment) or run more repetitions; with request-level data, a bootstrap confidence interval on the p95 difference gives a more formal answer.

LoadTester's run comparison and regression detection work against a baseline run you choose, so the baseline should be a representative run from this protocol, not whichever run happened to be fastest. Scheduled tests at the same time each day remove time-of-day effects from the comparison. Plan limits apply to how long each run can be (up to 20 minutes on Pro), which is enough for the warm-up plus measurement window described above at moderate rates.

Diagnosis table

How to re-test and confirm runs are now comparable

After applying the protocol, repeat the noise measurement: five runs of the same build, interleaved over a day if your risk is time-of-day effects. What should change: achieved RPS equals the target in every run, p95 spread falls (a pinned environment commonly gets it well under the original figure), and the per-minute p95 inside each run is flat after warm-up. Then run an A-B comparison on a deliberately slower build (add a fixed sleep to one endpoint) and check that your threshold catches it while repeated runs of the baseline pass. That proves the gate detects what you care about without flapping. How to read load test results covers what to look at within a single run, and performance regression testing covers wiring the comparison into releases.