Load Testing in Production vs Staging

Load test in staging when you need to push past the breaking point, compare builds, or test changes that could corrupt data. Load test in production when the question depends on things staging cannot copy: real data volume, warm caches, the full network path (CDN, WAF, load balancers) and third-party limits. Most teams need both: staging for aggressive and destructive tests, and small, bounded production tests at off-peak hours to confirm that staging numbers hold in the real environment.

The decision in one question

Ask: would the answer change if the test ran in the real environment? If the result depends mostly on your code (a query got slower, a new endpoint allocates too much memory), staging gives a reliable answer at low risk. If it depends on the environment (how much traffic the CDN absorbs, whether the database plan changes at 400 million rows, whether the payment provider rate-limits you), staging will give a confident answer to a different question.

This page is about choosing where to test. Once you decide to touch production, the operational safety steps (blast radius, runbook, stop conditions) are in the production load testing checklist.

What staging can and cannot tell you

The four gaps that matter most

• Data size. A 2 GB staging database fits in the buffer pool; a 600 GB production database does not. Index scans that are memory-only in staging become disk reads in production. Sorting, pagination and COUNT(*) queries are the usual surprises. See load test data for seeding realistic volumes.
• Caches. Production caches are warm with real access patterns. A staging test either hits a cold cache (too pessimistic) or hammers the same few keys (too optimistic).
• Topology. Production often has a CDN, a WAF, more load balancer hops, multiple availability zones, different instance types and different connection limits. Each one changes latency and can be the bottleneck. Load testing behind Cloudflare covers one common case.
• Third parties. Payment, email, search and auth providers behave differently in sandbox mode, and live endpoints have rate limits that you might hit at your target load. In production you also must not send real emails or charges from a test.

Production testing techniques, from safest to riskiest

Off-peak runs against read paths

Pick the lowest-traffic window from your metrics (not a guess), test read-only or idempotent endpoints, and ramp in steps with automatic stop conditions. This is the simplest production test and answers most "will the real stack handle X RPS" questions. Make sure your test traffic is identifiable: allowlist the generator's IPs and tag requests with a header so you can filter them out of analytics.

Shadow traffic (mirroring)

Copy real production requests to a second deployment and discard its responses. Users never see the shadow's output, but it processes real request shapes at real rates. Options include nginx's mirror directive (ngx_http_mirror_module), Envoy's request mirror policies, Istio's mirror and mirrorPercentage fields on a VirtualService, and capture-and-replay tools such as GoReplay. Caveats: mirrored writes must not reach real data stores, and the mirror adds load to shared dependencies unless isolated. Shadowing measures a new version under real traffic; it does not easily push beyond real traffic levels unless you replay at a multiplier.

Synthetic tenants and test accounts

Create dedicated accounts, organizations or stores that exist in production but are flagged as synthetic. Tests can then exercise write paths (sign-ups, orders, uploads) against real infrastructure. Side effects such as payment, email and webhooks are routed to no-op or sandbox providers by the synthetic flag. This needs application support, and cleanup is part of the design.

Small slices and canaries

Route a slice of production capacity (one pod, one zone, one canary) and drive it to high utilization with real plus synthetic traffic. This measures per-instance capacity in the real environment while limiting the blast radius to that slice. Load balancer weights or header-based routing select the slice.

Full-scale production load test

Generating peak or above-peak load across the whole production system, usually before a known event such as a launch or a sale. Highest realism and highest risk. It needs everything in the checklist, people watching, and a stop button. The Black Friday load testing guide covers this scenario.

Risk matrix

Use the table to pick the lowest-risk row that can answer your question. If the question is "is the new build slower", the first row is enough. If it is "can our real edge take 2,000 RPS at 3 a.m. without the WAF blocking us", only a production row answers it.

A practical combined approach

1. Every merge or nightly, in staging: a short regression test comparing p95 and error rate against a baseline (see performance regression testing).
2. Before significant releases, in staging: stress and breakpoint tests to learn where and how it fails.
3. Monthly or before launches, in production: an off-peak, step-ramped read-path test to the expected peak plus margin, with auto-stop thresholds and a person watching.
4. Calibrate: compare the production result with the staging result at the same RPS. The ratio (for example, production p95 is 1.4x staging) tells you how much to trust future staging-only numbers. Recheck it whenever infrastructure changes.

With LoadTester, production runs only target domains you have verified with a DNS TXT record or a well-known file, worker source IPs are published for WAF allowlisting, and p95 or error-rate thresholds can stop a run automatically when breached. Those controls reduce, but do not replace, the planning above.

Signs your staging results are misleading

Even without a production test, a few signals in staging results suggest they will not transfer:

• The database barely registers. CPU and disk reads on the database stay near idle while the app tier saturates. In production the database is often the first bottleneck, so the staging data set is probably too small or too cache-friendly.
• Cache hit ratio far from production. Compare the test's Redis or CDN hit ratio with production dashboards. A 99% hit ratio in staging against 80% in production means the test is measuring the cache.
• Latency much lower than production at much higher load. If staging serves 1,000 RPS at p95 = 40 ms and production struggles at 300 RPS with p95 = 250 ms, something structural differs: data, topology, instance types or noisy neighbors.
• No third-party calls in traces. Mocked dependencies remove latency that real users always pay. Add realistic delays to mocks, or account for them separately.
• Different scaling config. Fixed replica counts in staging and autoscaling in production (or the reverse) make throughput numbers incomparable.

Any one of these does not make staging useless. It means staging numbers are good for comparing builds against each other and poor as a forecast of production capacity.

Account for the error budget

A production test that causes errors spends the same error budget as a real incident. Rough math, assuming evenly spread traffic: a 28-day window is 40,320 minutes. A test that causes 5% errors on all traffic for 10 minutes fails 5% × 10 / 40,320 ≈ 0.0012% of the window's requests. Against a 99.9% SLO (a 0.1% budget) that is about 1.2% of the budget, which is acceptable. A test that degrades into 50% errors for an hour fails about 0.074% of the window's requests, roughly three quarters of the budget. Decide in advance how much budget a test may spend and set the auto-stop error threshold to match. SLA vs SLO vs SLI and the error budget calculator cover the math.

Choosing the environment: checklist

1. Write the question the test must answer.
2. List which of data size, caches, topology and third parties the answer depends on.
3. If none: staging. If any: plan a bounded production run as well.
4. If the test writes or triggers side effects: staging, or production with synthetic tenants and no-op integrations.
5. Pick the lowest-risk row of the risk matrix that answers the question.
6. For production, follow the production checklist and set an error budget limit for the run.
7. Record the staging-to-production ratio so future staging results are interpretable.