Test Data for Load Tests

Load test data needs three properties: enough distinct users and records that caches, locks and rate limits behave as they do in production; a database large enough that query plans and memory use match; and a lifecycle (seed, run, verify, clean up) that never touches real customer data. A test where 1,000 virtual users log in as the same account and request the same product ID mostly measures your cache, not your system.

Why test data changes the result

The same script with different data can produce results that differ by an order of magnitude. Data affects the test through four mechanisms:

• Cache hit ratio. Ten product IDs fit in any cache, so every request is a hit. Production traffic spread across 200,000 products is not.
• Contention. Many VUs updating the same cart, account or inventory row serialize on a row lock. Production users mostly touch their own rows.
• Per-user limits. Rate limits, session limits and fraud rules are usually keyed by user or IP. One account at 500 RPS gets throttled; 500 accounts at 1 RPS do not. If throttling is what you want to test, do it deliberately (see rate limit testing).
• Data volume. Query plans, index sizes and buffer pool hit ratios depend on table size. A query that is 2 ms on 50,000 rows can be 400 ms on 50 million rows if the plan or memory fit changes.

The first and last points push results in opposite directions, which is why a test with both a tiny database and a tiny key set can look plausible while being wrong on both counts.

How many unique users and accounts you need

Size the account pool from what the test does with sessions and state:

Worked example: a 15-minute test at 300 RPS where 10% of iterations are sign-ups needs 300 × 900 × 0.1 = 27,000 unused email addresses. A test peaking at 2,000 VUs doing logged-in checkout needs at least 2,000 accounts, and 4,000 to 10,000 if you want the per-account request rate to look like real users. Use the concurrent users calculator to estimate peak VUs first. Login-heavy tests have their own pitfalls (hashing cost, lockout rules), covered in authentication load testing.

Key distribution and cache-hit realism

With uniformly random keys and a warm cache that holds C items, the steady-state hit ratio is roughly C / K for K distinct keys (and 100% when K ≤ C). That makes the size of your key pool the main lever on cache behavior:

Production access is rarely uniform. A few popular items get most traffic (roughly a Zipf-like distribution), so real hit ratios are higher than C / K. The practical approach:

1. Read the production hit ratio from your cache metrics (Redis keyspace_hits / keyspace_misses, CDN cache-hit reports, database buffer pool stats).
2. Export the top N keys by traffic plus a random sample of the long tail.
3. Weight the test's key selection so popular keys appear more often, then check that the test's hit ratio lands within a few points of production.

If you cannot match it, at least bracket it: one run with a hot key set (best case) and one with uniform keys over the full catalog (worst case). The truth sits between them.

Data volume vs production

Seed the test database to production size, or at least to the size at which the working set no longer fits in memory the way it does in production. Things to match:

• Row counts of the large tables (orders, events, sessions, audit logs), within the same order of magnitude.
• Distribution, not just count. One customer with 2 million orders behaves differently from 2 million customers with one order each. Pagination and "my orders" queries depend on it.
• Index and table bloat if production has it. A freshly loaded table is perfectly packed; a production table after years of updates is not.
• Statistics. Run ANALYZE (PostgreSQL) or the equivalent after seeding so the planner sees realistic row estimates.

Where the test runs matters here too. Production vs staging covers when a small staging database invalidates the result.

Seeding scripts

Generate synthetic data with the database itself where possible; it is much faster than inserting through the API. A PostgreSQL example creating 2 million users and 20 million orders with a skewed distribution:

-- users: predictable emails so the load test can log in
INSERT INTO users (id, email, password_hash, is_synthetic, created_at)
SELECT g,
       'lt-user-' || g || '@loadtest.example',
       '$2b$10$...precomputed-hash-for-test-password...',
       true,
       now() - (random() * interval '3 years')
FROM generate_series(1, 2000000) AS g;

-- orders: skewed so some users have many orders (power-law-ish)
INSERT INTO orders (user_id, total_cents, status, created_at, is_synthetic)
SELECT 1 + floor(2000000 * power(random(), 3))::int,
       (500 + random() * 20000)::int,
       (ARRAY['paid','shipped','refunded'])[1 + floor(random() * 3)::int],
       now() - (random() * interval '3 years'),
       true
FROM generate_series(1, 20000000);

ANALYZE users; ANALYZE orders;

Precompute one password hash and reuse it so seeding does not spend hours in bcrypt. Then export the credentials the load generator needs:

\copy (SELECT email, 'TestPassword!1' AS password FROM users
       WHERE is_synthetic ORDER BY random() LIMIT 10000)
  TO 'users.csv' WITH CSV HEADER

.example is a reserved top-level domain (RFC 2606), so these addresses can never deliver mail to a real person if an email integration is accidentally left on.

CSV feeders in JMeter, k6 and Locust

JMeter: CSV Data Set Config

Add a CSV Data Set Config with Filename: users.csv, Variable Names: email,password, Ignore first line: True, and reference ${email} in samplers. Two settings decide uniqueness: Sharing mode "All threads" hands each line to exactly one thread in turn, and Recycle on EOF False plus Stop thread on EOF True prevents reuse when single-use data runs out. In distributed mode each JMeter engine reads its own copy of the file, so split the file per engine to avoid duplicates.

k6: SharedArray

import http from 'k6/http';
import exec from 'k6/execution';
import { SharedArray } from 'k6/data';
import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';

const users = new SharedArray('users', () =>
  papaparse.parse(open('./users.csv'), { header: true }).data
);

export default function () {
  // unique across the whole test, not per VU
  const u = users[exec.scenario.iterationInTest % users.length];
  http.post('https://staging.example.com/api/login',
    JSON.stringify({ email: u.email, password: u.password }),
    { headers: { 'Content-Type': 'application/json' } });
}

SharedArray loads the file once and shares it read-only across VUs, which keeps memory flat with large files. Using iterationInTest gives each iteration a different row; drop the modulo for single-use data and stop the test when rows run out.

Locust: a shared iterator

import csv, itertools
from locust import HttpUser, task, between

with open("users.csv") as f:
    ROWS = list(csv.DictReader(f))
POOL = itertools.cycle(ROWS)   # use iter(ROWS) for single-use data

class Shopper(HttpUser):
    wait_time = between(1, 3)

    def on_start(self):
        self.creds = next(POOL)
        self.client.post("/api/login", json=self.creds)

    @task
    def orders(self):
        self.client.get("/api/orders")

Locust has no built-in feeder, and in distributed mode each worker process loads its own copy, so give each worker a different slice of the file. For other tools and for hosted services, the general pattern is the same: per-VU or per-iteration rows, no accidental reuse of single-use data.

Idempotency and cleanup

Write-heavy tests leave data behind and can be distorted by retries. Plan for both:

• Mark everything. An is_synthetic flag, a dedicated tenant, or a recognizable email domain lets you exclude test data from reports and delete it safely.
• Unique idempotency keys. If your API supports an Idempotency-Key header, generate a fresh UUID per logical operation. Reusing a key turns every request after the first into a cached replay, which is much faster than real processing and inflates throughput.
• Delete in batches. DELETE FROM orders WHERE is_synthetic on 20 million rows can lock tables and bloat the write-ahead log. Delete in batches of 10,000 to 50,000 rows, or drop and recreate a dedicated test schema.
• Reset between comparable runs. A regression comparison is only fair if both runs start from the same data. Restore from a snapshot rather than letting each run add more rows.
• Disable real side effects. Route email, SMS, payments and webhooks for synthetic accounts to sandbox or no-op providers. Checkout load testing covers payment sandboxes in more detail.

PII and compliance rules

• Do not copy production personal data into test environments. Under GDPR and similar laws, a copy is still processing personal data, with all the access, retention and breach obligations that implies, and test environments are rarely protected like production.
• Prefer fully synthetic data generated with SQL as above or with libraries such as Faker. It is safe to share, commit and reset.
• If you need production shape, copy the structure and distributions (row counts, key popularity, value lengths), not the values. Masking or pseudonymizing real records is better than raw copies, but treat masked data as sensitive unless it has been reviewed.
• Keep credentials for test accounts out of public repositories, even synthetic ones, if those accounts exist in production.
• Logs and reports leak data too. Request bodies captured in failed-sample logs or shared result links should contain only synthetic values.

Test data checklist

1. Count distinct accounts needed: at least peak VUs for logged-in flows, total iterations for single-use flows.
2. Size the key pool so the test's cache hit ratio matches production, or bracket it with best and worst cases.
3. Seed large tables to production order of magnitude with realistic skew, then run ANALYZE.
4. Use synthetic data only, on reserved domains such as .example, with an is_synthetic marker.
5. Feed rows per iteration (CSV Data Set Config, SharedArray, shared iterator) and prevent reuse of single-use data.
6. Generate fresh idempotency keys, disable real side effects, and restore from a snapshot before comparison runs.
7. Clean up in batches and confirm no synthetic data remains in analytics. Note the data setup in your load test plan.