High CPU Under Load at Modest RPS

When CPU hits 100% at a request rate that should be easy, the server is spending far more CPU per request than you think, usually in serialization, regex, TLS handshakes on connections that are not reused, synchronous logging, garbage collection or deliberately slow password hashing. Do not guess: compute CPU milliseconds per request, run a sampling profiler (perf, async-profiler, py-spy, Node's --cpu-prof) while the load test holds a steady rate, and fix the widest frames in the flame graph. If CPU looks moderate but latency is still bad inside a container, check CFS throttling before anything else.

What the symptom looks like

The typical pattern in a load test: CPU climbs roughly linearly with RPS and reaches 90-100% at, say, 150 RPS on a 4-core box, p95 rises steeply from that point, and the database and network are nearly idle. Common accompanying lines:

# top / htop: one process at ~400% on 4 cores, load average above core count
%Cpu(s): 97.1 us,  1.8 sy,  0.0 ni,  0.4 id,  0.0 wa,  0.0 hi,  0.4 si,  0.3 st

# Kubernetes
kubectl top pod api-7d9f   CPU(cores): 1998m   (limit: 2)

# Node.js event loop blocked
Warning: event loop lag 850ms   (from your monitoring, e.g. perf_hooks.monitorEventLoopDelay)

# Gunicorn worker killed because it was busy on CPU too long
[CRITICAL] WORKER TIMEOUT (pid:42)

Two variants need different handling. User CPU high (us) means your code or runtime is busy; profile it. System CPU high (sy, si) means the kernel is busy with syscalls, context switches, networking or connection setup; that often points to connection churn or tiny writes. High st (steal) means the hypervisor is giving your CPU to someone else, which is an infrastructure problem, not a code problem.

Start with CPU milliseconds per request

This one number tells you whether the problem is "too much work per request" or "too many requests":

CPU ms per request = (cores × utilization × 1000) / RPS

Worked example: 4 cores at 95% busy serving 150 RPS gives (4 × 0.95 × 1000) / 150 ≈ 25 ms of CPU per request. For a JSON API that reads a few rows, 1-5 ms is a more typical budget, so something is costing 5-20 times more than it should. If instead you measure 2 ms per request at 1,800 RPS, the code is fine and you need more cores; the server capacity calculator covers that case.

Measure at a fixed arrival rate so RPS does not drift while you sample (see throughput vs latency). Then measure per endpoint: one endpoint at 200 ms of CPU mixed into many cheap ones will dominate the average.

Profile while the test holds steady load

Profiling an idle process tells you nothing. Start the load test, wait until RPS and CPU are flat, then take a 30-60 second sample.

# Any native or JIT process on Linux: sample at 99 Hz with call stacks
perf record -F 99 -g -p $(pgrep -n myservice) -- sleep 30
perf report --stdio | head -60
# (for flame graphs: perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg)

# JVM: async-profiler (v3 ships the asprof launcher)
asprof -d 30 -e cpu -f /tmp/cpu-flame.html <pid>

# Python: py-spy attaches without restarting the process
py-spy record -o /tmp/cpu.svg --pid <pid> --duration 30
py-spy top --pid <pid>            # live view, like top for functions

# Node.js: writes a .cpuprofile on exit; open it in Chrome DevTools
node --cpu-prof --cpu-prof-dir=/tmp/prof server.js

# Go: built-in pprof endpoint
go tool pprof -http=:8081 http://localhost:6060/debug/pprof/profile?seconds=30

Read the flame graph for wide frames, not tall ones. A frame's width is its share of CPU samples. The usual suspects show up by name: JSON.stringify, jackson.databind, json.dumps, re.match/RegExp, SSL_do_handshake/ECDSA_sign/RSA, bcrypt, logging appenders, or GC threads.

Most likely causes, ranked

1. Password hashing in the test path

bcrypt, scrypt, Argon2 and PBKDF2 are slow on purpose. Each +1 on the bcrypt cost factor doubles the work, and frameworks commonly default to a cost of 10-12. If a hash costs 250 ms of CPU on one core (measure yours; it varies by hardware and cost), an 8-core server can verify at most 8 / 0.25 = 32 logins per second, no matter how fast everything else is. A load test where every VU logs in on every iteration measures the hash function, not your app. Measure it directly:

node -e "const b=require('bcrypt');const h=b.hashSync('x',12);console.time('c');b.compareSync('x',h);console.timeEnd('c')"
python3 -c "import bcrypt,time;h=bcrypt.hashpw(b'x',bcrypt.gensalt(12));t=time.time();bcrypt.checkpw(b'x',h);print(time.time()-t)"

Fix the test, not the cost: log in once per VU, reuse the token or cookie, and model login as its own realistic share of traffic. Authentication load testing covers that design.

2. TLS handshakes without connection reuse

A full TLS handshake costs far more CPU than encrypting a typical response, and with RSA certificates the server's private-key operation is the expensive part. If the client or a proxy opens a new connection per request, the server pays it on every request. Confirm by counting handshakes vs requests: in nginx, log $ssl_session_reused and $connection_requests; on the box, ss -s shows connection churn and TIME_WAIT counts climbing. Fixes: enable keep-alive on every hop (keepalive_timeout, upstream keepalive), make sure the load generator reuses connections like real browsers do, enable session resumption, and prefer ECDSA certificates. Also check the reverse: if your test opens one connection per VU but real users come through a CDN, you may be testing a handshake pattern that never happens in production.

3. Serialization

Serializing big objects to JSON is often the single largest cost in an API. Signs: JSON.stringify, Jackson or json.dumps frames, and CPU per request that scales with response size. Fixes: return fewer fields, paginate, cache serialized output for hot read endpoints, use faster encoders (orjson in Python, avoiding reflection-heavy mappers), and stop serializing the same ORM graph twice.

4. Regex and string work

A regex compiled inside a hot loop, or one with nested quantifiers that backtracks on long input, can burn whole cores. Look for re.compile called per request, user-agent parsing per request, and validation of large bodies. Precompile, cache results, bound input lengths.

5. Logging

Debug-level logging, synchronous appenders writing to a slow disk, and JSON-formatting every log line can cost more than the request. Check the log level in the test environment, switch to async appenders (Logback AsyncAppender, buffered handlers), and sample high-volume logs.

6. Garbage collection

High allocation rates show as GC threads in the profile. On the JVM, enable -Xlog:gc*:file=/tmp/gc.log and look at pause frequency and total GC time. A container JVM without -Xmx uses MaxRAMPercentage, which defaults to 25% of the container limit, so a 2 GiB pod gets a 512 MiB heap and collects constantly. Set -XX:MaxRAMPercentage=70 or an explicit -Xmx. In Go, GODEBUG=gctrace=1 shows GC frequency; GOGC and GOMEMLIMIT tune it.

Containers: CPU throttling looks like slowness, not like 100%

With a Kubernetes CPU limit, the kernel's CFS quota gives the container a budget per 100 ms period. A limit of 2 CPUs is 200 ms of CPU time per 100 ms period. A process with 8 busy threads can spend that budget in 25 ms of wall time and then gets no CPU at all for the remaining 75 ms. Averaged over a minute, kubectl top shows a moderate number; per request, p99 jumps by tens of milliseconds.

# cgroup v2, from inside the container
cat /sys/fs/cgroup/cpu.max      # "200000 100000" = 2 CPUs per 100 ms period
cat /sys/fs/cgroup/cpu.stat     # nr_periods, nr_throttled, throttled_usec

# Prometheus (cAdvisor): fraction of periods throttled
rate(container_cpu_cfs_throttled_periods_total[1m])
  / rate(container_cpu_cfs_periods_total[1m])

If more than a few percent of periods are throttled under load, the limit is cutting your latency. Fixes: raise or remove the CPU limit (keep the request), match runtime thread counts to the limit (GOMAXPROCS, JVM -XX:ActiveProcessorCount, worker counts), or run fewer, larger replicas. The Kubernetes load testing page covers limits and requests more broadly.

Diagnosis table

How to re-test and confirm

Re-run the same fixed-RPS step-ramp you used to find the problem, for example 50, 100, 150, 200, 250 RPS at 2 minutes each, and compare two numbers per step: CPU ms per request and p95. A real fix lowers CPU ms per request at every step, which moves the point where latency bends (see why latency increases under load) to a higher RPS. If CPU per request drops but the knee does not move, a different resource is now the bottleneck: check the database pool and network next.

Take a second profile at the new peak to confirm the wide frame is gone and to find the next one. Keep the test's request mix identical between runs, including the share of logins, or you are comparing different workloads. In LoadTester, saving the run as a baseline and comparing the next one against it shows the p95 change per step without manual spreadsheet work.