Memory Grows During a Soak Test: Leak, Cache or Fragmentation

Memory that grows during a soak test is one of three things: a real leak (objects stay reachable and accumulate with every request), a cache or buffer that is filling toward a limit it may not have, or allocator fragmentation where the process holds freed memory it cannot hand back. Tell them apart by comparing RSS with heap used and by watching the low points of the garbage-collection sawtooth: if the post-GC floor keeps rising at a steady load, take two heap snapshots a few minutes apart and diff them. If heap is flat but RSS climbs, the problem is native memory or fragmentation, not your objects.

What the symptom looks like

On a dashboard, memory rises slowly across the run. In logs and events, the end state is one of these:

# Kubernetes: kubectl describe pod api-5f7c
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

# Kernel log (dmesg / journalctl -k)
Memory cgroup out of memory: Killed process 31337 (java) total-vm:6123456kB, anon-rss:2091520kB

# JVM
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: GC overhead limit exceeded

# Node.js
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

# Python (usually no message: the kernel kills it, exit 137)

Exit code 137 is 128 + 9: the process got SIGKILL, typically from the kernel's OOM killer when the cgroup limit is hit. A runtime-level error (Java heap space, Node's heap limit) is different: the runtime's own heap cap was reached while the container still had room. That distinction matters, because the first can be native memory and the second is always managed heap. Under load, an OOM kill often surfaces at the proxy first as a burst of 502 Bad Gateway errors while the pod restarts.

Leak, cache growth or fragmentation

All three look like "memory goes up". Their shapes differ:

• Leak: grows in proportion to work done. Double the RPS and the slope doubles. Never levels off. Post-GC floor rises steadily.
• Cache growth: grows fast at first and then flattens once the cache holds the working set, as long as the cache is bounded. An unbounded cache keyed by something with high cardinality (user ID, full URL with query string, request ID) behaves like a leak.
• Fragmentation / allocator retention: managed heap is flat, RSS climbs or stays high after load stops. Common with glibc malloc in multi-threaded processes, which creates many arenas and does not always return freed pages.

Step one: RSS vs heap

Plot two lines for each process: resident memory as the OS sees it, and heap used as the runtime sees it.

# Kubernetes / cAdvisor (what eviction and OOM decisions track)
container_memory_working_set_bytes{pod=~"api-.*"}
container_memory_rss{pod=~"api-.*"}

# Inside the box
ps -o pid,rss,vsz,cmd -p <pid>
grep -E 'VmRSS|RssAnon|RssFile' /proc/<pid>/status

Runtime heap metrics: JVM jvm_memory_used_bytes{area="heap"} (Micrometer) or jcmd <pid> GC.heap_info; Node process.memoryUsage() (heapUsed vs rss); Go runtime.MemStats or /debug/pprof/heap; Python tracemalloc.get_traced_memory().

Heap snapshots by runtime

Take one snapshot after warm-up, one 10-15 minutes later at the same steady load, and a third after another interval. Objects whose counts grow across all three are your candidates; this three-snapshot approach filters out normal churn.

JVM

jcmd <pid> GC.class_histogram | head -30      # quick: instance counts by class, run twice and diff
jcmd <pid> GC.heap_dump /tmp/heap-1.hprof     # full dump; open in Eclipse MAT, compare dominator trees
-Xlog:gc*:file=/tmp/gc.log:time,uptime        # post-GC heap per collection
-XX:NativeMemoryTracking=summary   then   jcmd <pid> VM.native_memory summary

Native Memory Tracking shows non-heap growth (threads, metaspace, direct buffers, code cache). A heap dump pauses the JVM, so take it from one replica, not all of them.

Node.js

node --heapsnapshot-signal=SIGUSR2 server.js
kill -USR2 <pid>     # writes a .heapsnapshot in the working directory; load in Chrome DevTools > Memory, use "Comparison"

Typical Node leaks: listeners added per request and never removed (watch for MaxListenersExceededWarning), closures held by timers, module-level Maps used as caches.

Python

import tracemalloc
tracemalloc.start(25)
snap1 = tracemalloc.take_snapshot()
# ... later, under the same load ...
snap2 = tracemalloc.take_snapshot()
for stat in snap2.compare_to(snap1, "traceback")[:10]:
    print(stat)

For native extensions, memray tracks allocations below the Python layer. Common Python leaks: global lists or dicts appended per request, functools.lru_cache(maxsize=None) on functions called with unique arguments.

Go

curl -s localhost:6060/debug/pprof/heap > heap1.pb.gz
# later
curl -s localhost:6060/debug/pprof/heap > heap2.pb.gz
go tool pprof -base heap1.pb.gz heap2.pb.gz     # shows only what grew
curl -s localhost:6060/debug/pprof/goroutine?debug=1 | head   # goroutine leaks look like memory leaks

When heap is flat and RSS keeps climbing

Check, in this order:

1. Thread or goroutine count. ls /proc/<pid>/task | wc -l. Each thread has a stack; a pool that grows per request will look like a leak.
2. Off-heap buffers. JVM direct buffers, Netty pooled buffers, Node Buffers (counted in external/arrayBuffers, not heapUsed).
3. glibc arenas. glibc malloc creates up to 8 arenas per core on 64-bit systems, and freed memory in each arena may not return to the OS. Test with MALLOC_ARENA_MAX=2 in the environment, or run with jemalloc via LD_PRELOAD. If RSS flattens, it was fragmentation, and the setting is the fix.
4. Page cache in the working set. Writing large files inside the container counts toward its memory. Logs written to a local file under load are a common example.

Designing the soak test so the answer is clear

• Constant arrival rate at a realistic level (around normal peak, not maximum). Varying load creates varying memory and hides slopes. With constant RPS, slope per minute converts directly to bytes per request.
• Exclude warm-up. JIT, connection pools and caches fill during the first minutes. Measure the slope after that.
• Do the per-request math. Example: memory floor rises 180 MiB over 20 minutes at 300 RPS = 360,000 requests, so roughly 520 bytes retained per request. With a 512 MiB headroom and production traffic of 100 RPS, that is about 2.9 hours to an OOM kill, which matches "it restarts every few hours" reports.
• Per-request leaks scale with requests, not time. A 20-minute run at 3× normal rate exercises as many requests as an hour at normal rate. Time-driven leaks (timers, scheduled jobs, token refresh every N minutes) need real wall-clock duration; see how long a load test should run.
• Add a cool-down and a second cycle. Stop load for 5 minutes, then repeat. A leak ratchets up again from a higher starting point; allocator retention reuses what it kept and stays flat.
• Exercise the full mix. Leaks often live in a rarely hit path such as error handling, file upload or a specific report. Include error cases in the scenario.

The soak testing page covers which metrics to trend during the run. In LoadTester a single run is at most 20 minutes on Pro, which suits per-request leaks at a higher steady rate, and a scheduled test can repeat it to catch regressions across builds. For true multi-hour wall-clock soaks, run k6, Locust or Gatling from your own infrastructure; they handle long runs well.

Set runtime caps below the container limit

A leak should end in a readable error, not a silent kill. If the runtime's own heap cap is above (or missing relative to) the container limit, the kernel kills the process first and you lose the stack trace, the heap dump and the GC log tail. Leave room for non-heap memory: thread stacks, metaspace or code cache, native buffers and the allocator's own overhead.

# JVM in a 2 GiB container: heap about 70%, dump on heap OOM
-XX:MaxRAMPercentage=70 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps

# Node.js in a 1 GiB container: old space well under the limit
node --max-old-space-size=700 server.js

# Go 1.19+: soft memory limit makes the GC work harder before the hard limit
GOMEMLIMIT=900MiB

Keep the Kubernetes memory request equal to the limit for services you are soak testing, so the pod is not evicted early by node pressure for reasons unrelated to your code. When the cap is set this way, a leak shows up as GC time rising and p95 creeping up for a while before the runtime throws its own out-of-memory error, which is exactly the drift you want the soak test to catch. Restart counts on Kubernetes are worth graphing next to memory for the same reason.

Diagnosis table

How to re-test and confirm

Re-run the same steady-rate soak with the same scenario and the same warm-up exclusion. The fix is confirmed when the post-GC floor is flat (slope within noise) for the full measured window and the second cycle after cool-down starts where the first one ended. Compare the bytes-per-request figure before and after: it should fall to near zero, not just shrink. Also check that p95 latency stays flat across the run; a leak often shows as slowly worsening latency from rising GC work before it ever causes a kill. How to read load test results covers spotting that kind of drift.