502 Bad Gateway Under Load: How to Find the Cause

A 502 Bad Gateway under load means a proxy (nginx, an ALB, Cloudflare, an ingress controller) tried to get a response from your app and got either nothing usable or a closed connection. Under load the usual causes are, in order: app processes crashing or being restarted (often OOM kills or worker timeouts), a keep-alive idle-timeout mismatch where the app closes connections the proxy still thinks are open, and the app running out of workers or listen-queue space so new connections are refused. The proxy's error log tells you which one in a single line.

What a 502 actually tells you

A 502 is never generated by your application. It is generated by the hop in front of it, and it means "I am a gateway, and the upstream gave me an invalid response or no response at all." That distinction matters because a 502 rate that rises with load is almost always about connections and processes, not slow code. Slow code produces 504 Gateway Timeout instead: the upstream is alive, just late.

Roughly, the proxy returns 502 when one of these happens:

• The TCP connect to the upstream is refused (nothing listening, or the listen queue is full on a Unix socket).
• The upstream closes or resets the connection before sending a complete response header.
• The upstream sends something that is not valid HTTP, or a header block larger than the proxy's buffer.
• Every upstream server is marked down, so there is nothing to connect to.

Because the proxy is the one that notices, its error log is your primary evidence. The app log often shows nothing at all for the failing requests, since they never reached a handler.

Most likely causes, ranked by how often they show up

1. Upstream process crashes or restarts mid-request. OOM kills, Gunicorn/Unicorn worker timeouts that kill the worker, an unhandled exception that takes down a Node process, a Kubernetes liveness probe that fails under load and restarts the container. Every in-flight request on that process becomes a 502.
2. Keep-alive idle-timeout mismatch. The proxy reuses a pooled connection at the exact moment the app decides it has been idle too long and closes it. Low, steady 502 rate (often 0.01-1%) that appears at moderate load and does not scale with CPU.
3. Worker or connection exhaustion. PHP-FPM pm.max_children reached, a full listen backlog, too few Gunicorn workers. New connections are refused or reset.
4. Deploys and scaling events during the test. Pods terminated without draining, targets deregistered while still receiving traffic, autoscaler adding instances that are not ready yet.
5. Protocol issues. Response headers too large for proxy_buffer_size, an app speaking HTTP/1.0 or closing early on large POST bodies. Rarer, and usually not load-dependent.

Reading the exact log strings

Grep the proxy's error log for the window of the test. These are the nginx messages that map to a 502, with what each means:

# count 502 causes during the test window
grep -E 'prematurely closed|Connection refused|Connection reset|no live upstreams|too big header' \
  /var/log/nginx/error.log | sed -E 's/.*\] [0-9#]+: \*[0-9]+ //' | cut -c1-80 | sort | uniq -c | sort -rn

AWS Application Load Balancer

The ALB does not have an error log, but it has two useful signals. In CloudWatch, HTTPCode_ELB_502_Count rising while HTTPCode_Target_5XX_Count stays flat means the load balancer generated the 502, not your app. TargetConnectionErrorCount rising means the ALB could not open connections to targets. In the access logs, a line with elb_status_code 502 and target_status_code of - means the target never returned a response on that connection.

Cloudflare

If your origin returns a 502, Cloudflare passes it through. When Cloudflare itself cannot get a valid response from the origin it uses its own 52x codes instead: 520 (empty or unexpected response, the Cloudflare equivalent of "prematurely closed"), 521 (connection refused), 522 (TCP connect timed out). See load testing behind Cloudflare for the full list.

Kubernetes ingress-nginx

Same nginx strings, in the controller pod logs: kubectl logs -n ingress-nginx deploy/ingress-nginx-controller | grep -c 'prematurely closed'.

Cause 1: upstream crashes, restarts and OOM kills

If 502s come in bursts, look for a process dying at the same timestamp. The burst size roughly equals the number of requests that were in flight on the dead process.

# kernel OOM killer (bare metal / VM)
dmesg -T | grep -iE 'out of memory|killed process'
#  Out of memory: Killed process 23817 (node) total-vm:..., anon-rss:...

# Kubernetes
kubectl get pods -w                       # RESTARTS column climbing during the test
kubectl describe pod api-7d9f... | grep -A3 'Last State'
#  Last State: Terminated
#    Reason:   OOMKilled
#    Exit Code: 137

# Gunicorn worker killed for exceeding --timeout
#  [CRITICAL] WORKER TIMEOUT (pid:4127)

Fixes depend on which one you found:

• OOM: measure memory per request under load (concurrency multiplies buffers, ORM result sets, JSON serialization). Either raise the memory limit or cap concurrency per process. For Node, set --max-old-space-size below the container limit so V8 collects garbage before the kernel kills the process.
• Worker timeout: a Gunicorn sync worker that takes longer than --timeout (default 30 s) is killed, and every request on it becomes a 502. The real problem is the slow request; raising the timeout only hides it.
• Liveness probes: a probe that hits the same saturated worker pool will fail under load and restart a healthy-but-busy container, turning a slowdown into an outage. Give liveness probes a generous timeoutSeconds and failureThreshold, and keep the expensive checks in the readiness probe.

Cause 2: keep-alive idle-timeout mismatch

This is the most misdiagnosed 502. The rule: the server side of a connection must keep idle connections open longer than the client side. The proxy is the client of your app, so the app's idle timeout must exceed the proxy's.

When the order is wrong, this happens: the app closes an idle connection after 5 s; at almost the same instant the proxy picks that connection from its pool and writes a request into it. The request lands on a socket that is closing, the proxy gets a FIN or RST, and returns 502. It is a race, so the rate is low and random, and it gets more likely at moderate load (lots of pooled connections that go briefly idle) rather than at peak.

Node behind an ALB is the classic case: the ALB will hold a connection idle for up to 60 s, Node closes it after 5 s. Fix it in the app:

// Node.js: app must outlive the ALB's 60 s idle timeout
const server = app.listen(3000);
server.keepAliveTimeout = 65_000;   // > ALB idle timeout
server.headersTimeout   = 66_000;   // must be > keepAliveTimeout

# Gunicorn behind an ALB
gunicorn app:app --workers 4 --keep-alive 65

If nginx sits between the ALB and the app and pools upstream connections, set nginx's upstream idle timeout shorter than the app's:

upstream app {
    server 127.0.0.1:3000;
    keepalive 64;              # idle connections cached per worker
    keepalive_timeout 30s;     # shorter than the app's keepAliveTimeout
}
server {
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

nginx retries a failed request on the next upstream (proxy_next_upstream error timeout is the default), but since 1.9.13 it does not retry non-idempotent requests such as POST unless you add non_idempotent. That is why keep-alive 502s often show up only on POST endpoints in your load test results.

Cause 3: worker and connection exhaustion

When every worker is busy, new connections queue in the kernel's listen backlog. When the backlog fills, a TCP socket usually stops completing handshakes (which looks like a timeout), while a Unix socket returns EAGAIN immediately, which nginx turns into a 502. PHP-FPM over a Unix socket is the textbook example:

# php-fpm log
WARNING: [pool www] server reached pm.max_children setting (20), consider raising it

# check the listen queue on the socket
ss -xl | grep php
#  u_str LISTEN 129 128 /run/php/php-fpm.sock ...   (Recv-Q 129 > backlog 128 = full)

Raising pm.max_children is only correct if you have the memory: children × average RSS must fit in RAM with headroom. Otherwise you trade 502s for OOM kills. Size it from measurement:

# average PHP-FPM child RSS in MB
ps -o rss= -C php-fpm8.2 | awk '{s+=$1; n++} END {print s/n/1024 " MB avg, " n " procs"}'

Little's Law gives you the number of workers you actually need: at 200 RPS and 150 ms average service time, 200 × 0.15 = 30 requests are in service at any moment, so 20 children will saturate. See Little's Law for load testing for the full method. Also check listen.backlog in the pool config and net.core.somaxconn; the effective backlog is the smaller of the two.

If the server side is fine but you see refused connections everywhere, also rule out file-descriptor limits: "too many open files" errors on the app make accept() fail.

Cause 4: deploys and autoscaling during the test

If 502s line up with a rollout or a scale-in, the cause is connection draining. In Kubernetes, the pod receives SIGTERM while endpoints are still being removed from the load balancer and kube-proxy, so for a few seconds new requests still arrive at a process that is shutting down. The common fix is a short preStop sleep so the pod keeps serving until it has been removed everywhere:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "15"]
terminationGracePeriodSeconds: 45

On an ALB, check the target group's deregistration delay is longer than your slowest request. In LoadTester you can push a deploy marker to the test timeline (POST /api/annotations/push) so a 502 burst can be lined up against the rollout instead of guessed at.

Diagnosis table

How to re-test and confirm the fix

Do not re-run the same spike and eyeball the result. Use a test shape that isolates the cause you fixed:

• Keep-alive fix: run a steady, moderate load (say 30-50% of capacity) for 10 minutes with variable think time, so pooled connections regularly go idle. The 502 count should drop to exactly zero, not "lower". Any remaining 502 means another hop still has the wrong ordering.
• Crash/OOM fix: step-ramp in fixed-RPS mode (for example 50, 100, 150, 200 RPS, 2 minutes each) and watch memory per process and restart count alongside the error rate. The step where 502s used to start should now pass with zero restarts.
• Worker exhaustion: same step-ramp; the expected change is that overload now shows up as rising latency first (queueing) instead of an immediate jump in 502s. Track p95 together with errors, as explained in p95 vs p99 latency.

Use an arrival-rate test rather than a fixed number of VUs for this. With closed-loop VUs, the generator slows down as soon as the backend does, so you may never reach the request rate at which the crash happened (see open vs closed workload models). In LoadTester, an error-rate threshold such as "errors < 1%" with auto-stop ends the run as soon as 502s cross the line, which keeps a regression test from hammering a broken deploy for the full duration.