Connection Reset by Peer During Load Testing

"Connection reset by peer" (ECONNRESET, or "socket hang up" in Node.js) means the other end of a TCP connection sent a RST packet and tore the connection down, usually before a response arrived. Under load the most common causes are a keep-alive race where the server closes an idle connection just as the client reuses it, a process crashing or restarting, a proxy, load balancer, rate limiter or WAF deliberately resetting connections, and kernel queues or tables overflowing. Capture who sent the RST with tcpdump, check netstat -s and dmesg, and the cause is usually clear within minutes.

What a reset actually is

A normal TCP close is a FIN handshake: each side says "I am done sending." A reset is an abort: one side sends a RST segment, and the other side's next read or write fails immediately. The kernel sends a RST when:

• A packet arrives for a connection or port that does not exist (the process died, or the port is closed).
• An application closes a socket that still has unread data in its receive buffer.
• An application closes with SO_LINGER set to zero, which some proxies and rate limiters do on purpose.
• A middlebox (load balancer, firewall, WAF) decides to kill the connection.

How it looks in different clients:

A close cousin is EPIPE / "broken pipe": you wrote to a connection the peer had already reset. Same causes, different timing.

Most likely causes, ranked

1. Keep-alive race at the idle timeout. Low, random reset rate at moderate load, often only on the first request after a pause.
2. The server process crashed, restarted or was OOM-killed. Resets in bursts, matching restart timestamps.
3. Deliberate resets by a rate limiter, WAF, or connection limit on a load balancer, ingress or CDN. Resets start at a sharp threshold of RPS or connections, often only from the generator's IPs.
4. Server closes early with unread request data, for example rejecting a large upload with 413 and closing before reading the body.
5. Kernel queue and table overflows: listen backlog, SYN backlog, orphaned sockets, conntrack. These mostly cause drops and timeouts, but can produce resets depending on settings.
6. The load generator itself: a client with its own idle timeout or connection pool limit that aborts connections.

Step one: find out who sent the RST

Do not guess. Capture resets on the client and on the server at the same time. If the server's capture shows it sending the RST, the server (or its kernel) did it. If the client sees a RST that the server never sent, something in between did: a load balancer, NAT gateway, firewall or WAF.

# capture only RST packets on port 443, with timestamps
tcpdump -ni any 'tcp port 443 and (tcp[tcpflags] & tcp-rst != 0)' -c 200

# typical output
# 14:02:11.381 IP 10.0.2.8.443 > 10.0.1.4.51234: Flags [R], seq 39182..., win 0, length 0

Then look at the packets just before the RST for that same connection (capture the full flow with -w and open it in Wireshark). A RST right after the client sends a request on a connection idle for exactly N seconds is the keep-alive race. A RST after the client sends a request body larger than some limit points at early close. A RST as the very first response to a SYN means nothing was listening.

Cause 1: the keep-alive race

The client keeps a pool of idle keep-alive connections. The server closes connections idle longer than its keep-alive timeout. If the client's idle timeout is equal to or longer than the server's, there is a window where the client picks a connection the server is closing at that exact moment. The request arrives at a closed socket, the server's kernel answers with RST, and the client sees ECONNRESET.

The rule: the client's idle timeout must be shorter than the server's. Applied to every hop, that means idle timeouts get longer as you go deeper toward the app. Defaults that commonly break this:

• Node.js server.keepAliveTimeout: 5 s. Behind an AWS ALB (60 s idle), the ALB is the client and outlives the server. Set keepAliveTimeout to 65 s and headersTimeout slightly higher.
• Gunicorn --keep-alive: 2 s. Same problem behind any proxy that pools connections.
• HTTP clients in your app (calling other services) with idle timeouts longer than the downstream server's. Configure the client pool to drop idle sockets earlier, for example 4 s against a 5 s server.

The same race produces 502s when the client is a proxy; the 502 under load guide has the nginx upstream config. A tell-tale sign of this cause: the reset rate is higher at moderate, bursty load than at full, continuous load, because connections go idle more often.

Cause 2: listen and SYN backlog overflow

When connections arrive faster than the application calls accept(), they queue in the kernel. Two queues matter: the SYN queue (half-open connections, sized by net.ipv4.tcp_max_syn_backlog) and the accept queue (completed connections waiting for the app, sized by the smaller of the app's listen() backlog and net.core.somaxconn).

What happens on overflow depends on settings. With the default net.ipv4.tcp_abort_on_overflow = 0, the kernel silently drops the client's final ACK when the accept queue is full, so the client sees retransmits and slow connects (often stepping at 1 s and 3 s) rather than resets. With tcp_abort_on_overflow = 1 it sends a RST instead. Either way, check the counters:

# cumulative counters since boot; run before and after the test and compare
netstat -s | grep -i listen
#   18342 times the listen queue of a socket overflowed
#   18342 SYNs to LISTEN sockets dropped

# or with nstat (shows deltas since last call)
nstat -az TcpExtListenOverflows TcpExtListenDrops

# per listening socket: Recv-Q = current accept queue, Send-Q = its limit
ss -ltn 'sport = :443'
#  State   Recv-Q  Send-Q  Local Address:Port
#  LISTEN  512     511     0.0.0.0:443

# SYN floods at high connection rates also show up in the kernel log
dmesg -T | grep -i 'syn flooding'
#  TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies.

Fix by raising both the kernel limit and the application's own backlog; the effective value is the smaller of the two.

# /etc/sysctl.d/90-backlog.conf
net.core.somaxconn = 8192            # default 4096 since kernel 5.4, 128 before
net.ipv4.tcp_max_syn_backlog = 8192

# nginx: backlog defaults to 511 on Linux
listen 443 ssl backlog=8192;

// Node.js: listen(port, host, backlog), default 511
server.listen(3000, '0.0.0.0', 8192);

A bigger backlog only buys time during bursts. If the accept queue is full continuously, the app is out of capacity and needs more workers or instances.

Cause 3: conntrack and other kernel tables

Hosts using iptables/nftables with stateful rules, Docker's NAT, or Kubernetes kube-proxy in iptables mode track every connection in the conntrack table. When it fills, new packets are dropped:

dmesg -T | grep conntrack
#  nf_conntrack: nf_conntrack: table full, dropping packet

# usage vs limit
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
conntrack -S | head          # per-CPU stats: look at drop, insert_failed, early_drop

Dropped packets mostly surface as connect timeouts and retransmits, but connections whose state is lost mid-stream can be answered with resets, so mixed timeouts and resets on a NAT'd host should send you here. Raise the limit and shorten how long finished connections are tracked:

net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

Two other kernel messages that mean the kernel is resetting connections itself: TCP: too many orphaned sockets (raise net.ipv4.tcp_max_orphans or find why so many connections are closing without being fully shut down) and TCP: out of memory -- consider tuning tcp_mem. For a quick overview of socket state during the test, ss -s shows totals for established, orphaned and time-wait sockets in one line.

Cause 4: rate limiters, WAFs and connection limits

A load test from a few IPs looks a lot like an attack. Many edge components respond by closing connections rather than returning a status code:

• nginx return 444; closes the connection without a response, which clients report as "empty reply" or "socket hang up".
• Some WAFs and DDoS protection layers send RSTs to clients that exceed a connection or request rate.
• Load balancers and ingress controllers with per-client connection caps, and cloud firewalls with connection-tracking limits, reset or drop new connections above the cap.

Signals: resets begin at a clean threshold (say, exactly at 100 RPS per source IP), the server's own capture shows no RST, and requests that do succeed are fast. Check the WAF or CDN event log for the generator's IPs. For Cloudflare-fronted sites see load testing behind Cloudflare; for testing the limiter itself on purpose, see rate limit testing. LoadTester publishes its worker IPs at https://app.loadtester.org/api/ips so you can allowlist them in a WAF rule before a test.

Cause 5: early close with unread data

If the server rejects a request (auth failure, 413 body too large) and closes the socket without reading the rest of the request body, the kernel sends a RST because unread data is discarded. The client, still uploading, gets ECONNRESET or EPIPE and may never see the status code. This shows up in load tests of upload or large-POST endpoints. Fix it on the server by draining the body (or a bounded amount of it) before closing, and on the client by sending Expect: 100-continue for large bodies.

Diagnosis table

How to re-test and confirm

Pick the test shape that exercises the cause you fixed:

• Keep-alive race: moderate, bursty load with think time longer than the old server idle timeout (for example 6-10 s against a 5 s server) for 10 minutes. Reset count should go to zero. A pacing calculator such as the think time and pacing calculator helps you set this up deliberately.
• Backlog / SYN overflow: a spike test that jumps from baseline to 3-5x in seconds. Compare nstat deltas for TcpExtListenOverflows before and after the fix; the delta should be zero and connect time should stay flat during the spike.
• Conntrack: run long enough to fill the table at the old settings (high connection churn without keep-alive), and watch nf_conntrack_count plateau well below nf_conntrack_max.
• WAF / rate limits: after allowlisting, re-run the same ramp and confirm the error rate no longer jumps at the old threshold; the WAF log should show the test IPs matched by your skip rule.

Read errors by type, not as one percentage. A run with 0.3% resets and 0% 5xx tells a very different story from the reverse; how to read load test results covers separating them.