"Too Many Open Files" During Load Testing

"Too many open files" (EMFILE, errno 24) means a process hit its per-process file-descriptor limit, and every TCP socket counts as a file. On Linux the default soft limit is 1024, so a load generator or server handling more than about a thousand concurrent connections fails at socket(), accept() or open(). Find which side is failing, check the real limit in /proc/<pid>/limits, and raise it where that process is started (shell, systemd unit, container runtime or nginx config), not only in /etc/security/limits.conf.

What EMFILE and ENFILE mean

Every open file, socket, pipe, epoll instance and eventfd a process holds uses one file descriptor. The kernel enforces two different limits:

• EMFILE (errno 24), "Too many open files": this process reached its own RLIMIT_NOFILE. By far the common case.
• ENFILE (errno 23), "Too many open files in system": the whole machine reached fs.file-max. Rare on modern kernels, where the default is very large.

During a load test, descriptors are mostly sockets. One HTTP/1.1 keep-alive connection is one fd on the client and one on the server. A reverse proxy uses two per proxied request (client side plus upstream side). Add log files, listening sockets, pipes and epoll instances, and a process with a 1024 limit runs out somewhere around 900-1000 concurrent connections.

First, find out which side is failing

The same message can come from the load generator or from the system under test, and the fix is in different places. The error text tells you which.

A simple rule: if the error is on accept, it is the server. If it is on connect, dial or socket while opening outbound connections, it is the client, which is either your load generator or your app calling a dependency.

When the generator fails, your results are invalid, not just noisy. The generator records errors that never reached the server, and the error rate you report says nothing about the system you meant to test.

Check the limit the process actually has

ulimit -n in your shell only shows your shell's limit. A daemon started by systemd, Docker or a supervisor has its own. Always check the running process:

# the limits the running process really has
cat /proc/$(pgrep -o nginx)/limits | grep 'open files'
#  Max open files            1024                 524288               files
#                            ^ soft               ^ hard

# how many it is using right now
ls /proc/$(pgrep -o nginx)/fd | wc -l

# watch it during the test, per process
watch -n1 'for p in $(pgrep nginx); do echo "$p $(ls /proc/$p/fd | wc -l)"; done'

# system-wide: allocated, unused (always 0 on modern kernels), max
cat /proc/sys/fs/file-nr
#  14560   0   9223372036854775807

lsof -p <pid> | wc -l works too but overcounts, because it also lists memory-mapped files and the current directory that do not consume descriptors. Counting /proc/<pid>/fd is exact.

The soft limit is what is enforced. The hard limit is the ceiling an unprivileged process may raise its own soft limit to. On most current systemd distributions the default is soft 1024, hard 524288 for services; many runtimes (Go since 1.19, the JVM by default on Linux) raise their soft limit to the hard limit at startup, which is why a Java service may show 524288 while a Python one shows 1024.

Raise it in the place that starts the process

Interactive shells and SSH sessions

# current shell only, up to the hard limit
ulimit -n 65536

# persistent for login sessions: /etc/security/limits.conf (applied by pam_limits)
loadtest   soft   nofile   65536
loadtest   hard   nofile   65536

limits.conf only applies to sessions that go through PAM (logins, SSH, su). It does not affect systemd services, which is the most common reason "I raised the limit and nothing changed".

systemd services

# systemctl edit nginx   (creates a drop-in override)
[Service]
LimitNOFILE=65536

# then
systemctl daemon-reload && systemctl restart nginx
cat /proc/$(pgrep -o nginx)/limits | grep 'open files'

nginx: two separate limits

nginx has its own connection cap independent of the OS limit. worker_connections is the maximum simultaneous connections per worker, and it counts upstream connections too, so a proxying worker serves roughly worker_connections / 2 clients. worker_rlimit_nofile lets the master raise each worker's fd limit without touching systemd.

worker_processes auto;
worker_rlimit_nofile 65536;      # per-worker RLIMIT_NOFILE

events {
    worker_connections 16384;    # stock configs often ship 768 or 1024
}

Keep worker_rlimit_nofile comfortably above worker_connections (about 2x when proxying, since cache files and logs also use descriptors).

Docker and Kubernetes

# docker run
docker run --ulimit nofile=65536:65536 myimage

# docker compose
services:
  api:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

Kubernetes has no per-pod ulimit field. Containers inherit the container runtime's defaults (configured in containerd or CRI-O on the node), and those defaults have changed between versions. Check from inside the pod with kubectl exec <pod> -- sh -c 'ulimit -n', and either fix it at the runtime level or raise the soft limit in the entrypoint up to the hard limit.

System-wide ceilings

# /etc/sysctl.d/90-fd.conf
fs.file-max = 2097152     # total fds on the host (ENFILE)
fs.nr_open  = 1048576     # max any single process's hard limit may be set to (default 1048576)

sysctl --system

On current kernels fs.file-max is usually already huge, so you rarely need to touch it. You only need fs.nr_open if you want a per-process limit above 1,048,576.

How many descriptors you actually need

Size the limit from the concurrency you expect, not from a number copied from a blog. A worked example for an nginx reverse proxy in front of an app:

• Target: 20,000 concurrent client connections (for example, 20,000 VUs each holding a keep-alive connection).
• nginx with 4 workers: 5,000 client connections per worker, plus up to 5,000 upstream connections when every request is in flight, so worker_connections needs at least 10,000; set 16,384 for headroom.
• worker_rlimit_nofile: 32,768 or more.
• The app behind it only sees nginx's upstream connections. With an upstream keepalive pool, that is far fewer than 20,000.

To estimate concurrent connections from a request rate, use Little's Law: connections in use ≈ RPS × average connection hold time. 2,000 RPS at 250 ms is 500 busy connections; a keep-alive pool holds more, idle, on top. The concurrent users calculator does this arithmetic.

The related trap: ephemeral port exhaustion

Once you raise the fd limit, the next wall on the client side is often source ports. Every outbound TCP connection needs a unique tuple of (source IP, source port, destination IP, destination port). When one client IP opens many connections to the same destination, it runs out of source ports and connect() fails with EADDRNOTAVAIL:

connect: cannot assign requested address          # errno 99, EADDRNOTAVAIL
dial tcp 10.0.0.5:443: connect: cannot assign requested address   # Go
Error: connect EADDRNOTAVAIL 10.0.0.5:443           # Node.js

The math with Linux defaults: net.ipv4.ip_local_port_range is 32768 60999, which is 28,232 ports. A connection closed by the client sits in TIME_WAIT for 60 seconds (a kernel constant; tcp_fin_timeout does not change it). So one client IP opening a new connection per request to a single destination sustains roughly 28,232 / 60 ≈ 470 new connections per second before it runs out.

# how many sockets are in TIME_WAIT
ss -tan state time-wait | wc -l
ss -s                      # summary, including timewait count

# current range
sysctl net.ipv4.ip_local_port_range

Fixes, in order of preference:

1. Use keep-alive. A generator or proxy that reuses connections needs a handful of ports, not thousands. This is also the realistic choice for most browser and API traffic.
2. Fix proxy-to-upstream connections. nginx without an upstream keepalive pool opens a new connection to the app per request, so the proxy host exhausts ports even when clients use keep-alive. Add keepalive, proxy_http_version 1.1 and proxy_set_header Connection "".
3. Widen the range: net.ipv4.ip_local_port_range = 10240 65535 (make sure no local service listens in that range).
4. Allow TIME_WAIT reuse for outgoing connections: net.ipv4.tcp_tw_reuse = 1 (the default, 2, enables it only for loopback). It relies on TCP timestamps. Do not look for tcp_tw_recycle; it was removed in Linux 4.12.
5. Add source IPs or generator machines, which multiplies the available tuples.

On the load generator side

Tools like k6, wrk, Gatling, Locust and JMeter all hit this when you push concurrency on one machine, and their docs recommend raising ulimit -n before large runs. A checklist for a self-run generator box:

ulimit -n 250000
sysctl -w net.ipv4.ip_local_port_range="10240 65535"
sysctl -w net.ipv4.tcp_tw_reuse=1
# then confirm from inside the tool's process, not just the shell
cat /proc/$(pgrep -n k6)/limits | grep 'open files'

Also watch the generator's CPU: a generator above roughly 80% CPU starts adding its own latency to your results (see distributed load testing for when to split the load across machines). A hosted generator such as LoadTester removes this half of the problem, since descriptor limits, port ranges and worker capacity on the sending side are managed for you; your tuning is then limited to the server side.

Diagnosis table

How to re-test and confirm

Run a concurrency step-ramp that crosses the old ceiling: for example VU steps of 500, 1,000, 2,000, 4,000, each held for 2 minutes, while watching the fd count of the server process (and of the generator if you run your own). Expected results after the fix:

• No EMFILE or EADDRNOTAVAIL errors at the step that previously failed.
• The fd count tracks concurrency and plateaus at each step, then falls back after the test ends.
• The new bottleneck, if any, shows up as rising latency or CPU rather than instant connection errors. That is the real capacity limit and the number worth reporting; see how many users can my website handle.

For port exhaustion, run a fixed-RPS test above 470 new connections per second with keep-alive disabled, then enabled, and compare the ss -s TIME_WAIT counts. With keep-alive the count should stay in the low hundreds.