504 Gateway Timeout Under Load

A 504 Gateway Timeout under load means a proxy or load balancer waited for your application's response longer than its timeout (60 s by default for nginx's proxy_read_timeout and the AWS ALB idle timeout, 100 s for Cloudflare, which reports it as 524) and gave up. Your app is still alive but requests are spending too long queued or blocked: usually waiting for a database connection, a lock, or a free worker. The fix is to find what the requests are waiting on and then order the timeouts so each inner layer gives up before the layer in front of it.

What a 504 means, and how it differs from a 502

A 504 is produced by the gateway, not by your app. The gateway connected to the upstream, sent the request, and then nothing came back within the configured time. Compare that with a 502 Bad Gateway, where the connection was refused, closed, or returned garbage. As a rule of thumb:

• 502: something died or a connection broke. Usually fast to fail.
• 504: something is slow. Fails at exactly the timeout value, so the latency of failed requests clusters at 60.0 s, 100 s, or whatever your limit is.

That clustering is your first diagnostic. Plot the response time of the 504s from your load test. If they all sit at one value, you know which hop timed out. If some are at 30 s and some at 60 s, two different layers are timing out.

Why 504s appear only under load: queueing

An endpoint that answers in 80 ms at idle can produce 60-second timeouts at peak without a single slow query, because of queueing. Little's Law says the number of requests in the system equals arrival rate times time in the system. Once arrivals exceed the rate a resource can serve, the queue grows without bound and wait time grows with it.

A worked example: a service has 20 database connections and each query holds a connection for 50 ms. Maximum throughput through the pool is 20 / 0.05 = 400 queries per second. At 380 QPS the pool is at 95% utilization and requests wait noticeably. At 450 QPS, 50 extra requests per second pile up; after 60 seconds, 3,000 requests are queued, the newest one would wait about 7.5 s for a connection, and the queue keeps growing. Nothing is "slow", yet every request eventually crosses the 60 s line. For the math behind this curve see why latency increases under load and Little's Law for load testing.

Most likely causes, ranked

1. Connection pool exhaustion (DB, Redis, HTTP client to another service). Requests wait for a pooled connection; the pool timeout is often longer than the proxy timeout.
2. Slow queries and lock contention. A query that is fine alone becomes slow when 200 copies run at once, or a hot row lock serializes writes.
3. Worker/thread pool saturation. All app workers are busy, so new requests queue in the accept backlog or the server's internal queue.
4. Slow downstream dependency. A third-party API or internal service with its own capacity limit, called without a short timeout.
5. Mis-ordered timeouts. Not a root cause, but it decides whether users see a clean, fast error or a 60 s hang followed by a 504.

How to tell which one it is

Confirm which hop timed out

# nginx error log, the message behind a 504
upstream timed out (110: Connection timed out) while reading response header from upstream,
  client: 203.0.113.7, server: api.example.com, request: "GET /orders HTTP/1.1",
  upstream: "http://10.0.3.14:8080/orders"

# "while connecting to upstream" instead means the TCP connect itself timed out
# (proxy_connect_timeout), which points at a full backlog or network problem.

On an ALB, HTTPCode_ELB_504_Count rises and the access log shows elb_status_code 504 with target_status_code - because the target never returned a response within the idle timeout (the target-side timing fields are typically logged as -1).

Look for pool-exhaustion errors in the app

Most pools log a recognizable message when a caller gives up waiting:

If you see none of these but the pool timeout is 30 s and the proxy timeout is 60 s, check whether requests are already being cut off by the proxy before the pool error can fire. That is the timeout-ordering problem in action.

Check the database while the test runs

-- PostgreSQL: who is waiting, and on what
SELECT wait_event_type, wait_event, state, count(*)
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY 1,2,3 ORDER BY 4 DESC;

-- Lock waits specifically
SELECT pid, now() - query_start AS waiting, left(query, 80)
FROM pg_stat_activity
WHERE wait_event_type = 'Lock'
ORDER BY waiting DESC;

Many rows with wait_event_type = 'Lock' means contention; many active sessions with wait_event_type of IO or no wait event mean the queries themselves are expensive. Many idle in transaction sessions mean the app holds connections while doing other work, which starves the pool. On MySQL, SHOW PROCESSLIST and performance_schema.data_lock_waits (8.0) give the same picture.

Check worker saturation

If the DB looks calm, the queue is in front of the app. Signs: app CPU near 100% on every core, or (for thread-per-request servers) active threads pinned at the maximum. For Node, event-loop lag above tens of milliseconds means the single thread is saturated. For a listen backlog filling up, see the ss -ltn checks in connection reset by peer under load.

The timeout ladder: inner layers must give up first

Every layer between the user and the database has a timeout. The rule is that timeouts should get shorter as you move inward:

client > CDN / load balancer > reverse proxy > app request timeout > DB statement timeout

If an outer layer times out first, it returns a 504 to the user while the inner layers keep working on a request nobody will receive. Under load that is the worst possible behavior: the wasted work keeps the pool busy, so the next request waits longer, times out too, and the system collapses into 100% 504s with the database still at full CPU. When the inner layer times out first, the request fails quickly with a specific error, the connection returns to the pool, and the app can return a deliberate 503 or a fallback.

# nginx
location /api/ {
    proxy_pass http://app;
    proxy_connect_timeout 5s;    # TCP connect; default 60s
    proxy_read_timeout   30s;    # between two reads, not total; default 60s
    proxy_send_timeout   30s;
}

-- PostgreSQL, per role so reporting jobs can keep longer limits
ALTER ROLE app_user SET statement_timeout = '8s';
ALTER ROLE app_user SET lock_timeout = '2s';
ALTER ROLE app_user SET idle_in_transaction_session_timeout = '30s';

# HikariCP (Spring Boot)
spring.datasource.hikari.maximum-pool-size=20
spring.datasource.hikari.connection-timeout=3000   # ms to wait for a connection

Note that nginx's proxy_read_timeout is the time between two successive reads, not the total request time. A response that streams a byte every 50 s will never trip a 60 s timeout.

Cloudflare: 524 vs 504

Behind Cloudflare, a timeout at the edge does not look like a 504. If Cloudflare opened a connection to your origin and received no HTTP response within 100 seconds, it returns 524 A Timeout Occurred. If the TCP connection to the origin could not be established at all, you get 522 Connection Timed Out. A 504 seen through Cloudflare was most likely generated by your own infrastructure (an ALB or nginx behind Cloudflare timed out first at 60 s) and passed through. So a load test returning 504s through Cloudflare points at your origin's proxy, while 524s point at a request that took over 100 s end to end. More in load testing behind Cloudflare.

Fixes that remove the queue, not just the error

• Size the pool from Little's Law, then check the DB can take it. Required connections = peak queries per second × average hold time. 400 QPS × 0.05 s = 20 connections busy on average, so a pool of 20 is at 100% utilization; give it headroom (30) and confirm the DB's max_connections covers pool size × number of app instances.
• Shorten connection hold time. Do not hold a DB connection across an HTTP call to another service. Kill idle in transaction sessions.
• Fix the expensive query. EXPLAIN (ANALYZE, BUFFERS) the top statements from pg_stat_statements ordered by total_exec_time. A missing index often shows up only under concurrency because the sequential scans compete for I/O and buffer cache.
• Remove hot-row locks. Counters and inventory rows updated by every request serialize all writers. Batch, shard the counter, or use SELECT ... FOR UPDATE SKIP LOCKED for queue-style tables.
• Fail fast on overload. Bound the queue: a short pool acquire timeout plus a 503 with Retry-After is better than a 60 s hang. Clients and load balancers can react to a fast 503; a 504 at 60 s has already burned a worker for a minute.
• Put timeouts on outbound calls. Every HTTP client to a dependency gets a timeout shorter than your own request timeout.

Diagnosis table

How to re-test and confirm

504s are a queueing problem, so test with a shape that shows the queue forming. Run a step-ramp in arrival-rate (fixed RPS) mode, for example 100, 200, 300, 400, 500 RPS, holding each step for 2-3 minutes. A closed-loop VU test hides queueing because VUs wait for slow responses and stop sending; see open vs closed workload models.

What should change after a fix:

• Pool sizing / query fixes: the step at which p95 latency starts climbing moves to a higher RPS. Errors at the old breaking step go to zero.
• Timeout ordering: at overload, the failures change from 504 at 60 s to fast 503s (or pool errors) at 2-5 s, and the DB stops being busy within seconds of the test ending.
• Lock fixes: Lock waits disappear from pg_stat_activity during the test, and throughput keeps rising with RPS instead of flattening.

Watch p95 and p99 per step rather than the run average; the average hides the step where the queue starts (see how to read load test results). In LoadTester, a threshold like p95 < 800 ms with auto-stop ends the ramp at the first step that breaches, which gives you the capacity number directly.