Database Connection Pool Exhausted Under Load

A "connection pool exhausted" error under load means every connection in your application's pool was checked out and no request gave one back before the pool's wait timeout ran out. The cause is almost always hold time, not pool size: connections held by slow queries, by transactions that wrap HTTP calls, or by leaked sessions that never return. Measure how long each request holds a connection, size the pool with Little's Law (connections = transactions per second × hold time), and make sure the sum of all pools across all replicas fits under the database's max_connections.

What the error looks like

The wording depends on the pool and on which side gave up. These are the strings people search for, grouped by where they come from.

Application-side pool (the pool ran dry)

# HikariCP (Spring Boot default pool)
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available,
  request timed out after 30000ms (total=10, active=10, idle=0, waiting=37)

# SQLAlchemy
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached,
  connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)

# node-postgres (pg.Pool with connectionTimeoutMillis set)
Error: timeout exceeded when trying to connect

Recent HikariCP versions append pool statistics to the message. active=10, idle=0, waiting=37 is the whole diagnosis in one line: all 10 connections busy, 37 threads queued behind them. The 30000ms and timeout 30.00 are the defaults: HikariCP's connectionTimeout is 30 seconds with maximumPoolSize 10, and SQLAlchemy's QueuePool defaults to pool_size=5, max_overflow=10, pool_timeout=30.

Database-side (the server ran out of slots)

FATAL:  sorry, too many clients already
FATAL:  remaining connection slots are reserved for non-replication superuser connections   (PostgreSQL 15 and older)
FATAL:  remaining connection slots are reserved for roles with the SUPERUSER attribute      (PostgreSQL 16+)
FATAL:  remaining connection slots are reserved for roles with privileges of the "pg_use_reserved_connections" role  (16+)

PostgreSQL's max_connections defaults to 100, of which superuser_reserved_connections (default 3) are kept for superusers. PostgreSQL 16 added reserved_connections (default 0) for roles granted pg_use_reserved_connections, which is why the message wording changed. If you see the "reserved" message, ordinary roles already hit the ceiling; the next connection attempt gets "too many clients already".

Most likely causes, ranked

1. Hold time grew under load. A query that takes 5 ms idle takes 80 ms when the database is busy, so each request holds its connection 16 times longer and the same pool supports 16 times less throughput. Exhaustion is the symptom; database saturation is the cause.
2. Work done while holding a connection. A transaction (or Spring's open-session-in-view, or a request-scoped session) stays open while the code calls an external API, renders a template or waits on a queue. The connection sits idle-in-transaction, unavailable to anyone.
3. Pool sized for one replica, multiplied by many. 12 pods × maximumPoolSize 20 = 240 connections against max_connections = 100. Fine at 3 pods, broken after the autoscaler adds more.
4. Leaks. A code path that gets a connection and never returns it (exception before close(), a session not closed in a background task). Each leaked connection permanently shrinks the pool, so the failure appears after N requests, not at a particular RPS.
5. Pool genuinely too small. Real, but less common than the four above. Little's Law tells you quickly.
6. Lock waits. Many requests updating the same row (a counter, an inventory row, a "last seen" timestamp) queue on a row lock while each one holds a connection.

Sizing with Little's Law

A pool is a queue with a fixed number of servers. The average number of connections in use is:

connections busy = transactions per second × average hold time (seconds)

Worked example: an endpoint receives 400 RPS, each request runs one transaction, and the connection is held for 25 ms from checkout to return. Average busy connections = 400 × 0.025 = 10. A pool of 10 would run at 100% utilization, which means queueing on every burst. Give it headroom so the average sits around 60-70% busy: 10 / 0.65 ≈ 15 connections. The connection pool size calculator does this math with replicas and database limits included, and Little's Law for load testing explains the formula.

Now apply load. If hold time rises from 25 ms to 120 ms because the database is busy, the same 400 RPS needs 48 connections. No reasonable pool covers that, and adding connections to a saturated database makes each query slower still. That feedback loop is why "just raise the pool size" so often makes things worse.

How to confirm each cause

Measure hold time and wait time from the pool

HikariCP exposes Micrometer metrics in Spring Boot: hikaricp_connections_active, hikaricp_connections_pending, hikaricp_connections_acquire_seconds (wait to get one) and hikaricp_connections_usage_seconds (hold time). Plot usage time against RPS during the test. If usage p95 climbs with load, the database or the code inside the transaction is slowing down. If pending rises while usage stays flat, the pool really is too small.

For SQLAlchemy, log the pool state periodically during the run:

import logging, threading
def log_pool(engine):
    logging.info("pool %s", engine.pool.status())  # "Pool size: 5  Connections in pool: 0 Current Overflow: 10 Current Checked out connections: 15"
    threading.Timer(5, log_pool, [engine]).start()

Look at what the database thinks the connections are doing

-- Connection count by state, per application
SELECT application_name, state, count(*)
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY 1, 2 ORDER BY 3 DESC;

-- Who is holding a transaction open while doing nothing
SELECT pid, application_name, now() - xact_start AS xact_age, now() - state_change AS idle_for, left(query, 80)
FROM pg_stat_activity
WHERE state = 'idle in transaction'
ORDER BY xact_age DESC LIMIT 20;

-- Lock waits
SELECT pid, wait_event_type, wait_event, left(query, 80)
FROM pg_stat_activity WHERE wait_event_type = 'Lock';

Many rows in idle in transaction with ages of hundreds of milliseconds or more point to cause 2: code holding a transaction across non-database work. Many active rows with wait_event_type = 'Lock' point to row contention. Mostly active with no waits and high database CPU means the database is saturated, so look at pg_stat_statements for the top queries by total_exec_time.

Check for leaks

Turn on HikariCP leak detection during the test (the lowest accepted value is 2000 ms; the default 0 disables it):

spring.datasource.hikari.leak-detection-threshold=10000

Any connection held for longer than 10 s produces a Connection leak detection triggered warning with the stack trace of the code that borrowed it. The other signature of a leak: after the load test ends and traffic drops to zero, active stays above zero and never recovers until restart.

Add up every pool

SHOW max_connections;
SELECT count(*) FROM pg_stat_activity;   -- current total
-- then: replicas x pool max + migrations + cron jobs + admin tools <= max_connections - reserved

Fixes, with config

Shorten hold time first

• Move HTTP calls, file I/O and message publishing outside the transaction. Read what you need, commit, then do the slow work.
• In Spring Boot, disable open-session-in-view so a connection is not held for the whole request: spring.jpa.open-in-view=false.
• Fix the slow queries that pg_stat_statements shows (missing indexes, N+1 patterns). Halving query time halves the connections needed.
• Kill abandoned transactions at the server so a bug cannot hold connections forever: ALTER ROLE app SET idle_in_transaction_session_timeout = '30s';

Size the pool from the math, and cap total connections

# Spring Boot / HikariCP: example for ~400 RPS at 25 ms hold time per replica
spring.datasource.hikari.maximum-pool-size=15
spring.datasource.hikari.minimum-idle=15
spring.datasource.hikari.connection-timeout=2000   # fail fast, below the HTTP timeout
spring.datasource.hikari.max-lifetime=1500000      # a little under any DB/LB connection limit

# SQLAlchemy
engine = create_engine(url, pool_size=15, max_overflow=0, pool_timeout=2, pool_pre_ping=True)

A short acquisition timeout does not cure exhaustion, but it changes the failure from a 30-second hang (which upstream turns into a 504 gateway timeout) into a fast, countable error. Also match the worker pool: if Tomcat runs 200 threads against a pool of 15, 185 threads can queue for connections. That may be what you want, but decide it deliberately.

Put PgBouncer in front when replicas multiply

When many replicas each need a pool, a transaction-mode pooler lets thousands of client connections share a small number of server connections. PgBouncer's defaults are pool_mode = session, default_pool_size = 20 and max_client_conn = 100, so a fresh install gives you none of the multiplexing until you change the mode:

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 40        ; server connections per user/database pair
reserve_pool_size = 5
reserve_pool_timeout = 3
max_prepared_statements = 200 ; protocol-level prepared statements in transaction mode

Transaction mode returns the server connection at commit, so session state does not survive: SET, session advisory locks, LISTEN and temp tables break. Prepared statements work in transaction mode only on recent PgBouncer versions with max_prepared_statements set. Watch SHOW POOLS; in the admin console: cl_waiting above zero and a growing maxwait mean clients are queued in PgBouncer instead.

Fix contention at the data model

For hot-row updates, batch increments, shard the counter across N rows, or move it to Redis. Keep lock-holding transactions as short as the code allows.

Diagnosis table

How to re-test and confirm the fix

Re-run with a fixed arrival rate, not a fixed number of virtual users. Closed-loop VUs slow down when responses slow down, so they back off just as the pool fills and hide the problem; an open model keeps pushing the RPS that caused it (see open vs closed workload models). Use a step-ramp, for example 100, 200, 300, 400, 500 RPS for 2 minutes each, and record at each step:

• Pool active, pending and connection usage time p95.
• Database CPU and the count of idle in transaction sessions.
• Request p95 and error rate.

What should change: usage time should stay roughly flat until the database itself nears saturation, pending should stay at zero through your target RPS, and when you do push past capacity the failure should appear as rising latency followed by fast acquisition-timeout errors, never as 30-second hangs. After the run, active must drop to zero within seconds. If the problem was the sum of pools, repeat the test with the replica count your autoscaler reaches at peak, not the count you start with. In LoadTester you can set a threshold such as p95 < 400 ms with auto-stop so a regression run ends as soon as the pool starts queuing; the latency under load guide covers how to find where that knee sits.