How to Load Test a FastAPI App

To load test a FastAPI app, first check how each endpoint runs. async def endpoints run on the worker's single event loop, so any blocking call inside them stalls every request on that worker. Plain def endpoints run in a threadpool limited to 40 threads by default. Then size Uvicorn workers to CPU cores, and keep the SQLAlchemy pool (5 + 10 overflow per worker by default) under your database's connection limit. While the test runs, compare latency with CPU. High latency at low CPU means something is blocking or a pool is waiting, not that you need more servers.

How FastAPI handles concurrent requests

FastAPI runs on Starlette under an ASGI server, usually Uvicorn. Each worker process has one asyncio event loop. FastAPI decides how to call your code based on how you declare it:

• async def endpoints and dependencies are awaited directly on the event loop. While one awaits I/O, the loop serves other requests. Thousands of concurrent requests per worker are possible if all I/O is async.
• def endpoints and dependencies are run, per the FastAPI docs, "in an external threadpool that is then awaited". That pool is AnyIO's default thread limiter, which allows 40 threads at once per process. The 41st concurrent sync call waits for a free thread.

Python's GIL still applies. Only one thread runs Python bytecode at a time per process, so neither model makes CPU-bound code faster. For CPU work you need more processes.

What usually breaks first

Blocking calls inside async def

This is the most common FastAPI performance bug, and a load test shows it right away. Code like this looks harmless:

@app.get("/quote")
async def quote(symbol: str):
    r = requests.get(f"https://pricing.internal/{symbol}", timeout=5)  # blocks the loop
    return r.json()

While requests.get waits 80 ms, the event loop cannot run anything else. One worker now serves about 12 requests per second, one at a time, and CPU stays near idle. The same happens with time.sleep, psycopg2, sync SQLAlchemy sessions, boto3 or large file reads inside async def. Fix it by using an async client (httpx.AsyncClient, asyncpg, aiobotocore), declaring the endpoint with def, or wrapping the call in await run_in_threadpool(...).

To find it in a running app, start asyncio debug mode in staging (PYTHONASYNCIODEBUG=1). It logs Executing <Task ...> took 0.183 seconds for any step that holds the loop longer than 100 ms.

Threadpool exhaustion

If most endpoints are def and each waits 200 ms on a database, one worker handles at most about 40 / 0.2 = 200 requests per second through the threadpool. Past that, requests queue for a thread and latency rises with no error. Sync dependencies count too: a def get_db() dependency takes a thread even when the endpoint is async.

Connection pool timeouts

SQLAlchemy's QueuePool and the async engine's pool both default to pool_size=5, max_overflow=10, pool_timeout=30. An async worker can easily have 100 requests in flight, all waiting for one of 15 connections:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00

With asyncpg used directly, asyncpg.create_pool defaults to min_size=10, max_size=10. More in database connection pool exhausted.

Keep-alive behind a load balancer

Uvicorn's --timeout-keep-alive defaults to 5 seconds. Behind an AWS ALB (60 s idle timeout) or another proxy that keeps upstream connections longer, Uvicorn closes connections the balancer still considers open, which causes intermittent 502s. Set it above the balancer's idle timeout.

Server settings to check

# VM with 4 cores
uvicorn app.main:app --host 0.0.0.0 --port 8000 \
  --workers 4 --timeout-keep-alive 75 --no-access-log

# FastAPI CLI (wraps Uvicorn)
fastapi run app/main.py --workers 4

# Gunicorn as process manager (uvicorn-worker package)
gunicorn app.main:app -k uvicorn_worker.UvicornWorker -w 4 \
  --keep-alive 75 --timeout 30 --max-requests 5000 --max-requests-jitter 500

In Kubernetes, run one worker per container and scale replicas. Readiness, restarts and CPU accounting then work per process. Turning off the access log (or making it async) removes a write per request from the hot path. Ship logs from your proxy instead.

Raising the threadpool limit

from contextlib import asynccontextmanager
import anyio

@asynccontextmanager
async def lifespan(app):
    anyio.to_thread.current_default_thread_limiter().total_tokens = 100
    yield

app = FastAPI(lifespan=lifespan)

Raise it only if your def endpoints are waiting on I/O, and make the database pool large enough to serve the extra threads. Otherwise you have just moved the queue.

Async SQLAlchemy engine

engine = create_async_engine(
    "postgresql+asyncpg://app@db/app",
    pool_size=20,
    max_overflow=0,        # fixed size makes the DB total predictable
    pool_timeout=5,        # fail fast under overload
    pool_pre_ping=True,
    pool_recycle=1800,
)

Total connections = workers × replicas × (pool_size + max_overflow). Four workers × 5 replicas × 20 = 400, which is far above PostgreSQL's default max_connections of 100. Run the numbers in the connection pool size calculator, or add PgBouncer in transaction mode (and then disable asyncpg's statement cache, or configure prepared statement support, because transaction pooling breaks named prepared statements by default).

FastAPI tuning table

Pydantic cost and background tasks

Pydantic v2 validates and serializes with a Rust core, and for typical request bodies its cost is small. It adds up on responses. An endpoint that returns 5,000 nested objects through response_model validates and serializes every one. On a single worker that can mean tens of milliseconds of CPU per request, and at that point the endpoint is CPU-bound regardless of how async it is. Paginate large lists, return only the fields clients use, and check with py-spy top --pid <worker> during the test. If pydantic_core or JSON encoding dominates the profile, that is your target. High CPU under load covers the general method.

BackgroundTasks run in the same process after the response is sent. They do not appear in your response latency, but they use the same event loop, threadpool and database pool. Under load, an endpoint that schedules a 300 ms email task per request quietly halves the capacity of everything else. If a background task does heavy work or must not be lost on restart, move it to a real queue.

What to put in the test scenario

• Every endpoint style in your app: async endpoints, sync endpoints, and endpoints with sync dependencies. Each one fails differently.
• The largest realistic responses (maximum page size, full detail objects) to expose serialization cost.
• Endpoints that call other services, against real or realistically slow stubs. A stub that answers in 1 ms hides blocking calls.
• Auth the way clients do it: get OAuth2 or JWT tokens once per virtual user and send Authorization: Bearer. Token issuing (password hashing) belongs in a separate, smaller test. See authentication load testing.
• Writes that trigger background tasks, so their hidden cost shows up in the results.

Most FastAPI apps are pure APIs, so the general advice in API load testing on payload variety and test data applies too.

How to read FastAPI-specific signals

If p95 climbs steadily with load and none of these stand out, work through latency increases under load resource by resource.

Worked example: sizing one service

An illustrative FastAPI service must handle 600 RPS at peak. Its main endpoint is async def, uses asyncpg through SQLAlchemy, spends 4 ms of CPU per request in Python and Pydantic, and holds a database connection for 12 ms.

• CPU: one worker can do about 1 ÷ 0.004 = 250 RPS at 100% CPU. Planning for 70% gives 175 RPS, so 600 RPS needs 4 workers (on 4 cores).
• Connections: 150 RPS per worker × 0.012 s = 1.8 connections busy on average. A fixed pool of 5 per worker leaves room for bursts, and 4 × 5 = 20 connections in total is comfortable for the database.

Run the test to that target. If p95 is fine but CPU is at 40% and latency is still higher than the 4 ms + 12 ms estimate, something is waiting, often a blocking call or a sync dependency. That is exactly the gap the load test is meant to reveal.

Pre-test checklist

1. Every async def endpoint audited for blocking libraries (requests, psycopg2, sync SQLAlchemy, boto3, time.sleep).
2. Workers set from CPU cores or one per container; --reload off.
3. --timeout-keep-alive above the load balancer's idle timeout.
4. Database pool sizes multiplied out across workers and replicas and checked against max_connections.
5. Threadpool limit reviewed if you rely on def endpoints.
6. Access logging off or asynchronous; debug middleware off.
7. Largest realistic responses and auth tokens included in the scenario.
8. py-spy available on one host for live profiling.

In LoadTester, run an exact-RPS ramp with a p95 threshold and auto-stop. Repeat it after each fix (blocking call removed, pool resized) and compare the runs to see how far the ceiling moved.