Autoscaling Too Slow for Traffic Spikes

Autoscaling is too slow for a spike whenever the time from "traffic rose" to "new capacity is serving" is longer than your system can queue the excess. That time is the sum of metric lag, the autoscaler's evaluation interval, scheduling or instance launch, application startup and readiness, and it is commonly one to several minutes. Every second of it, the excess arrival rate piles up as backlog: excess RPS × seconds of lag = requests that wait, time out or fail. Fix it by keeping buffer capacity for the spike you expect, scaling ahead of known peaks, scaling on a demand metric (RPS, queue depth) instead of CPU, cutting startup time, and shedding load cleanly while capacity catches up.

What the symptom looks like

In a spike test or a real launch, the pattern is consistent: errors and p95 jump within seconds of the spike, the replica count starts rising a minute or more later, and by the time capacity has caught up the damage is done. Typical evidence:

# kubectl get hpa -w
NAME   REFERENCE        TARGETS    MINPODS   MAXPODS   REPLICAS
api    Deployment/api   cpu: 41%/70%   4      40        4
api    Deployment/api   cpu: 99%/70%   4      40        4
api    Deployment/api   cpu: 99%/70%   4      40        6
api    Deployment/api   cpu: 97%/70%   4      40        9

# kubectl get events
Warning  FailedScheduling  pod/api-6c..  0/12 nodes are available: 12 Insufficient cpu.
Normal   TriggeredScaleUp  pod/api-6c..  pod triggered scale-up: [{eks-workers 12->15 (max: 30)}]
Warning  Unhealthy         pod/api-6c..  Readiness probe failed: HTTP probe failed with statuscode: 503

# Front door during the lag
upstream timed out (110: Connection timed out) while reading response header from upstream
HTTP 503 / 504 bursts, then recovery

Serverless shows the same shape with different words: cold-start "Init Duration" lines in Lambda logs, throttling errors when a concurrency limit is reached, or Cloud Run requests queued while new instances start.

Where the minutes go

Break the delay into stages and measure each one. Here is a Kubernetes example with illustrative numbers; yours will differ, which is exactly why you measure them.

The HPA itself is not the slow part by default. Its default scale-up behavior has no stabilization window and allows adding the larger of 100% of current replicas or 4 pods every 15 seconds; the 300-second stabilization window applies only to scale-down. The slow parts are around it. On VMs the equivalent stages are the CloudWatch alarm period and evaluation, the instance launch, boot and user-data, and the instance warmup before the group counts it.

The hidden multiplier: CPU-based scaling cannot see past 100%

The HPA computes desiredReplicas = ceil(currentReplicas × currentMetric / target). CPU utilization cannot exceed 100% of the request, so with a 70% target the most a single decision can ask for is 100/70 ≈ 1.43 times the current replicas, no matter how big the spike is. Worked example: 4 replicas, traffic triples. CPU pegs at 100%, and the HPA asks for ceil(4 × 100/70) = 6. Those 6 also peg, it asks for 9, then 13. Each step waits for new pods to be ready and for metrics to update, so reaching the 12-13 replicas the traffic actually needs takes three rounds of the whole startup pipeline.

A demand metric fixes this. If you scale on requests per second per pod (via a Prometheus adapter or KEDA), a tripling of traffic produces a request for three times the replicas in one decision.

The backlog math

backlog ≈ (arrival rate − capacity) × time to new capacity

Worked example: each pod handles 150 RPS at acceptable latency. You run 6 pods (900 RPS capacity) for a normal 600 RPS. A campaign drives traffic to 1,800 RPS within 20 seconds. Excess = 1,800 − 900 = 900 RPS. If time to useful new capacity is 100 seconds, the excess is 900 × 100 = 90,000 requests that cannot be served on time. No reasonable queue absorbs that: with a 30-second client timeout, most of them fail, and the clients' retries add more load (see retry storms under load). If the stepwise CPU behavior above stretches the lag to 4 minutes, it is 216,000.

The same formula tells you what to do. Either shrink the excess (buffer capacity: run more pods so the excess is smaller or zero) or shrink the time (faster signal, faster start). With 12 replicas at baseline, the spike's excess would be zero.

How to confirm which stage is slow

• Metric lag: overlay your load test's RPS with the HPA's view (kubectl get hpa -w or the kube_horizontalpodautoscaler_status_current_metrics series). The delay between the two curves is your signal lag.
• Decision to pod Running: kubectl get events --sort-by=.lastTimestamp shows SuccessfulRescale, Scheduled, Pulling, Pulled, Started with timestamps. FailedScheduling and TriggeredScaleUp mean you waited for a node.
• Running to Ready: compare the container start time with the Ready condition time in kubectl get pod -o yaml. Then check your app's own startup log.
• Stepwise growth: if replicas grow by ~1.3-1.5x per round under a CPU target, you are hitting the 100% ceiling described above.

Put deploy and scale events on the load test timeline. In LoadTester you can push an annotation to a running test (POST /api/annotations/push with your API token) from a script that watches the HPA, so the replica changes appear on the same chart as RPS and p95.

Fixes

1. Buffer capacity sized from the spike you expect

Set minReplicas and the target utilization so normal peak plus the expected spike fits in current capacity for the lag duration. Lowering the CPU target from 80% to 50% both adds standing headroom and lets each HPA decision step up by 2x instead of 1.25x. For node-level headroom, run low-priority placeholder pods that the scheduler evicts when real pods need the space, so new replicas do not wait for a node.

2. Scale before the spike when you know when it comes

Product launches, campaigns, sales and daily peaks are predictable. KEDA's cron scaler sets a replica floor for a time window:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api
spec:
  scaleTargetRef:
    name: api
  minReplicaCount: 4
  maxReplicaCount: 60
  triggers:
  - type: cron
    metadata:
      timezone: Europe/Zagreb
      start: 0 17 * * 5      # Friday 17:00
      end: 0 23 * * 5
      desiredReplicas: "30"
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      query: sum(rate(http_requests_total{service="api"}[1m]))
      threshold: "120"       # target RPS per replica

On AWS Auto Scaling groups, scheduled actions and predictive scaling serve the same purpose, and warm pools keep pre-initialized instances ready. For Black Friday-style events, scale up hours ahead and keep scale-down slow.

3. Scale on demand, not on CPU

RPS per pod, in-flight requests or queue depth reflect the full spike in one reading. Keep CPU as a secondary trigger. Shorten the rate window used for the metric (a 1-minute rate reacts faster than 5 minutes) and check your metrics pipeline's scrape interval.

4. Make each new replica useful sooner

• Smaller images and pre-pulled images on nodes (a DaemonSet that pulls the image works).
• Lazy initialization: do not load every cache or open every connection before serving.
• A startupProbe for slow boots plus a fast, accurate readiness probe (periodSeconds: 2-5), so pods join as soon as they can serve.
• Allow faster scale-up explicitly if you have changed the defaults; for example a Pods policy of 10 per 15 s for small deployments.
• Serverless: provisioned concurrency (Lambda) or minimum instances (Cloud Run) for the baseline; see serverless load testing.

5. Shed load cleanly during the lag

Something will exceed capacity eventually. Make it fail fast instead of timing out: cap concurrency per pod, return 503 with Retry-After when the queue is full, and rate limit at the edge.

# nginx: cap per-client rate, reject excess quickly instead of queueing forever
limit_req_zone $binary_remote_addr zone=api:10m rate=50r/s;
location /api/ {
    limit_req zone=api burst=100 nodelay;
    limit_req_status 503;
}

Fast rejections keep latency for admitted requests low, which keeps CPU-based metrics meaningful and keeps clients from piling on retries.

Diagnosis table

How to re-test and confirm

Run the same spike profile in arrival-rate mode, starting from the replica count you actually run at that time of day, not from an already-scaled deployment. A good shape: 5 minutes at baseline, a jump to the target RPS within 10-30 seconds, then hold for at least as long as your measured time-to-capacity plus a few minutes. Measure three things: time from spike start to p95 back under threshold, total failed requests during the transition, and the replica curve. A fix should cut the first two; a buffer-capacity fix should make them close to zero. Keep in mind that a single test's RPS is bounded by the load generator (LoadTester's largest plan runs up to 2,000 RPS per test), so pick a spike ratio relative to your baseline that you can actually generate. The Kubernetes load testing guide covers the broader cluster setup.