A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
Scenario
Marketing ran a campaign, traffic multiplied within a minute or two, and the service fell over — despite a horizontal pod autoscaler that everyone assumed was the safety net. The post-incident picture: the HPA was set to scale on CPU at around 70% utilization, a new pod took close to a minute to become ready (image pull, startup, warm-up), and the cluster itself was running close to its node capacity, so some new pods had nowhere to schedule until the cluster autoscaler added machines — a process measured in minutes, not seconds. By the time capacity arrived, the existing pods had already saturated, latency exploded, and clients were retrying, making everything worse.
The Quick Fix on the Table
In the retro, a colleague proposes raising maxReplicas way up. If the autoscaler was allowed to go higher, the reasoning goes, it would have scaled out of trouble. It is a one-line YAML change and it closes the action item.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- The ceiling was never the problem — the reaction time was. The outage happened in the window between the spike arriving and new capacity becoming useful. Metrics scrape interval, HPA sync period, pod scheduling, image pull and readiness add up to minutes of lag; a sharp spike does its damage in seconds. A higher cap changes nothing about that window.
- The cluster couldn't have hosted the pods anyway. With nodes near capacity, extra replicas just sit in
Pendingwhile the cluster autoscaler provisions machines. RaisingmaxReplicaswithout node headroom is permission to queue, not capacity to serve. - Overloaded pods lie to the autoscaler. A saturated service can wedge on queueing, connection limits or GC before CPU cleanly crosses the threshold, and pods that crash under load restart and reset the signal. The trigger itself was late, not just the ceiling low.
- Retry storms outrun any scaler. Once latency climbed, clients retried and multiplied the offered load. No replica count wins an arms race against uncontrolled retries; that has to be handled at admission, not at scale-out.
- It converts an outage risk into a cost incident. An unbounded ceiling plus a metrics anomaly or a retry loop is how you wake up to a cluster that scaled to hundreds of replicas overnight. The cap exists for a reason.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Reconstruct the timeline first. From metrics, line up: when traffic rose, when the HPA reacted, when pods were scheduled, when they became ready, and when nodes were added. This turns "autoscaling failed" into concrete numbers — usually several minutes of end-to-end lag — that dictate the fixes.
- Cut pod time-to-ready. Slim the image, tune startup, and make the readiness probe honest (ready means "can serve", not "process exists"). Every second shaved off startup shrinks the vulnerable window.
- Keep warm headroom instead of scaling from zero margin. Run more baseline replicas (lower the CPU target so steady-state utilization leaves room), and keep node headroom with a cluster overprovisioner — low-priority placeholder pods that get evicted the moment real workloads need the space, so scale-up doesn't wait on EC2.
- Protect the service at the front door. Add load shedding and admission control: concurrency limits per pod, request timeouts, rate limiting at the ingress or API gateway, and fast rejection (
429/503withRetry-After) once queues fill. Serving 80% of users normally beats serving 100% of users a 30-second timeout. - Tame the retry amplification. Enforce exponential backoff with jitter and retry budgets in clients and service mesh policy, and consider circuit breakers so a struggling backend isn't hammered into the ground.
- Pre-scale for known events. A marketing campaign is not a surprise — it has a launch time. Put a step in the campaign runbook to scale up in advance (a scheduled
kubectl scaleor a temporaryminReplicasbump), and make marketing-to-engineering notification a standing process. - Then revisit the ceiling. With honest startup times, warm headroom and load shedding in place, set
maxReplicasfrom a load test, not from fear. Now the number means something.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "We never hit the max replica ceiling — we hit the reaction-time floor. The outage was over before the pods the HPA already asked for were ready to serve."
- "Autoscaling is a cost optimization for gradual load changes, not a defense against step-function spikes. The defense against spikes is warm capacity and load shedding."
- "Raising the cap on a cluster with no node headroom just queues pods in Pending. If we do only one thing, overprovisioned node headroom buys more than any HPA setting."
- "Without admission control, the retry storm multiplies load faster than any scaler can add capacity. Shedding load early keeps the service alive long enough for scaling to catch up."
- "This campaign had a scheduled start time. The cheapest fix on the list is a runbook line that says 'scale up before the email goes out'."