A 30-Second Blip Became a 2-Hour Outage. More Retries?
Scenario
The payment service hiccupped for about thirty seconds. The platform then stayed down for two hours. The post-incident graphs tell the story: the moment the blip started, the payment service was hit with more than twenty times its normal traffic — and almost none of it was new user activity. It was retries. The mobile client retries failed requests three times. The API gateway retries three times. Two internal services in the call path retry three times each — none of them with backoff. Every layer retried independently, multiplying the load at each hop, and the payment service could never get its head above water long enough to recover. The retries were the outage.
The Quick Fix on the Table
"Raise the retry counts from 3 to 5 everywhere — that way requests keep trying until the service comes back." The logic is that the blip simply outlasted the retries, so more attempts means eventual success.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- Layered retries multiply, they don't add. Three retries at each of four layers isn't 12 attempts — it's up to 3×3×3×3, hundreds of downstream requests per original user action. Raising each layer to 5 pushes the worst case toward 5⁴ = 625. The proposal upgrades the amplifier that caused the incident.
- Recovery load is what kept the service down. The payment service fixed its 30-second problem quickly; it then spent two hours being kicked back down by its own callers every time it tried to stand up. More retries make that recovery window strictly harder to escape.
- Synchronized retries arrive as waves. Without jitter, every client that failed at time T retries at T+interval — as a single coordinated spike. The service faces walls of traffic with idle gaps between them, the worst possible load shape for recovery.
- Retries hold resources hostage upstream. Every retrying request pins a connection, a thread, and the user's patience at each layer above. Longer retry chains propagate the payment service's problem upward into gateway and client-side resource exhaustion — spreading the outage, not containing it.
- It misreads what retries are for. Retries paper over transient, isolated failures — one dropped packet, one timed-out call. Against a saturated dependency, retries are indistinguishable from a DDoS run against yourself.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Collapse the retry topology first. Decide, per call path, the one layer that owns retries — usually the outermost client for user-facing flows, or the innermost caller with the most context. Every other layer passes failures through. This one architectural decision removes the exponent.
- Exponential backoff with full jitter, everywhere retries remain. Delay grows per attempt (1s, 2s, 4s...) and each delay is randomized across its full range so clients desynchronize. Most HTTP clients and SDKs support this natively — it's configuration, not invention.
- Cap total attempts with a retry budget. Enforce a service-wide budget — for example, retries may add at most 10% to normal request volume (a token-bucket per client works). When the budget is exhausted, fail fast. This bounds worst-case amplification no matter how bad the day gets.
- Add circuit breakers on the payment-service callers. After a threshold of consecutive failures the breaker opens and calls fail immediately without touching the network; a half-open probe tests recovery gently. The service gets the quiet window it needs to stand up — the exact thing it never got during the incident.
- Give the service a way to say "not yet." Return
429/503withRetry-Afterunder load, and make clients honor it. Load shedding at the server plus obedient clients turns a meltdown into a brownout. - Cut timeouts to match reality. Long timeouts hold connections open and delay failure detection. Set timeouts slightly above healthy p99 latency per hop so failing calls release resources quickly and the breaker gets a fast signal.
- Prove it and watch it. Replay the incident in a load test (kill the payment service for 30 seconds, watch amplification) to verify the fixes, then add standing metrics for retry rate, breaker state, and budget consumption with alerts — retry amplification should be a dashboard number, not a post-mortem discovery.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "The graphs show the payment service took 20x traffic that was almost entirely retries. The blip lasted 30 seconds; the retries lasted two hours. Raising counts from 3 to 5 upgrades the exact mechanism that hurt us."
- "Retries stack multiplicatively across layers — four layers of 3 is potentially hundreds of calls per user action, and four layers of 5 is worse by another order of magnitude. We need to delete layers of retries, not deepen them."
- "Every time the service tried to recover, its own callers knocked it back down. A circuit breaker gives it the quiet window to stand up — more retries guarantee it never gets one."
- "With backoff, jitter, and a retry budget, genuinely transient failures still get retried and succeed — we lose nothing on normal days. What we cap is the disaster amplification on bad days."
- "This is textbook metastable failure: the outage sustained itself after the trigger ended. Any fix that adds load during recovery fails that test; every fix I'm proposing sheds load during recovery."