3 AM Outage: Roll Back or Fix Forward?
Scenario
It is a little after 3 AM and your pager goes off: the checkout service error rate has jumped from a fraction of a percent to nearly 10% in the space of a few minutes. Roughly twenty minutes earlier, a new release went out. The previous version ran cleanly for over a week before that. A rollback to it would take about two minutes through your deploy pipeline. You do not yet know what the root cause is — only that errors started climbing shortly after the deploy, and that every failed request is a customer who could not pay you.
The Quick Fix on the Table
A teammate pops into the incident channel: they think they see the bug and can push a small patch — "give me a few minutes." Fixing forward feels productive, avoids "undoing" the release, and if the patch works you get both service restoration and the new feature in one move. Nobody has to admit the deploy was bad.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- You'd be debugging in production with customers as the test suite. "I think I see it" is a hypothesis, not a diagnosis. If the patch is wrong — or right about one bug but blind to a second — the outage extends by a full write-review-build-deploy cycle each attempt.
- The clocks are not comparable. Rollback has a known, rehearsed duration: about two minutes to a version with nine days of clean production history. A forward fix has an unknown duration written by a person who was asleep twenty minutes ago.
- 3 AM is the worst time to trust your own judgment. Code review is thin, staging validation gets skipped "just this once," and a tired engineer under pressure is exactly the profile that ships the second bug on top of the first.
- Every extra minute is measurable revenue loss. Checkout errors are the most expensive kind of error. The incident cost is dominated by duration, and rollback minimizes duration with near-certainty; a patch merely might.
- Fixing forward destroys your clean comparison. Once a patch lands on top of the broken release, you are diagnosing two changes instead of one. Rolling back preserves the simple before/after signal for the post-incident investigation.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Restore service first, understand later. The incident priority is mean time to recovery, not root cause. Declare the decision explicitly in the channel: "We are rolling back to the last known-good version now."
- Check for rollback blockers before pulling the trigger. The one thing that legitimately complicates rollback is state: did the release run a schema migration, change a message format, or start writing data the old version cannot read? A quick check of the release notes and migration history (
git logon the migrations directory, or your migration tool's status output) answers this in under a minute. - Execute the rollback through the pipeline, not by hand. Use the same mechanism you always use —
kubectl rollout undo deployment/checkout, re-deploying the previous tag, or your CD tool's rollback action — so the result is a known artifact, not an improvised one. - Verify recovery against the original signal. Watch the error rate that paged you return to baseline, and confirm with a synthetic checkout transaction. Do not close the incident on "deploy finished"; close it on "customers can pay again."
- Preserve the evidence. Capture logs, traces and a copy of the bad build before things scale down or rotate away, so the daytime investigation starts with data instead of folklore.
- Fix forward in daylight. The teammate's patch idea is not wasted — it becomes the starting hypothesis for a properly reviewed, tested fix that redeploys through the normal pipeline the next day.
- Feed the lesson back into the pipeline. If errors took six minutes to page and twenty to act on, look at deploy-time canary analysis or automatic rollback on error-budget burn, so the next bad release reverts itself before a human wakes up.
The honest caveat: forward fix is the right call in a minority of cases — when rollback is impossible (an irreversible migration already ran, or the old version has a worse bug, such as the security flaw the release was fixing) or when the fix is genuinely trivial and faster than the rollback itself, like reverting a single config flag. The discipline is to treat forward fix as the exception that must justify itself, never the default.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Rollback is a two-minute operation to a version with nine days of clean history. The patch is an unknown number of minutes to a version with zero minutes of history. At 9% checkout errors, I'll take the certain two minutes."
- "We don't need to know the root cause to stop the bleeding — we know the errors started right after the deploy, and that correlation is all a rollback needs."
- "If the patch guess is wrong, we've extended the outage by a full build-and-deploy cycle and we'll be having this same conversation twenty minutes from now, with more lost revenue behind us."
- "Rolling back isn't throwing the fix away — it buys us the time to write that same patch tomorrow with review, tests and a clear head, then ship it through the normal pipeline."
- "The only thing I want to verify before rolling back is that this release didn't run an irreversible migration. If it did, then we talk about fixing forward — as a constraint, not a preference."