Your Primary Region Is Down. Fail Over Now?
Scenario
Your primary cloud region is having a very bad day. Users are seeing errors, dashboards are red, and the provider's status page has upgraded from green to a carefully worded yellow. You have prepared for this: a warm standby in a second region, fed by an asynchronous database replica running a few seconds behind. But the situation is messier than the DR diagram: the primary isn't cleanly dead — it's degraded. Some traffic still succeeds, some AZs may be fine, and critically, the primary database may still be accepting writes.
In the war room, the pressure peaks and someone makes the call everyone is thinking: "Flip DNS to the standby. Now. Every minute we debate is downtime." Decisiveness feels like leadership; hesitation feels like failure.
The Quick Fix on the Table
Immediately point DNS at the standby region and declare failover complete. One change, instant sense of action, and the error graphs should start recovering within the TTL.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- A degraded primary plus a promoted standby equals split-brain. If the primary database is still accepting writes while you promote the replica, you now have two databases that each believe they are authoritative, diverging with every second. DNS caching and long-lived connections guarantee some clients keep writing to the old primary long after the flip. Reconciling divergent write histories is one of the worst problems in operations — routinely worse than the outage itself.
- Async replication means the flip itself loses data. The replica is seconds behind; promotion silently discards whatever hadn't shipped. That may be acceptable — but it must be a decision, made knowing the current lag, not a side effect nobody measured at flip time.
- "Warm" standby is a hypothesis until verified. Is the standby scaled for full production traffic? Are secrets current, feature flags synced, dependent third-party allowlists updated, background workers and cron jobs ready — and pointed at the right database? An unverified failover can convert a partial outage in one region into a total outage in two.
- You may be failing over into the same blast radius. If the root cause is a global control-plane issue, a bad deploy of yours, or an upstream dependency, the standby inherits the problem — and you've spent your one DR move and added migration chaos on top.
- Nobody planned the way back. Failover is half a procedure. Without a failback plan — how the old primary gets demoted, resynced, and validated — you'll run indefinitely on a region with no DR remaining, having spent your redundancy to buy the appearance of speed.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Scope the outage in minutes, not vibes. Establish: which AZs/services are affected, is your impact total or partial, what do the provider's status page and your own synthetic checks from outside say, and — decisively — can the failure be mitigated inside the region (shift AZs, shed load, roll back a deploy) faster than a regional failover?
- Check the replication state before anything else touches the database. Measure current replica lag concretely (e.g.
aws rds describe-db-instances/ CloudWatchReplicaLag, orpg_last_wal_receive_lsn()vs replay). This number is your data-loss bill for promotion — get it in writing in the incident channel. - Fence the primary before promoting the standby. The non-negotiable ordering: stop writes at the old primary first — set it read-only, revoke the app's write credentials, or block writes at the security-group/proxy level — and only then promote the replica. If the primary is unreachable, fence at every layer you can still reach. One writable database at a time, always.
- Fail over by runbook, with roles. Execute the tested procedure: promote the replica, repoint application config, scale the standby to production capacity, verify health checks and a real end-to-end transaction, then shift traffic — DNS with low TTLs or, better, a routing-layer switch that isn't hostage to resolver caching. One person commands, one executes, one communicates.
- Declare the decision criteria out loud. A simple pre-agreed threshold ends the war-room debate: e.g. "if full-region impact exceeds 15 minutes with no in-region mitigation and replica lag is under X seconds, we fail over." If your organization doesn't have this yet, the incident commander sets it explicitly now — the point is that the flip happens on criteria, not adrenaline.
- Plan failback before you relax. Once stable on the secondary: rebuild replication in the reverse direction, reconcile any orphaned writes from the fenced primary (you kept its WAL/logs), and schedule a controlled failback — or formally accept the secondary as the new primary and rebuild DR the other way.
- Afterwards, make it boring. The real lesson of a scary failover is that it was scary. Quarterly game days that execute the actual runbook — including fencing and failback — are what turn the next regional outage into a procedure instead of a gamble.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "The primary isn't dead — it's degraded, which is the dangerous case. If it's still taking writes when we promote the replica, we get two authoritative databases diverging in real time, and untangling that will take longer than this outage."
- "Ten minutes of verification isn't hesitation, it's the difference between a controlled failover and a coin flip: outage scope, replica lag, standby readiness, fencing. Then we flip with confidence instead of hope."
- "The replica is async. Promotion deletes whatever hasn't shipped — let's read the lag number out loud and decide to accept it, rather than discover it in tomorrow's reconciliation."
- "Order matters more than speed: fence the old primary first, promote second, shift traffic third. DNS caching means clients will hit the old region for a while no matter what we do — fencing is what makes that harmless."
- "If we fail over, I also want the failback plan stated before we start — otherwise we're spending our only DR move with no route home, and running the business on a region with zero redundancy."