Finance Wants Everything on Spot Instances. Including the Database.
Scenario
Finance has mandated a 40% cut in cloud spend, and a consultant's slide deck has the answer: Spot instances, up to 70% cheaper, interruption rates quoted below 5%. Your estate is about forty on-demand EC2 instances: a web tier behind a load balancer, CI runners, async workers, and a self-managed PostgreSQL primary with two asynchronous read replicas (typically 1–3 seconds behind) and nightly snapshots. The platform lead is on board with moving everything — database included — arguing that a Spot interruption is just a reboot, and the systems survive reboots.
The Quick Fix on the Table
Flip the whole fleet to Spot in one migration. Maximum discount, one project, finance target hit in a quarter. Interruptions are rare, the replicas provide redundancy anyway, and "treat it like a reboot" sounds like operational maturity.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- A Spot reclaim is not a reboot. A reboot is planned, graceful and returns the same machine. A Spot interruption is a two-minute warning followed by termination, at a time the market chooses. For a stateless web pod that is a shrug; for a Postgres primary it is an unplanned failover under a countdown.
- Asynchronous replication means reclaim = data loss. The replicas run 1–3 seconds behind. Lose the primary and you lose every transaction in that window — on a system taking orders or payments, that is silently vanished customer data, not degraded performance. Nightly snapshots make the worst case up to 24 hours, not 3 seconds.
- Interruptions correlate; percentages don't protect you. "Under 5%" is a fleet average. When capacity tightens in a pool, the primary and the replicas in the same instance family can be reclaimed within minutes of each other. The redundancy argument assumes independent failures; Spot reclaims are not independent.
- Failover under a two-minute timer is where outages are born. Promotion, DNS/endpoint switchover, connection storms from reconnecting apps, and a possible split-brain if the old primary comes back — doing that routinely, triggered by the Spot market rather than by your maintenance calendar, converts a cost project into a reliability lottery.
- The database is a rounding error in the savings anyway. A handful of DB instances out of forty carry a small share of the bill but effectively all of the data-loss risk. The risk-to-savings ratio on that slice is the worst in the whole proposal.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
Say yes to the 40% target — and get there by matching purchase model to workload tolerance instead of applying one model to everything.
- Classify workloads by interruption tolerance. Stateless and retry-safe (web tier behind the ALB, CI runners, idempotent async workers) can die anytime; stateful (Postgres primary and replicas, anything with a local disk that matters) cannot. Write the list down — it is the contract for the whole migration.
- Move the tolerant majority to Spot aggressively. Web tier and workers via Auto Scaling groups or EKS with mixed instance policies, diversified across many instance types and AZs (
capacity-optimizedallocation), keeping a small on-demand base so a market-wide reclaim can't take the tier to zero. CI runners are the perfect Spot citizens — a killed build simply reruns. - Handle the two-minute warning properly. Consume the interruption notice (EventBridge / instance metadata), drain connections from the load balancer, let workers finish or requeue jobs, and make jobs idempotent so a mid-task reclaim is safe. On EKS,
aws-node-termination-handleror Karpenter does the choreography. - Cut the database bill with committed-use pricing, not Spot. The database runs 24/7 and is the definition of steady-state — exactly what Reserved Instances / Savings Plans exist for: 40–60% off with zero added risk. Right-size the instances first, and consider whether managed RDS with its own HA story beats self-managed economics once engineer time is priced in.
- Sweep the rest of the bill. Delete unattached EBS volumes and idle load balancers, downshift storage classes, schedule non-prod environments off outside working hours, and shrink over-provisioned instances. These unglamorous items routinely find 10–15% on their own.
- Present the blended math to finance. Spot on ~70% of the fleet at ~60–70% off, RIs/Savings Plans on the stateful core at ~40%+, plus hygiene savings — the composite typically lands at or beyond the 40% target, with the data-loss risk left at zero. Frame the alternative honestly: the Spot-everything delta on the DB slice is a few percent of the bill against the cost of one data-loss incident.
- Prove it before declaring victory. Run chaos-style interruption drills on the Spot tiers (terminate instances on purpose), watch error rates stay flat, and put the savings on a dashboard finance can see monthly.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "I'm not pushing back on the 40% — I'm signing up for it. I'm pushing back on taking data-loss risk to capture the last few percent, when committed-use pricing gets the database its discount risk-free."
- "Our replicas are one to three seconds behind. A Spot reclaim of the primary means every transaction in that window is gone — that's customer orders, not an inconvenience."
- "The 'under 5%' interruption rate is an average, and reclaims cluster: when a pool tightens, the primary and its replicas can be taken within minutes of each other. Our redundancy math assumes failures that don't arrive together."
- "Spot is a fantastic deal for the 70% of our fleet that can die anytime — web tier, CI, workers. That's where the discount is large, safe, and immediate."
- "A reboot is us choosing the time; a reclaim is the market choosing it, with two minutes' notice, possibly during Black Friday. Those are not the same event, and the database is the one system that knows the difference."