"Spot instances save up to 90%!" — that's the marketing ceiling; 60–70% is what we actually saw. Still real money. Until all 50 of them get reclaimed during a traffic spike.

Black Friday timeline:

  • 06:00 - Traffic starts climbing
  • 08:00 - Auto-scaling adds spot instances
  • 10:00 - All systems nominal
  • 11:30 - AWS reclaims 50 spot instances simultaneously
  • 11:31 - Auto-scaling tries to spin up on-demand (at roughly 3x our spot price)
  • 11:32 - Insufficient capacity in our AZ
  • 11:33 - 15 minutes of partial outage

The problem:

We were 100% spot for cost savings. No baseline of on-demand instances, a single instance type in a single AZ, and nothing listening for interruption warnings. When AWS needed the capacity back, we had no fallback.

What we fixed:

  • Baseline: 30% on-demand for guaranteed capacity
  • Spot fleet diversified across instance types and AZs, using the price-capacity-optimized allocation strategy so the fleet draws from the deepest capacity pools
  • Handle the 2-minute interruption notice and EC2 rebalance recommendations: drain connections, deregister from the load balancer, and start replacements proactively
  • Graceful degradation when capacity is limited

Getting the on-demand/spot/Savings Plans mix right for a given workload profile is bread-and-butter cloud cost optimization work — the discount is only real if you survive the interruptions.

Lesson: Spot for batch jobs? Perfect. Spot for serving production traffic without a fallback? You're gambling.

← Back to Lessons Learned
Tags: #AWS #CostOptimization #SpotInstances #HighTraffic