Set up multi-region database replication "for disaster recovery."

What we paid for:

  • Primary in us-east-1, read replicas in eu-west-1 and ap-southeast-1
  • ~$8,000/month all-in: roughly $5,300 for two always-on replica instances with storage, $1,800 in cross-region replication traffic (a write-heavy workload ships every WAL byte to both regions), and $900 in snapshots and backup storage

What we actually used:

  • 100% of reads and writes served from us-east-1
  • 0 failover tests in 18 months
  • No DR runbook
  • No one who knew how to trigger failover

The realization:

On paper we had an RTO of minutes (promote a replica) and an RPO of seconds. In practice both numbers were fiction: never tested means never known. If us-east-1 had gone down, our real RTO was "however long it takes to learn failover during an outage" — unbounded.

What we did:

  • Reduced to a single read replica in us-east-1 for HA within the region
  • Cross-region DR via cold backups: snapshots plus continuous WAL archiving to an S3 bucket replicated to us-west-2
  • Wrote actual runbooks and started quarterly DR tests

The tradeoff — stated explicitly:

This was a conscious downgrade of region-loss recovery, not a free lunch:

  • RTO before: minutes on paper, never validated — actual value unknown. After: ~4 hours to restore the snapshot and replay WAL in us-west-2; last game day measured 3h20m.
  • RPO before: seconds on paper, never validated. After: ≤15 minutes, bounded by the WAL archive interval.
  • Cost: $8,000/month → $1,200/month.

The business signed off because for this workload a region-wide outage costing a few hours was acceptable; paying $6,800/month extra for untested "minutes" was not. If your workload genuinely can't tolerate hours, keep the warm replica — but then you must test promoting it. That sizing conversation is the core of the cloud cost and architecture reviews we run.

The quarterly test:

Each quarter: restore the latest snapshot into us-west-2, replay WAL to a target timestamp, point a staging stack at the restored database, run the smoke suite, and record the wall-clock time as the measured RTO. The runbook gets corrected on the spot, and a different engineer runs it each time so the knowledge doesn't live in one head. The same discipline of paying only for what you validate shows up across our cloud cost-cutting guide.

Result: $8,000/month → $1,200/month, with DR numbers we've actually measured.

Lesson: DR that's never tested isn't DR. It's just expensive. A cheaper, tested recovery beats an expensive, hypothetical one.

← Back to Lessons Learned
Tags: #Cloud #CostOptimization #Architecture #MultiRegion