Cross-AZ Traffic Bill Shock
Our Kubernetes services were chatty. Too chatty. Cross-AZ data transfer: $8K/month.
The architecture:
- 20 microservices
- Distributed across 3 AZs (good for HA)
- Services call each other frequently
- Each request: average 5 inter-service calls
The math:
- Cross-AZ data transfer: $0.01/GB (each direction)
- 2/3 of calls cross AZ boundaries
- Average request size: 50KB
- 1M requests/day × 5 calls × 50KB × 2/3 cross-AZ
- = ~160GB/day cross-AZ = ~$50/day = $1,500/month per service
Solutions considered:
- ❌ Single AZ: Loses high availability
- ✅ Topology-aware routing: Prefer same-AZ pods
- ✅ Reduce payload sizes (compression, pagination)
- ✅ Batch API calls where possible
- ✅ Service mesh with locality-aware load balancing
Result: 60% reduction in cross-AZ traffic with Istio locality-aware routing.
Lesson: High availability has a cost. Know what it is.