Cross-AZ Traffic Bill Shock
Our Kubernetes services were chatty. Too chatty. Cross-AZ data transfer: ~$8K/month.
The architecture:
- 20 microservices
- Distributed across 3 AZs (good for HA)
- Services call each other frequently
- Each request: average 5 inter-service calls
The math:
- Cross-AZ data transfer: $0.01/GB in each direction, so $0.02/GB effective
- With pods spread evenly across 3 AZs, roughly 2/3 of calls cross an AZ boundary
- Worked example for one user-facing flow: 1M requests/day × 5 calls × 50KB payloads ≈ 250GB/day of east-west traffic, ~167GB of it cross-AZ ≈ $100/month
- That sounds harmless — until you add every flow: request and response payloads, retries, gRPC streaming, log shipping, and Kafka replication across all 20 services
- Measured with VPC Flow Logs: ~13TB/day cross-AZ ≈ $260/day ≈ $8K/month
Solutions considered:
- Single AZ — rejected: it eliminates the cost by eliminating high availability
- Topology-aware routing — prefer same-AZ endpoints for service-to-service calls
- Reduce payload sizes — compression and pagination on the chattiest APIs
- Batch API calls where possible
- Service mesh with locality-aware load balancing
Result: 60% reduction in cross-AZ traffic with Istio locality-aware routing.
Lesson: High availability has a cost. Know what it is. Data transfer is one of the line items we audit first in cloud cost reviews, because it hides inside "networking" and nobody owns it.