Cross-AZ Traffic Bill Shock
Our Kubernetes services were chatty. Too chatty. Cross-AZ data transfer: ~$8K/month.
The architecture:
- 20 microservices
- Distributed across 3 AZs (good for HA)
- Services call each other frequently
- Each request: average 5 inter-service calls
The math:
- Cross-AZ data transfer: $0.01/GB in each direction, so $0.02/GB effective
- With pods spread evenly across 3 AZs, roughly 2/3 of calls cross an AZ boundary
- Worked example for one user-facing flow: 1M requests/day × 5 calls × 50KB payloads ≈ 250GB/day of east-west traffic, ~167GB of it cross-AZ ≈ $100/month
- That sounds harmless — until you add every flow: request and response payloads, retries, gRPC streaming, log shipping, and Kafka replication across all 20 services
- Measured with VPC Flow Logs: ~13TB/day cross-AZ ≈ $260/day ≈ $8K/month
Solutions considered:
- Single AZ — rejected: it eliminates the cost by eliminating high availability
- Topology-aware routing — prefer same-AZ endpoints for service-to-service calls
- Reduce payload sizes — compression and pagination on the chattiest APIs
- Batch API calls where possible
- Service mesh with locality-aware load balancing
Result: 60% reduction in cross-AZ traffic with Istio locality-aware routing.
Lesson: High availability has a cost. Know what it is. Data transfer is one of the line items we audit first in cloud cost reviews, because it hides inside "networking" and nobody owns it.
Incidents like this are preventable. Our AWS Managed Services team watches client environments 24/7 with AI-assisted monitoring and senior engineers on the pager — so stories like this stay in the archive.