Debug Logging Cost Explosion
AWS bill spiked 400% overnight. Cause: debug logging statement deployed to production.
The chain of events:
- Developer added debug logging for an investigation
- Investigation finished, logging stayed
- PR review missed it
- Hot code path: 50K requests/second
- Each request: 3 debug log lines
- CloudWatch ingestion: 💸💸💸
The math:
- 50K req/sec × 3 logs × 500 bytes = 75MB/sec
- 75MB/sec × 86,400 sec/day = 6.5TB/day
- CloudWatch ingestion: $0.50/GB = $3,250/day
What we implemented:
- CI check for debug log patterns in production code
- CloudWatch alarms for log ingestion rate spikes
- Weekly cost anomaly reviews
- Budget alerts at 120% of baseline
Lesson: One console.log() can cost more than a junior dev's salary.