"Let's log everything in production, just in case." That sentence ended in a CloudWatch bill just over $50,000 for a single month. Cost war stories usually skip the arithmetic, which is how bad advice spreads — so here is ours in full.

The chronic problem: a log-everything culture

  • Every HTTP request body, including large payloads
  • Every database query with bound parameters
  • Health-check responses every 10 seconds, per pod
  • No retention policy on any log group

That baseline ran about 500GB a day. At CloudWatch's $0.50/GB ingestion rate, that is roughly $7,500 a month — plus about $5,400 a month in storage after a year of accumulation at $0.03/GB-month, because nothing ever expired. Call the chronic cost $13,000 a month: bad, stable, and invisible inside a growing AWS bill. Ninety-nine percent of those logs were never read.

The acute problem: one leftover debug statement

A developer added three debug log lines to trace an issue on an internal fan-out path, finished the investigation, and left the logging in. Code review missed it. The public API served around 3,000 requests per second, but each request triggered six to eight internal calls — so the instrumented code path executed roughly 20,000 times per second at peak.

  • 20K executions/sec × 3 lines × ~500 bytes ≈ 30MB/sec
  • ≈ 2.6TB/day ≈ $1,300/day in ingestion alone
  • Left running for a full month: ≈ $39,000

Add the $13,000 chronic baseline and you have the $50K month. Nothing exotic happened — it was multiplication nobody did before deploying.

The fixes

  • Structured logging with enforced levels. INFO is the production floor — a WARN-only policy sounds thrifty but strips out exactly the context you need mid-incident. DEBUG runs only behind sampling (about 1% of requests) or a runtime toggle.
  • 7-day retention in CloudWatch. Anything older ships to S3 and is queried with Athena at a fraction of the cost.
  • A CI check that flags debug-level logging added to hot production paths.
  • Alarms on log ingestion rate plus budget alerts at 120% of baseline — the acute incident would have paged us on day one instead of surfacing on the invoice.
  • A weekly cost anomaly review. Boring by design, and the habit that catches everything the alarms miss. We treat spend as a first-class signal in our observability and AIOps practice for exactly this reason.

Result: roughly $50,000/month down to $3,200/month, with better signal than before. The broader playbook is in our cloud cost-cutting guide.

Lesson: Logs are priced per gigabyte, so do the multiplication before you ship the log line. One forgotten debug statement can out-cost an engineer's salary.

← Back to Lessons Learned
Tags: #AWS #CostOptimization #Logging #Monitoring