A promotion ended. The cache didn't get the memo. Customers kept buying $500 items at the $350 promo price for four more hours.

What happened:

  • Admin ended the promotion, restoring the regular price in the database
  • Cache TTL was 6 hours
  • No cache invalidation on update
  • API served the cached (promo) price
  • 327 orders at the expired price
  • Roughly $49,000 in unintended discounts before anyone noticed

Root cause:

The admin panel updated the database directly. The cache layer had no idea anything changed.

The fix:

  • Write-through cache: updates go through cache layer
  • Event-driven invalidation: database changes trigger cache clear
  • Shorter TTL for price-sensitive data (5 minutes)
  • Manual cache flush button for admins

Lesson: Cache invalidation is hard because it requires you to know everywhere data can change. One missed path = stale data disaster. For the longer version of how this goes wrong even with invalidation in place, see our cache invalidation nightmare.

← Back to Lessons Learned

Incidents like this are preventable. Our AWS Managed Services team watches client environments 24/7 with AI-assisted monitoring and senior engineers on the pager — so stories like this stay in the archive.

Tags: #Caching #DataConsistency #Debugging #Redis