Stale Cache Data Disaster
Price update didn't propagate to cache. Customers bought $500 items for $50 for 4 hours.
What happened:
- Admin updated product price in database
- Cache TTL was 6 hours
- No cache invalidation on update
- API served cached (old) price
- 327 orders at wrong price
- $47,000 revenue loss
Root cause:
The admin panel updated the database directly. The cache layer had no idea anything changed.
The fix:
- Write-through cache: updates go through cache layer
- Event-driven invalidation: database changes trigger cache clear
- Shorter TTL for price-sensitive data (5 minutes)
- Manual cache flush button for admins
Lesson: Cache invalidation is hard because it requires you to know everywhere data can change. One missed path = stale data disaster.