Redis saved our database. Until it killed it.

What happened:

  • Traffic spike: 50K requests/second against a single Redis instance
  • That rate alone wouldn't have hurt — Redis handles far more in simple GET/SET ops — but we were storing multi-hundred-KB serialized objects and running a few O(N) commands per request
  • Redis hit 100% CPU; responses slowed from sub-millisecond to hundreds of milliseconds
  • Client timeouts fired — and our cache client treated every timeout as a cache miss
  • The full 50K requests/second fell through to the database as a thundering herd
  • Database died within 30 seconds

Root cause:

Two compounding mistakes. We put expensive payloads on a single hot instance, and we had no circuit breaker: when Redis got slow, "timeout equals miss" meant we didn't degrade gracefully — we redirected the entire load onto the one system the cache existed to protect.

The fix:

  • Circuit breaker pattern: if Redis is slow, serve stale data or fail the request — don't fall through to the database
  • Distinguish timeouts from misses in the cache client; only true misses may query the database
  • Redis Cluster for horizontal scaling; smaller values, no O(N) commands on hot paths
  • Connection pooling with timeouts, and load shedding when approaching capacity

Capacity-testing the cache tier — not just the app tier — is a standard step in our DevOps and reliability engagements for exactly this reason.

Lesson: Your cache is not just a performance optimization. It's a critical component. When it fails, you need a plan that doesn't involve destroying your database.

← Back to Lessons Learned
Tags: #Redis #Caching #Database #Performance