Our rate limiter stored state in Redis. Redis went down. Rate limiting stopped. The abusive traffic it had been quietly absorbing did not.

The timeline:

  • 14:00 - Redis master taken down for maintenance; failover didn't kick in as expected
  • 14:01 - Rate limiter couldn't read/write state; fail-open: all requests allowed
  • 14:02 - The scraper and bot traffic the limiter had been shedding — several times our legitimate load — hit the API unthrottled
  • 14:03 - API servers saturated
  • 14:05 - Complete outage

The design flaw:

We framed it as a binary choice: "fail-open" (allow everything) or "fail-closed" (block everything). We picked fail-open and never designed the middle option — keep limiting, just with cruder local state.

Why we made that choice:

"We don't want Redis issues to block legitimate users." Reasonable — full fail-closed would have turned every Redis blip into a self-inflicted total outage. But fully open was the other extreme.

What we should have done:

  • In-memory fallback rate limiter per instance
  • Conservative limits when shared storage is unavailable — degraded enforcement, not none
  • Redis replication with automatic failover, tested before it's needed
  • Rate limiting at multiple layers (CDN, API Gateway, App), so no single store is load-bearing

Walking through how each control behaves when its dependencies die is a core part of our SecOps reviews — most teams have never asked the question until an incident answers it.

Lesson: A rate limiter that fails fully open is not a control, and one that fails fully closed is an outage generator. Design the degraded mode: local state, conservative limits, multiple layers.

← Back to Lessons Learned
Tags: #Redis #RateLimiting #API #Security