"There are only two hard things in computer science: cache invalidation and naming things." The line is usually credited to Phil Karlton at Netscape. After two production incidents in the same year, we stopped treating it as a joke.

Incident one: the cache key that became a data exposure

"Let's add caching to make it faster." We cached an API response that contained user-specific data — name, email address, order history. The cache key did not include the user ID. For six hours, every user who hit that endpoint saw the first user's data.

We initially filed it as a correctness bug. It was not: it was a data-exposure incident. One person's personal data was served to everyone, which put us squarely in privacy-review territory. We had to purge the cache, reconstruct from access logs who could have seen what, and assess our notification obligations — a day of legal and engineering work caused by one missing field in a key. If you cache anything user-scoped, treat a malformed key as a security failure mode, not a performance bug, and run it through your security operations process like any other exposure.

Incident two: the invalidation fan-out we couldn't track

Months later, a different system. Our invalidation logic on a profile update looked thorough:

  • Invalidate the user cache
  • Invalidate the user's posts cache
  • Invalidate followers' feeds
  • Invalidate the search index and recommendation cache

What we forgot: the user's comments on other people's posts (old display name), notification history (old avatar), and the email preview cache (old bio). Users reported "ghost" versions of themselves for days.

The realization

As a system grows, the number of places a piece of data is cached grows faster than your ability to track them. Both incidents were the same disease: nobody could enumerate every cache, so nobody could reason about keys or invalidation completely.

What we do now

  • A central cache registry. Every cache — Redis, CDN, in-process — gets an entry: owner, key schema, TTL, invalidation triggers, acceptable staleness. A new cache does not pass design review without one. This is the single highest-leverage change we made.
  • Versioned cache keys. Bumping a version prefix retires a whole namespace without hunting individual keys.
  • Targeted invalidation over global busting. Flushing everything on a user-data change sounds safe, but it trades staleness for a thundering herd: every request misses at once and stampedes the database. Version prefixes, jittered TTLs, and request coalescing keep invalidation from becoming its own outage.
  • Accepted staleness where it genuinely doesn't matter — documented in the registry, not decided ad hoc.

Questions we ask before adding any cache

  • Is the data user-specific, and does the key prove it?
  • How often does it change, and what does stale data cost?
  • Can we enumerate every invalidation path today?
  • What is the blast radius if the key is wrong?

Lesson: If you can't list every place a piece of data is cached, you can't reliably invalidate it. And the fastest cache is still the one you don't need — a cache can even add latency — so optimize the query first and cache only what you've measured.

← Back to Lessons Learned
Tags: #Caching #Architecture #Security #Debugging