Every Deploy Hammers the Database for 10 Minutes. Bigger DB?
Scenario
Your e-commerce API deploys several times a day, and every deploy follows the same painful script: the deploy pipeline wipes the entire Redis cache, and for the next ten minutes the primary Postgres instance pins at 90–95% CPU while p99 latency goes from a comfortable ~80ms to multiple seconds, with some requests timing out entirely. Outside those windows the database idles at 25–30% CPU with a cache hit rate around 97%. The expensive product and pricing queries that the cache normally absorbs all land on the database at once, many of them duplicates of each other. There is no request coalescing, and every key carries the same flat TTL.
The Quick Fix on the Table
A teammate wants to double the database instance size. The reasoning: it is only money, the spike is only ten minutes, and a machine twice the size will simply shrug it off. No code changes, no risk, problem gone.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- You'd pay 24/7 for a problem that lasts 40 minutes a day. The database is at 25–30% CPU the rest of the time. Doubling it means paying roughly double, around the clock, to absorb four self-inflicted ten-minute spikes — the most expensive possible way to buy headroom you almost never use.
- It probably doesn't even work. A 97% hit rate means the database normally sees ~3% of read traffic. After a full flush it briefly sees closer to all of it — a 20–30x surge. Twice the CPU does not absorb a 30x surge; it turns a ten-minute crisis into a slightly shorter crisis, and connection limits, lock contention and I/O saturate before CPU generosity saves you.
- The stampede grows with the business. More traffic and more products mean a bigger thundering herd on every deploy. Sizing the database for the stampede is a treadmill: you will be back for the next instance size in a quarter.
- Most of the spike is redundant work. Hundreds of concurrent requests recomputing the same product and pricing queries is pure waste. Buying hardware to do duplicate work faster is optimizing the wrong loop.
- It leaves the real hazard in place. A pipeline that runs
FLUSHALLmeans any deploy — or any incident that restarts Redis — detonates the same bomb. Bigger hardware hides the fuse; it doesn't remove it.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Stop flushing the whole cache on deploy. Interrogate why
FLUSHALLis in the pipeline at all. Almost always the honest need is "invalidate entries whose shape changed." Version the cache keys (embed a schema/app version in the key prefix, e.g.v42:product:123) or delete only the affected prefixes, so a deploy invalidates what changed and nothing else. This one change removes most of the problem. - Add request coalescing for cache misses. On a miss, let one request compute the value while the others wait for that result instead of all hitting Postgres — a per-key lock/singleflight in the application, or a short-TTL "computing" placeholder in Redis. This collapses N identical queries into 1 even during a genuine cold start.
- Warm the cache before it takes traffic. If some invalidation is unavoidable, run a warmer that pre-populates the top-N hottest keys (products, price lists) from access logs or a popularity table before the new version goes live — and roll deploys gradually so a cold instance receives a trickle, not the firehose.
- Jitter the TTLs. Uniform 15-minute TTLs make organic mini-stampedes too: keys populated together expire together. Add randomized spread (e.g. 15 min ± 20%) and consider serve-stale-while-revalidate for the hottest keys so expiry never means a synchronous miss storm.
- Put a floor under the database. Cap the application's connection pool and statement timeouts so that even in the worst case Postgres degrades gracefully instead of collapsing; a pooler such as
pgbouncerkeeps the herd queued at the door rather than trampling inside. - Verify with the numbers you already have. After the changes, the deploy-time graphs should show it directly: cache hit rate barely dips, DB CPU stays near baseline, p99 stays flat through a deploy. Keep the dashboard — it is the regression test for this fix.
- Then price capacity honestly. If steady-state load ever genuinely needs a bigger instance, buy it for that reason — a decision based on baseline utilization, not on a self-inflicted spike.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "We'd be paying double for the database 24 hours a day to survive 40 minutes of damage per day that our own pipeline causes. The cheap fix is to stop causing it."
- "At a 97% hit rate, flushing the cache multiplies database load by twenty to thirty times. Doubling the instance doesn't absorb that — it just moves the timeout line slightly."
- "Most of that spike is hundreds of requests recomputing identical product queries. Coalescing turns those hundreds of queries into one; no instance size does that."
- "Versioned cache keys give deploys exactly what FLUSHALL was trying to give them — fresh data — without throwing away the 97% of the cache that was still valid."
- "If we fix the stampede and the baseline still creeps up as we grow, I'll be the first to propose a bigger instance — sized from real steady-state load, not from a fire we set ourselves four times a day."