Pods Keep OOMKilling: Scale Up or Fix the Leak?
Scenario
A payments worker service gets OOMKilled every three to four hours under perfectly ordinary load. The memory graph is a textbook sawtooth: usage climbs steadily from startup under a 512Mi limit until the kernel kills the container, the pod restarts, and the climb begins again. Jobs are delayed around each kill but nothing fully breaks, so the pain is chronic rather than acute. Everyone quietly suspects a leak; the heap profiler that would confirm it has never been pointed at the service.
The Quick Fix on the Table
A colleague proposes the pragmatic-sounding bundle: raise the limit to 2Gi, add two more replicas, and put "investigate the leak" on the backlog for a future sprint. Kills stop today, payments flow, everyone moves on.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- Against a leak, a bigger limit buys hours, not a fix. The sawtooth shows steady growth regardless of ceiling height. Four times the memory means roughly four times longer between kills — the crash moves from every 3–4 hours to overnight and weekends, which is a worse time to crash, not a better outcome.
- More replicas multiply the problem. Every added replica is another copy of the same leak marching toward its own OOM kill, and another 2Gi of cluster memory reserved for garbage. You are horizontally scaling the defect.
- "Later" means never, and the evidence goes stale. Once kills stop appearing on the dashboard, the backlog ticket loses its urgency forever. Meanwhile the leak keeps shipping in every release, code drifts, and the eventual investigation starts from a colder trail.
- This is a payments worker being killed mid-work. Each OOM kill is a SIGKILL — no graceful shutdown, no cleanup. Unless every job is perfectly idempotent, the team is betting money-movement correctness on the exact property nobody has verified. Making kills rarer but heavier does not make that bet safer.
- The cost is permanent and invisible. Requests sized for the leak reserve node capacity around the clock. Multiply 2Gi by replicas by environments and the "free" fix has a real monthly price — paid indefinitely for holding garbage in RAM.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
Buy a little time cheaply, then spend it actually finding the leak — with the investigation scheduled now, not "a future sprint".
- Take a modest mitigation, deliberately. Raise the limit one step (e.g. 512Mi → 1Gi) to stretch the kill interval and calm the noise — explicitly labelled as anesthesia, not surgery, with the profiling work committed in the same breath.
- Confirm it is really a leak, not sized-out working set. Check
kubectl top podtrends and the container memory graphs against workload: does memory correlate with queue depth and fall when load falls (undersized working set / cache), or climb monotonically regardless (leak)? The sawtooth under normal load points to leak — verify it. Also sanity-check limits vs requests: memory limit should generally equal request for a service like this, so the pod's failure mode is predictable rather than node-pressure-dependent. - Point a profiler at one replica in production. Enable the runtime's native tooling —
pprofheap profiles for Go, async-profiler or heap dumps (jmap) for the JVM,tracemalloc/py-spy for Python, heap snapshots for Node — and capture profiles at intervals along the sawtooth. Diffing two profiles an hour apart usually names the allocation site outright. Continuous profiling (e.g. Parca or Pyroscope) makes this a standing capability instead of a one-off. - Look at the usual suspects while profiles collect. Unbounded in-process caches or maps keyed by job ID, listeners/goroutines/threads spawned per job and never released, response bodies or DB cursors not closed, and per-job clients created instead of reused. Recent diffs to the worker's job-handling path get first review.
- Fix, then prove it with the same graph that convicted it. After the patch, the memory curve should go flat (or gently oscillate with load) across a multi-day soak. The sawtooth's disappearance is the acceptance test.
- Right-size afterwards and add guardrails. Set requests/limits from the observed post-fix working set plus sensible headroom — likely well below 2Gi. Alert on sustained memory growth rate (e.g. climbing steadily over hours) rather than on kills, so the next leak is caught in review-time, not pager-time.
- Verify job idempotency regardless. The investigation surfaced a real risk: SIGKILL mid-payment-job. Whatever the leak outcome, confirm jobs are safely retryable — that hardening outlives this incident.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "The graph is a monotonic climb to the ceiling — at 2Gi it's the same climb with a higher ceiling. We'd move the crashes from every four hours to the middle of the night, which is not an improvement."
- "Adding replicas to a leaking service means running more copies of the leak. We'd be paying for extra cluster memory whose job is to hold garbage a little longer."
- "Every OOM kill is a hard SIGKILL on a payments job with no graceful shutdown. Fewer-but-bigger crashes doesn't make that safe — finding the leak and verifying idempotency does."
- "I'm fine taking one modest limit bump today as painkillers — but the profiling session goes on this sprint's board, because once the dashboard goes quiet, 'later' never comes."
- "A heap profile diff across two hours of that sawtooth will likely name the exact allocation site. That's a day of work against a permanent 4x memory bill and a crash we've merely rescheduled."