The Box Keeps OOM-Killing. Just Add a Huge Swapfile?
Scenario
A latency-sensitive service keeps getting shot by the kernel. Under peak load, memory on the VM spikes, the OOM killer picks the service process, and users see a burst of 5xx errors until the process restarts. The machine has limited RAM and no swap configured. Crucially, nobody has actually established why memory grows — it might be a legitimate working set that outgrew the box, a slow leak, or transient allocation spikes under load. All three look identical on a dying host.
A colleague offers the classic sysadmin reflex: "Add a 16GB swapfile and crank vm.swappiness up. The kernel will page to disk instead of killing the process. No more OOM kills, no more 5xx."
The Quick Fix on the Table
Provision a big swapfile and tune swappiness so the kernel prefers paging over killing. Two commands, no deploy, and the OOM messages disappear from dmesg almost immediately.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- It converts a fast, visible failure into a slow, invisible one. An OOM kill is brutal but honest: seconds of downtime, a clear kernel log line, a restart. A latency-sensitive service pushed into swap doesn't die — it thrashes: page faults turn nanosecond memory access into millisecond disk access, response times explode a thousandfold, and the service is effectively down while every health check still says "up."
- It diagnoses nothing. Working set too big, memory leak, or load spikes each demand a completely different fix. Swap treats all three identically — by hiding them. If it's a leak, 16GB of swap just extends the runway before the same crash, now preceded by hours of degraded latency.
- The OOM killer still arrives — later and angrier. Swap doesn't repeal memory limits; it defers them. When swap fills too, the kernel kills the process anyway, except now recovery starts from a machine that has been thrashing for an hour, with dirty pages to flush before anything responds.
- High swappiness on a latency service is self-sabotage. Raising
vm.swappinessinvites the kernel to page out even reclaimable-but-warm memory proactively. You are volunteering your p99 as a sacrifice before the emergency even starts. - It hides the capacity signal from the business. OOM kills are the metric that says "this workload no longer fits this instance." Muffling that signal with disk means the real conversation — right-sizing, fixing the leak, or shedding load — never happens.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Identify the consumer before touching knobs. Read the kernel's own report first —
dmesg | grep -i oomshows the per-process RSS table at kill time. Track growth over time withsmem,ps aux --sort=-rsssnapshots, or node-exporter memory metrics. Distinguish the three signatures: steady climb between restarts (leak), plateau above RAM (working set), spikes correlated with request bursts (load-driven). - If it's a leak, prove it and fix it in the app. Heap-profile with the runtime's tooling (pprof, async-profiler, tracemalloc, valgrind massif as fits the stack). A leak fixed in code beats any amount of infrastructure accommodation.
- If the working set genuinely grew, right-size the instance. RAM is what this workload needs; buy RAM, not disk pretending to be RAM. Compare the cost of one instance size up against the engineering hours already burned on OOM incidents — it usually wins instantly.
- Contain the blast radius with cgroups. Run the service under a systemd unit with
MemoryMax=(andMemoryHigh=for early throttling), so an errant allocation kills or throttles this service predictably instead of letting the global OOM killer choose a victim. Protect critical co-tenants by lowering their kill priority viaOOMScoreAdjust=. - Make the failure graceful instead of preventing it badly. Add memory-based load shedding or admission control in the service, cap queue depths and request body sizes, and alert on memory utilization at 80% — so humans act before the kernel does.
- Use a small swap deliberately, if at all. A modest swapfile (1–2GB) with low swappiness (
vm.swappiness=1–10) is defensible as a shock absorber for truly cold pages — paired with an alert on any sustained swap-in/swap-out activity (vmstatsi/so), because for this service, swapping is an incident.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "An OOM kill costs us seconds of downtime and tells us exactly what died. Thrashing into 16GB of swap costs us an hour of thousand-fold latency while every health check stays green — for a latency-sensitive service, that's the worse outage."
- "Swap doesn't answer the only question that matters: leak, working set, or spike? Each has a different fix, and the
dmesgOOM report plus a day of RSS tracking will tell us which we're in." - "If it's a leak, big swap just delays the same crash and adds hours of degraded service in front of it. If the working set outgrew the box, the honest fix is RAM — disk pretending to be RAM is paying for latency with more latency."
- "I'd rather bound the failure than blur it: a cgroup
MemoryMaxmakes the worst case predictable and protects everything else on the host, and an 80% memory alert gets a human there before the kernel." - "A small swapfile with swappiness near 1 as a shock absorber — fine, with an alert on any actual swapping. Sixteen gigabytes with high swappiness isn't a safety net; it's a muffler on the fire alarm."