Disk Is at 100% and the App Is Down. rm -rf?
Scenario
A single VM runs your application behind nginx. Suddenly the site is throwing 500s and the logs are full of "No space left on device" — the root filesystem is at 100%. There is no log rotation configured, and /var/log has been growing quietly for months during perfectly normal operation. The pressure is on: the app is down right now.
The Quick Fix on the Table
A colleague wants to skip the analysis: rm -rf /var/log, free the space in one shot, restart the service. "It always works." It is one command, and the outage clock is ticking.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- You delete the evidence mid-incident. The logs you are about to erase are the record of what happened, whether anything else went wrong, and what has been filling the disk. Post-incident review becomes guesswork.
- Deleting open files often frees nothing. If a running process still holds a log file open, the kernel keeps the space allocated after
rm. The disk stays at 100% and now the file is invisible tols— a strictly worse debugging position. - Removing the directory itself breaks services. syslog, nginx and many daemons expect
/var/logand its subdirectories to exist with specific ownership; nuking the tree can turn a disk-full outage into a services-won't-start outage. - You haven't confirmed logs are even the problem. Core dumps, a runaway cache, database bloat or an exhausted inode table can all present as "disk full". Deleting logs on a hunch may free 2% and leave you exactly where you started.
- Nothing prevents the rerun. Without rotation, the disk refills at the same rate and the same 3am page comes back in a few weeks.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Confirm the shape of the problem.
df -hshows which filesystem is full, anddf -irules out inode exhaustion masquerading as a full disk. - Find where the space actually is.
du -xh --max-depth=1 / | sort -h(orncdu -x /) walks down to the offending directory in a minute — verify it really is/var/log, and which files inside it. - Check for deleted-but-open files.
lsof +L1lists files that were removed while a process still holds them; if a previous "cleanup" already happened, this is where the missing gigabytes are, and the fix is restarting or signaling the holding process. - Truncate instead of deleting.
truncate -s 0 /var/log/app/huge.log(or> huge.log) frees the space immediately even for open files, keeps the file handle valid for the writing process, and preserves recent history if you copy the tail first. Compress older logs withgziprather than erasing them. - Restore service and verify. With headroom recovered, restart the affected services and confirm the app is healthy and writing logs normally again.
- Fix the cause: rotation. Configure
logrotatewith size caps, compression and retention for the offenders — and check journald'sSystemMaxUse— so growth is bounded by policy instead of disk size. - Add an early warning. A disk-usage alert at 80% turns the next occurrence into a calm ticket instead of an outage.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Two minutes of df and du isn't delay — it's making sure the one command we run actually frees space. rm on an open log file frees nothing and hides the file."
- "Truncating gets us the same free space as deleting, right now, without breaking file handles or losing the ability to see what happened."
- "Those logs are the flight recorder for this incident. Erasing them means the postmortem is fiction."
- "rm -rf restores service until the disk refills. Logrotate plus a disk alert means we never run this incident again — that's the fix worth defending."
- "If 'always works' were true we'd have rotation and no outage. The fact that we're down at 100% is proof the quick version has been failing us silently for months."