Pod Stuck in CrashLoopBackOff After a Deploy
Scenario
A deploy just went out and the new pods never reach Ready. kubectl get pods shows CrashLoopBackOff with the restart counter climbing every couple of minutes. The release contained two changes: a new application version and a new environment variable. The cluster has plenty of free CPU and memory, so this is not a scheduling or capacity squeeze — the container starts, dies, and Kubernetes dutifully restarts it into the same wall, backing off longer each time.
The Quick Fix on the Table
A colleague glances at the restarts and offers the classic: "give it more CPU and memory and it'll settle." Bump the resource block, redeploy, hope. It feels like action, requires no reading of logs, and sometimes — by pure coincidence — a redeploy shakes something loose and it "works".
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- It's a guess wearing a fix's clothes. CrashLoopBackOff is not a diagnosis — it only means "the container keeps exiting." Config errors, missing secrets, failed migrations, bad images, unreachable dependencies and panics on boot all produce the identical status. Changing resources addresses exactly one cause out of a dozen, chosen without evidence.
- The evidence says resources aren't it. The cluster has headroom, and genuine memory kills announce themselves:
OOMKilledin the container status and exit code 137. Nobody has looked. If the exit code is 1, all the memory in the fleet won't change it. - Two things changed; the fix ignores both. A new app version and a new environment variable landed together — the two most likely suspects by far. Raising limits investigates neither.
- Even a lucky "fix" is a loss. If a redeploy happens to succeed, you have learned nothing, permanently inflated the resource requests (paying for it on every node forever), and left the actual bug in the code path waiting for its next opportunity.
- It burns the cheapest resource you have. The diagnosis takes three kubectl commands and five minutes. Trial-and-error resource bumps take a redeploy cycle per guess — slower than just looking, and each cycle extends the broken deploy.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Read the crash, not the status.
kubectl logs <pod> --previousshows the output of the container that just died — usually the stack trace or fatal config error is right there. (Without--previousyou're looking at the fresh container that hasn't crashed yet.) - Get the exit code and reason.
kubectl describe pod <pod>→ Last State. Exit code 1: application error — go to the logs. 137 withOOMKilled: memory limit genuinely too low — the colleague earns the point. 127/126: bad command or entrypoint. Also scan Events for image pull failures, missing ConfigMaps/Secrets, and probe kill messages. - Interrogate what changed.
kubectl rollout history deployment/<name>and a diff of the manifests. The new env var is a prime suspect: typo in the name, wrong value, a reference to a Secret key that doesn't exist — or the new code requiring a variable that only some environments define. The new version is the other: a startup migration failing, or a dependency it can't reach. - Rule out probe murder. If logs look healthy and the app seems to boot, check whether the liveness probe is killing a slow-starting container — the describe Events will show probe failures before each restart. The fix there is a
startupProbeor realisticinitialDelaySeconds, not more CPU. - Restore service while you fix. If customers are affected,
kubectl rollout undo deployment/<name>to the previous version buys calm; the broken manifest is still in git for the investigation. - Fix the actual cause and redeploy through the pipeline. Correct the env var, add the missing secret, fix the migration — whatever the evidence named — then watch the rollout go Ready.
- Close the loop. Add a startup config-validation step (fail fast with a clear message when required vars are missing) and a CI check that manifests reference only existing ConfigMap/Secret keys, so this class of crash-loop dies in the pipeline next time.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "CrashLoopBackOff only tells us the container keeps dying — it doesn't say why. Two kubectl commands give us the exit code and the dying container's logs; let's spend five minutes on evidence before a redeploy on a guess."
- "If this were memory, the pod status would say OOMKilled with exit code 137. Until we've looked, raising limits is treating a symptom we haven't even confirmed exists."
- "This deploy changed exactly two things — the app version and a new env var. The odds are overwhelming the answer is in one of those, and neither is investigated by adding CPU."
- "Even if a resource bump happens to coincide with it settling, we've learned nothing, we pay for inflated requests on every replica forever, and the real bug is still in there."
- "If the logs do show an OOM kill, I'll raise the memory limit myself — with a number based on what the container actually used, not a doubling in the dark."