Someone Changed Infra by Hand. Terraform Plan Is a Mess.
Scenario
During a recent incident, an engineer went straight into the cloud console and adjusted things by hand — a couple of load balancer settings, some security group rules — to get the fire out. It worked, but nothing was written down and nothing went through version control. Days later, someone runs terraform plan and gets a wall of unexpected diffs: Terraform wants to revert settings on the load balancer and the security group back to what the code says. Nobody is sure which of the manual changes were the real fix and which were temporary experiments that should never survive. State and reality have parted ways, and every apply is now a loaded gun.
The Quick Fix on the Table
Two competing shortcuts get proposed in the same meeting. One camp says "just run terraform apply — the code is the source of truth, whatever it reverts obviously wasn't official." The other says "just run terraform refresh or update the code to match whatever is live — reality wins, the incident fix must be preserved." Both are one-command solutions, and both are wrong as blanket policies.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- Blind apply can re-trigger the incident. If one of those manual edits was the actual fix — say, a health check timeout or an allowed CIDR — reverting it to the coded value puts production straight back into the failure mode you just escaped.
- Blind "reality wins" launders unreviewed changes into the codebase. Absorbing every console edit into code or state ratifies changes nobody reviewed, including debugging leftovers like an overly-broad security group rule that was only meant to live for an hour.
- Drift is rarely one change. A messy plan usually mixes intentional fixes, abandoned experiments, and provider-side noise. Any single-command answer treats all three identically, which guarantees at least one wrong outcome.
- It does nothing about recurrence. Neither shortcut answers the real question: why could someone change production out-of-band without the change landing back in code? Fix the plan today and the same mess is back after the next incident.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Freeze applies first. Announce that nobody runs
terraform applyagainst this workspace until drift is reconciled — lock the workspace or pause the pipeline if your tooling supports it. - Capture the full diff. Run
terraform plan -refresh-onlyto see exactly what changed in reality versus state, and a normalterraform planto see what an apply would do. Save both outputs as the incident artifact. - Interview before you decide. Talk to the engineer who made the changes and pull the cloud audit trail (
CloudTrailor equivalent) to timestamp each modification. Classify every drifted attribute as "intentional fix," "temporary debug," or "unknown." - Reconcile per resource, not per plan. For changes that should stay: update the
.tfcode to match reality, then apply — the plan for those resources becomes a no-op. For changes that should go: leave the code as-is and let apply revert them, on your schedule and with eyes on dashboards. - Accept reality into state where needed. Use
terraform apply -refresh-onlyto sync state with live values you've decided to keep, so future plans are clean rather than noisy. - Verify with a clean plan. The exit criterion is simple:
terraform planshows no changes. Until then the reconciliation isn't done. - Close the barn door. Restrict console write access for humans (read-only by default, break-glass roles for incidents), add scheduled drift detection so divergence is caught in hours instead of weeks, and write a one-page emergency-change rule: console changes during incidents are allowed, but must be codified before the incident ticket closes.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Neither 'code always wins' nor 'reality always wins' is a policy — it's a coin flip. Some of those console edits were the fix and some were debris; only a per-resource review can tell them apart."
- "If we blind-apply and one of those settings was the incident fix, we've just scheduled a repeat of the outage with extra steps."
- "If we blind-absorb reality, we're merging unreviewed production changes into main without a pull request — that's the exact thing IaC exists to prevent."
- "The reconciliation takes an afternoon. Recovering from applying the wrong half of this plan takes a lot longer, and we'd be doing it during an outage."
- "Drift is a process failure, not a Terraform failure. Unless we add drift detection and a rule that emergency console changes get codified same-day, we'll be having this meeting again next quarter."