One Giant Terraform State Runs Everything
Scenario
Every piece of your cloud estate — networking, IAM, databases, Kubernetes clusters, application resources across environments — lives in one Terraform root module with one state file. A terraform plan takes ten minutes because it refreshes hundreds of resources, and the output is a wall of text where the one change you made hides among provider-side noise. Engineers have started avoiding infrastructure changes altogether, because a typo in an app-level variable can, in principle, produce a plan that touches the production database. Changes queue up, then land in big risky batches — exactly the pattern the avoidance was trying to prevent.
When the pain comes up in retro, a teammate suggests keeping the structure as-is: "The state works. We just need to be more careful — read every plan line by line, and only seniors apply." Restructuring state sounds risky, and nobody wants to be the person who broke prod moving it.
The Quick Fix on the Table
Keep the monolithic state and add process: mandatory plan reviews, senior-only applies, more caution. It requires zero engineering work up front and feels responsible — vigilance as architecture.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- "Be careful" doesn't scale and doesn't survive 2 AM. A ten-minute plan with 300 resource lines will be skimmed, not read — especially during an incident. Human review of noisy output is precisely the control that fails when it matters most.
- The blast radius stays maximal. As long as one state holds everything, every apply carries theoretical write access to everything. One provider bug, one bad refresh, one mis-scoped
-target, and the failure domain is your whole company. - Slow plans are a tax on every change, forever. Ten-minute refresh cycles make small, frequent, low-risk changes economically irrational. Teams respond by batching — and change failure rate rises with batch size.
- One state is one lock. A single state file means one team's apply blocks everyone else's. As the org grows, the monolith becomes a coordination bottleneck with a queue in front of it.
- Caution actively worsens the avoidance problem. The scenario's root issue is that people already fear changes. Adding ceremony raises the cost of change further, which means more deferral, bigger batches, and scarier applies — the exact spiral you were trying to exit.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Draw the target layout first. Split along ownership and rate-of-change boundaries, per environment: a slow-moving foundation stack (VPC, DNS, IAM), a platform stack (clusters, shared databases), and per-team/per-service application stacks. Each gets its own state and its own pipeline; a bad apply in one cannot touch the others.
- Wire cross-stack references read-only. Downstream stacks consume upstream outputs via
terraform_remote_statedata sources or provider data lookups (by tag/name). Consumers can read the VPC ID; they can never mutate the VPC. - Migrate resources without touching the cloud. State surgery is metadata-only when done right: pull a copy (
terraform state pull), then move resources between states withterraform state mv -state-out=..., or — inside a refactor — usemovedblocks so Terraform records renames instead of planning destroy/create. Import into the new stack withimportblocks where cleaner. - Prove each migration with a zero-change plan. The acceptance test for every extracted stack is
terraform planshowing No changes in both the new stack and the shrunken monolith. Nothing applies until both are clean. - Go incrementally, safest slice first. Extract one low-risk, well-bounded stack (e.g. monitoring or a single stateless service) end to end, including its pipeline and state backend key. Bank the pattern, then repeat. Never attempt the big-bang split.
- Add guardrails that don't depend on vigilance.
lifecycle { prevent_destroy = true }on stateful resources, separate state backends/prefixes per environment with distinct IAM, and CI policy checks (OPA/Sentinel/terraform plan -detailed-exitcodegates) that block destructive plans on protected resources automatically. - Measure the payoff. Track plan duration, applies per week, and plan-output size before and after. Sub-minute plans and single-digit-line diffs are what turn infrastructure changes back into routine work.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "'Be more careful' is asking humans to compensate for an architecture problem. The plan is ten minutes and hundreds of lines — the day someone skims it under pressure is the day it fails, and that day is coming."
- "The scary part isn't splitting state — it's that today every apply, from anyone, can theoretically reach the production database. State moves are reversible metadata edits; the monolith's blast radius is not."
- "Splitting is safer than it sounds:
terraform state mvandmovedblocks change bookkeeping, not infrastructure, and every step is validated by a 'No changes' plan before anything applies." - "Small states give us small plans, and small plans are the review process actually working — a three-line diff gets genuinely read, a three-hundred-line one gets scrolled."
- "We don't need a big bang. One stack extracted this sprint, pattern proven, then repeat. Meanwhile
prevent_destroyon the database goes in today — that guardrail costs one line."