Terraform Wants to Replace the Database. Apply Now?
Scenario
The change was supposed to be trivial: update one tag on the production RDS instance. But the Terraform plan comes back with the line every operator dreads — the database is marked for replacement, # forces replacement, destroy then create. It is mid-afternoon on a Tuesday, traffic is at its daily peak, and three engineers are pinging you because their branches are stacked behind this change. Hourly automated snapshots exist, and the pipeline's apply stage is one approval click away.
The temptation assembles itself: snapshots mean recoverable, the change is "just a tag," teammates are blocked, and Terraform itself presents replacement as a perfectly normal plan. One click and everyone is unblocked.
The Quick Fix on the Table
Approve the apply. Terraform says this is the state transition; snapshots are the safety net; and holding the queue for a tag change feels disproportionate.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- "Destroy and recreate" means real downtime and real data exposure. The instance is deleted, then a new one is provisioned — many minutes of hard outage at peak traffic, plus every write since the last hourly snapshot permanently gone. A tag is worth exactly zero of either.
- Snapshots are a floor, not a shield. Restoring produces a new instance: new endpoint, warm-up time, parameter/security-group reattachment, and up to an hour of lost transactions. "We have snapshots" describes how bad the bad day is, not whether to schedule one.
- A tag change should never force replacement — so the plan is telling you something else. Something besides the tag differs between config and reality: a changed immutable attribute, drift from a manual console edit, a provider version behavior change, or a renamed resource address. Applying without knowing which means executing a change you do not understand on your most stateful resource.
- Peer pressure is not an operational input. Three engineers waiting to rebase lose minutes; a destroyed production database loses customers. The asymmetry could not be steeper, and "people were waiting" has never once looked good in a postmortem.
- The click normalizes a lethal pattern. If the team learns that destructive plans on stateful resources get approved under mild pressure, the guardrail is gone permanently — the next one might be silently hidden in a 40-resource plan.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Stop the apply and say why, in one sentence. "The plan destroys the production database; nothing applies until we know what forces the replacement." Post it where the blocked engineers can see it — clarity dissolves most of the pressure.
- Find the forcing attribute. Read the plan carefully: Terraform annotates the exact argument with
# forces replacement. Useterraform plan -out=tfplanandterraform show -json tfplanto inspect it precisely. Typical culprits for RDS:identifier,engineversion downgrades,availability_zone, storage type transitions, or a resource address change making Terraform see a "new" resource. - Check for drift and address changes. Run
terraform plan -refresh-onlyto surface manual console edits, and check whether the resource was renamed or moved in a refactor. An address change is fixed with amovedblock orterraform state mv— bookkeeping, zero cloud impact. - Choose the non-destructive path for the real difference. If config drifted from reality, align the config to reality (or accept the drift into code). If an attribute genuinely must change, find the in-place route: many RDS changes are modifiable with
apply_immediately = falseduring a maintenance window, or via a blue/green deployment with a replica promotion — not a destroy. - Install the guardrails this near-miss just paid for. Add
lifecycle { prevent_destroy = true }to the database and every irreplaceable stateful resource, enable RDS deletion protection at the AWS level (deletion_protection = true), and add a CI check that fails any plan containing a delete on a protected resource list. - Unblock the team without the risk. The waiting engineers need the tag change merged, not the database replaced. Once the forcing attribute is neutralized and the plan shows an in-place update (
~not-/+), apply confidently — usually the same day.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "The plan doesn't say 'update a tag' — it says 'delete the production database and build a new one.' I'm not approving a delete I can't explain, at peak traffic, for a tag."
- "Snapshots cap the damage at 'up to an hour of customer transactions gone plus a long outage.' That's the disaster-recovery floor, not an acceptable cost for a Tuesday afternoon change."
- "A tag can't force replacement, so the plan is really telling us about drift, a rename, or a changed immutable attribute. Twenty minutes with
terraform showtells us which — and almost certainly gives us a zero-downtime path." - "The blocked branches cost us minutes. Getting this wrong costs an outage, data loss, and every future plan being distrusted. I'll take the minutes."
- "Whatever we find, two lines go in today:
prevent_destroyin Terraform and deletion protection on the instance — so this decision never again rests on someone's afternoon judgment."