Blue-Green Config Drift

Blue-green deployment worked perfectly. Except the green environment had different config.

The setup:

• Blue environment: running production
• Green environment: idle, ready for deploy
• Deploy to green, test, switch traffic

What went wrong:

A feature flag had been flipped in blue weeks earlier — directly, through the admin UI. Nobody touched green. When the next release deployed to green and traffic switched over, the flag silently reverted to its old value and users lost a feature they'd been relying on for weeks. No deploy failed, no alert fired; the environment was simply telling the truth about its own stale config.

Once we audited, we found more drift:

• Different database connection pool sizes
• Different log levels
• Different timeout values
• API keys rotated in blue only

The fix — two kinds of config, two homes:

The naive fix list — "put config in Git" and "use an external config store" — points in opposite directions until you separate what belongs where.

Versioned config lives in Git and ships with the release. Pool sizes, timeouts, log levels, resource limits: these are properties of a release, so they belong in the repo as rendered manifests applied identically to blue and green. A GitOps controller — Argo CD in our case, Flux works equally well — continuously diffs live state against Git, so an out-of-band edit to either environment shows up as drift within minutes instead of at cutover. This is the same discipline we treat as table stakes in CI/CD pipeline design.

Runtime toggles live in one shared external service. Feature flags must not exist per-environment at all. We moved them to a single flag service (an OpenFeature-compatible provider) that both colors read from, so a flag flip applies to whichever environment is serving traffic. Flag state has one home; blue and green are just consumers.

And a gate before every cutover. The pipeline diffs the rendered config of both environments and refuses to switch traffic on any delta:

# refuse to cut over if blue and green disagree
diff \
  <(kubectl get cm app-config -n blue -o jsonpath='{.data}') \
  <(kubectl get cm app-config -n green -o jsonpath='{.data}') \
  || { echo "config drift detected - aborting cutover"; exit 1; }

Secrets got the same treatment: rotated in one external store and synced into both environments by an operator, never rotated "in blue" by hand. The database side of blue-green has its own traps — see our blue-green database nightmare.

Lesson: Blue-green deployments require blue-green config management. An idle environment drifts by default; only automation notices.

---