Blue-Green Config Drift
Blue-green deployment worked perfectly. Except the green environment had different config.
The setup:
- Blue environment: running production
- Green environment: idle, ready for deploy
- Deploy to green, test, switch traffic
What went wrong:
- Feature flag updated in Blue
- Nobody updated Green
- Deployed new version to Green
- Switched traffic
- Feature flag reverted to old value
- Users lost a feature they'd been using for weeks
More config drift found:
- Different database connection pool sizes
- Different log levels
- Different timeout values
- API keys rotated in Blue only
The fix:
- Config as code (all config in repo)
- Deploy config separately from code
- Config sync check before traffic switch
- External config store (not per-environment)
Lesson: Blue-green deployments require blue-green config management.