Blue-green works great for stateless apps. Both environments share one database — and our schema changes made rollback impossible exactly when we needed it.

The situation:

  • Deploy new version to green
  • New version runs its DB migration on startup
  • Switch traffic to green
  • Bug discovered an hour later
  • Want to roll back to blue...
  • But blue can't read the new schema

The destructive migration:

-- The migration that shipped with the green release
ALTER TABLE users DROP COLUMN legacy_id;
ALTER TABLE users RENAME COLUMN user_id TO id;

Old code expects legacy_id and user_id. They're gone. The "instant rollback" that justified blue-green in the first place now requires restoring a backup or hand-writing a reverse migration during an incident — the worst possible time to be writing DDL.

The fix — expand/contract, worked over three releases:

Release N (expand). Add the new column without touching the old ones. Code dual-writes but still reads the old column:

ALTER TABLE users ADD COLUMN id BIGINT;
-- backfill in batches to avoid long locks
UPDATE users SET id = user_id WHERE id IS NULL AND user_id < 100000;
-- ...repeat per batch, then:
CREATE UNIQUE INDEX CONCURRENTLY users_id_key ON users (id);

Roll back at this point? Trivial — old code never sees the new column.

Release N+1 (migrate reads). Code now reads from id, still writes both columns. Run a parity check comparing the two columns before proceeding. Roll back? Also fine — the previous release still dual-writes, so neither column goes stale.

Release N+2 (contract). Only after N+1 has been stable and query logs confirm nothing references the old columns, drop them:

ALTER TABLE users DROP COLUMN legacy_id;
ALTER TABLE users DROP COLUMN user_id;

The destructive step still happens — but two releases after the risky code change, when a rollback of the app no longer depends on the columns being there.

Key principle:

  • Every schema change must be backward compatible with the running release
  • N-1 version must work with N schema — that is the rollback contract
  • Never delete or rename in the same release that stops using something

Config suffers the same shared-state problem between blue and green — we hit that too, in blue-green config drift.

Lesson: The database is the hardest part of blue-green because it isn't blue or green — it's shared. Plan migrations for it explicitly, as part of deployment automation rather than an afterthought bolted to app startup.

← Back to Lessons Learned

Incidents like this are preventable. Our AWS Managed Services team watches client environments 24/7 with AI-assisted monitoring and senior engineers on the pager — so stories like this stay in the archive.

Tags: #Deployment #BlueGreen #Database #Migration