Blue-Green Database Nightmare

Blue-green works great for stateless apps. Our database schema changes made rollbacks impossible.

The situation:

• Deploy new version to Green
• New version runs DB migration
• Switch traffic to Green
• Bug discovered
• Want to rollback to Blue...
• But Blue can't read the new schema!

The destructive migration:

-- Migration
                                    ALTER TABLE users DROP COLUMN legacy_id;
                                    ALTER TABLE users RENAME COLUMN user_id TO id;

Old code expects legacy_id and user_id. They're gone.

The fix - expand/contract pattern:

1. Expand: Add new column, keep old ones
2. Migrate: Deploy code that writes to both
3. Backfill: Copy data to new column
4. Contract: After stable, remove old column

Key principle:

• Schema changes must be backward compatible
• N-1 version must work with N schema
• Never delete/rename in same release

Lesson: Database is the hardest part of blue-green. Plan for it explicitly.

---

← Back to Lessons Learned