ConfigMap Typo Outage

Production down for 2 hours. Root cause: A single typo.

The change:

# ConfigMap update
data:
  DATABASE_CONNECITON_TIMEOUT: "30000"  # typo: CONNECITON

What happened:

• App looked for DATABASE_CONNECTION_TIMEOUT
• Didn't find it (typo), and no default was set
• The config loader silently passed 0 to the database driver
• In our driver — as in JDBC and libpq — a timeout of 0 means no timeout
• Queries against an already-slow replica hung indefinitely, the connection pool filled with stuck connections, and every request behind it queued until the service stopped responding

Why it wasn't caught:

• ConfigMaps are just YAML—no schema
• Kubernetes doesn't validate key names
• PR review missed the typo
• Integration tests used different config

The fixes:

• App validates required env vars at startup
• Fail fast if critical config is missing
• Schema validation for ConfigMaps in CI
• Canary deployments to catch issues early

Config schema checks are one of the cheapest gates you can add to a CI/CD pipeline — a few lines of validation against two hours of downtime.

Lesson: ConfigMaps are not code—treat them like they are.

---