Autoscaling That Killed the Database

Our Kubernetes cluster autoscaled perfectly. Too perfectly.

The incident:

• Traffic spike at 10 AM
• HPA scaled pods: 5 → 50
• Each pod: 20 database connections
• Total connections: 100 → 1,000
• PostgreSQL max_connections: 200
• "FATAL: too many connections" across every new pod

The cascade:

New pods couldn't connect, failed their health checks, and were restarted — and every restart opened a fresh burst of connection attempts against a database already at its limit. Within minutes the cluster was thrashing: pods cycling, connection storms on every cycle, and the original traffic spike still unserved.

The fix:

PgBouncer in front of PostgreSQL, running in transaction pooling mode: 1,000 application-side connections multiplexed onto roughly 100 real server connections. The mode choice matters. Session pooling pins one server connection per client for its lifetime, so it wouldn't have reduced our count at all. Transaction pooling releases the server connection after each transaction — that's where the 10:1 multiplexing comes from — but it breaks anything that assumes session state: session-level advisory locks, LISTEN/NOTIFY, session SET variables, and prepared statements on older drivers. We audited for those first; the one service using advisory locks got its own small session-mode pool.

Second, we stopped the HPA from stampeding. Scale-up is now rate-limited so downstream systems get time to absorb each step, and max replicas is capped at what the data tier can actually sustain:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - type: Pods
      value: 4
      periodSeconds: 60
maxReplicas: 30

On AWS, RDS Proxy is the managed alternative: same multiplexing idea, IAM auth, nothing to operate — at per-vCPU pricing, and with the caveat that session state "pins" connections and quietly disables the multiplexing you're paying for.

What we deliberately did not do:

Gate readiness probes on a database connection. It sounds sensible, but readiness on a shared dependency means that when the database blips, every pod goes unready at the same moment, the Service loses all its endpoints, and a partial degradation becomes a total outage — the exact cascade we were trying to stop. Probes should verify the pod, not vote on shared infrastructure; we cover that failure mode in health checks that lied.

Lesson: Autoscaling doesn't know about downstream limits. You must enforce them — in the pooler, in the HPA policy, and in capacity planning for the one component that doesn't autoscale. Sizing that chain end-to-end is core DevOps engineering work, not a tuning afterthought.

---