High Availability That Wasn't

We had 3 replicas across 3 nodes. We thought we were highly available. We weren't.

The setup:

• 3 replicas of each service
• 3 Kubernetes nodes
• Pod anti-affinity: spread across nodes — done
• All nodes: us-east-1a — the part nobody checked

The incident:

• AWS us-east-1a partial outage
• All 3 nodes affected
• All pods evicted
• No capacity in the zone
• Complete service outage

Why it happened:

• Cheaper instances in us-east-1a
• Auto-provisioner defaulted to single AZ
• Nobody checked node distribution

The fix:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-service

Pod-level constraints only help if nodes actually exist in multiple zones: we also reconfigured the node groups and cluster autoscaler to span three AZs, so the scheduler had somewhere to spread to. Verifying that redundancy holds at every layer — pod, node, zone — is a standard checkpoint in our cloud architecture reviews.

Lesson: Replicas without zone distribution is not high availability.

---