Ingress Controller Single Point of Failure

We had HA for every service. Except the one thing that routes traffic to all of them.

The setup:

• 20 services with 3+ replicas each
• Multi-AZ deployment
• Pod disruption budgets
• NGINX Ingress Controller: 1 replica

The incident:

• Ingress controller pod OOMKilled
• 30 seconds to reschedule
• With no NGINX pod running, the cloud load balancer had no healthy targets — clients got connection failures and 503s from the LB, not even a clean 502
• Every service behind it was unreachable at once

Why just 1 replica?

• "It's just infrastructure, it never fails"
• Default Helm chart value: 1
• Nobody changed it

The fix:

controller:
  replicaCount: 3
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
          topologyKey: kubernetes.io/hostname

Also:

• PodDisruptionBudget with minAvailable: 2
• Proper resource requests/limits
• HPA for traffic spikes

It's the same blind spot as replicas that all landed in one availability zone: redundancy you haven't verified isn't redundancy.

Lesson: Your ingress controller IS your availability. Treat it that way.

---