Health Checks That Lied

Our pods reported healthy but couldn't serve traffic. How?

The health check:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 10

The handler:

@GetMapping("/health")
public String health() {
    return "OK";  // always returns OK
}

The app would start, lose its database connection, time out against the cache — and keep answering "OK". Kubernetes kept every pod in rotation, the load balancer kept routing, and users got errors from pods the platform swore were fine.

The first fix made it worse:

The obvious patch — wire database, cache, and external API checks into /health — went into the same liveness probe. Three days later the database failed over, unavailable for about 30 seconds. Every pod's liveness probe failed within one probe cycle, and the kubelet did what liveness failures demand: it killed and restarted all 40 pods. Each restart reopened connection pools and warmed caches against a database that was mid-recovery. The health checks themselves became the overload — a restart storm hammering the exact dependency everyone was waiting on. A 30-second blip turned into a 20-minute outage.

The distinction that matters:

• Liveness: "Is this process wedged?" Checks the process only — deadlocks, a stuck event loop. Never shared dependencies. Failure means restart, and restarting 40 pods doesn't fix a broken database.
• Readiness: "Can this pod serve right now?" Failure removes the pod from Service endpoints without restarting it; it rejoins when the check passes. Dependency checks belong here if anywhere — but gate on shared dependencies with care, because if the database is down for everyone, all pods go unready together and you serve nothing. Often a degraded response beats no endpoints at all.
• Startup: covers slow initialization so the liveness probe doesn't kill a pod that's still booting.

What we run now:

livenessProbe:
  httpGet:
    path: /health/live     # process only, no dependencies
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /health/ready    # pod-local checks, e.g. warm caches
    port: 8080
  periodSeconds: 5
  failureThreshold: 2
startupProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 5
  failureThreshold: 30

@GetMapping("/health/ready")
public ResponseEntity<HealthStatus> readiness() {
    HealthStatus status = healthService.check();
    return status.canServe()
        ? ResponseEntity.ok(status)
        : ResponseEntity.status(503).body(status);
}

Dependency health moved out of probes entirely, into metrics and alerts where humans — not the kubelet — decide what to do. The related cascade, where restarts flood a recovering database with connections, has its own write-up in autoscaling that killed the database; getting probe semantics right is a standard item in our Kubernetes and DevOps reviews.

Lesson: A health check that doesn't check health is worse than none — and a liveness probe that checks too much will burn the cluster down to prove it.

---