Node ran out of disk. All pods evicted. New pods scheduled on same node. Evicted again. Repeat until someone woke up.

The timeline:

  • 2:00 AM: Node crosses the kubelet's default eviction threshold (nodefs.available < 10%)
  • 2:05 AM: Kubelet evicts pods; node gets the node.kubernetes.io/disk-pressure taint
  • 2:10 AM: With the pods (and their logs) gone, disk usage drops back under the threshold
  • 2:15 AM: After the eviction-pressure-transition-period (5 minutes by default) the taint clears — and the scheduler puts the pods right back on the same node, which now advertises the most free resources
  • 2:25 AM: Logs regrow, threshold crossed again, evictions repeat
  • 2:50 AM: Alert fires (finally), two full loops later

Root cause:

  • Application logging to container filesystem
  • No log rotation configured
  • ephemeral-storage limits not set
  • The DiskPressure taint clears when usage drops below the threshold — not when the underlying log problem is fixed — so the same node kept accepting the same pods

The fix:

resources:
  requests:
    ephemeral-storage: "1Gi"
  limits:
    ephemeral-storage: "2Gi"

Plus:

  • Log to stdout (collected by Fluentd)
  • Configure container log rotation
  • Monitor node disk usage with alerts at 70% — well before the kubelet acts

With ephemeral-storage limits in place, a runaway logger gets its own pod evicted instead of taking down the node. Node-level disk alerts are part of the baseline monitoring we set up in every DevOps engagement.

Lesson: Set ephemeral-storage limits. Always.

← Back to Lessons Learned
Tags: #Kubernetes #Reliability #ResourceManagement #Debugging