Resolved

Kubernetes pods silently OOMKilled for 14 hours without alerts

Kubernetes
Monitoring
Grafana
Prometheus
Alerting
Pod Failures

A critical monitoring dashboard was developed to detect pod failures, including OOMKill events, crash loops, and memory exhaustion in real-time. Previously, such failures went undetected for prolonged periods.

A Kubernetes pod experienced 100 OOMKill terminations (exit code 137) over 14 hours without alerting, highlighting critical gaps in monitoring. The incident exposed four key failure modes:

Issue

  • Undetected OOMKilled pods: No alerts for pods terminated by the kernel’s OOM killer.
  • Crash loops: Containers stuck in CrashLoopBackOff without visibility.
  • Pre-OOM memory conditions: No monitoring of memory usage approaching limits.
  • High restart counts: Indicator of unstable or misconfigured pods.

Operators only discovered the issue after investigating unrelated metric gaps, underscoring the need for proactive pod-level failure detection.

Culprit

The root cause was a combination of missing monitoring and default Kubernetes behavior:

  1. Automatic restarts: OOMKilled pods restarted silently without alerts.
  2. No Prometheus queries: Critical metrics like kube_pod_container_status_last_terminated_exitcode were unmonitored.
  3. Dashboard limitations: Existing tools focused on cluster metrics, not pod failure patterns.
  4. Memory threshold blindness: No visualization of memory usage relative to pod limits.

Solution

A real-time health alerts dashboard was implemented in Grafana, featuring:

Alert Panels

  1. OOMKilled Pods: Real-time count of pods with exit code 137, with color-coded thresholds (Green=0, Yellow=1+, Red=10+).
  2. High Restart Counts: Tracks pods with restart counts exceeding 5.
  3. Failed Pods: Monitors pods in Failed phase.
  4. CrashLoopBackOff: Highlights pods stuck in crash loops.

Diagnostic Panels

  1. Restart Count Table: Sortable by restart frequency, with color-coding (Green=0, Yellow=5+, Red=20+).
  2. Termination Exit Codes: Focuses on OOMKilled events (exit code 137) with contextual details.
  3. Memory Usage % of Limit: Time-series trends showing memory usage as a percentage of the limit, with 100% as the OOM threshold.

Impact

  • Detection time reduced from 14 hours to <30 seconds.
  • Namespace-wide coverage for all pods.
  • Granular alerts at the pod/container level, including diagnostics for root causes.

The solution now provides immediate visibility into critical pod failure modes, ensuring proactive incident response.