Kubernetes pods silently OOMKilled without monitoring alerts

Issue

The DCGM exporter pod, responsible for collecting GPU metrics, was silently terminated due to out-of-memory (OOM) conditions (exit code 137) for an extended period of 14 hours without detection. This resulted in the disappearance of GPU monitoring data from Grafana dashboards. The absence of proactive monitoring for critical pod failures, including OOMKills, crash loops, or image pull errors, contributed to this undetected failure.

Culprit

Two primary factors led to the undetected failure:

1. Insufficient Memory Limits: The DCGM exporter pod was configured with a memory limit of 256Mi, which proved insufficient for its operations, following a similar pattern observed in a previous API OOMKill incident.
2. Missing Monitoring Infrastructure: There were no alerts or dashboards in place to detect OOMKilled pods or other critical pod failures, leading to a lack of visibility into pod health.

As a result, the pod entered a CrashLoopBackOff state with 100 restarts over the 14-hour period, yet this failure remained unnoticed until it became apparent that GPU metrics were no longer being displayed.

Solution

To address these issues, the following improvements were implemented:

Memory Limit Adjustment

The memory limits for the DCGM exporter pod were doubled to ensure sufficient resources:

• Memory Requests: Increased from 64Mi to 128Mi.
• Memory Limits: Increased from 256Mi to 512Mi.

Health Alerts Dashboard

A comprehensive Grafana dashboard was created to monitor pod health and provide actionable alerts:

Dashboard Components

1. Critical Alerts Row: Monitors for OOMKilled pods, crash loops, failed pods, and pods with high restart counts (>5 restarts).
2. Actionable Details Panel: Displays a table with pod names, failure reasons, and namespaces, color-coded by severity for quick identification of issues.
3. Memory Pressure Monitoring: Tracks memory usage as a percentage of the limit to prevent future OOMKills.

Query Improvements

False positive queries were fixed by ensuring accurate detection of active failures, such as using == 1 to count only pods in a failed state.

Dashboard Configuration

• Auto-refresh: Set to 30 seconds for real-time updates.
• Time Range: Displays the last 1 hour of data.
• Accessibility: The dashboard is accessible at [IP] and is titled name-health-alerts.

Outcome

The improved monitoring infrastructure now provides immediate visibility into pod failures, including OOMKills and crash loops, enabling proactive troubleshooting. This ensures that critical failures are detected and addressed promptly, minimizing downtime and improving overall system reliability.