Resolved

Grafana dashboard displaying stale metrics from deleted Kubernetes pods

Kubernetes
Prometheus
Grafana
kube-state-metrics
Monitoring
DevOps
Metrics
Dashboard

Grafana health dashboard showed outdated metrics for OOMKilled and CrashLoopBackOff events from terminated pods | Root cause was kube-state-metrics retaining data until Prometheus garbage collection

In a Kubernetes monitoring setup using Prometheus and Grafana, the health dashboard displayed stale metrics for critical pod failures (e.g., OOMKilled, CrashLoopBackOff) even after the affected pods were deleted. This led to false positives, making it difficult to distinguish between current issues and historical data.

Issue

The Grafana dashboard incorrectly showed non-zero values for critical failure metrics (e.g., OOMKilled events) even though no active issues existed. The stale data persisted from previously terminated pods, which had experienced failures but were no longer running. This caused confusion by presenting outdated metrics as current problems.

Culprit

The root cause was kube-state-metrics continuing to export metrics for deleted pods until Prometheus performed garbage collection (which can take hours). The original queries in the dashboard only checked for the existence of failure metrics without verifying whether the pod was currently running. For example:

  • A query might count all pods that had ever experienced an OOMKilled event, regardless of their current state.

Solution

Query Modifications

The dashboard queries were updated to include a join with the kube_pod_start_time metric, which only exists for currently running pods. This ensures that only metrics from active pods are included:

  • Added conditions to filter queries by the existence of kube_pod_start_time to confirm the pod is running.
  • Added fallback logic (or vector(0)) to ensure metrics return zero when no active pods match the criteria.

Implementation Details

  1. Updated Queries: Modified all relevant queries to focus exclusively on currently running pods and their metrics.
  2. Dashboard Reload: Applied the changes to the dashboard configuration and forced a reload to ensure the updated queries were used.
  3. Cache Clearance: Cleared the Grafana cache to reflect the updated metrics immediately.

This solution ensures that the dashboard now displays metrics only for currently existing pods, eliminating stale data while preserving historical records for forensic analysis when needed.

Key Insights

  • kube-state-metrics retains data for deleted pods until Prometheus performs garbage collection.
  • Queries should filter for currently running pods to avoid displaying stale metrics.
  • Adding fallback logic (or vector(0)) ensures queries return zero when no active pods match the criteria, preventing false positives.