Issue

The Grafana health dashboard displayed ”❌ Failed Pods: 15” without specifying which pods were actually failing. This false alarm lacked actionable details, making it ineffective. Upon investigation, it was revealed that the dashboard was showing false positives due to incorrect PromQL queries against kube-state-metrics.

Culprit

The root cause was a misunderstanding of how kube-state-metrics exports pod phase metrics. The service exports all possible phases for every pod with binary values (1=current state, 0=not current state).

For example:

pod: s3-email-cleanup, phase="Pending" = 1 (actual state)
pod: s3-email-cleanup, phase="Running" = 0
pod: s3-email-cleanup, phase="Failed" = 0 (not failed)
pod: s3-email-cleanup, phase="Succeeded" = 0
pod: s3-email-cleanup, phase="Unknown" = 0

The incorrect query counted all metrics with the label phase="Failed" regardless of their value:

# ❌ WRONG - counts all pods (15 pods × 1 metric = 15 "failed")
count(kube_pod_status_phase{namespace="name", phase="Failed"})

This created false positives by counting metrics where phase="Failed" existed (value=0) instead of only counting pods that were actually in a Failed phase (value=1).

Solution

Fixed False Positive Queries

The PromQL queries were corrected to only count metrics where the value was 1:

# ✅ CORRECT - counts only actually failed pods
count(kube_pod_status_phase{namespace="name", phase="Failed"} == 1)

The same fix was applied to queries for CrashLoopBackOff.

Added Actionable Details Panel

A new “⚠️ Pods with Issues (Waiting State)” table panel was added, displaying:

Pod name
Failure reason (color-coded: 💥 CrashLoopBackOff, 📦 ImagePullBackOff)
Container name
Namespace

Using the query:

kube_pod_container_status_waiting_reason{namespace="name"} == 1

Dashboard Layout Restructuring

Moved the new issues table to position y=4 (just below stats).
Reorganized related panels to positions y=10 and y=18.

Actual Issue Discovery

The investigation revealed the real problem: 2 CronJob pods were stuck in ImagePullBackOff, trying to pull a non-existent name/api:latest image from Docker Hub. These were shown as “Pending” phase with ImagePullBackOff waiting reasons, not “Failed” phase as initially indicated.

Key Lesson

Always filter kube-state-metrics phase/reason queries with == 1 to avoid false positives from metrics that exist but have value=0. The dashboard now adheres to UX principles:

Shows counts and details.
Makes issues actionable.
Color-codes by severity.
Prioritizes critical issues visually.