Resolving Grafana Dashboard Metrics Issues in Kubernetes Monitoring Stack

Issue

Grafana dashboards displayed “No data” across all panels for Kubernetes pod metrics, including both pod status and resource utilization graphs. After applying fixes, pod status metrics (e.g., kube_pod_status_phase) began displaying correctly, but CPU and memory graphs remained empty due to a 403 Forbidden error when Prometheus attempted to scrape the kubelet /metrics/cadvisor endpoint. This resulted in empty panels for container_memory_working_set_bytes and container_cpu_usage_seconds_total.

Culprit

Missing Metrics Sources

1. kube-state-metrics Absence: No component was providing Kubernetes object status metrics, such as pod phases or replica counts.
2. cAdvisor Access Blocked: Prometheus was unable to scrape container-level metrics due to:
   • K3s kubelet configuration not exposing the /metrics/cadvisor endpoint by default.
   • Insufficient RBAC permissions for the Prometheus ServiceAccount to access kubelet metrics.
   • Authentication/authorization failures when Prometheus attempted to scrape the kubelet endpoint, resulting in a 403 Forbidden error.

Architecture Gaps

The monitoring stack relied on two distinct metrics pipelines:

• Functioning Path: kube-state-metrics → Prometheus → Grafana (for pod status metrics).
• Broken Path: kubelet cAdvisor → Prometheus → Grafana (for CPU/memory metrics).

The 403 error specifically indicated that Prometheus was unable to authenticate or authorize itself when accessing the kubelet /metrics/cadvisor endpoint.

Solution

Part 1: kube-state-metrics Deployment

To resolve the pod status metrics issue, the kube-state-metrics component was deployed to provide pod status information. This included:

• RBAC Configuration: Created a ClusterRole with permissions to access Kubernetes resources like pods, nodes, and services.
• Deployment: Deployed kube-state-metrics v2.10.1 with the necessary permissions and resource limits.
• Prometheus Integration: Configured Prometheus to scrape metrics from the kube-state-metrics service.

Result: Pod status panels in Grafana now populate with accurate kube_pod_status_phase metrics, showing “Up” or “Down” states.

Part 2: cAdvisor Configuration

To address the CPU/memory metrics issue, the following steps were taken:

1. Prometheus Scrape Configuration: Updated the Prometheus configuration to attempt scraping the cAdvisor endpoint using HTTPS, TLS configuration, and bearer tokens.
2. RBAC Enhancement: Enhanced the Prometheus ServiceAccount’s RBAC permissions to include access to kubelet metrics and proxy resources.

However, the configuration returned a 403 Forbidden error when attempting to scrape the cAdvisor endpoint directly from the kubelet.

Next Steps for Complete Fix

To fully resolve the cAdvisor integration issue, the following actions are required:

1. K3s-Specific Kubelet Configuration: Investigate and update the K3s kubelet arguments to enable the exposure of the cAdvisor endpoint. This may include adding the --kubelet-arg="authentication-token-webhook=true" flag to the k3s server configuration.

2. Alternative cAdvisor Deployment: Deploy cAdvisor as a DaemonSet in the Kubernetes cluster, allowing Prometheus to scrape metrics without relying on the kubelet endpoint. This involves:

   • Creating a DaemonSet for cAdvisor with appropriate volume mounts for host paths (/rootfs, /var/run, /sys, /var/lib/docker).
   • Configuring Prometheus to scrape metrics from the newly deployed cAdvisor service.

3. RBAC Enhancement: Further refine the RBAC permissions for the Prometheus ServiceAccount to ensure full access to the required resources.

By implementing these steps, the CPU and memory metrics should become available in Grafana dashboards, completing the monitoring stack functionality.