Resolved

Grafana shows no data for Kubernetes CPU and memory metrics

Kubernetes
Prometheus
Grafana
RBAC
Monitoring
cAdvisor
DevOps
K3s

Prometheus returns 403 Forbidden when scraping kubelet cAdvisor endpoints, causing missing CPU and memory metrics in Grafana dashboards | Fixed by adding nodes/metrics RBAC permission to Prometheus ServiceAccount

Working with a Kubernetes (K3s) monitoring stack where Prometheus scrapes cAdvisor metrics for container resource usage, visualized through Grafana dashboards. The system uses Prometheus Operator with custom RBAC configurations to access kubelet endpoints.

Issue

Grafana dashboards displayed “No data” for CPU and memory usage panels while pod status panels worked correctly. Prometheus targets reported 403 Forbidden errors when scraping the kubernetes-nodes-cadvisor endpoint, preventing collection of container resource metrics for various pods such as API, vLLM, PostgreSQL, and Redis. This resulted in missing time-series data for resource utilization graphs despite the pods being operational.

Culprit

The issue was caused by a missing nodes/metrics RBAC permission in the Prometheus ClusterRole configuration. Although the RBAC included permissions for nodes and nodes/proxy, it lacked the specific nodes/metrics permission required to access the kubelet’s /metrics/cadvisor endpoint. K3s enforces strict permission checks for these metrics, returning an HTTP 403 error when the permission is absent.

Solution

RBAC Permission Addition

  • Added the nodes/metrics resource permission to the Prometheus ClusterRole configuration.
  • This allowed Prometheus to access the cAdvisor metrics endpoint, resolving the 403 Forbidden errors.

Deployment Workflow

  • Applied the updated RBAC configuration.
  • Restarted Prometheus to ensure the new permissions were recognized.

Verification Results

  • Prometheus target health improved from down to up for the kubernetes-nodes-cadvisor job.
  • CPU and memory metrics became available for all monitored pods, including API pods, vLLM services, PostgreSQL, and Redis.
  • Grafana dashboards now displayed real-time data and historical trends for resource utilization.

This fix ensured that all monitoring panels in Grafana correctly showed container-level metrics, enabling comprehensive visibility into resource usage across the Kubernetes cluster.