Kubernetes cluster lacks GPU monitoring for NVIDIA RTX 4060 Ti metrics

Issue

The Kubernetes cluster lacked GPU monitoring capabilities despite hosting NVIDIA RTX 4060 Ti GPUs. While CPU and memory metrics were visible in Grafana, critical GPU-specific data was unavailable, including:

• GPU temperature (necessary for cooling and performance optimization)
• GPU power consumption (required for energy monitoring)
• GPU memory utilization (needed for capacity planning)
• Historical trends (essential for anomaly detection)

Although the nvidia-smi command provided these metrics locally, they were not integrated into the broader monitoring stack.

Culprit

Three key components were missing from the monitoring architecture:

1. Absence of DCGM Exporter: The NVIDIA Data Center GPU Manager (DCGM) exporter, responsible for collecting GPU metrics, was not deployed.
2. Missing Prometheus Scrape Configuration: No configuration existed to scrape GPU metrics from the exporter’s metrics endpoint.
3. No Grafana Dashboard: There was no visualization layer dedicated to GPU-specific metrics.

The existing setup, which relied on Prometheus and Grafana, only captured Kubernetes pod metrics, node metrics, and application metrics, but had no integration with NVIDIA’s monitoring tools.

Solution

DCGM Exporter Deployment

The NVIDIA DCGM Exporter was deployed as a Kubernetes DaemonSet with privileged access to GPU devices. This allowed it to collect and expose GPU metrics in a format compatible with Prometheus. Key features of the deployment included:

• Use of a specialized runtime class for proper GPU access.
• Automatic pod discovery for efficient metric collection.
• Exposure of metrics on a dedicated port for Prometheus scraping.
• Restriction to GPU-equipped nodes only.

Prometheus Configuration

A new scrape job was added to the Prometheus configuration to target the DCGM Exporter’s metrics endpoint. This enabled Prometheus to collect GPU-specific data, including temperature, power usage, and memory utilization.

Grafana Dashboard Implementation

A dedicated GPU monitoring dashboard was created in Grafana, featuring:

• Current Status: Real-time metrics for GPU temperature, power consumption, memory usage, and utilization, with configurable thresholds.
• Historical Trends: Timeseries graphs for temperature, power usage, and memory allocation over time.

The dashboard used Prometheus queries to fetch and display metrics, ensuring parity with the nvidia-smi command-line tool.

Verification Results

All GPU metrics became immediately available in Prometheus, with values matching those from the nvidia-smi command. The solution provided:

• Real-time GPU monitoring integrated with the existing Prometheus/Grafana stack.
• Historical data retention for trend analysis.
• Alerting capabilities based on Grafana thresholds.
• Kubernetes-native deployment for scalability.
• Exact metric consistency with NVIDIA’s native monitoring tool.