A Kubernetes-managed DCGM (NVIDIA Data Center GPU Manager) exporter experienced recurring crashes due to insufficient memory allocation, disrupting GPU telemetry collection for Grafana dashboards.
Issue
The DCGM exporter pod crashed repeatedly with exit code 137 (OOMKilled) after approximately 14 hours of operation. This prevented GPU metrics such as temperature, power consumption, and memory utilization from being reported to Grafana. Logs indicated successful initialization of DCGM and NVML, followed immediately by termination due to out-of-memory constraints.
Culprit
The 256Mi memory limit assigned to the DCGM exporter pod was inadequate for sustained GPU metrics collection. Analysis revealed:
- Memory spikes during metric initialization, a known behavior of DCGM.
- Historical data from a similar incident in October 2025 showed that increasing the memory limit from 256Mi to 512Mi resolved comparable OOMKilled errors.
- The original configuration used overly restrictive memory settings:
- Memory requests: 64Mi
- Memory limits: 256Mi
Solution
Memory Allocation Adjustment
The memory configuration was updated to double the allocation:
- Memory requests: Increased from 64Mi to 128Mi
- Memory limits: Increased from 256Mi to 512Mi
Deployment Process
- The updated configuration was applied to the DCGM exporter deployment.
- The existing pod was terminated to enforce the new memory limits, triggering a restart.
Verification
Post-deployment checks confirmed the resolution:
- The pod remained in a running state without further OOMKilled errors.
- Logs no longer contained OOM-related termination messages.
- GPU metrics (temperature, power, memory utilization) were restored to the Grafana dashboard.
This change ensured continuous GPU telemetry collection without further disruptions.