Resolved

Helm deadlock during install with insufficient CPU resources

Helm
Kubernetes
DevOps
FluxCD
GitOps
Secret Management
Azure AD
Monitoring

HelmRelease stuck in install loop due to pods failing to schedule from excessive CPU requests (600m requested vs 14m available) | Resolved by deleting stuck release and letting GitOps recreate with optimized values

Issue

A Helm installation process for a monitoring solution in a Kubernetes environment encountered a deadlock due to excessive CPU resource requests (600m total) being assigned to Grafana and Prometheus pods. The cluster node had insufficient free resources (only 14m available out of 1900m total capacity), causing the pods to fail scheduling. The Helm reconciliation process stalled indefinitely, preventing updates to the resource requests even after the configuration was revised in Git to a more conservative allocation (250m total).

Culprit

The deadlock arose from three key factors:

  1. Resource Misallocation: The initial CPU requests (600m) exceeded the available node capacity, leading to pod scheduling failures.
  2. Helm Design Limitation: Helm cannot modify a release during an in-progress install, even if the configuration is updated externally.
  3. Lack of Pre-Flight Validation: There was no preliminary check to verify resource availability before pod scheduling.

The result was a permanent deadlock where the installation could not proceed, and updates could not be applied.

Solution

To resolve the deadlock, the stuck Helm release was deleted, allowing FluxCD to recreate it with the updated, lower CPU requests (250m). This approach succeeded because:

  • The deletion canceled the blocked installation attempt.
  • FluxCD automatically synced with the latest Git configuration, applying the corrected resource constraints.
  • The revised installation succeeded as the updated requests fit within the available node capacity.

Additionally, the system implemented a more secure approach to secret management by integrating Azure Key Vault for storing sensitive credentials. Secrets were dynamically injected into Grafana using Azure AD OAuth and read-only volume mounts, ensuring secure and production-grade handling of credentials.

Resource optimization was achieved by reducing CPU requests in the Helm configuration, and further audits were recommended to address ongoing CPU starvation issues caused by high node utilization (99% capacity).