Resolved

Single-node AKS cluster stuck at 99 percent CPU with pending pods

Kubernetes
AKS
Azure
DevOps
Resource Optimization
Helm
GitOps
Prometheus
Redis

Azure Kubernetes Service single-node cluster at capacity with Prometheus, Alertmanager and Redis pods stuck in Pending state due to overprovisioned resource requests | Solution reduces CPU allocation from 99 percent to 70 percent by optimizing Helm chart resource requests

Issue

A single-node Azure Kubernetes Service (AKS) cluster, designed for production-grade monitoring and policy enforcement, encountered severe resource constraints. Despite the cluster having 82% of its CPU capacity unused, pods remained in a pending state due to overly aggressive resource requests from default Helm chart configurations. This led to a deadlock where neither old high-resource pods could be terminated nor new low-resource pods could be scheduled.

Culprit

The primary issue stemmed from Helm chart defaults that were optimized for multi-node clusters, causing misalignment with the single-node environment. The cluster’s actual CPU usage was significantly lower than the requested resources, creating an artificial capacity bottleneck. For example, Prometheus was requesting 200m CPU while only using 50m, leading to unnecessary resource waste and scheduling failures.

Solution

The issue was resolved through the following steps:

  1. Resource Optimization: Resource requests for all services (Prometheus, Alertmanager, Redis, Grafana, and Gatekeeper) were significantly reduced while retaining production-grade architecture. This adjustment freed up substantial CPU and memory capacity, ensuring efficient utilization of the available resources.

  2. Deadlock Resolution: The Kubernetes scheduling deadlock was addressed by either forcefully terminating old high-resource pods or configuring Helm to use a Recreate strategy with forced upgrades. This ensured that new pods could be scheduled without waiting for old ones to terminate.

  3. Storage Configuration Fix: A misconfigured storage class for Redis was corrected to use the default AKS storage class, resolving pending pod issues related to storage.

As a result, the cluster’s CPU allocation dropped from 99% to 70%, freeing up capacity for additional workloads while maintaining optimal performance. This optimization provided a cost-effective solution without requiring hardware upgrades.