Resolved

AKS single-node cluster CPU exhaustion from 2-replica managed components

Kubernetes
AKS
Azure
DevOps
Performance
Infrastructure
Terraform
FluxCD

Azure Kubernetes Service AKS-managed components default to 2 replicas on single-node clusters, consuming 50%+ CPU capacity | Critical bottleneck with only 30m CPU available for workloads

Issue

A single-node Azure Kubernetes Service (AKS) cluster experienced severe CPU exhaustion, with 98.4% utilization due to unnecessary high-availability replicas for AKS-managed components. This left only 1.6% of CPU capacity (30m) available for workloads, preventing the scheduling of any pod requiring more than this minimal allocation. The cluster’s CPU usage was dominated by AKS-managed components running 2 replicas each, despite having only a single node, which is unsupported for such configurations.

Culprit

AKS defaults all managed components (e.g., Gatekeeper, CoreDNS, metrics-server, konnectivity-agent) to 2 replicas for high availability, assuming multi-node production clusters. On a single-node setup, this creates:

  • Wasted CPU allocation: Each AKS-managed component consumes resources redundantly, as it cannot operate in a highly available manner on a single node.
  • No control via standard Kubernetes tools: The replicas are managed by AKS and cannot be adjusted using Kubernetes manifests, GitOps tools (e.g., FluxCD), or Terraform configurations.
  • Design mismatch: AKS’s default configuration is optimized for production environments with 3+ nodes, not for single-node development clusters.

Solution

Immediate Mitigations

  1. Disable Unnecessary Features: Temporarily disable components like Gatekeeper to reduce CPU overhead.
  2. Constraint Autoscaling: Prevent accidental scaling of workloads by enforcing minimum and maximum replica limits at 1.

Structural Solutions

  1. Scale to Multiple Nodes: Increase the cluster to 2 nodes to meet high-availability requirements and double available CPU capacity.
  2. Document Constraints: Clearly document the limitations of single-node clusters in deployment guides and READMEs to prevent misconfigurations.

Cost Considerations

  • Single-Node Cluster: Incurs high CPU overhead (~98.4% utilization) but remains cost-effective for minimal workloads.
  • Two-Node Cluster: Increases capacity to 3800m CPU but doubles costs (~$215-225/month vs. $125-135/month for a single node).

Tradeoffs

  • Memory Utilization: Remains underutilized (~42% available) despite the CPU constraints.
  • Proof-of-Concept Limitation: Accept the CPU overhead as a temporary constraint while developing features.

The solution involves balancing resource constraints with cost efficiency while ensuring the cluster remains functional for development purposes.