Resolved

AKS cluster creation fails with SystemPoolSkuTooLow error

Azure
AKS
Kubernetes
DevOps
Infrastructure
Terraform
Cloud
VM

Azure Kubernetes Service cluster deployment fails when system node pool uses VM SKU with insufficient vCPUs (1) or RAM (4GB) | Error occurs during AKS provisioning with DC1ds_v3 (1 vCPU, 8GB RAM) despite meeting memory requirements

Issue

A Kubernetes cluster deployment using Azure Kubernetes Service (AKS) failed during Terraform execution due to insufficient vCPU and RAM requirements for the system node pool. Specifically, the system node pool, which hosts critical Kubernetes components like coredns, metrics-server, and kube-proxy, requires a minimum of 2 vCPUs and 4GB of RAM. The deployment attempted to use a VM SKU with insufficient vCPU capacity, despite meeting RAM requirements, resulting in a failure.

Culprit

The root cause of the failure is AKS’s strict hard minimum requirements for system node pools:

  • vCPU: ≥2 (required for reliable operation of critical services).
  • RAM: ≥4GB (required for memory-intensive tasks).

The system node pool is responsible for hosting essential Kubernetes components, including:

  • coredns: DNS resolution for cluster services.
  • metrics-server: Continuous metrics collection and reporting.
  • kube-proxy: Network routing and proxy operations.
  • tunnelfront: Ingress traffic management.

The configuration issue arose from specifying an insufficient VM SKU in Terraform, which did not meet the vCPU requirement despite exceeding the RAM requirement.

Solution

VM SKU Upgrade

The solution involved upgrading the VM SKU to meet the minimum vCPU requirement while balancing cost considerations. Key actions included:

  1. Replacing the Insufficient SKU: Upgraded from a VM with 1 vCPU to one with 2 vCPUs to satisfy the system node pool requirements.

  2. Cost vs. Requirements Tradeoff Analysis: Evaluated alternatives to balance cost and performance:

    • Higher-tier SKUs: Met requirements but increased costs significantly.
    • Optimal SKUs: Met requirements at a lower cost but may require quota adjustments.
    • Alternative Solutions: Considered less expensive options like Azure Container Instances (ACI) for non-production environments.

Architectural Recommendations

To ensure compliance with AKS requirements and optimize costs:

  1. Separate System and User Pools: Configure distinct node pools for system (meeting minimum requirements) and user workloads (using cost-effective SKUs).

  2. Quota Management: Request quota increases for optimal SKUs (e.g., B-series VMs) if restricted by default quotas.

  3. Use of Alternative Solutions: For non-production environments, explore cost-effective alternatives like ACI, acknowledging the tradeoff of reduced Kubernetes features.

Final Configuration

The deployment was successfully completed using a VM SKU that meets the minimum vCPU and RAM requirements. Post-deployment verification confirmed the system node pool was operational and hosting critical Kubernetes components.

Cost Considerations

The upgrade increased the total monthly cost due to the higher-tier SKU, but it ensured reliable operation of critical cluster services. Cost optimization strategies, such as using optimal SKUs or alternative solutions, were considered to mitigate financial impact.