Resolved

AKS single-node cluster fails to schedule pods due to insufficient CPU

Kubernetes
AKS
DevOps
Infrastructure
Scheduling
Resource Management
CPU
Cluster

Kubernetes jobs in a single-node AKS cluster encountered scheduling failures due to insufficient CPU resources, resulting in pods remaining in Pending state. Resource constraints required adjustments to workload requests and long-term architectural improvements.

Issue

Kubernetes jobs entered a Pending state with the error 0/1 nodes are available: 1 Insufficient cpu, indicating that the single-node cluster had no available CPU capacity to accommodate new workloads. Specifically, a bloomberg-indexing job failed to schedule while a continuously running service (mcp-server) consumed the available CPU resources, causing complete cluster saturation.

Culprit

The single-node Azure Kubernetes Service (AKS) cluster lacked sufficient CPU capacity to handle both:

  • Continuous workloads: A service (mcp-server) requesting a portion of the node’s CPU.
  • Batch workloads: A job (bloomberg-indexing) requiring additional CPU resources.

Kubernetes rejected the job due to hard resource constraints, and preemption was not possible as all existing pods were non-preemptible. This resulted in a bottleneck where neither workload could be scheduled concurrently.

Solution

Immediate Mitigation

Temporary measures included scaling down non-critical deployments to free up resources for critical workloads.

Long-term Configuration Adjustments

  1. Reduced CPU requests: Lowered the CPU allocation for batch jobs to better fit within cluster capacity constraints.

Architectural Improvements

  • Implemented resource quotas to prevent future saturation.
  • Considered multi-node clusters for production environments to:
    • Enable proper pod scheduling and preemption.
    • Isolate critical services from batch jobs.
    • Scale horizontally during resource contention.
  • Monitored and adjusted resource limits to align with workload demands.