Resolved

AKS single-node cluster hits volume attachment limit with pending pods

Kubernetes
AKS
DevOps
Storage
PersistentVolume
Helm
Monitoring
Infrastructure

Single-node Azure Kubernetes Service cluster reached 4-volume attachment limit causing pods to remain in Pending state with 'max volume count exceeded' error | Required disabling persistent storage for monitoring stack to free capacity

Resolving Volume Attachment Limits in Single-Node AKS Clusters

Issue

A single-node Azure Kubernetes Service (AKS) cluster reached its volume attachment limit, preventing the mcp-server pod from scheduling. The cluster already hosted three persistent volumes (Grafana, Prometheus, and Alertmanager) and required an additional volume for the mcp-server, exceeding the node’s constraint of 4 volume attachments. This caused the pod to remain stuck in a Pending state, blocking critical application deployment.

Culprit

Azure AKS enforces a hard limit of 4 volume attachments per node for standard VM types. The architecture used 5 persistent volumes:

  • 3 PVCs for monitoring (Grafana, Prometheus, Alertmanager).
  • 1 PVC for Redis (8Gi).
  • 1 PVC for mcp-server (50Gi).

The Helm configuration for monitoring components (e.g., Grafana, Prometheus) was set to use persistent storage by default, unaware of the node’s volume constraints. This resulted in an unschedulable pod due to the volume limit being exceeded.

Solution

To resolve the issue, disabled persistent storage for monitoring components and replaced it with ephemeral storage (emptyDir). This approach:

  • Reduced volume attachments from 5 to 2 (Redis and mcp-server PVCs).
  • Maintained monitoring functionality using in-memory storage for non-critical data.
  • Ensured the mcp-server pod could schedule successfully by freeing up volume attachment slots.

Outcome: The mcp-server pod was successfully scheduled, and monitoring continued to function using ephemeral storage, adhering to the node’s volume attachment constraints.