Resolved

Kubernetes resource exhaustion preventing application deployment

Kubernetes
Resource Optimization
DevOps
Performance
CPU Management
AKS

A Kubernetes cluster running on a single-node setup consumed nearly all CPU resources, leaving insufficient capacity for deploying the MCP server application. Optimizations were required to free up resources and enable deployment.

Issue

A single-node Kubernetes cluster (with a CPU limit of 1900m) was operating at 98.4% CPU utilization, leaving only 30m of CPU available for new workloads. This prevented the deployment of the MCP server application, which required 1500m of CPU when configured for high availability. The cluster hosted:

  • Kubernetes-managed components consuming 50% of CPU.
  • Over-provisioned monitoring stacks using significantly more resources than needed.
  • Additional services that collectively exhausted the cluster’s capacity.

Culprit

1. Fixed Overhead from Kubernetes-Managed Components

The cluster was forced to run highly available (HA) configurations for system components (e.g., DNS, monitoring) even on a single-node setup, consuming 50% of the total CPU without any actual redundancy benefit.

2. Over-Provisioned Resource Requests

Resource requests for monitoring and other components were set arbitrarily high without measuring actual usage. For example, a monitoring component might request 50m CPU while only consuming 14m in practice, leading to wasteful allocation.

3. MCP Server Configuration for High Availability

The MCP server was originally configured for a high-availability production environment with three replicas, each requesting 500m CPU. This resulted in a total request of 1500m CPU, which was impossible to deploy on the single-node cluster.

Solution

1. Right-Sizing Monitoring Components

  • Actual Usage Measurement: Monitored real CPU consumption using cluster metrics.
  • Adjusted Requests: Set resource requests to 1.5× the actual usage to ensure stability while freeing up capacity.
  • Result: Reduced monitoring stack CPU usage by 53%, freeing up significant resources.

2. Optimizing MCP Server Deployment

  • Single-Replica Deployment: Reduced the MCP server to a single replica to fit within the cluster’s remaining capacity.
  • CPU Request Reduction: Lowered CPU requests from 500m per replica to 100m.
  • Autoscaling Constraints: Disabled autoscaling to prevent accidental scaling that could exhaust resources.
  • Result: Reduced MCP server CPU requirements by 93%, enabling deployment within the available resources.

3. Redis Configuration Adjustment

  • Standalone Mode: Configured Redis to run in standalone mode (single replica) to minimize resource usage.
  • Result: Redis CPU usage remained optimal with no additional overhead.

Verification Process

  • Resource Usage Validation: Confirmed actual CPU and memory usage post-optimization to ensure stability and headroom.
  • Deployment Confirmation: Verified successful deployment of the MCP server with the adjusted resource requests.

Final State:

  • Total CPU Usage: Reduced from 98.4% to 61.6%, freeing up 730m CPU (38.4% headroom).
  • Memory Usage: Remained within safe limits (45% of available memory).
  • MCP Server Deployable: Successfully deployed with 100m CPU request, meeting operational requirements.