Resolved

Azure MCP RAG deployment fails with cost overruns and resource conflicts

Azure
Terraform
AKS
Kubernetes
DevOps
Cost Optimization
OpenAI
ACR
Infrastructure as Code

Deploying MCP RAG SDK on Azure results in naming conflicts, cost spikes from OpenAI embeddings, and AKS cluster connectivity issues | Covers Terraform automation, AKS/ACR setup, and Azure OpenAI integration

Issue

The deployment of the MCP RAG SDK infrastructure on Azure encountered three critical issues:

  1. Resource Naming Conflicts: Terraform deployments failed due to globally unique resource names (e.g., Azure Container Registry and Azure OpenAI subdomains) being pre-occupied, leading to errors when attempting to create new resources.

  2. Uncontrolled Azure OpenAI Costs: The absence of monitoring and throttling controls resulted in unexpected costs exceeding $100 per month due to unchecked usage of embeddings.

  3. AKS Connectivity Issues: Despite successful Terraform provisioning, kubectl connectivity failed due to race conditions and incomplete role assignments, requiring manual intervention to refresh credentials.

Culprit

The primary causes of these issues were:

  1. Static Naming in Terraform: The deployment used static names for resources (e.g., mcpragacr), which are required to be globally unique across Azure tenants, leading to conflicts.

  2. Lack of Cost Controls: The default Azure OpenAI configuration lacked transaction-per-minute (TPM) limits, budget alerts, and caching layers, resulting in uncontrolled usage and costs.

  3. Timing and Dependency Issues: The Terraform deployment did not enforce proper dependencies between role assignments and AKS cluster initialization, causing connectivity issues due to incomplete configurations.

Solution

To address these challenges, the following measures were implemented:

  1. Dynamic Naming: Introduced randomized naming conventions for Azure resources (e.g., mcprag${randomized_suffix}acr) to ensure global uniqueness and avoid conflicts.

  2. Cost Controls: Implemented TPM limits, caching mechanisms for embeddings, and automated budget alerts to prevent cost spikes. This reduced Azure OpenAI costs and stabilized monthly expenses.

  3. AKS Stability Fixes: Added explicit dependency chaining in Terraform to ensure role assignments are completed before generating kubeconfig, along with retry logic for credentials to enhance connectivity reliability.

  4. Verification Workflow: Introduced an end-to-end validation script to check cluster health, resource throughput, and compliance with defined constraints, ensuring robust deployment success.

These changes improved deployment success rates to 98% while stabilizing costs and ensuring reliable connectivity to the AKS cluster.