Resolved

Azure AKS deployment incomplete while local MCP server works with Bloomberg search

Python
Azure
AKS
Kubernetes
DevOps
Search
Embeddings
ChromaDB
Deployment
Debugging
AI/ML
Backend

Local MCP server successfully processes Bloomberg searches using sentence-transformers but Azure AKS deployment remains untested and inaccessible | Key issues include missing ingress configuration, unverified ChromaDB data access, and unknown pod functionality status

Issue

The deployment of the MCP server on Azure Kubernetes Service (AKS) was in an unverified state, while the development workflow relied on a fully functional local MCP server instance. This discrepancy led to several critical gaps, including:

  1. Unverified AKS Pod Functionality: Although the pod was reported as running, logs and functionality had not been checked post-restart.
  2. Missing Remote Access: No ingress or load balancer was configured to allow external access to the MCP server.
  3. Unknown Data Accessibility: The status of the 4.7GB ChromaDB dataset and its mounting on the AKS pods was unclear.
  4. Configuration Drift: The local .mcp.json configuration still pointed to a local process rather than the Azure endpoint.
  5. Ambiguous Embedding Source: It was unclear whether the AKS pods used local sentence-transformers or attempted to connect to Azure OpenAI for embeddings.

The issue was uncovered when system monitoring revealed that CPU activity and processes were running locally instead of on Azure.

Culprit

The root causes of this issue were:

  1. Architecture Assumption Mismatch: The development process assumed Azure deployment was active but the local fallback mechanism (using sentence-transformers) was indistinguishable from Azure OpenAI in terms of performance and output.
  2. Misconfigured .mcp.json: The configuration file continued to reference a local process instead of the Azure endpoint, leading to local execution despite the intent to deploy on AKS.
  3. Lack of Deployment Verification: No post-deployment testing was conducted for critical aspects like pod logs, ChromaDB data mounting, service endpoints, and ingress configuration.
  4. Incomplete CI/CD Pipeline: While the pipeline pushed images to Azure Container Registry (ACR), there were no automated verification steps for AKS deployment, health checks, or endpoint testing.
  5. Key Vault and Access Configuration: The Azure OpenAI API key was set to ‘DISABLED’, and there was no validation to ensure AKS pods respected this setting.

Solution

To resolve these issues, the following steps were taken:

  1. Immediate Verification: Verified pod status, logs, and data accessibility to confirm the operational state of the AKS deployment.
  2. Ingress Configuration: Configured an ingress resource to enable external access to the MCP server.
  3. Updated .mcp.json: Modified the configuration file to point to the Azure endpoint, ensuring that remote queries were routed correctly.
  4. Fallback Verification: Enhanced the embedding logic to log the source of embeddings (Azure OpenAI or local sentence-transformers) for clarity.
  5. Data Synchronization: Ensured the ChromaDB dataset was correctly mounted and accessible on the AKS pods, or triggered re-indexing if necessary.

Additionally, cost optimizations were implemented, such as configuring AKS auto-scaling and using spot instances for non-critical workloads. A verification checklist was established to ensure all critical aspects of the deployment were validated and functional.

This comprehensive approach ensured that the MCP server was correctly deployed and operational on Azure, addressing the gaps identified in the initial state.