Resolved

AKS pod stuck in CrashLoopBackOff with tree_sitter dependency error

Kubernetes
AKS
Docker
GitOps
Python
DevOps
Azure
CI/CD
Embeddings
NLP

Kubernetes pod fails to start with ModuleNotFoundError for tree_sitter | GitOps pipeline broken due to registry mismatch between GitHub Actions and Azure ACR | Embedding dimension conflicts between Azure OpenAI and local sentence-transformers

Resolving AKS Deployment Issues and Completing GitOps for MCP Server

Issue

The AKS deployment of the MCP server experienced CrashLoopBackOff, causing pods to fail repeatedly for over 85 minutes. Three critical issues were identified:

  1. Missing Dependencies: The Docker image lacked critical dependencies (e.g., tree_sitter and related packages), causing immediate pod crashes.
  2. Registry Mismatch: The CI/CD pipeline pushed images to GitHub Container Registry (ghcr.io), while Kubernetes pulled from Azure ACR (mcpragacrj3j893.azurecr.io), breaking automated deployments.
  3. Embedding Dimension Conflict: Bloomberg data was indexed using 384-dimensional sentence-transformers embeddings, while Azure OpenAI generated 1536-dimensional embeddings, causing dimension mismatch errors during search operations.

Culprit

Missing Dependencies

The Docker image was built before critical dependencies (tree_sitter and related packages) were added to the project configuration (pyproject.toml), resulting in a functional gap between development and production environments.

Registry Mismatch

The GitHub Actions workflow was configured to push exclusively to GitHub Container Registry, while Kubernetes deployment manifests referenced Azure ACR. This misalignment prevented automated deployments from receiving updated images.

Embedding Dimension Conflict

The system used sentence-transformers for local embedding generation (384 dimensions) but relied on Azure OpenAI for alternative embeddings (1536 dimensions). This mismatch caused ChromaDB to reject vector similarity searches due to incompatible dimensionality.

Solution

Fixed Docker Image with Complete Dependencies

Rebuilt the Docker image to include all necessary dependencies, ensuring compatibility between development and production environments. The updated image now supports:

  • tree_sitter and related language parsers.
  • sentence-transformers with PyTorch dependencies.
  • ChromaDB and all indexing requirements.

Unified CI/CD Pipeline for Azure ACR

Modified the GitHub Actions workflow to push images to both GitHub Container Registry and Azure ACR, ensuring alignment between CI/CD pipelines and Kubernetes deployment manifests.

Embedding System Fallback Architecture

Implemented a dual-mode embedding system that dynamically selects between local sentence-transformers (384 dimensions) and Azure OpenAI (1536 dimensions) based on configuration. This ensures compatibility with existing Bloomberg data while enabling future flexibility.

Outcome: The AKS deployment is now stable, GitOps workflows are fully aligned, and the embedding system operates consistently without dimension conflicts.