Resolved

Kubernetes init container crash with CrashLoopBackOff and embedding dimension mismatch

Kubernetes
DevOps
AI/ML
Embeddings
ChromaDB
Python
Debugging
Azure

Azure Kubernetes job failed with 100% error rate due to embedding dimension mismatch (384 vs 1536) | Init container stuck in CrashLoopBackOff searching for missing script file

Working with Kubernetes deployments and ChromaDB vector databases to index 106,600 Bloomberg articles using sentence-transformers for local embedding generation. The deployment involved transferring 3.5GB of vector data to Azure PVC and managing GitOps workflows with Flux.

Issue

Two critical failures occurred during deployment:

  1. Embedding Dimension Mismatch: Azure Kubernetes job failed with 100% error rate due to a mismatch between expected (1536 dimensions) and actual (384 dimensions) embedding dimensions.
  2. Init Container Crash: Pod startup was blocked by an init container stuck in CrashLoopBackOff state, repeatedly searching for a non-existent script.

Embedding Job Failure

The Bloomberg indexing job consistently failed due to dimension mismatches between local and cloud environments, resulting in 7,804 failed attempts during vector insertion into ChromaDB.

Pod Startup Block

Pods remained stuck in the initialization phase due to the init container failing to locate a script that was no longer required.

Culprit

Dimension Mismatch

The Azure environment expected embeddings from a different model (likely Azure OpenAI), producing 1536-dimensional vectors, while the local indexing used the sentence-transformers/all-MiniLM-L6-v2 model, which generates 384-dimensional vectors. This discrepancy caused all job attempts to fail.

Missing Init Script

The deployment configuration referenced a script (index-default-repo.sh) that was no longer needed after switching to local indexing. The absence of this script caused the init container to fail repeatedly.

Solution

Local Indexing Workaround

  1. Generated embeddings locally using the correct model to match the expected dimensions.
  2. Created a ChromaDB instance locally and stored 921,657 embedding chunks.
  3. Compressed and transferred the data to Azure PVC for use in the cloud environment.

Deployment Fixes

  1. Removed the problematic init container from the deployment configuration.
  2. Applied changes via GitOps workflow to ensure consistency across environments.
  3. Forced pod restarts to apply the changes and resolve the startup issues.

Performance Impact

  • Local indexing: Completed in 34 minutes for 106,600 articles.
  • Data transfer: Took approximately 20 minutes for the compressed data.
  • Total resolution time: Approximately 1 hour, achieving a 100% success rate after fixes.