Kubernetes job fails with CPU exhaustion and ChromaDB client mismatch

Working with Azure Kubernetes Service (AKS) to deploy a Bloomberg financial news indexing pipeline (120K articles) using hybrid search (vector + BM25 + RRF fusion), with ChromaDB for persistent storage and Azure OpenAI for embeddings.

Issue

The Kubernetes job failed to schedule due to five sequential technical problems:

1. CPU exhaustion on a single-node cluster, preventing pod scheduling
2. ChromaDB client architecture mismatch, where the HTTP server model failed to start
3. Dependency import error for the tree_sitter module, blocking script execution
4. Environment variable name mismatch between the Job YAML (AZURE_OPENAI_KEY) and script expectation (AZURE_OPENAI_API_KEY)
5. Network policy blocking egress to external repositories

Culprit

CPU Exhaustion

A single-node AKS cluster lacked sufficient resources due to:

• Default CPU requests exceeding available capacity
• Resource competition from the mcp-server deployment
• Absence of a scaling strategy for batch jobs

ChromaDB Client Architecture

The HTTP server model was incompatible with the Kubernetes Job lifecycle, requiring direct file system operations for persistent volume access.

Import Dependency Chain

The ArticleChunker import triggered an unnecessary dependency chain, leading to a ModuleNotFoundError for tree_sitter.

Environment Variable Mismatch

Configuration drift occurred between the Kubernetes Job YAML and script expectations, causing API key retrieval failures.

Network Policy

Default-deny NetworkPolicy blocked all egress traffic, including DNS resolution, HTTPS connections to Hugging Face, and fallback HTTP attempts.

Solution

Resource Allocation

• Scaled down non-critical deployments.
• Reduced CPU requests in the job specification.
• Implemented resource monitoring for future deployments.

ChromaDB Client Architecture

Switched from the HTTP server model to a PersistentClient approach, removing server startup logic from the Job manifest.

Dependency Isolation

Inlined the ArticleChunker logic directly into the indexing script to avoid unnecessary imports.

Environment Variable Alignment

Standardized variable naming across Kubernetes Secrets and Python scripts to ensure consistency.

Network Policy Configuration

Created targeted egress rules to allow DNS, HTTP, and HTTPS traffic for the indexing job.

GitOps Workflow Enforcement

Mandated a standardized workflow requiring all changes to pass through GitOps pipelines, preventing direct configuration drift.