Resolved

Kubernetes pod crashes and network policy issues in production email pipeline

Kubernetes
K3s
DevOps
Networking
Memory Management
OOMKilled
NetworkPolicy
Pod Communication
Infrastructure

API pods crashed due to memory exhaustion and network policies incorrectly blocking pod-to-pod communication. Memory limits were increased, and network policies were corrected to resolve these issues.

API pods in the production email processing pipeline experienced crash-looping with OOMKilled errors and network policies incorrectly blocking pod-to-pod communication. Despite the appearance of correct configurations, these issues disrupted the email processing workflow.

Issue

  1. Memory Exhaustion: API pods crashed due to insufficient memory limits (512Mi), which were exceeded by the combined usage of two uvicorn workers and fast-agent MCP processes.
  2. Network Policy Failures: Network policies using empty podSelector blocked all pod-to-pod communication in the K3s environment, preventing essential service interactions.
  3. Stale Network Rules: Persistent iptables rules from previous configurations caused policies to fail until pods were restarted.

Culprit

  1. Insufficient Memory Allocation: The original memory limits were too restrictive for the workload, leading to continuous pod restarts due to OOMKilled errors.
  2. Network Policy Syntax Error: The use of empty podSelector in network policies resulted in an empty ipset, effectively blocking all pod communication in the K3s environment.
  3. Stale iptables Rules: K3s’s kube-router did not automatically flush stale iptables rules, requiring manual pod restarts to apply new network policies.

Solution

Memory Configuration

  • Increased memory limits from 512Mi to 1Gi to accommodate the workload, reducing memory usage from 96% to 47% utilization.

Network Policy Fix

  • Revised network policy syntax to allow pod-to-pod communication within the namespace.
  • Added explicit egress rules to ensure connectivity to PostgreSQL, vLLM, DNS, and other necessary services.

Operational Fixes

  • Pod restarts were required to flush stale iptables rules and apply updated network policies.
  • Connectivity verification was performed to ensure all pods could communicate successfully.

Remaining Issue

  • Application-layer issue: The fast-agent response extraction problem persisted, where the .send() method returned tool call JSON instead of the processed message. Further investigation is required to determine the correct method for retrieving the final response.

The fixes resolved the critical infrastructure issues, ensuring stable operation of the email processing pipeline while maintaining proper pod communication and memory management.