Kubernetes network policies block pod-to-pod API connectivity

Issue

API services in the Kubernetes cluster experienced a complete connectivity failure, with Cloudflare tunnels returning errors such as ERR Unable to reach the origin service despite successful readiness probes. This resulted in a false positive monitoring scenario where the system appeared healthy but was actually blocking traffic, leading to a halt in email processing.

Culprit

The root cause was stale iptables rules in the kube-router that persisted after NetworkPolicy deletions. These rules included REJECT all with icmp-port-unreachable, which blocked all pod-to-pod traffic except Kubernetes health checks. This created an invisible firewall that allowed monitoring probes to pass while blocking all other communication, including traffic from Cloudflare tunnels.

Solution

Immediate Fix

All NetworkPolicies were removed using kubectl delete networkpolicies --all, which flushed the corresponding iptables rules and immediately restored connectivity. Connectivity was verified by testing API endpoints directly.

Code Corrections

1. Fast-Agent API Fix: Updated the API to use the correct method for sending requests, ensuring proper communication between components.
2. Uvicorn Configuration: Temporarily reduced the number of Uvicorn workers to 1 to avoid potential resource contention, pending further investigation.

Security Remediation Plan

1. Minimal NetworkPolicy: Implemented a minimal NetworkPolicy to restore basic security while ensuring connectivity.
2. Verification Process: Established a process to test connectivity after each policy application, ensuring that changes did not reintroduce connectivity issues.
3. Persistent Issue Tracking: Addressed lingering issues, such as JSON response formatting, separately from connectivity problems.

The solution successfully restored connectivity and established a secure and verified configuration for the Kubernetes cluster.