Resolved

Cloudflare tunnel fails to route emails to Kubernetes API service

Kubernetes
Cloudflare
AWS SES
SNS
Networking
DevOps
API
Email Processing

External emails via AWS SES/SNS not reaching Kubernetes API service due to Cloudflare tunnel misconfiguration | NodePort vs internal service port routing issues in K3s cluster

Issue

The email routing pipeline in the Kubernetes cluster failed silently in production, despite initial tests appearing successful. External emails processed through AWS SES/SNS were not triggering AI responses, as the Cloudflare tunnel could not properly route traffic to the internal API service. The tunnel was attempting to access the API via internal DNS (api-service.name.svc.cluster.local:30800), which is invalid for NodePort services.

Culprit

Misconfigured NodePort Routing

The Cloudflare tunnel was incorrectly configured to use the NodePort (30800) via internal DNS, a setup that is unsupported. NodePorts are only accessible through the host IP, not internal cluster DNS.

Service Port Ignored

The API service’s internal port (8000) was overlooked in favor of the NodePort, despite the tunnel running inside the cluster. Internal DNS resolution should have used the service port (8000), not the NodePort.

Missing Network Policies

The Cloudflare tunnel pod lacked explicit network permissions to communicate with the API service on the required port (8000), causing silent failures in routing.

Solution

Tunnel Reconfiguration

The Cloudflare tunnel was updated to route traffic to the internal service port (8000) instead of the NodePort, enabling proper DNS resolution within the cluster.

Network Policy Updates

Network policies were modified to allow the tunnel pod to communicate with the API service on ports 7844 (tunnel port) and 8000 (API service port).

Service Discovery Validation

The architecture was validated to support:

  • External Access: NodePort (30800) for host-based traffic.
  • Internal Routing: Service port (8000) for cluster-internal communication via DNS.

Flow Restoration

The complete email pipeline was restored: Email → AWS SES → SNS → Cloudflare Tunnel → API Service → vLLM/MCP → Response.

Key Technical Decisions

  1. Tunnel Placement: Deployed inside the cluster to utilize internal DNS.
  2. Port Separation: Maintained NodePort for external access while using service ports internally.
  3. Scalability: Internal service routing automatically adapts to pod scaling.
  4. Security: Network policies restrict tunnel-to-API communication to required ports.