Resolved

Cloudflare Tunnel connectivity failure due to protocol restrictions

Cloudflare
Kubernetes
Networking
Security
DevOps
Webhooks

Cloudflare Tunnel pod failed to establish connections due to restrictive Kubernetes network policies blocking the preferred QUIC protocol. The issue was resolved by configuring the tunnel to use an alternative protocol that complied with existing network policies.

A Cloudflare Tunnel pod was running but unable to establish connections with Cloudflare edge servers, causing a complete failure of the SNS webhook architecture. This disrupted the external email processing pipeline and resulted in connection refusal errors at the ingress layer.

Issue

  • The Cloudflare Tunnel pod was unable to establish connections with Cloudflare edge servers, leading to a failure in the SNS webhook architecture.
  • External health checks for the webhook endpoint failed, breaking the entire email processing pipeline.
  • The tunnel relied on a specific protocol that was not permitted by the existing network policies.

Culprit

The Kubernetes network policy only allowed egress on standard ports (DNS, HTTPS, internal API) but explicitly blocked the QUIC protocol, which Cloudflare Tunnel preferred for its connections. This resulted in:

  • A preference hierarchy that prioritized the blocked QUIC protocol over alternative protocols.
  • Inability to fall back to HTTP/2 due to missing configuration to force its use.
  • Zero-trust network architecture enforcing strict egress policies that did not account for the tunnel’s specific requirements.

Solution

Protocol Configuration

To resolve the issue, the Cloudflare Tunnel was configured to use the HTTP/2 protocol instead of QUIC:

  • Forced HTTP/2 usage: Updated the tunnel configuration to explicitly use HTTP/2, which complied with the existing network policies.
  • Maintained security: The solution adhered to security constraints while enabling connectivity through the allowed HTTPS (443) egress rule.

Verification

  • Tunnel logs: Verified successful connection establishment in the tunnel logs.
  • Webhook endpoint testing: Confirmed that the webhook endpoint returned HTTP 200 responses upon successful connection.

Architecture Preservation

  • Webhook-only ingress: Maintained the webhook-only ingress pattern, ensuring secure communication between SNS and the tunnel endpoint.
  • Zero outbound polling: Preserved the architecture that prevents K3s pods from initiating external connections.
  • Selective external access: Ensured that only necessary external services (e.g., S3 HTTPS) were permitted.
  • Network policy compliance: Kept the existing network policies intact without adding exceptions for the tunnel.

This solution resolved the connectivity issue while maintaining the security and architecture constraints of the system.