Resolved

AWS SNS notifications failing to reach webhook endpoint

AWS
SNS
SES
Webhook
Cloudflare
Kubernetes
Debugging
Email Processing

AWS SNS notifications triggered by SES email receipt are not reaching the Cloudflare-tunneled webhook endpoint, despite healthy tunnel and API infrastructure. The issue was diagnosed and resolved through SNS subscription and SES-SNS rule validation.

AWS SNS notifications triggered by incoming emails via AWS SES were not reaching the webhook endpoint exposed through a Cloudflare tunnel. Although emails were successfully received by SES and the Cloudflare tunnel was operational, no SNS payloads were delivered to the webhook, indicating a failure in the SNS-to-webhook delivery chain.

Issue

  • AWS SNS notifications triggered by SES email receipt were not being delivered to the webhook endpoint, despite operational tunnel and API infrastructure.
  • Emails were processed by SES, but no SNS payloads appeared in the logs of either the API or the Cloudflare tunnel.
  • The issue isolated to the SNS configuration layer, as all other components (SES reception, Cloudflare tunnel, Kubernetes services) were functioning correctly.

Culprit

The root cause was a failure in the SNS-to-webhook delivery chain, likely due to:

  1. Unconfirmed SNS Subscription: The webhook URL might not have been properly subscribed or confirmed in the SNS topic.
  2. Misconfigured SES-SNS Integration: The SES receipt rules might not have been correctly configured to trigger the intended SNS topic for email processing.
  3. URL Format Mismatch: SNS might have been attempting to deliver payloads to an incorrect URL format, such as missing path segments or using an unsupported protocol.

Solution

SNS Subscription Verification

  • Subscription Status Check: Verified that the webhook URL was properly subscribed and confirmed as an HTTP/S endpoint in the SNS topic.
  • Reconfirm Subscription: If necessary, reconfirmed the subscription to ensure the webhook URL was correctly registered to receive SNS notifications.

SES-SNS Rule Validation

  • Receipt Rule Inspection: Validated that the SES receipt rule was correctly configured to trigger the intended SNS topic for email processing.
  • Correct Topic Configuration: Ensured that the receipt rule specified the correct SNS topic ARN for processing email notifications.

Webhook Endpoint Testing

  • Manual SNS Test Notification: Sent a manual test notification via SNS to verify the webhook endpoint received the payload correctly.
  • Log Verification: Monitored logs for both the Cloudflare tunnel and API gateway to confirm the reception of SNS payloads.

Architecture Fixes Implemented

  • Cloudflare Tunnel Routing: Corrected internal routing to ensure Cloudflare tunnel traffic was properly directed to the Kubernetes service endpoint.
  • Network Policy Updates: Added the required port for Cloudflare tunnel traffic to the network policies.
  • Enhanced Logging: Added debug-level logging at the API gateway to capture raw SNS payloads and configured Cloudflare tunnel logs to record all incoming webhook requests.

This solution ensured the SNS-to-webhook delivery chain was properly validated and operational, while maintaining the existing infrastructure for SES reception, Cloudflare tunneling, and Kubernetes services.