Resolved

AWS SNS webhook 429 errors cause 30 minute email delivery delays

AWS SNS
Rate Limiting
Python
Redis
Webhooks
Performance
Backend
Email Processing

Rate limit of 10 requests/minute on SNS webhook triggers AWS retry backoff, delaying email processing by 15-30 minutes | Solution increases limit to 300/min while maintaining abuse protection

Issue

AWS SNS webhook responses returned HTTP 429 errors during email bursts (50-150 emails/minute), triggering AWS’s exponential retry backoff mechanism. This resulted in delays of 15-30 minutes for legitimate email processing, despite the system’s promise of 5-minute response times. The low rate limit of 10 requests/minute was insufficient to handle production traffic patterns, causing emails to become stuck in the SNS retry queue.

Culprit

The root cause was a hardcoded rate limit of 10 requests per minute in the .env configuration for the AWS SNS webhook. This limit was enforced using Python’s SlowAPI library with Redis-backed storage, which limited requests per IP address. When AWS SNS (which uses a limited set of source IPs) sent bursts of 50-150 emails per minute, all requests beyond the 10/minute threshold were rejected, triggering AWS’s aggressive retry mechanism. The exponential backoff resulted in significant delays, creating a reliability issue.

Solution

To address this issue, the following changes were implemented:

Increased Rate Limit

The rate limit was increased to 300 requests per minute to accommodate realistic email bursts while maintaining abuse protection. This change:

  • Handles up to 150 simultaneous emails without delays.
  • Supports a sustained load of approximately 5 emails per second.
  • Maintains protection against abuse by setting a maximum of 18,000 emails per hour.

Architecture-Aware Rate Limiting

The solution recognized that:

  • The HTTP layer (300/min limit) should not be the bottleneck.
  • The actual bottleneck is the vLLM processing (~1 email/minute with full orchestration).
  • Natural rate limiting exists at multiple stages:
    • Email worker poll interval (5 seconds).
    • vLLM processing time (2-10 seconds per email).
    • Total capacity of ~50-60 emails/hour sustained.

Monitoring Implementation

Critical alerts were added to monitor:

  • Dead Letter Queue (DLQ) messages: Tracks emails that fail after 23 days of retries.
  • HTTP 429 responses: Monitors for rate limit violations on the /sns-webhook endpoint (should be rare with the new limit).
  • Email processing latency: Alerts if processing takes longer than 5 minutes, indicating a potential backlog.

Verification Testing

A simulation was conducted to validate the new rate limit:

  • Test Scenario: Sent 150 requests from a single IP to simulate AWS SNS behavior.
  • Expected Result: All 150 requests were accepted immediately, staying within the 300/minute limit.

This solution ensures that legitimate traffic is processed quickly while maintaining the system’s reliability. The 300/minute limit provides sufficient headroom for AWS SNS traffic patterns without compromising protection against abuse. The overall principle remains focused on reliability over speed, ensuring all legitimate traffic is accepted quickly while processing occurs at the system’s sustainable rate.