SNS webhook timeouts and email loss in synchronous processing

Issue

The original email processing architecture relied on synchronous handling, where AWS SNS webhooks triggered AI processing (using vLLM) before storing results in PostgreSQL. This design led to critical reliability failures:

1. SNS Webhook Timeouts: AI processing (taking 2–5 seconds) frequently exceeded the 15-second timeout limit for SNS webhooks.
2. Permanent Email Loss: If AI processing failed, emails were lost without any recovery mechanism.
3. Database Write Ordering: The system wrote to the database after AI processing, meaning a failure in AI processing would leave the email unrecorded in the database.
4. Lack of Observability: There was no way to track pending or failed emails in the system, making debugging and recovery challenging.

The original flow blocked on AI processing before writing to the database, creating a single point of failure. If any step failed, the entire email was lost.

Culprit

The root cause was the synchronous processing architecture, where:

• Webhook handlers performed blocking AI operations before database storage.
• Database writes occurred last in the sequence, after AI processing completed.
• No transactional safety mechanisms were in place, leading to partial failures and inconsistent system states.
• There was no retry mechanism for transient failures, exacerbating reliability issues.

Solution

Asynchronous Store-First Architecture

The solution involved adopting a store-first architecture that prioritizes database storage before AI processing. Key components of this approach include:

1. Immediate Database Storage: Emails are stored in PostgreSQL with a pending status immediately upon receipt via SNS webhook, ensuring no data loss even if AI processing fails.

2. Database Schema Enhancements: The schema was extended to include fields for tracking processing status (pending, completed, retry), retry count, timestamps, and error messages. This enables robust monitoring and retry logic.

3. Background Worker Implementation: A separate worker process polls the database for pending emails, processes them asynchronously, and updates their status. If processing fails, the system automatically retries (up to 3 attempts) with a 5-minute delay between retries.

4. Technical Improvements:

   • Reduced Webhook Latency: Webhook response time decreased from 2–5 seconds to under 100 milliseconds, as the system only needs to store the email in the database.
   • Zero Email Loss: The store-first approach guarantees that all emails are recorded, even if AI processing fails.
   • Full Observability: The database now tracks all pending, failed, and completed emails, providing complete visibility into the system’s state.
   • Scalability: The worker architecture supports horizontal scaling, allowing multiple workers to process emails concurrently.
   • Retry Mechanism: Automatic retries with exponential backoff (5-minute delay) ensure transient failures are handled gracefully.

The solution maintains the existing AI processing logic while introducing reliability guarantees through architectural changes, such as the store-first approach and background workers. This ensures high availability and resilience without requiring changes to the AI processing code itself.