AWS SES email processing pipeline fails with notification format mismatch

Issue

The email processing pipeline failed with two critical symptoms:

1. Notification Format Mismatch: The webhook handler crashed when receiving AWS SES notifications in a direct format ("notificationType": "Received") instead of the expected SNS-wrapped format.
2. Environment Breakage: Deployment failed silently when critical environment variables like VLLM_URL and DATABASE_URL were missing, resulting in connection errors for the vLLM integration.

Culprit

Notification Format Problem

The AWS SES documentation was ambiguous, leading to the incorrect assumption that notifications would arrive in an SNS-wrapped format. However, the actual payload used a direct SES format, causing the handler to fail. The expected SNS format was:

{
  "Type": "Notification",
  "Message": "{SES payload}"
}

While the actual received format was:

{
  "notificationType": "Received",
  "mail": { ... },
  "receipt": { ... }
}

Environment Variable Loss

During deployment, Kubernetes configmaps were not regenerated, leading to missing or improperly formatted environment variables. This caused silent failures in the vLLM client initialization, specifically:

• Missing VLLM_URL without the required http:// prefix.
• Undefined DATABASE_URL, which is necessary for PostgreSQL connection strings.

Solution

SES Notification Handling

The webhook parser was updated to handle both SES and SNS notification formats. The parse_event function now checks for the presence of a Type field in the event to determine the format and processes it accordingly.

Environment Configuration

1. Added Protocol to vLLM URL: The VLLM_URL environment variable was configured explicitly with the http:// protocol to ensure proper connection.

   env:
     - name: VLLM_URL
       value: "http://vllm-service.name.svc.cluster.local:8000"

2. Regenerated Configmaps: A post-deployment script was implemented to regenerate Kubernetes configmaps from the .env file, ensuring all necessary environment variables are correctly applied.

Critical Fixes

1. Cloudflare Tunnel Routing: The internal Kubernetes DNS resolution was configured to correctly route requests to api-service.name.svc.cluster.local:8000.

2. Database Verification: A PostgreSQL health check endpoint was added to provide real-time status and metrics:

   {
     "status": "healthy",
     "processed_emails": 42,
     "last_processed": "2025-09-28T16:26:55Z"
   }

3. Graceful Degradation: A four-layer fallback mechanism was implemented:

   • Primary vLLM integration
   • Fallback to a local LLM
   • Template responses
   • Generic error messages

The solution ensured a robust end-to-end email processing pipeline with verified reliability, successfully handling complete workflows from email reception to response delivery.