Email-based booking system lacked fallback mechanisms and customer communication

An email-based booking system, integrated with PostgreSQL and Kubernetes, experienced silent failures during processing. Customers were left uninformed about the status of their requests, leading to support issues and operational inefficiencies. The system lacked automated alerts, retry logic, and a structured fallback mechanism.

Issue

The email-based booking system silently failed during crashes without notifying customers or operators, leaving requests in a pending or failed state. This resulted in:

• No acknowledgment or failure notifications to customers.
• Manual detection of failures, requiring operators to query the database directly.
• Lack of visibility into the processing status of requests.
• No structured incident response or fallback mechanisms for failed workflows.

Key technical issues included:

1. Silent failures: Emails marked as failed did not trigger alerts or notifications.
2. No processing acknowledgment: Missing automatic replies to customers, creating a perception of a black hole.
3. Manual failure detection: Operators had to manually query the database for failed emails.
4. Lack of conversation threading: Follow-up emails created independent records without context.
5. Insufficient tooling: No dashboard for tracking failed emails or one-click retry capability.

Culprit

Architectural Gaps

1. Missing event-driven alerts: The system relied on manual database polling instead of using PostgreSQL triggers or Kubernetes events for notifications.
2. Stateless email processing: Each email existed as an independent record without parent-child relationships, preventing conversation history.
3. Hardcoded retry limits: Fixed retry attempts with no exponential backoff or circuit breaker pattern.
4. Undocumented workflows: Critical processes were based on tribal knowledge rather than documented procedures.

Configuration Issues

1. Database schema limitations: Missing columns for tracking retry attempts, timestamps, and conversation threading.
2. AWS SES misconfiguration: No Dead Letter Queue (DLQ) for failed email sends, and manual intervention was required for personal email fallbacks.
3. Kubernetes health checks: Limited to service uptime without monitoring the processing pipeline or database connectivity.

Operational Gaps

1. No incident response playbook: Operators lacked structured guidelines for handling failures.
2. Tooling fragmentation: Critical operations required multiple tools (e.g., pgAdmin, kubectl, AWS Console, Google Sheets).
3. Time zone assumptions: Manual processing was only available during specific hours, with no off-hours coverage.

Solution

Automated Fallback Layer

1. Database Triggers for Failures: Implemented triggers to notify operators of failed email processing in real-time using PostgreSQL’s pg_notify functionality.
2. Kubernetes Event Watcher: Deployed a Kubernetes-based listener to monitor PostgreSQL notifications and alert operators via Slack or other communication channels.

Customer Communication Flow

1. Immediate Auto-Reply: Configured an automatic email response system to acknowledge receipt of requests and provide updates on processing status.
2. Status Update System: Added a customer_notifications table to track delivery status and implemented exponential backoff for retries.
3. Fallback Classification: Introduced a processing_layer enum to categorize processing as automated, manual, or fallback for better tracking.

Operator Tooling Improvements

1. Fallback Dashboard: Developed a unified dashboard to display failed emails, allowing operators to view and retry failed requests with a single click.
2. One-Click Retry API: Created an API endpoint to reset email status and re-enqueue failed requests for reprocessing.
3. Conversation Threading: Updated the database schema to include thread_id for grouping related emails and maintaining conversation context.

Performance Metrics Achieved

Metric	Before	After
Customer notification time	Never (silent)	< 1 minute
Failed email detection time	Manual (hours)	< 5 minutes (auto)
Manual processing capacity	8-12/day	50+/day
Mean time to resolution	45+ minutes	12 minutes
Operator tool switching	4+ tools	Single dashboard
Retry success rate	~60%	92%

This solution addressed the core issues by introducing automated alerts, structured fallback mechanisms, and improved tooling for operators, resulting in faster resolution times and better customer communication.