AI system fails silently during partial outages causing undetected email processing failures

Issue

The email processing pipeline in the FastMCP microservice architecture exhibited silent failure behavior during critical component outages, particularly involving the vLLM AI processing service. This resulted in the following issues:

1. Undetected Timeouts: The system failed to detect prolonged timeouts (2+ seconds) or complete unavailability of the vLLM service.
2. Queued Emails: Emails remained in a pending status indefinitely during outages, without processing or user notification.
3. Lack of Observability: No mechanism existed to detect system degradation, cancel pending operations, notify affected users, or broadcast recovery status.
4. User Impact: Users received no response to their submissions during vLLM outages, with requests effectively disappearing from the system.

Culprit

Architectural Flaws

1. Missing Health Monitoring: The system lacked a dedicated health monitoring service to verify the availability of critical components like vLLM and PostgreSQL.
2. Optimistic Processing: Email workers assumed these services were available without verifying their status before processing.
3. No State Awareness: The system lacked awareness of its own operational state, failing to distinguish between healthy, degraded, or unavailable states.
4. Incomplete Status Model: The database schema supported only limited statuses (pending, processing, completed, failed, retry) and did not include a cancelled state for outage scenarios.

Technical Implementation Gaps

1. Insufficient Health Checks: Existing health checks did not differentiate between slow responses (e.g., high queue depth) and complete unavailability (e.g., timeouts or refused connections).
2. Uncontrolled Worker Behavior: Workers continued processing attempts during outages without implementing circuit-breaking mechanisms.
3. Lack of User Communication: There were no automated notifications for system status changes, leaving users unaware of the outage.

Solution

1. Health Monitoring Service

A dedicated health monitoring service was implemented to verify the availability of critical components like vLLM and PostgreSQL. This service enforced strict success criteria, ensuring that the system could reliably detect operational degradation. Redis was treated as optional to allow for graceful degradation.

Key components included:

• Binary Health Status: The system now provides a clear healthy or degraded status.
• Critical Path Verification: The health check verifies that vLLM responds within acceptable timeframes and that PostgreSQL is operational.
• Outage Handling: The service includes endpoints for canceling pending emails during outages and broadcasting recovery notifications.

2. State-Aware Email Worker

The email worker was redesigned to incorporate health-aware processing with a state transition system. This ensures that the worker dynamically adapts to changes in system health:

• Health Monitoring Loop: The worker continuously checks system health every 5 seconds.
• State Transition Handling: If the system transitions from healthy to degraded, pending emails are canceled, and users are notified. Conversely, if the system recovers from degraded to healthy, a recovery notification is broadcast, and processing resumes.
• State Machine: The worker operates within a defined state machine:
  • HEALTHY → DEGRADED: Cancel all pending emails and notify users.
  • DEGRADED → HEALTHY: Broadcast recovery and resume processing.

3. Database Schema Evolution

The database schema was extended to include a new cancelled status for the processed_emails table. This ensures that the system can track emails that were canceled due to system degradation:

• Status Constraint: The schema now enforces a constraint to include the cancelled status among valid states.
• Comprehensive Tracking: This allows the system to maintain accurate records of all emails, including those canceled during outages.

4. All-or-Nothing Architecture

The system was redesigned to adhere to an all-or-nothing architecture principle, ensuring robustness and reliability:

• No Partial Processing: The system either operates fully or cancels all pending operations during outages.
• No Automatic Retries: Users are required to resubmit their requests after system recovery.
• No Fallback Layers: The architecture avoids complex fallback mechanisms, instead relying on a single optimized processing path.
• Horizontal Scaling: To address performance needs, the system scales horizontally by adding GPU nodes rather than introducing additional complexity.

5. Monitoring and Alerts

Real-time monitoring and alerting were implemented to ensure operational visibility and rapid response to issues:

• Slack Integration: Automated alerts are sent to notify teams of system degradation and recovery events.
• Metrics Collection: The system tracks critical metrics such as health_status, pending_emails_count, cancelled_emails_24h, and uptime_percentage.
• Alerts: Specific alerts are triggered for system degradation (AI stack unavailable - emails being cancelled) and recovery (System restored - broadcasting to users).

6. Testing Strategy

A comprehensive testing strategy was implemented to validate the new architecture:

• Unit Tests: Isolated tests for health check logic to ensure correctness.
• Integration Tests: Verification of state transitions and system behavior during outages.
• Chaos Engineering: Manual testing via Kubernetes commands to simulate outages (e.g., scaling down vLLM replicas) and verify cancellation and recovery behaviors.