Resolved

Resilient fallback system for horse matching during AI service outages

Database
SQL
Fallback System
Backend
Rule-Based Processing
System Resilience

Parent requests for horses fail when primary AI-based services are unavailable. Implements a database-driven fallback using SQL queries and manual confirmation workflows.

When primary AI services (e.g., natural language processing and horse selection algorithms) are unavailable, the horse matching system must degrade gracefully to ensure core functionality remains accessible. This solution introduces a database-only fallback that relies on SQL queries and rule-based processing to maintain basic operations.

Issue

The horse matching system becomes entirely non-functional when AI-dependent components fail, such as:

  • Natural language parsing of parent requests
  • Horse selection algorithms
  • Automated confirmation workflows

This design flaw leaves users unable to make requests if advanced services experience downtime, as there is no alternative path for processing requests using only database resources.

Culprit

The system was originally designed with a ‘happy path’ approach, where all critical operations relied on AI services. When these services fail (e.g., due to unavailability of AI models or backend services), the entire pipeline collapses because:

  • No fallback mechanism exists for parsing email requests.
  • Horse selection depends entirely on AI-driven ranking.
  • Confirmation processes require automated tooling.

Solution

A resilient fallback system was implemented to replace AI components with database-driven logic. This approach ensures 80% functionality during outages while requiring minimal manual intervention.

Database-Driven Request Processing

The solution replaces AI dependencies with three key phases:

  1. Rule-Based Email Parsing: Extracts request details (e.g., dates, locations) using predefined patterns. For example:

    • Defaults to a future date if no explicit booking time is provided.
    • Matches locations against a predefined list of valid cities.

  2. Parent Context Retrieval: Uses SQL queries to fetch relevant user data (e.g., email, subscription status) from the database.

  3. Horse Selection Algorithm: Implements SQL-based ranking to identify suitable horses, prioritizing:

    • Horses with prior interactions with the parent.
    • Horses with high acceptance and completion rates.
    • Availability during the requested timeframe.

Confirmation Workflow

To ensure accuracy, the system introduces a manual confirmation process:

  • Generates a 6-digit random code for each request.
  • Sends a confirmation email to the parent, requiring an explicit reply with the code.
  • Uses separate codes for horse verification.
  • Enforces a 24-hour timeout for parent responses.

Key Improvements

  • Zero AI Dependency: All processing relies on SQL and rule-based logic.
  • Graceful Degradation: Maintains core functionality during service disruptions.
  • Explicit Manual Confirmation: Reduces errors by requiring user acknowledgment.
  • Schema Enhancements: Added metrics like accept_rate and completion_rate to improve selection logic.

This approach transforms the system into a resilient architecture capable of operating independently of AI services while preserving essential features.