Resolved

Resolving duplicate email responses in an AI-powered booking system

AI/ML
Backend
Debugging
Email
Workflow
Python
Tool Calling

An AI-based booking system was generating duplicate email responses, with the first containing correct information and the second containing raw technical data. The issue was resolved by adjusting tool parser configurations and addressing evaluator logic.

An AI-powered booking system deployed in Kubernetes was generating duplicate email responses for a single booking request. The first email correctly confirmed the booking, while the second email (sent shortly afterward) contained raw technical data and a false failure message, causing confusion for customers.

Issue

The system generated two emails per booking request:

  • First email: Correct customer-facing confirmation (e.g., “Confirmed! [Name] will babysit…”).
  • Second email: False failure message with raw JSON tool parameters, exposing internal technical details.

This issue created customer confusion and highlighted an underlying flaw in the system’s workflow.

Culprit

Primary Issue (Fixed)

The root cause was an incorrect tool parser configuration for the AI model, which caused tool calls to be returned as plain text instead of structured data. Specifically:

  • The tool parser was misconfigured for the AI model in use (e.g., hermes instead of the correct parser for the model).
  • This resulted in broken agent workflows and unintended retry attempts.

Secondary Issue (Unresolved)

The system’s evaluator logic incorrectly marked successful responses as needing improvement, triggering unnecessary retry iterations. Evidence included:

  • Logs showing the first iteration completing successfully with the correct response.
  • The second iteration generating a broken response with raw JSON data.

Hypothesized causes for the secondary issue included:

  • Overly strict evaluator criteria (e.g., mismatched time formats).
  • Improper loop termination logic in the agent workflow.
  • Response extraction capturing all iterations instead of just the final successful one.

Solution

Fixed AI Tool Configuration

  • Updated the tool parser to the correct configuration for the AI model in use.
  • This ensured tool calls were properly structured, enabling the agent workflow to complete successfully.
  • The corrected configuration allowed for proper response generation and email delivery.

Network Policy Adjustment

  • Enabled HTTPS egress in the Kubernetes network policy to allow email sending to external services.
  • This resolved issues with email delivery through the AI system’s email service integration.

Pending Fixes

  1. Evaluator Logic: Investigated the criteria used by the evaluator to determine if a response needed improvement. The goal is to ensure only truly problematic responses trigger retries.
  2. Loop Termination: Verified the logic controlling the agent workflow iterations to ensure proper termination after a successful response.
  3. Response Extraction: Ensured that only the final successful response is processed and sent to the customer.

This solution resolved the immediate issue of duplicate emails while addressing the underlying causes of incorrect retry behavior.