Resolved

Handling indefinite hangs and timeouts in LLM API calls

Python
AI/ML
API
AsyncIO
Error Handling
Performance
Debugging

LLM API calls in a Python-based video generation pipeline were hanging indefinitely due to unreliability in timeout enforcement, leading to cryptic errors and pipeline failures. The issue was resolved by implementing robust timeout mechanisms and retry logic.

A Python-based video generation pipeline using an asynchronous LLM client (AsyncOpenAI) encountered critical issues during API calls. The system relied on concurrent API interactions for both generating and validating text snippets, with ChromaDB RAG context retrieval to enhance accuracy.

Issue

The pipeline suffered from two primary failures:

  1. Indefinite hangs: LLM API calls failed to enforce timeouts, causing the generator and validator components to freeze indefinitely.
  2. Cryptic errors: API responses containing content=None triggered unhelpful NoneType errors, preventing clear debugging and recovery.

The system required manual intervention (e.g., ctrl+c) to terminate, with validation for specific snippets consistently hanging while others completed successfully. The ChromaDB RAG context (comprising three chunks of ~2000 characters each) was confirmed as not contributing to the issue, with total validation prompts remaining well within token limits (~8k-15k characters).

Culprit

The issue stemmed from three interrelated problems:

1. Unreliable Timeout Enforcement

The AsyncOpenAI client’s built-in timeout parameter did not propagate to the asyncio event loop, allowing API calls to hang indefinitely without recovery. This lack of enforcement created unpredictable delays in the pipeline.

2. Missing Retry Logic

The absence of retry mechanisms meant that a single API failure would cause the entire pipeline to terminate abruptly, without any attempt to recover or retry the call.

3. Poor Error Handling

The system lacked validation for content=None responses, leading to cryptic NoneType errors instead of clear and actionable failure messages. This obscured the root cause of the problem and made debugging difficult.

Solution

To address these issues, the following improvements were implemented:

1. Timeout Enforcement with Explicit Limits

The solution involved replacing the unreliable AsyncOpenAI timeout with a coroutine-level timeout enforcement using asyncio.wait_for(). This ensured that API calls could not exceed a predefined duration (e.g., 90 seconds).

2. Retry Wrapper Implementation

A retry wrapper function (call_llm_with_retry) was created with the following parameters:

  • 3 retry attempts: A fixed number of retries to balance resilience and performance.
  • 10-second delay: A fixed delay between retries to avoid overwhelming the API with rapid successive calls.
  • 90-second timeout per attempt: Ensures each attempt is constrained by a strict deadline.
  • Explicit None content validation: Checks for content=None responses and raises clear error messages instead of allowing cryptic exceptions.

3. Improved Error Handling

The system now includes:

  • Explicit checks for None content before accessing response fields, preventing NoneType errors.
  • Clear error messages that include contextual details (e.g., finish_reason) to aid debugging.
  • Fail-fast principle: The system aborts after reaching the maximum number of retries, avoiding indefinite hangs.

Timeout Calculation

  • Best-case scenario: 0-90 seconds (immediate success on the first attempt).
  • Worst-case scenario: 290 seconds (3 attempts × 90 seconds + 2 delays × 10 seconds).

Modules Updated

The generator and validator modules were updated to incorporate these improvements, ensuring consistent behavior across all components of the pipeline.

Outcome

The implemented fixes resolved the issues of indefinite hangs and cryptic errors, resulting in:

  • Reliable timeout enforcement: API calls now adhere to strict time constraints.
  • Robust retry logic: The system can recover from transient failures without manual intervention.
  • Clear error messages: Debugging is streamlined with actionable feedback.
  • Stable pipeline execution: The video generation pipeline operates reliably, with reduced risk of hangs or failures.

These changes ensured a more resilient and maintainable system for LLM-based video generation.