Resolved

LLM model comparison reveals repetition and timeout issues in hybrid validation

AI/ML
LLM
Python
API
Performance
Debugging
NLP
Validation

Testing Mistral vs Qwen models for video script generation uncovered severe repetition patterns and missing API timeouts | Hybrid validation caused over-correction while pure Qwen showed baseline repetition issues

Testing two large language models—Mistral and Qwen—for automated video script generation revealed critical issues in a hybrid validation system. The evaluation uncovered severe repetition patterns and process timeouts, particularly in configurations combining both models.

Issue

The hybrid model configuration (Mistral for generation and Qwen for validation) introduced two significant problems:

  1. Severe Content Repetition: Qwen-based configurations produced scripts with identical phrases repeated excessively (e.g., the phrase “type hints define schema, docstring provides description” appeared five or more times in a single script).
  2. Process Hanging: The refinement pipeline stalled indefinitely during the second iteration due to unconfigured API timeouts, requiring manual intervention to resolve.

Culprit

Repetition Root Causes

  • Qwen’s Over-Correction: The model’s focus on technical correctness led to repetitive verification patterns, even during pure generation tasks.
  • Feedback Loop Mismatch: Mistral’s generated outputs were over-corrected by Qwen’s validation feedback, resulting in redundant explanations where corrections were repeated unnecessarily.
  • Lack of Refinement Guidance: There were no instructions to limit corrections to a single mention, leading to repetitive phrasing.

Timeout Issue

The absence of timeout configurations in the asynchronous API clients caused indefinite hanging when the model’s response exceeded expected durations. While the HTTP fetcher was configured with a timeout, the API clients for both generation and validation lacked such safeguards, leading to stalling.

Solution

Model Configuration

A pure Mistral configuration was selected for both generation and validation tasks due to:

  • Absence of repetition in test outputs.
  • Consistent quality across all generated scripts.
  • Natural integration of feedback when self-validating.
  • Production-ready baseline performance.

Timeout Implementation

Timeouts were added to the API clients, along with enhanced error handling to address stalling issues. This included:

  • Configuring a 60-second timeout for all API requests.
  • Implementing detailed error messages to distinguish between timeout errors and other failures.

Validation Parameters

Several parameters were adjusted to improve the validation process:

  • Maximum Tokens: Constrained feedback length to prevent excessive output.
  • Temperature: Set to zero to ensure deterministic outputs.
  • Token Limits: Established clear limits for both generation and refinement processes.
  • Iteration Control: Limited the number of refinements to prevent infinite loops.

These changes resolved the repetition issues and eliminated process hangs, resulting in a more reliable and efficient script generation pipeline.