Resolved

LLM content analysis produces inconsistent results despite fixed inputs

Python
AI/ML
LLM
Configuration
Debugging
Backend
Determinism
NLP

Hardcoded temperature parameter in Python LLM client ignored .env configuration | Caused random quality variance and non-deterministic theme identification

Issue

The content analysis system in the video generation pipeline produced inconsistent results for identical inputs, including hallucinated tags (e.g., irrelevant themes like ‘middleware’ appearing in outputs) and missing valid tags across repeated runs. Despite configuring the system for deterministic behavior (e.g., setting TEMPERATURE=0.0 in the environment), the output remained probabilistic, as if the system were using a higher temperature value.

Culprit

The inconsistency stemmed from a hardcoded parameter within the system’s configuration logic. A critical setting that should have been controlled through centralized configuration (e.g., environment variables) was instead hardcoded as a fixed value. This bypassed the intended configuration system, causing the system to ignore user-defined settings and behave unpredictably.

Solution

Centralized Configuration

The hardcoded parameter was replaced with a reference to the centralized configuration system. This ensured that all components of the pipeline adhered to the same settings, eliminating discrepancies between intended and actual behavior.

Technical Impact

By aligning the system with its configured settings:

  • Deterministic outputs: Identical inputs now produced identical results.
  • Eliminated hallucinations: Random theme generation was removed, reducing false positives in tag identification.
  • Improved reliability: The system now consistently enforces user-defined parameters.

Verification

Testing confirmed the fix: repeated runs with identical inputs produced identical theme sets, demonstrating that the system now respects configuration settings. This resolved both the inconsistency issue and related downstream problems in script generation.

Broader Fixes

This fix was part of a broader effort to eliminate hardcoded parameters across the system, addressing similar issues in other components. Such changes ensure that all configuration is managed through a centralized system, reducing the risk of unintended behavior.