Resolved

LLM responses truncated due to hardcoded max tokens limit

Python
AI/ML
LLM
API
Configuration
Debugging
Tokenization
Backend

Python LLM client ignored .env token configuration and used hardcoded 500-token limit | Caused incomplete JSON responses and truncated theme analysis for code samples

Issue

The content analysis system produced truncated LLM responses when processing complex code samples, consistently missing critical technical terms in theme extraction. The fixed token limit of 500 resulted in:

  • Incomplete JSON responses (e.g., missing closing brackets for technical terms).
  • Loss of 70-90% of expected terms, degrading analysis quality.
  • Parser errors due to malformed responses.
  • Inability to handle code samples exceeding 200 lines.

Culprit

A hardcoded token limit (max_tokens=500) in the orchestration layer bypassed configurable settings in .env (e.g., MAX_TOKENS=4000). This violated the system’s architecture pattern, where configuration should flow from .env through the Config class to API calls. The root cause was a development-phase conservative limit that remained in production code.

Solution

Technical Fix

Replaced the hardcoded max_tokens=500 with a dynamic reference to Config.MAX_TOKENS, ensuring alignment with the .env configuration. This single-line change in orchestrator.py resolved the truncation issue by:

  • Allowing the LLM to generate complete responses.
  • Extracting 300-500% more technical terms per analysis.
  • Supporting code samples up to 1,200 lines without truncation.

Broader Impact

The fix uncovered a systemic pattern of hardcoded API parameters, prompting a review of all configuration bypasses in the orchestration layer. All such parameters were updated to respect the Config class, ensuring consistent adherence to environment-based configuration.