Resolved

AI-generated code documentation introduces hallucinated concepts

AI/ML
Documentation
Python
LLM
Code Generation
Technical Writing
Prompt Engineering
Quality Assurance

LLM-generated closing sections in technical documentation incorrectly introduced concepts not present in the actual code | Caused by vague prompt constraints and missing validation criteria

Issue

The AI-generated closing sections in code documentation frequently introduced technical concepts not present in the actual codebase, creating ‘hallucinated’ content. Examples included:

  • Discussing ‘FastMCP Middleware Timing’ in code containing only @mcp.tool decorators.
  • Referencing ‘authentication’ and ‘server composition’ in simple arithmetic functions.
  • Introducing ‘performance optimization’ patterns in code with no timing or benchmarking logic. This resulted in 50% of generated documentation containing inaccurate technical information, risking developer misguidance.

Culprit

Three interconnected issues caused hallucinations:

  1. Overly vague prompt constraints allowed the LLM to interpret ‘ecosystem context’ broadly, introducing unrelated concepts from its training data.
  2. Missing validation criteria in the evaluator prompt meant no check was in place to ensure closing sections aligned with the actual code.
  3. Non-deterministic theme analysis used a hardcoded temperature setting, causing random and sometimes incorrect theme tagging.

Solution

Part 1: Strengthened Prompt Constraints

Explicitly restricted the LLM to only discuss concepts present in the provided code, including:

  • Examples of forbidden hallucinations (e.g., ‘middleware’ in code with no middleware).
  • Clear rules to tie closing sections back to actual code patterns.

Part 2: Added Validation Criterion

Introduced a new evaluator criterion to automatically fail if closing sections introduced concepts not found in the code, with examples of common hallucinations.

Part 3: Fixed Non-Deterministic Analysis

Replaced the hardcoded temperature setting with a configurable, deterministic approach to ensure consistent theme identification.

This transformation eliminated hallucinations entirely, achieving 0% hallucination rate in test runs while maintaining consistent documentation quality.