Resolved

LLM feedback loop fails to improve outputs despite good ratings

Python
AI/ML
LLM
Feedback Loop
FastMCP
Prompt Engineering
Generative AI
System Design

Multi-turn conversation system where LLM iterations all received 'GOOD' ratings without improvement | Root cause was missing conversation context showing previous attempts and evaluator feedback

Issue

The LLM-based content generation system produced stagnant outputs with identical ‘GOOD’ ratings across multiple iterations, failing to incorporate evaluator feedback. Generated scripts remained unchanged (e.g., script_01_iter1_good.md through script_01_iter4_good.md), showing no observable improvement.

Culprit

The core problem was the lack of conversation context in the LLM’s input. The system failed to provide:

  1. Previous attempts: The LLM did not see its own prior iterations.
  2. Structured feedback: Evaluator comments were not presented in a way that allowed the LLM to recognize specific issues or improvement instructions.
  3. Multi-turn architecture: The system relied on single-turn OpenAI completions, appending feedback as plain text rather than framing it as a dialogue.

Solution

Multi-Turn Conversation Architecture

Implemented FastMCP’s ctx.sample() with SamplingMessage objects to create a structured conversation history:

  • Message structure: Each iteration includes the user’s initial request, the LLM’s previous attempt, and the evaluator’s feedback as distinct turns.
  • Improvement focus: Feedback is framed as explicit improvement instructions, enabling the LLM to iterate meaningfully.

Key Changes

  1. Generator Updates: Introduced a multi-turn sampling function (call_llm_with_sampling()) to replace direct OpenAI calls, maintaining context across iterations.
  2. Evaluation Loop: Tracked previous attempts and feedback, ensuring the LLM receives complete conversation history for each generation attempt.
  3. Temperature Control: Used deterministic settings (temperature = 0) for consistent evaluation outputs.

Expected Outcome

The revised architecture enables iterative refinement by:

  • Iteration 1: Starting with a baseline attempt (e.g., ‘FAIR’).
  • Subsequent Iterations: Incorporating feedback to produce progressively improved outputs (e.g., ‘GOOD’ → ‘EXCELLENT’).

The solution ensures the LLM leverages prior attempts and evaluator feedback as part of a structured dialogue, enabling meaningful improvements in generated content.