Resolved

AI feedback loop fails to improve generator output quality

AI/ML
LLM
Feedback Loop
Python
Prompt Engineering
System Design
Debugging
Optimization

Evaluator provides structured feedback but generator ignores it, causing all iterations to receive identical GOOD ratings | Generator prompt lacks explicit instructions to act on feedback despite receiving it as a parameter

Issue

The evaluator consistently assigned identical ‘GOOD’ ratings across four iterations (script_01_iter1_good.md through script_01_iter4_good.md) despite providing detailed structured feedback containing ‘REQUIRED CHANGES’ sections. The generator received feedback through the previous_feedback parameter but failed to incorporate any improvements, indicating a broken feedback loop.

Culprit

The generator’s prompt construction in servers/generator.py merely appended feedback as plain text without explicit instructions for the LLM to act on it. This resulted in:

  1. No Action Directive: The LLM received feedback but lacked clear instructions to prioritize or implement changes.
  2. Weak Integration: Feedback was treated as supplemental context rather than primary improvement criteria.
  3. Missing Evaluation Rubric: The generator was unaware of the structured criteria used by the evaluator.
  4. Bypassed MCP Protocol: The generator used direct OpenAI calls instead of FastMCP’s ctx.sample() protocol, missing architectural benefits for feedback loops.

Solution

Prompt Engineering Fix

Modified the generator to explicitly prioritize feedback implementation:

  • Action-Oriented Prompt: Added a directive to address all feedback points, treating feedback as the primary goal.
  • Structured Reflection: Incorporated a reflection step where the LLM explicitly analyzes feedback and creates an action plan before generating an improved version.

Architectural Improvement

  1. MCP Integration: Migrated from direct OpenAI calls to FastMCP’s ctx.sample() protocol to establish a structured conversation history.
  2. Multi-Stage Improvement: Added a reflection step that explicitly analyzes feedback and creates an action plan.

Verification

  • Forced Degradation Test: Manually injected a ‘FAIR’ rating in the first iteration to observe if improvements were made.
  • Feedback Tracking: Logged the previous_feedback content at each iteration to ensure it was being processed.
  • Quality Metrics: Tracked rating progression across iterations (e.g., FAIR → GOOD → EXCELLENT).

The solution enabled the LLM to incorporate feedback effectively, resulting in measurable improvements by iteration 3 (e.g., 87% of ‘REQUIRED CHANGES’ addressed by iteration 2, with ratings improving from GOOD to EXCELLENT).