Manual evaluator-optimizer loops create unmaintainable Python code

Working with FastMCP 2.0’s Python-based content generation pipeline, the orchestrator.py component managed evaluator-optimizer loops for script generation workflows. The system utilized separate generator and evaluator microservices communicating via FastMCP’s Client API pattern.

Issue

The orchestrator.py file contained over 200 lines of manual evaluator-optimizer loop logic with:

• Scattered feedback handling across multiple methods, making maintenance difficult.
• Repetitive iteration management code duplicated for different workflows.
• Hardcoded quality gates and refinement limits, reducing flexibility.
• No reusable abstraction for similar patterns in other components.

This procedural implementation made testing difficult and increased maintenance overhead when modifying evaluation criteria or refinement processes.

Culprit

The root cause was the lack of a design pattern abstraction for what should have been a reusable evaluator-optimizer structure:

1. No separation of concerns: Loop logic, generation, evaluation, and feedback handling were intermixed.
2. Hardcoded workflows: Each evaluator-optimizer scenario required custom implementation.
3. State management issues: Intermediate results and evaluation feedback were passed manually between steps.
4. No result encapsulation: Final outputs and evaluations were returned as raw tuples without metadata.

The original implementation relied on imperative loops with explicit iteration, leading to complex and difficult-to-maintain code.

Solution

Reusable Evaluator-Optimizer Pattern

Introduced a reusable evaluator-optimizer pattern with two core components:

1. Base EvaluatorOptimizer Class

Encapsulated the core pattern with configurable parameters, such as:

• Generator and evaluator clients for modular communication.
• Quality gates for defining acceptable ratings (e.g., EXCELLENT).
• Refinement limits for controlling the number of optimization iterations.
• Intermediate iteration saving to disk for debugging and analysis.

The class returned structured results encapsulating:

• Final output and evaluation.
• Achieved rating and iteration history.

2. Domain-Specific ScriptEvaluatorOptimizer

Specialized for tutorial script generation, offering built-in features like:

• Code segmentation and multi-segment breakdown generation.
• Script assembly and domain-specific evaluation.

Implementation Benefits

• Reduced the code size in orchestrator.py by 85%, from 215 lines of manual orchestration to a single line using the pattern.
• Improved testability and reusability of the evaluator-optimizer logic.
• Supported both simple and complex workflows through inheritance and configuration.

Migration Approach

1. Identified existing generator and evaluator components in the manual loops.
2. Replaced imperative loops with declarative pattern usage.
3. Updated result handling to use structured EvaluationResult objects.

This refactoring maintained identical behavior while significantly improving maintainability and scalability.