Twitter bot generates irrelevant zen koan replies to technical questions

Working with FastMCP 2.13.0 and ChromaDB for vector storage, a Twitter/X engagement bot classified tweets and generated context-appropriate replies using the OpenRouter API for LLM inference. The system processed tweets through a quality-evaluation pipeline with lane-specific generation strategies.

Issue

The engagement bot generated cryptic zen koan replies for all tweets, regardless of content type, including technical questions such as “Anyone explored FastMCP yet?” which required factual answers. The system exhibited:

• Multiple API calls per tweet without visible progress.
• Frequent rate limit hits due to lack of visibility into processing.
• High skip rates (90%+) due to an overly strict EXCELLENT-only quality bar.
• No real-time logging of tweet content, classifications, or generated replies.
• Lost subprocess logs, with critical information not surfacing to the main process.

Culprit

Single-Mode Processing Pipeline

All tweets were processed identically:

1. Uniform classification: Technical discussions, questions, and praise were all treated as opportunities for zen koan replies.
2. Strict quality gate: Only replies rated EXCELLENT were accepted, ignoring the context of different tweet types.
3. No lane differentiation: No distinction was made between types of tweets, leading to inappropriate responses.

Logging Architecture Flaw

• FastMCP subprocess isolation: Logs generated using logger.info() were captured by FastMCP’s stdio handler but never surfaced to the main process.
• No stderr utilization: Critical updates during long-running LLM operations were not logged to sys.stderr, leading to lost information.

Solution

Two-Lane Classification System

Introduced a system to classify tweets into different lanes based on content analysis:

• ZEN lane: For generic tweets, requiring a GOOD or EXCELLENT rating.
• DISCUSS lane: For technical discussions, requiring an EXCELLENT rating with up to 5 refinement attempts.

Lane-Specific Quality Bars

Lane	Quality Required	Max Refinements	Style
ZEN	GOOD or EXCELLENT	1	Paradoxical koans
DISCUSS	EXCELLENT only	5	Concise dev terminology

Real-Time Visibility

Implemented stderr-based logging to ensure critical updates were visible:

• Used sys.stderr to log tweet content, classifications, and replies, bypassing FastMCP’s stdio capture.
• Forced immediate output with sys.stderr.flush() to ensure logs were visible in real-time.

Prompt Modularization

Restructured prompts into dedicated files for classification, generation, and evaluation, improving maintainability and clarity:

• Classification rules: Defined in classify.txt.
• Technical reply style: Specified in discuss_generate.txt with constraints for conciseness and technical language.
• Quality criteria: Defined in discuss_evaluate.txt.
• Koan generation: Defined in zen_generate.txt.

Key Style Constraints:

• Concise and technical: Short, punchy, and focused on developer terminology.
• Character limits: Max 250 characters, with an ideal range of 100-180 characters.
• Use of abbreviations: Such as ctx, async, and config.

The solution improved the accuracy and relevance of replies while ensuring real-time visibility into the processing pipeline, reducing skip rates and enhancing overall system reliability.