ChromaDB hardcoded collection names prevent multi-topic document retrieval

Issue

The script generation system was unable to retrieve relevant documentation when switching between topics (e.g., FastMCP to Pydantic) via the --topic command-line argument. Despite the topic specification, all queries defaulted to FastMCP documentation, resulting in low-quality outputs. For example, Pydantic-related scripts were validated against FastMCP documentation chunks, leading to mismatched context and incorrect analysis.

Culprit

The system relied on hardcoded references in multiple files, enforcing FastMCP-specific behavior. Key issues included:

1. Fixed collection names: ChromaDB collections were hardcoded to use FastMCP documentation exclusively.
2. Path restrictions: File paths were locked to FastMCP documentation directories.
3. Terminology dependencies: Validation prompts included FastMCP-specific terminology, causing misalignment with other topics.
4. Lack of abstraction: The architecture lacked a mechanism to dynamically resolve topic-specific documentation sources.

Solution

To enable multi-topic support, the system was refactored with the following improvements:

Topic Configuration Layer

A centralized configuration system was introduced to manage topic-specific settings, including:

• Collection names: Unique identifiers for each topic’s ChromaDB collection.
• Documentation paths: Flexible paths to source documentation for each topic.
• File patterns: Support for different file extensions (e.g., .md, .mdx).
• Terminology: Topic-specific terminology to ensure accurate validation and generation.

Dynamic ChromaDB Indexing

The system was updated to:

• Initialize all configured ChromaDB collections at startup.
• Index documents only when collections are empty, ensuring up-to-date and relevant data.
• Store collections in a global dictionary for easy access based on the topic.

Topic-Aware Processing

Key functions were modified to accept a topic parameter, enabling:

• Validation: Dynamic selection of the appropriate ChromaDB collection for validation based on the specified topic.
• Generation: Topic-specific terminology in prompts and dynamic collection lookup for generation.
• Orchestration: Propagation of the topic parameter through all tool calls to maintain consistency.

Expanded Documentation Acquisition

To support Pydantic-specific queries, the system was enhanced to:

• Acquire sparse documentation: Focused on extracting only relevant documentation files (e.g., 87 .md files for Pydantic).
• Fetch discussions: Utilized GitHub’s GraphQL API to retrieve discussions and answers for comprehensive coverage.

Resulting Collections

The system now supports multiple collections with diverse documentation sources:

• FastMCP: 758 chunks from 181 .mdx files.
• Pydantic: 4033 chunks from 87 documentation files and 2180 discussions.

This approach resolved the topic-switching issue, ensuring relevant documentation is retrieved for accurate script validation and generation while maintaining scalability for additional topics.