Issue

The Bloomberg article indexing job failed due to a missing dependency (tree_sitter) that was unintentionally required during the import of the ArticleChunker class. Despite the dependency being excluded from the article processing pipeline, the import chain triggered a cascading requirement for tree_sitter, which was needed for AST-based code parsing in the CodeChunker class.

Culprit

The root cause was an implicit dependency chain created by module-level imports in the __init__.py file. The architecture intended to separate concerns between article text processing (lightweight) and code parsing (requiring tree_sitter), but the design failed to enforce this separation at the import boundary. Specifically:

The ArticleChunker class was designed to handle text-based chunking and should not depend on tree_sitter.
The __init__.py file imported all chunking classes at the module level, creating an unintended dependency chain.
Importing ArticleChunker triggered the import of CodeChunker, which in turn required tree_sitter.

Solution

The issue was resolved by inlining the ArticleChunker implementation directly into the indexing script (index_bloomberg_articles.py), thereby eliminating the problematic import chain entirely. This approach:

Removed all external dependencies for the article processing pipeline, ensuring no reliance on tree_sitter or other RAG-specific modules.
Preserved the identical API for backward compatibility.
Maintained self-contained execution, allowing the script to run with minimal dependencies (only langchain for text splitting).
Avoided import side effects by bypassing the __init__.py entirely.

This solution strictly enforced the separation of concerns between article and code processing pipelines, ensuring lightweight execution for article indexing while maintaining full functionality for code chunking in other contexts.

Future improvements include a more robust module structure to prevent similar dependency issues and better separation of concerns.