Resolved

ChromaDB semantic search misses relevant content in large documents

ChromaDB
RAG
Python
Semantic Search
Vector Database
NLP
Document Processing
Retrieval-Augmented Generation

RAG implementation truncates retrieved documents to 2000 chars, losing critical information located beyond the cutoff | Fix implements proper pre-indexing chunking strategy for accurate semantic retrieval

Issue

The ChromaDB-based Retrieval-Augmented Generation (RAG) pipeline failed to retrieve complete documentation from large Markdown files, leading to loss of critical context for code validation. The system was designed to provide context using semantic search but truncated retrieved documents to the first 2000 characters. This caused incomplete information when important details—such as those for specialized decorators like @mcp.tool()—appeared beyond the truncation limit, resulting in inaccurate validation outcomes.

Culprit

The problem stemmed from a mismatch between indexing and retrieval strategies:

  1. Indexing granularity: Documents were indexed as full files (exceeding 10,000 characters) but retrieved as fixed 2000-character chunks.
  2. Post-retrieval truncation: Semantic search identified relevant files, but the application layer arbitrarily truncated the retrieved content, discarding 80% or more of the document.
  3. Missing pre-processing: ChromaDB stored entire files, but no pre-chunking step was applied, forcing the application to perform ad-hoc truncation during usage.

Solution

To resolve this, a standard RAG chunking pattern was implemented with three key changes:

1. Pre-Indexing Chunking

Documents were divided into consistent 2000-character chunks before indexing. This ensured that retrieval operations could target specific segments of the original content, preserving all relevant information.

2. Metadata-Enhanced Retrieval

Retrieved chunks were labeled with metadata, including their position within the original file. This allowed the system to reference specific sections of the documentation, improving retrieval precision.

3. Full Chunk Utilization

Post-retrieval truncation was eliminated entirely. Instead of arbitrarily cutting off content, the system utilized complete chunks (2000 characters each) to ensure no information was lost during context preparation.

Key improvements:

  • Indexing efficiency: Increased indexing time marginally (from ~20 seconds to ~30 seconds) as a one-time cost.
  • Retrieval accuracy: Improved precision by targeting specific document sections, particularly for features documented mid-file or beyond the original 2000-character limit.
  • Debuggability: Added chunk numbering to metadata for easier troubleshooting.
  • Context consistency: Maintained a total context size of ~6000 characters while ensuring all relevant content was included.

Validation impact:

  • 100% accuracy for features documented at the start of files (unchanged).
  • ~40% accuracy improvement for features documented mid-file.
  • Complete resolution for features documented beyond the original 2000-character truncation limit.