Resolved

Unvalidated modular RAG backend may replace monolithic production system

Python
AI/ML
Backend
RAG
Modular Architecture
Validation
Testing
Migration

Discovered an untested modular RAG backend with advanced features (hybrid search, AST chunking) that could replace the current monolithic production system | Need validation to determine if adoption is viable or if refactoring existing code is safer

Issue

The engineering team faces a critical architectural decision between two divergent RAG backend implementations:

  1. A monolithic production system with proven deployment but missing advanced features like hybrid search and AST-aware chunking.
  2. An undiscovered modular system with advanced features but untested and unvalidated, raising concerns about production readiness.

The core challenge is assessing the production viability of the modular system to avoid either:

  • Wasting resources refactoring the monolithic system if the modular alternative is viable, or
  • Adopting an untested system that risks production failures.

Key risks in the modular system include:

  • Suspiciously large code files (e.g., chunking logic, embeddings).
  • Missing dependencies or configuration issues.
  • Untested advanced features (e.g., hybrid search combining vector and BM25 ranking).

Culprit

The divergence stems from parallel development without synchronization:

  1. Organizational silos: Two teams developed separate systems in isolation.
  2. Technical debt: The monolithic system grew without modular boundaries, while the modular system lacked validation.
  3. Missing integration points: Dependencies (e.g., tree-sitter for AST parsing) were undeclared, and configurations were unreconciled.
  4. Feature implementation risks: Advanced features like hybrid search and AST chunking were untested despite having extensive test suites.

Solution

A structured 3-day validation sprint is proposed to objectively assess the modular system before committing to either path:

Day 1: Environment Validation

  • Install dependencies and verify test suite completeness.
  • Capture critical metrics like dependency resolution success and import errors.

Day 2: Blocker Resolution

  • Fix import errors and circular dependencies.
  • Prioritize unit and integration tests to achieve >70% unit test pass rate.
  • Validate configurations and authentication flows.

Day 3: Feature Validation & Decision

  • Test advanced features like hybrid search and AST chunking.
  • Analyze test coverage and resolve critical blockers.
  • Apply a decision matrix based on test pass rates and coverage:

    • 80% pass rate: Proceed with adoption.

    • 70-80%: Adopt with minor fixes.
    • 50-70%: Adopt with major fixes.
    • <50%: Fall back to the monolithic system.

Contingency Plans

  1. Adoption Path: Gradual deployment with feature flags and monitoring.
  2. Fallback Path: Refactor the monolithic system incrementally, cherry-picking viable components.
  3. Hybrid Approach: Deploy the modular system for new features while retaining the monolithic system for core operations.

Key Decisions

  • Dependency Management: Resolve missing dependencies (e.g., tree-sitter bindings) and pin critical libraries.
  • Test Strategy: Prioritize unit tests, then integration and end-to-end tests.
  • Performance Validation: Conduct load testing to ensure acceptable latency under production conditions.

Expected Outcomes

The sprint aims to provide a 10x return on investment by either:

  • Saving development time if adoption succeeds, or
  • Preventing wasted effort if adoption fails.

A structured approach minimizes risk while enabling informed decision-making.