Testing gaps in CLI tool due to infrastructure bypass in FastMCP system

A Python-based FastMCP orchestration system uses a CLI tool for development testing, bypassing critical production infrastructure components like SNS webhooks, PostgreSQL queues, and email worker retry logic. This creates significant testing gaps, as infrastructure resilience and end-to-end processing cannot be validated.

Issue

The CLI tool bypasses key production components, resulting in:

1. No validation of infrastructure resilience: Database retries, queue processing, and retry logic are not tested.
2. Absence of network latency simulation: Critical for detecting issues like HTTP-layer deduplication failures.
3. False positives in testing: AI orchestration and tool interactions appear functional, but infrastructure-level failures (e.g., queue timeouts) remain undetected until production.

This discrepancy leads to sub-millisecond test responses, masking potential production failures.

Culprit

The CLI tool (cli_test.py) directly communicates with the server via stdio, bypassing the production stack entirely:

• Direct stdio communication: Skips the full infrastructure pipeline (AWS SES → SNS webhook → PostgreSQL queue → email_worker).
• Rapid development focus: Prioritized low-latency testing for core AI orchestration logic over infrastructure validation.
• Architectural simplification: Enabled multi-user testing via from_email parameters without simulating network conditions.

Solution

A three-tiered testing strategy was proposed to address these gaps:

1. Core Logic Testing (Current Implementation)

• Use case: High-frequency (100x/day) testing of AI orchestration logic and tool interactions.
• Latency: Sub-millisecond (direct stdio).
• Coverage: Validates middleware stack and vLLM inference but does not test infrastructure components.

2. Optional HTTP Proxy Layer (Future Extension)

• Use case: Simulate network latency and HTTP overhead for partial infrastructure testing.
• Latency: 200-500ms (HTTP proxy overhead).
• Implementation: Introduce a proxy layer to standardize communication paths and support future multi-channel requirements.
• Benefits: Unified gateway for future channels (e.g., WhatsApp, SMS).

3. Full Stack Validation (Production Simulation)

• Use case: Weekly infrastructure validation to test the complete pipeline.
• Latency: 5-15 seconds (full queue processing).
• Coverage: Validates SNS webhooks, PostgreSQL queues, and retry logic.
• Tooling: Simulate real email sending or SNS payloads to replicate production conditions.

Key Tradeoffs

Approach	Latency	Complexity	Infrastructure Fidelity
CLI stdio	~1ms	Low	None
HTTP Proxy	200-500ms	Medium	Partial
Full Stack	5-15s	High	Complete

This approach maintains the current CLI for core logic testing while documenting a pathway for future infrastructure validation. Critical infrastructure testing remains reserved for full stack simulations.