Tech visualization misclassifies common English words as frameworks

Working with a Python-based tech network visualization system that used semantic classification (sentence-transformers/Qwen3-Embedding-4B) to categorize technology terms, integrated with an Astro/D3.js frontend for rendering framework relationships. The system processed log files to generate JSON networks showing technology usage patterns across projects.

Issue

The semantic classification system produced false positives by misclassifying common English words (“to”, “the”, “from”) as technology frameworks, while the frontend visualization only rendered 2 of 8 available categories with inconsistent color mappings. Performance suffered from loading a 4B-parameter embedding model for simple term classification, with JSON generation taking minutes instead of seconds.

Culprit

Over-engineered Semantic Approach

The classify_tech_terms_by_similarity() function used sentence-transformers with Qwen3-Embedding-4B (4096-dimensional embeddings) to calculate cosine similarity against known frameworks, requiring three independent similarity thresholds (FRAMEWORK_SIMILARITY_THRESHOLD, LIBRARY_SIMILARITY_THRESHOLD, ENGLISH_WORD_THRESHOLD). This created:

1. False positives on generic English words
2. 4-6 second classification latency per term
3. Complex threshold tuning requirements

Frontend/Backend Mismatch

The Python backend generated 8 distinct categories (frontend, backend, database, etc.) while FrameworkNetworkCanvas.astro only rendered “frameworks” and “libraries” due to hardcoded filtering:

techTypes.includes(d.type)  // Only matched 2 of 8 possible types

Color mappings in the Canvas component used different RGB values than the legend, with hardcoded fallbacks for uncategorized terms.

Solution

Hardcoded Classification Replacement

Removed all semantic classification code (486 lines reduced to 486 lines after cleanup) and implemented a comprehensive hardcoded lookup system:

FRAMEWORKS = {
    'frontend': ['React', 'Astro', 'TypeScript', 'Tailwind', 'Shadcn'],
    'backend': ['Python', 'FastAPI', 'Flask', 'Django', 'Node.js'],
    'database': ['PostgreSQL', 'Redis', 'SQLite', 'MongoDB'],
    # ...7 total categories with 68+ terms
}

Dynamic Frontend Integration

1. resolved.astro: Modified to load categories dynamically from JSON node.group field
2. FrameworkNetworkCanvas.astro:
   • Updated filter from techTypes.includes() to d.type !== 'log'
   • Added 7-category color mapping matching backend output:

   .status-dot.frontend { background: #1b9e77; }
   .status-dot.backend { background: #d95f02; }
   /* ...7 total categories */

3. Performance: Classification became instant (regex lookup) with sub-second JSON generation

Example Output Transformation

Before (semantic):

Detected 89 "frameworks" (including "to", "the", "from")
Processing time: 4m12s

After (hardcoded):

Detected 68 unique frameworks
Top 10: Python (156x), Docker (89x), FastAPI (67x)
Processing time: 0.8s