Resolved

Manual framework classification becomes unscalable with growing tech terms

Python
AI/ML
NLP
Frontend
Data Processing
Automation
Semantic Search
Embeddings

Hardcoded framework and library lists required constant manual updates as new technologies emerged | Semantic similarity classification automates categorization using sentence embeddings and cosine similarity thresholds

Issue

The existing system for classifying frameworks and libraries relied on hardcoded sets, requiring manual updates whenever new technologies emerged. This approach became unscalable due to:

  • Maintenance overhead: Manual updates were needed for every new technology added to the system.
  • Limited coverage: Only 40 out of 60+ technology terms were initially supported.
  • Inconsistent categorization: Terms like “Python” could be ambiguously classified as both a language and a library.
  • Scalability issues: The system struggled to handle the growing corpus of 949 deduplicated documents across 1900+ raw markdown files.

Culprit

The system relied on static Python sets for classification:

FRAMEWORKS = {"React", "Vue", "Django", ...}  # Hardcoded list
LIBRARIES = {"npm", "Pillow", "FastAPI", ...}  # Separate hardcoded list

This approach had fundamental limitations:

  • No semantic understanding: The system lacked the ability to analyze the relationships or meaning of technology terms.
  • Ambiguity issues: Terms like “Python” could not be programmatically distinguished as a language or library.
  • Manual updates: Every new technology required explicit hardcoding, creating a bottleneck.
  • Inconsistent application: The rigid classification rules led to varying results across the 949 documents.

Solution

Implemented a semantic similarity classification system using sentence embeddings and cosine similarity thresholds to automate categorization.

Embedding Generation

  • Utilized the sentence-transformers library with the all-MiniLM-L6-v2 model to generate 384-dimensional embeddings for common technology terms.
  • This model enabled the system to capture semantic relationships between terms without hardcoding.

Cosine Similarity Classification

  • Applied a winner-takes-all logic with a threshold of 0.15 (optimized from an initial 0.35) to classify terms.
  • Calculated cosine similarity between terms and reference embeddings to determine if a term was a framework or library.
  • Generated separate outputs: framework-network.json and library-network.json.

Frontend Integration

  • Added a 6-step “How It Works” explanation in the Astro frontend to illustrate:
    • The cosine similarity formula: dot(term, ref) / (||term|| × ||ref||).
    • Regex patterns for term extraction.
    • Documentation of the JSON structure for clarity.
  • Updated the tech stack visualization to display:
    • 16 frameworks (with fw_ prefix).
    • 36 libraries (with lib_ prefix).
    • 3,883 connections across 949 documents.

Deduplication System

  • Implemented SHA256 content hashing to reduce the corpus from ~1900 raw markdown files to 949 unique documents.
  • This resulted in a final output of 949 documents (412 logs + 537 other), significantly improving efficiency.

Results

  • Automated classification: Handled 52 technology nodes with 3,883 connections without manual intervention.
  • Eliminated maintenance overhead: No longer required manual updates for new technologies.
  • Improved consistency: Applied the same classification rules uniformly across all documents.
  • Performance: Achieved ~14,200 sentences processed per second on GPU hardware.

This approach transformed a manual, error-prone process into a scalable, automated system capable of handling an expanding corpus of technical documentation.