Manual framework and library classification becomes unscalable with growing tech terms

Issue

The portfolio website’s network visualization for frameworks and libraries relied on a manually curated classification system. As the technology ecosystem expanded, maintaining this system became unscalable due to:

• The need to manually classify each new technology term (e.g., “Next.js”, “Vitest”, “Terraform”).
• Inconsistent classification criteria, leading to ambiguous categorizations for terms like “Docker” or “npm”.
• A hardcoded approach that lacked adaptability to new or emerging technologies.

The system also failed to distinguish between multi-category technologies and forced binary categorization, excluding or misclassifying tools that fit multiple roles.

Culprit

The original implementation used a static, hardcoded set of frameworks in a Python-based classification system. Key limitations included:

1. Manual Maintenance: Every new technology required manual updates to the classification set.
2. Inconsistent Criteria: No standardized or objective method for determining framework vs. library status.
3. Binary Categorization: Terms like “Python” (language vs. ecosystem) or “Docker” (platform/tool) were impossible to classify accurately with a rigid binary system.
4. Scalability Limits: The system struggled with the rapid growth of new technologies, creating significant overhead for developers.
5. Output Constraints: The generated framework-network.json file only supported frameworks, excluding libraries entirely or forcing them into an incorrect category.

Solution

To address these challenges, a semantic similarity-based pipeline was implemented using the sentence-transformers/all-MiniLM-L6-v2 model. The solution replaced manual classification with a model-driven approach, offering the following improvements:

• Semantic Classification Architecture: The system compares embeddings of each technology term against reference vectors for “framework” and “library”. Terms are classified based on cosine similarity scores, enabling nuanced categorization.

• Dynamic Output System: The pipeline now generates two separate JSON files:

  • framework-network.json: Contains terms with higher similarity to the “framework” reference.
  • library-network.json: Contains terms with higher similarity to the “library” reference. Both files maintain the bipartite network structure but with automated, consistent classification.

• Configurable Thresholds: Similarity thresholds for classification (default: 0.35) can be adjusted, allowing flexibility for different classification criteria.

• Expanded Technology Coverage: The system now supports a comprehensive list of technologies across categories such as frontend, backend, databases, testing, DevOps, and package managers, ensuring broader coverage.

• Performance Optimizations: The all-MiniLM-L6-v2 model was selected for its balance between accuracy and speed (~14,200 sentences processed per second on GPU). Techniques like embedding caching and batch processing were implemented to improve efficiency.

• Frontend Integration: The Astro components were updated to:

  • Dynamically load both framework and library datasets.
  • Include a toggle switch for switching between visualizations.
  • Maintain the existing network visualization logic while supporting dual datasets.

This solution reduced manual classification effort by 90% while improving consistency and scalability. Future enhancements include:

• Adjustable similarity thresholds for finer-grained control.
• Support for multi-label classification to handle ambiguous technologies.
• Expansion of reference terms (e.g., “web framework” vs. “JavaScript library”) to refine categorization.