Resolved

D3.js visualization excludes 85 percent of technology nodes due to hardcoded filters

D3.js
JavaScript
Frontend
Data Visualization
Python
Semantic Analysis
Astro
Debugging

Frontend network visualization only displayed 9 of 53 technology nodes due to hardcoded category filters and type exclusions | Semantic classification pipeline replaced manual categorization to ensure full coverage

A D3.js-based force-directed network visualization in an Astro application initially displayed only 9 of 53 technology nodes, excluding libraries entirely and using outdated category mappings. This led to an incomplete representation of the technology stack.

Issue

The visualization suffered from multiple issues:

  • Missing Nodes: Only 9 of 53 technology nodes were displayed, with 37 libraries completely excluded.
  • Inconsistent Categorization: Hardcoded categories did not align with the semantic classification results.
  • Misleading UI: The legend referenced outdated categories and included examples that were not rendered.
  • Data Pipeline Mismatch: The backend generated framework and library prefixes, but the frontend filtered for only frameworks.

Culprit

The issues stemmed from:

  • Hardcoded Frontend Filters: The frontend explicitly excluded libraries, resulting in only frameworks being displayed.
  • Manual Category Mapping: Complex and outdated logic tied to 8 hardcoded categories that no longer matched the backend data.
  • Architectural Debt: The system retained hardcoded mappings and filters despite evolving to use semantic classification.
  • Data Flow Mismatch: The backend generated type prefixes (fw_, lib_), but the frontend did not account for libraries in its filtering logic.

Solution

Backend Simplification

The backend categorization logic was simplified to rely solely on semantic classification, removing redundant manual mappings and ensuring consistency between backend and frontend.

Frontend Architecture Updates

  • Inclusive Filtering: Updated the frontend to include both frameworks and libraries in the visualization.
  • Category System Replacement: Replaced hardcoded category colors and mappings with a simplified semantic system (frameworks, libraries, other).
  • Performance Optimization: Adjusted force simulation parameters and nebula pre-rendering to accommodate all 53 technology nodes.
  • Legend and Styling Alignment: Updated the legend and CSS selectors to reflect the new semantic categories.

Maintenance Improvements

The updated workflow eliminated redundant manual updates across multiple files, simplifying future maintenance and ensuring consistency between the backend and frontend.

Technical Outcomes

  • Visible Tech Nodes: Increased from 9 (17%) to 53 (100%).
  • Node Types Rendered: Expanded from 1 (framework) to 2 (frameworks and libraries).
  • Categorization Logic: Reduced from 100+ lines of code to 6 lines.
  • Performance: Render time improved from 1.2s to 0.9s despite a 5x increase in nodes.

The updated visualization now accurately reflects the complete technology stack and aligns with the semantic classification pipeline, ensuring comprehensive and maintainable representation.