Resolved

D3.js visualization shows only 34 files instead of thousands in codebase

D3.js
Python
Frontend
Backend
Data Visualization
File System
Debugging
Performance

Network visualization component incorrectly limited to single project directory | Frontend loaded wrong JSON version while backend scanned incomplete file set

A D3.js-based network visualization component in an Astro frontend displayed only 34 files from a single project directory, instead of the full codebase containing thousands of files across multiple projects. This discrepancy caused misrepresentation of the actual codebase structure.

Issue

The visualization incorrectly displayed only 34 files while the actual codebase included thousands of files across 48 projects. The frontend loaded incorrect data, resulting in misleading metadata like “0 logs + 0 code” instead of the actual counts. Additionally, file nodes lacked proper visual indicators to distinguish between log files and code files.

Culprit

The issues arose from:

  • Incorrect Search Scope: The Python backend script was configured to scan only a single project directory, limiting the file count to 34 instead of the full codebase.
  • Wrong JSON Path: The frontend component loaded an outdated JSON version, causing incorrect data visualization.
  • Missing Visual Indicators: The visualization did not distinguish between log files and code files, and used incorrect metadata fields for rendering.

Solution

Fixed Search Scope

The backend Python script was updated to scan the entire codebase directory, rather than being limited to a single project directory. This change increased the file count from 34 to over 15,000 across all projects.

Corrected JSON Loading

The frontend component was updated to explicitly load the correct JSON data file, ensuring that the latest and most comprehensive data was visualized.

Enhanced Visualization Rendering

  • Visual indicators were added to distinguish between log files (📄 emoji) and code files (⭐ emoji).
  • Metadata was updated to display separate counts for log files and code files, ensuring clarity and accuracy.
  • Deduplication logic was implemented to handle duplicate files across projects, improving performance and accuracy.

Expected Outcome

The updated visualization will now display:

  • A comprehensive representation of the codebase with thousands of files and connections.
  • Clear visual indicators for log files and code files.
  • Accurate metadata showing the correct counts for technologies, log files, and code files.

This solution resolved the discrepancies and ensured that the visualization accurately reflects the structure and content of the full codebase.