Resolved

Codebase technology references lack human-readable documentation index

Python
Documentation
Code Analysis
Markdown
DevTools
Automation
File Processing
Data Visualization

Existing JSON network graphs from code scanning lack a searchable markdown index showing which files mention specific technologies | Needed human-readable reference for 285+ technologies across 867 files

The project involved creating a Python-based human-readable documentation index generator to complement existing JSON network graphs for technology references in a codebase. The goal was to scan 1,189 markdown files and produce a sorted, deduplicated index of 285 technologies, including file-level references for easy navigation.

Issue

The existing script, generate_framework_network.py, produced JSON network graphs for D3.js visualizations but lacked a searchable index that mapped technologies to their respective source files. Developers faced significant challenges:

  • No centralized index: Required manual searches across 930+ unique files to locate technology references.
  • Duplicate processing: The script reprocessed 253 identical files without deduplication.
  • False positives: Ambiguous terms (e.g., “Less”, “Rich”) were incorrectly classified as technologies.
  • No file-level mapping: While the network graph showed relationships, it did not indicate where these technologies were referenced.

Culprit

The original script had technical limitations that prevented the generation of a usable index:

  • Output format: JSON network graphs did not include file path references.
  • Lack of deduplication: No mechanism (e.g., SHA256 hashing) to skip identical files.
  • Thresholding issues: Technologies mentioned only once were included, creating noise.
  • No sorting or filtering: The output was unsorted, and false positives were not excluded.
  • Context loss: File-level associations were discarded during JSON conversion.

Solution

A new script, generate_tech_file_list.py, was developed to address these limitations:

Core Architecture

The solution extended the existing script’s functionality by:

  • Deduplicating files: Using SHA256 hashing to skip 253 duplicate files, ensuring only unique files were processed.
  • Deduplicating technologies: Removing duplicate technology references per file to avoid redundancy.
  • Filtering false positives: Excluding ambiguous terms (e.g., “Less”, “Rich”) and common programming constructs (e.g., Python built-ins like “Set”, “List”).
  • Setting configurable thresholds: Requiring a minimum of 2 mentions per technology to reduce noise.

Key Features

  • Alphabetical sorting: Technologies and file paths were sorted for readability.
  • File counts: Each technology entry included the number of files mentioning it.
  • Markdown compatibility: The output was formatted for GitHub/GitLab rendering, with consistent path separators across operating systems.
  • Performance: Scanned 1,189 files in ~60 seconds, with 93% of files containing technology mentions.

Output Structure

The generated Markdown index included:

  • A header section with the total number of technologies indexed.
  • A dedicated section for each technology, listing the files where it was mentioned.
  • File paths sorted alphabetically for easy navigation.

This approach ensured easy discoverability, reduced manual effort, and scalable documentation for developers navigating the codebase.