A system for analyzing framework mentions in markdown files initially missed 82% of project documentation, leading to inaccurate results and duplicates. This caused overcounting and skewed technology stack representation.
Issue
The framework network analysis system processed only 351 log files, missing 82% of actual project documentation. When expanded to recursive scanning, 1,986 files were found, with 41% being exact duplicates. This led to:
- Overcounting of framework mentions.
- Skewed connection statistics.
- Inaccurate technology stack representation.
- Performance overhead from processing identical content.
Culprit
The issues were due to:
- Limited Scanning Scope: Only searched for log directories, missing other documentation.
- No Deduplication Mechanism: Identical content was treated as separate files, causing duplicates.
- Conversational Term Pollution: Generic terms were included, skewing technology detection.
Solution
Recursive Markdown Scanning
The file scanning scope was expanded to recursively scan all markdown files across projects, excluding unnecessary directories like .venv and node_modules.
SHA256 Content Deduplication
A deduplication mechanism was implemented using SHA256 hashing to identify and eliminate duplicate content across files, reducing the dataset from 1,986 to 950 unique documents.
Framework Detection Refinement
Non-technical terms were removed from the framework detection, and a minimum mention threshold was set to improve accuracy. The categorization logic was updated to handle 80 frameworks, up from 48.
Performance Optimized Pipeline
The processing pipeline was optimized to handle the expanded dataset efficiently, ensuring quick extraction and accurate results.
Visualization Improvements
The visualization was enhanced with GPU-accelerated rendering, improved interactivity, and alignment with portfolio voice, ensuring clarity and relevance of the presented data.