Issue
A script (generate_framework_network_v2.py) for processing 7,784 code files and 600+ technology regex patterns suffered severe performance degradation. Initial attempts with either multiprocessing for I/O-bound tasks or threading for CPU-bound tasks failed due to serialization overhead and GIL limitations, respectively.
Symptoms:
- File reading phase: 7,784 files took >100 seconds instead of expected 1.7 seconds.
- Regex phase: Only 1 CPU core utilized despite 32 cores available.
Culprit
Backend Mismatch
- I/O-bound tasks with multiprocessing (loky): Serialization overhead dominated I/O time.
- CPU-bound regex with threading: Python’s Global Interpreter Lock (GIL) restricted true parallelism.
Root Cause
Monolithic backend usage (e.g., Parallel(n_jobs=32, backend='loky') for both phases) exacerbated bottlenecks.
Solution
Hybrid Backend Strategy
- Threading for I/O-bound tasks: File operations release the GIL during I/O waits, avoiding serialization overhead.
- Loky for CPU-bound regex: Separate processes bypass the GIL, enabling parallelism across all 32 cores.
Performance Results
| Phase | Work Type | Backend | Time Before | Time After | Speedup |
|---|---|---|---|---|---|
| File Reading | I/O-bound | Threading | >100s | 1.7s | 58x+ |
| Regex Extraction | CPU-bound | Loky | N/A* | 27.7s | - |
| Total | Hybrid | >100s | ~30s | 137x |
*N/A: Original threading approach would have taken ~50 minutes for regex phase (single-core).
Additional Optimizations
- File Deduplication: SHA256 hashing reduced 1,033 duplicate files.
- Visualization Improvements: Force-directed layout optimized for 300 iterations, hiding irrelevant nodes.
- Project Filtering: Excluded outdated projects and manual API providers.