Resolved

WhisperX memory leak and watcher race condition causes OOM crashes

Python
PyTorch
WhisperX
Memory Leak
Multiprocessing
Race Condition
OOM
Performance
DevOps
AI/ML

Progressive memory accumulation from WhisperX PyTorch models combined with watcher process race conditions leads to system freezes and SIGKILL crashes | Memory grows 3-6GB per video until 128GB RAM exhaustion while watchers preemptively kill each other's rendering processes

Issue

The system experienced progressive memory accumulation and frequent crashes after prolonged operation, leading to out-of-memory (OOM) errors and process conflicts. Videos failed to process correctly, resulting in system freezes and inefficient resource usage. The system’s multiprocessing architecture introduced race conditions that exacerbated the issue, causing processes to interfere with each other and exacerbate memory leaks.

Culprit

The issue was caused by three primary factors:

  1. Memory Leaks in Resource Management: Speech recognition models were loaded but never explicitly released after use, causing progressive memory accumulation in both GPU and CPU memory. Over time, this led to significant resource exhaustion, even on systems with substantial memory capacity.

  2. Race Conditions in Process Coordination: The system attempted to coordinate processes by preemptively terminating others before acquiring locks. This approach led to destructive interference, where processes would incorrectly terminate each other, causing failures and wasted resources.

  3. Hardcoded Resource Thresholds: Monitoring intervals and maximum resource thresholds were fixed and overly aggressive, leading to unnecessary CPU overhead and insufficient memory buffers to accommodate legitimate memory usage.

Solution

Explicit Resource Cleanup

To address memory leaks, the system was updated to explicitly release all loaded resources after use. This included forcing garbage collection and clearing cached memory, ensuring that large objects were properly freed and not retained unnecessarily.

Lock-Based Process Coordination

The coordination mechanism was redesigned to prioritize acquiring locks before performing any actions. This ensures that only one process handles a task at a time, preventing race conditions and destructive interference. The new approach relies on exclusive access mechanisms that are more robust and reliable.

Configurable Resource Monitoring

Resource thresholds and monitoring intervals were made configurable, allowing the system to adapt to different hardware configurations and usage patterns. This reduces unnecessary overhead and provides sufficient memory buffers to accommodate normal operations without triggering premature alerts.

Verification and Impact

The changes resulted in stable memory usage, elimination of race conditions, and improved system reliability. The system now operates for extended periods without encountering memory-related failures, and processes coordinate effectively to avoid conflicts.

Performance Improvements

  • Memory Usage: Stable and predictable, with no progressive accumulation.
  • Process Stability: Reduced failures due to race conditions and improved reliability.
  • System Uptime: Extended operational periods without crashes or OOM errors.
  • Resource Efficiency: Optimized monitoring and reduced CPU overhead.