WhisperX memory leak causes OOM crashes in video processing automation

A Python-based video processing automation system using WhisperX for speech recognition encountered severe memory leaks, leading to out-of-memory (OOM) crashes. The system relied on PyTorch models for processing videos sequentially, but improper memory management caused cumulative leaks that exhausted system resources.

Issue

The system would freeze and crash after processing approximately 20 videos sequentially, leaking 3-6GB of memory per video. Despite processing videos one at a time, systems with 128GB RAM would eventually exhaust memory capacity, resulting in unresponsive behavior. The automation script (run_all_watchers.sh) would become unresponsive after around 1 hour of operation.

Culprit

The root cause was improper memory handling of PyTorch models in WhisperX. Specifically:

• PyTorch models were loaded into CUDA memory for each video without explicit cleanup.
• Python’s garbage collector does not automatically release GPU memory allocations, leading to persistent leaks.
• CUDA tensors retained internal references even after Python variables went out of scope.
• Memory usage compounded with each video: 20 videos × 3-6GB = 60-120GB of accumulated leakage.

Solution

Memory Cleanup Protocol

A structured cleanup process was implemented after each video processing iteration:

1. Explicit Model Deletion: All model objects were explicitly deleted using del statements to break Python references.
2. Forced Garbage Collection: gc.collect() was called to ensure Python objects were released.
3. CUDA Cache Clearing: GPU memory was explicitly freed using torch.cuda.empty_cache() and torch.cuda.synchronize() to release CUDA allocations.
4. Verification Logging: Confirmation logs were added to verify memory cleanup completion.

Secondary Fixes

• Watcher Process Management: Removed aggressive process killing logic to prevent race conditions, relying instead on a robust lock mechanism (video_lock.py) for exclusive access control.
• Testing Validation: Created a test script (test_memory_cleanup.sh) to verify memory stability across sequential videos.

Performance Impact

The cleanup process added approximately 10 seconds of overhead per video but ensured long-term memory stability. The system now reliably processes videos without OOM crashes, even during extended 24/7 automation sessions.

Key Results:

• Memory usage stabilized at 13-15GB (previously increased by 3-6GB per video).
• GPU memory release confirmed via nvidia-smi.
• Watcher processes cooperated seamlessly using the lock mechanism.
• No more system freezes or OOM crashes during automation.