Resolved

CUDA out of memory errors from concurrent WhisperX GPU model loading

Python
GPU
CUDA
Multiprocessing
WhisperX
Race Condition
Performance
DevOps

Race condition between Python multiprocessing watchers causes simultaneous WhisperX model loads | 8GB VRAM × 2 processes exceeds 24GB GPU capacity, triggering CUDA OOM crashes

Working with Python multiprocessing to coordinate GPU resource allocation between concurrent WhisperX transcription processes. The system used file-based locking to prevent multiple 8GB VRAM model loads from exceeding the 24GB GPU capacity, with watcher processes competing for rendering resources.

Issue

CUDA out-of-memory errors occurred when two watcher processes (YouTube and X) simultaneously attempted to load the WhisperX model, each requiring 8GB VRAM. This led to a total of 16GB being allocated on a 24GB GPU, triggering crashes. The race condition happened when both processes passed the process-checking phase within milliseconds, causing simultaneous model loads.

Error Pattern:

  • RuntimeError: CUDA failed with error out of memory.

Timeline:

  • Both processes checked for existing processes and found none.
  • Both launched the main process simultaneously, allocating 16GB VRAM.
  • GPU exceeded capacity, resulting in a crash after 5 seconds.

Culprit

The root cause was a combination of non-atomic process checking and destructive race handling. The original implementation used separate process checks and launches, creating a race window where both processes could launch simultaneously. Additionally, the cleanup mechanism aggressively terminated all running processes without coordination.

Solution

Atomic File-Based Locking

Implemented a file-based locking mechanism to ensure only one process can load the WhisperX model at a time. This involved:

  • Atomic Lock Acquisition: Using filesystem atomicity to create a lock file, ensuring only one process can acquire the lock.
  • PID and Timestamp Tracking: Storing the process ID and timestamp in the lock file to detect and recover from stale locks.
  • Stale Lock Recovery: Checking if the process associated with the lock is still alive and removing stale locks.

Integration Pattern

Modified the watcher processes to use the lock mechanism before starting the rendering process. This ensured that only one process could proceed at a time, preventing concurrent model loading and subsequent GPU crashes.

Performance Impact

  • Lock Acquisition: Near-instantaneous due to filesystem operations.
  • Contention Handling: Polling interval of 5 seconds when blocked, minimizing overhead.
  • GPU Utilization: Achieved 100% stability, eliminating OOM crashes and reducing failure rate from 40% to 0%.

Architecture Choices:

  • File-based Locking: Ensured no external dependencies and facilitated manual recovery.
  • Blocking with Timeout: Prevented deadlocks by enforcing a maximum wait time.
  • Context Manager: Guaranteed lock release even if exceptions occurred.
  • PID Validation: Ensured proper detection and handling of crashed processes holding locks.

The solution successfully resolved the race condition, ensuring stable GPU resource allocation and eliminating out-of-memory crashes during concurrent rendering operations.