Resolved

CUDA out of memory error in WhisperX alignment on 16GB GPU

Python
PyTorch
CUDA
WhisperX
AI/ML
Performance
Debugging
Memory Management

WhisperX large-v2 and alignment models loaded simultaneously exceed 16GB GPU memory during long video processing (15+ minutes) | Fixes sequential model loading and memory cleanup in PyTorch/CUDA

Issue

CUDA Out-of-Memory (OOM) errors occurred during WhisperX alignment for videos longer than 15 minutes (918+ seconds). Despite a GPU with 16GB of total capacity, PyTorch failed to allocate 5.61 GiB of memory when only 5.55 GiB was free. This resulted in the error: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.61 GiB. GPU 0 has a total capacity of 15.57 GiB of which 5.55 GiB is free.

The error consistently occurred during the wav2vec2 forward pass at the whisperx.align() step, even though sufficient GPU capacity was theoretically available.

Culprit

The issue stemmed from two concurrent memory consumption problems:

  1. Simultaneous Model Loading: The large-v2 WhisperX model (~3GB) and the alignment model (~5.6GB) were loaded simultaneously, exceeding the 16GB GPU capacity. This caused a total memory requirement that surpassed the available GPU resources.

  2. Unreleased TTS Pipeline: The KPipeline instance for Kokoro TTS (~0.3GB) was never explicitly deleted after TTS generation, leading to residual memory consumption that further constrained available GPU resources.

Solution

Sequential Model Loading

The system was modified to load models sequentially. The large-v2 model was deleted before loading the alignment model. This involved explicit deletion of the model using Python’s garbage collection and clearing the CUDA cache to ensure proper memory release.

Kokoro TTS Cleanup

To prevent residual memory consumption, explicit deletion of the KPipeline instance was implemented after TTS generation. This ensured that all resources were properly released.

Post-TTS GPU Reset

After TTS generation, garbage collection and explicit CUDA cache clearing were enforced to reset GPU memory usage and prepare for subsequent processing stages.

Result: Memory usage was reduced to sequential peaks (~5.6GB max), enabling the processing of 15+ minute videos on 16GB GPUs. The fix maintained WhisperX’s accuracy while preventing OOM errors through controlled memory allocation.