Resolved

CUDA out of memory errors with WhisperX and TTS pipelines in multi-language video generation

Python
AI/ML
GPU
CUDA
Memory Management
TTS
WhisperX
Font Rendering
CJK
Multilingual

GPU memory exhaustion and missing CJK character rendering when generating Japanese videos with mixed English code segments | Solves memory leaks from improper pipeline cleanup and font support for Japanese/Korean/Chinese text

Working with Python-based video generation pipelines using WhisperX for transcription and Kokoro TTS for multi-language voice synthesis, the system implemented Japanese language support with proper CJK font rendering while managing GPU memory constraints in a 14GB VRAM environment.

Issue

The system encountered critical failures when generating Japanese videos with English code segments:

  1. CUDA Out of Memory errors during WhisperX processing, with only 2.6GB free when 3.5GB was required.
  2. Japanese characters rendering as boxes due to missing CJK font support.
  3. English code pronounced in Japanese when using the Japanese TTS pipeline for all segments.

Culprit

Memory Exhaustion

Three root causes combined to exhaust GPU memory:

  1. Double KPipeline allocation: Both text and code pipelines were created without proper cleanup.
  2. No GPU cleanup between stages: TTS models remained in GPU memory during WhisperX processing.
  3. Simultaneous WhisperX model loading: Both the transcription and alignment models were loaded together with a default batch size of 16.

Font Rendering Failure

The system lacked CJK font support, missing ranges for:

  • CJK Unified Ideographs.
  • Hiragana and Katakana scripts.
  • Hangul characters.

Voice Routing Issue

The single-language pipeline architecture routed all segments, including English code, through the Japanese pipeline, causing incorrect pronunciation.

Solution

GPU Memory Management

  1. Single Pipeline Architecture: Created a conditional English pipeline only when necessary.
  2. Aggressive GPU Cleanup: Ensured all pipelines and models were deleted and GPU memory was cleared after TTS generation.
  3. Staged WhisperX Processing: Reduced batch size to lower peak memory usage and staged processing for better memory management.

CJK Font Support

  1. Dual Font System: Implemented automatic detection and selection between Latin and CJK fonts.
  2. Per-Character Font Selection: Dynamically selected the appropriate font based on character ranges.
  3. Environment-Based Configuration: Used environment variables to specify font paths for Latin and CJK fonts.

Multi-Language Voice Routing

Implemented a dual pipeline system to route segments based on type and language:

  • Code Segments: Used the English pipeline if needed.
  • Text Segments: Used the appropriate pipeline based on language and segment type.

The solution improved memory efficiency and resolved font and voice routing issues, ensuring proper multi-language video generation with CJK support and optimized GPU usage.