Japanese text renders as boxes in video output with WhisperX alignment failures

Issue

A Python-based automated video generation pipeline encountered rendering issues when processing mixed Japanese-English content. Japanese text rendered as unreadable boxes (□□□□) while English code segments displayed correctly. Additionally, WhisperX alignment failed for multilingual audio, misclassifying Japanese characters as ‘non-speakable’ and producing incorrect character timings.

Culprit

Font Rendering Failure

The font loading system prioritized the first available font (LiberationMono-Regular.ttf) without considering language-specific requirements. Since this font lacks CJK glyphs (Unicode ranges 0x3040-0x309F, 0x30A0-0x30FF, 0x4E00-0x9FFF), Japanese characters were substituted with placeholder boxes despite the system successfully loading the font.

Audio Alignment Failure

WhisperX’s alignment model, designed for single-language input, failed to handle mixed-language content. Japanese characters were excluded from the speakable_chars set, causing misalignment in character timing for language transitions.

Solution

Language-Aware Font Selection

A priority system was introduced to detect multilingual requirements and reorder font lists:

• CJK font prioritization: When processing Japanese, Chinese, or Korean content, fonts containing CJK glyphs (e.g., NotoSansCJK-Regular.ttc) are loaded before fallback fonts.
• Dynamic font switching: Ensures the appropriate font is selected based on language detection, eliminating glyph substitution issues.

Per-Segment Multilingual Alignment

Language-aware alignment was implemented to handle mixed-language audio:

• Segment-based alignment: Each segment is processed with a language-specific alignment model (e.g., Japanese or English).
• Unicode support: Expanded character-level language detection to include Japanese Unicode ranges, ensuring accurate timing adjustments.
• Dynamic model loading: WhisperX models are loaded on-demand based on segment language, improving alignment accuracy for multilingual content.

Verification Results

• Japanese text now renders correctly, with proper glyph display.
• Audio alignment accurately handles mixed-language segments, with character timing aligned per language.
• System logs confirm successful CJK font selection and alignment for multilingual content.