Resolved

WhisperX character timing causes multi-second delays at punctuation clusters

Python
WhisperX
AI/ML
Video Generation
Timing Alignment
Debugging
Performance
Text Processing

Visual-only characters like quotes and parentheses accumulate timing delays in WhisperX-based video generation | Consecutive punctuation creates 1+ second freezes despite correct audio alignment

Issue

Visual rendering delays occurred in automated video generation systems when processing consecutive punctuation or non-speakable characters. Despite continuous audio playback via WhisperX alignment, visual characters like quotes, parentheses, or symbols would accumulate delays of up to 1.25 seconds before appearing on screen. This issue was particularly noticeable in technical content with dense punctuation clusters or code blocks, where visuals appeared frozen while the audio progressed normally.

Culprit

The root cause was an accumulation mechanism for visual-only characters that incorrectly compounded timing estimates. The system attempted to align visual rendering with audio cues by tracking a visual_offset for non-speakable characters. However, this design failed in scenarios where:

  • Multiple visual characters appeared consecutively (e.g., """,)), leading to compounded delays.
  • Visual characters appeared at segment boundaries without following spoken words, preventing resets to the timing offset.
  • The set of speakable_chars included spaces and punctuation, causing misalignment when these were removed during processing.

Solution

Two key improvements were implemented:

  1. Speakable Character Set Refinement: The set of speakable_chars was restricted to letters and numbers only, eliminating spaces and punctuation. This ensured WhisperX alignment focused solely on characters with corresponding audio cues, preventing drift caused by non-speakable elements.

  2. Reduced Visual Duration: The default duration allocated to visual-only characters was halved from 250ms to 125ms. This reduced the worst-case delay for clustered punctuation from 1.25 seconds to 0.625 seconds while maintaining realistic typing visualization (~8 characters per second).

The revised system classifies characters as either speakable (handled by WhisperX for precise audio-visual alignment) or visual-only (estimated timing applied). The visual_offset resets upon encountering speakable characters, ensuring delays do not accumulate across non-speakable sequences. This approach improved synchronization in technical content while preserving natural visual rendering for typing effects.