Resolved

Text-to-video generator shows punctuation too fast with machine gun effect

Python
TTS
Video Generation
Audio Processing
WhisperX
Text-to-Speech
Timing Synchronization
Media Processing

Unpronounced characters in auto-generated videos appeared at unrealistic 80ms speeds instead of matching actual keystroke durations | Fixed by using dynamic timing from keystroke audio files

Issue

The video generation system was producing unnatural pacing for unpronounced characters, such as punctuation and symbols, which appeared too quickly on screen. This resulted in a visual effect that resembled rapid machine gun fire, where characters flashed by much faster than normal typing speeds, causing confusion and an unnatural viewing experience.

Culprit

The issue was caused by three primary technical limitations:

  1. Text-to-Speech Limitations: The text-to-speech engine did not pronounce certain characters, such as punctuation and symbols, leaving no associated timing data for these characters.

  2. Timing Alignment Gaps: The system used for timing alignment did not provide timing information for unpronounced characters, which are not spoken during the audio track.

  3. Hardcoded Duration: A fixed duration of 80 milliseconds was universally applied to all characters, including unpronounced ones, which is significantly faster than typical typing speeds.

Solution

Dynamic Duration Lookup

The system was updated to use dynamic durations based on audio data for keystrokes. Instead of applying a fixed duration, the system now looks up the specific audio duration associated with each character to ensure it matches the actual keystroke timing.

Pipeline Reordering

The execution order of the pipeline was adjusted to load character-specific audio data before extracting timing information. This ensures that all necessary data is available when needed to accurately render unpronounced characters.

Character Classification

A classification system was implemented to distinguish between characters that are spoken and those that are not. Spoken characters use the timing data from the alignment system, while unpronounced characters use the audio durations for keystrokes. If a specific duration is not available for a character, a default duration is applied to maintain consistent rendering.

The solution ensures that unpronounced characters appear at a natural typing speed, improving the viewing experience and maintaining synchronization between visual and audio elements.