Resolved

TTS voice and text animation desync in auto-generated videos

TTS
Text-to-Speech
Animation
Video Generation
Python
Audio Processing
Timing Synchronization
WhisperX

Voice narration finishes before on-screen text animation completes due to added keystroke delays | System designed for text-to-follow-voice but visual delays create timing mismatch

Issue

The system experienced a synchronization mismatch between voice narration and on-screen text animation. Voice narration completed prematurely while the text animation, which included added delays for visual effects, was still in progress. This resulted in a disjointed viewing experience where the voiceover ended before the text animation finished, breaking immersion. The issue primarily manifested with characters that were visually animated but not spoken, such as punctuation marks.

Culprit

The root cause was an architectural assumption that text animation duration would not exceed voice narration duration. When visual delays were introduced for typing effects, these delays were applied without adjusting the voice narration speed. Specifically:

  1. Voice timing was derived from audio alignment for speakable characters only.
  2. Fixed delays (200-300 milliseconds) were added for visually animated characters that lacked corresponding audio cues.
  3. The total text animation duration exceeded the voice duration, leading to a desynchronization.

Solution

To resolve the synchronization issue, a dynamic adjustment algorithm was implemented to balance voice and text durations:

  1. Baseline Voice Generation: Initially generated voice narration at standard speed to measure its duration.
  2. Text Animation Duration Calculation: Estimated the total text animation duration by accounting for both speakable and visually animated characters.
  3. Speed Adjustment: Adjusted the voice narration speed dynamically to match the calculated text animation duration, ensuring synchronization.
  4. Fallback Handling: Implemented safeguards for cases where the adjusted speed fell outside acceptable limits.

This approach maintained the core design principle of aligning text with voice while dynamically compensating for added visual delays, resulting in a seamless synchronization of audio and visual elements.