Resolved

TTS and WhisperX character count mismatch causes keystroke desync

Python
TTS
WhisperX
Audio Processing
Text Alignment
Debugging
Video Generation
Timing Synchronization

Text-to-speech optimizer removed special characters before TTS processing but WhisperX aligned timings with original text | Resulted in keystroke sounds playing out of sync with displayed characters

Issue

The automated video generation pipeline experienced desynchronization between keystroke sounds and displayed characters after implementing a TTS optimizer. The issue worsened with special characters such as asterisks and quotation marks. For example, the text ‘hello world’ originally contained 15 characters, including special characters. The TTS optimizer removed these special characters, reducing the text to 11 characters ‘hello world’. This caused a mismatch where WhisperX generated timings for the 11-character audio, but the video pipeline attempted to display the original 15-character text, resulting in desynchronized keystroke sounds.

Culprit

The root cause was a character count mismatch between TTS processing and visual rendering. The pipeline followed these steps:

  1. Original text included special characters, totaling 15 characters.
  2. The TTS optimizer removed special characters, reducing the text to 11 characters.
  3. Kokoro TTS processed the optimized 11-character version.
  4. WhisperX generated timings based on the 11-character audio.
  5. The video pipeline attempted to display the original 15-character text using only the 11 timings.

Solution

Character Preservation

The TTS optimizer was updated to preserve all characters while retaining only safe transformations, such as case normalization. This ensured consistency between TTS input and visual output.

Pipeline Simplification

The pipeline was redesigned to guarantee a one-to-one character count match:

  1. Eliminated intermediate optimized text variables and used the original text consistently.
  2. Modified the timing extraction logic to work directly with the original text, ensuring synchronization.
  3. Removed redundant character-mapping logic to prevent mismatches.

Trade-offs and Design Principle

  • Kokoro now pronounces special characters literally, such as reading asterisks and quotation marks as-is.
  • The outcome was perfect audio-visual synchronization, where every character in the original text corresponded to a timing in the audio.
  • Verification confirmed no timing overlaps in test runs with complex text patterns.
  • A new design principle was established: The character count must remain identical between TTS processing and visual rendering when using WhisperX for timing alignment.