WhisperX character alignment scrambles text positions and boundaries

Issue

WhisperX’s character alignment system introduced visual corruption in rendered text, particularly in code segments and mixed-content scenarios. Two critical failures emerged:

1. Character Position Scrambling: Visual-only characters (e.g., parentheses, brackets, quotes) appeared in incorrect positions, disrupting readability. For example, get_average() rendered as getaverage_ ana()lyze, with characters displaced by up to 10 positions.

2. Cross-Segment Contamination: Text segments bled into adjacent segments, particularly at transitions between code and text. Segment boundaries were ignored during rendering, causing visual misalignment (e.g., "This demonstrates" split into "demo""n and strates).

The issues worsened progressively with each visual-only character encountered, leading to unreadable output.

Culprit

Three root causes combined to create the alignment failures:

1. Character Normalization Mismatch: The system mapped WhisperX’s transcribed characters directly to the original input text. WhisperX’s normalization process (e.g., removing parentheses, altering case) introduced mismatches between the timing data and the original characters, causing progressive scrambling.

2. Missing Segment Boundary Clipping: Character timings extended into pause regions between segments, resulting in visual delays and misalignment where characters appeared during incorrect audio playback segments.

3. Improper Separator Handling: Newline separators between code and text segments were removed, causing distinct content types to merge visually without proper separation.

Solution

A multi-faceted approach resolved the alignment issues:

1. Original Character Preservation

The system now preserves the original input characters while adopting WhisperX’s timing data. This ensures precise character-to-position mapping, eliminating scrambling caused by normalization mismatches.

2. Segment Boundary Clipping

Character timings are clipped to segment boundaries, preventing visual delays and misalignment. Characters are adjusted to render before segment pauses, ensuring accurate synchronization with audio playback.

3. Proper Separator Handling

Context-aware separators are reintroduced to visually distinguish between code and text segments while maintaining natural flow within text segments.

4. Visual Character Configuration

Visual-only characters (e.g., parentheses, brackets) are explicitly excluded from speakable characters and configured to render instantaneously, ensuring they appear in the correct positions without affecting timing.

Result: The system now correctly renders text and code segments with accurate audio-visual synchronization, eliminating scrambling and cross-segment contamination.