Keystroke audio produces robotic machine gun sounds instead of natural typing

Issue

The system for generating typing sounds in synthesized videos produced unnatural, rapid-fire audio effects resembling a ‘machine gun,’ rather than realistic human typing. Key problems included:

• Overlapping keystroke sounds cutting each other off during normal typing.
• Rapid sequences of sounds for newline and tab characters, disrupting the flow between code lines.
• Clustering of special character sounds (e.g., quotes, parentheses) within milliseconds, creating unnatural bursts.
• Missing realistic Shift/AltGr key sequences for German keyboard punctuation.
• Multiple sounds for consecutive spaces, particularly in code indentation.

Culprit

The issues stemmed from flaws in the audio processing pipeline:

1. Single-channel mixing: All keystroke sounds were combined on a single audio channel, leading to overlapping and abrupt cuts.
2. Hidden character sounds: Newline and tab characters triggered rapid-fire sounds between code lines, despite not being visually or audibly relevant.
3. Poor timing for visual-only characters: Special characters not spoken by the text-to-speech (TTS) system were assigned minimal timing, causing unnatural clustering.
4. Lack of Shift/AltGr sequences: German keyboard characters requiring Shift or AltGr modifiers were played as single sounds, lacking the realistic multi-step key sequences.
5. Missing space deduplication: Consecutive spaces, such as those in code indentation, generated multiple sounds without logical spacing.

Solution

To achieve more realistic typing audio, the system was refactored with the following improvements:

Multi-channel Audio Mixing

A distributed mixing architecture was implemented, distributing keystroke sounds across multiple parallel audio channels to prevent overlaps and abrupt cuts. This allowed for smoother sound transitions and natural overlapping of typing sounds.

Exclusion of Hidden Characters

Newline and tab characters were excluded from triggering sounds, eliminating rapid-fire sequences between code lines and improving the realism of typing effects.

Spaced Visual Character Timing

Visual-only characters, such as punctuation and symbols not spoken by the TTS system, were assigned a longer duration (80 milliseconds) to ensure they appeared naturally spaced and did not cluster unnaturally.

German Keyboard Sequences

Realistic typing sequences were implemented for German keyboard characters requiring Shift or AltGr modifiers. These sequences now include a press, key press, and release phase with appropriate spacing (100 milliseconds), mimicking human typing patterns.

Space Deduplication

Consecutive spaces were deduplicated to prevent multiple sounds for indentation or formatting spaces. This ensured clean and natural spacing in the audio output.

Result: The system now generates 99% fewer overlapping keystroke sounds while maintaining realistic typing rhythm and spacing. The improvements eliminated unnatural ‘brrrr’ effects and enhanced the realism of typing audio in synthesized videos.