Pygments syntax highlighting misaligns colors with code characters

Issue

The syntax highlighting in a Pygments-based code processing pipeline produced misaligned colors, where:

• Keywords like def appeared split across multiple colors (e.g., d=orange, e=orange, f=gray).
• Function names showed incorrect color blending (e.g., hell=yellow, o=orange).
• Operators like ( appeared in default gray instead of operator orange.
• This resulted in visual corruption, making the code unreadable despite correct token generation.

Culprit

The root cause was separator newline characters with lang=None being included in the tokenization input. When concatenating segments for Pygments, leading newlines were treated as:

1. A keyword token by the lexer state machine.
2. This caused subsequent characters to be tokenized incorrectly, leading to misclassified syntax elements.

Solution

Input Filtering

Added language-based filtering to exclude separator characters before tokenization:

• Filtered out characters with lang=None to ensure Pygments receives clean input.
• Constructed a filtered string for tokenization that excluded separator characters.

Dual-Pass Color Mapping

1. Code Characters: Mapped tokens to the filtered list of code characters.
2. Separators: Applied default coloring to separator characters separately.

Verification Results

Metric	Before	After
Tokenization Input	`‘
def hello(name):‘`	'def hello(name):'
’def’ Tokenization	❌ Split colors	✅ Uniform orange
’hello’ Classification	❌ Mixed types	✅ Pure function
Operator Colors	❌ Default gray	✅ Operator orange

The solution ensures that Pygments receives clean input without lexer-confusing characters, while maintaining proper color mapping for both code and separator elements. This resolves the misaligned syntax highlighting issue by addressing the root cause at the tokenization stage.