Resolved

Markdown colon labels lose styling after code blocks in Pygments

Python
Markdown
Pygments
Syntax Highlighting
Text Processing
Debugging
Tokenization
Frontend

Pygments token reordering causes markdown colon labels (e.g., 'Resources:') to render incorrectly after code blocks | Leading whitespace triggers token position misalignment in semantic analysis

Issue

A markdown processing system using Python’s Pygments library for syntax highlighting failed to consistently apply the correct turquoise color to colon-prefixed labels (e.g., ‘Resources:’, ‘Tools:’) when they appeared after code blocks. While these labels rendered correctly when placed before code blocks, they appeared in green (incorrect color) when following code blocks and leading whitespace (e.g., newlines).

This issue occurred specifically when:

  1. A code block was present earlier in the markdown.
  2. The subsequent text segment began with whitespace (e.g., newlines).
  3. The colon-prefixed label followed this whitespace.

Debugging revealed that Pygments’ tokenization process reordered tokens for whitespace-prefixed text, causing misalignment in the color mapping logic.

Culprit

The root cause was a token reordering bug in Pygments’ lexer. When processing text segments with leading whitespace (e.g., \nTools:), Pygments reordered the tokens during tokenization. This broke the position-based color mapping, as the system assumed tokens would appear in the order of the original string. Specifically:

  • Whitespace was processed after semantic tokens instead of before, due to reordering.
  • The color mapping logic relied on sequential token positions, which became misaligned.
  • This resulted in incorrect color application, particularly for labels after code blocks.

Solution

A three-part fix was implemented to address the token reordering and color mapping issues:

1. Whitespace Stripping

Leading whitespace was stripped from the text segment before tokenization to prevent Pygments from reordering tokens. The original whitespace count was preserved for accurate position mapping.

2. Offset Compensation

Token positions were adjusted to account for the stripped whitespace, ensuring semantic tokens aligned with their original character positions in the text. This included:

  • Applying default colors to leading whitespace.
  • Mapping stripped token positions back to their original character indices.

3. Phantom Whitespace Prevention

Bounds checking was added to prevent Pygments from generating tokens for non-existent characters, avoiding artificial token reordering.

Verification Results All test cases now render correctly:

  • Labels like ‘Tools:’ after code blocks now display in turquoise (previously green).
  • Multi-word labels (e.g., ‘Why This Slaps:’) are fully colored in turquoise.
  • Debug output confirms proper token sequencing, ensuring semantic tokens (e.g., colon-prefixed labels) are processed in the correct order.

The fix eliminated the color misalignment issue entirely while maintaining accurate rendering for all markdown segments.