Pygments lexer reorders tokens with leading whitespace in Markdown

The Pygments lexer in Python was used to process Markdown content with syntax highlighting, specifically handling colon-prefixed labels (e.g., “Tools:”, “Resources:”) that required custom semantic coloring. However, a token reordering issue caused inconsistent styling when these labels appeared after code blocks.

Issue

Markdown labels before colons (e.g., “Tools:”, “Resources:”) failed to render in turquoise when they appeared after code blocks, despite rendering correctly when they appeared before code blocks. This inconsistency caused the labels to appear in the default text color instead of the designated turquoise, creating a visual bug.

Culprit

The Pygments lexer reordered tokens when processing text segments containing leading whitespace, which disrupted the positional assumptions in the semantic analysis logic. When text segments contained leading newlines (e.g., ' Tools:'), Pygments returned tokens in reverse order:

• Input: ' Tools:'
• Expected: [Token(' '), Token('Tools:')]
• Actual: [Token('Tools:'), Token(' ')]

This reordering caused the semantic analysis function to miscalculate the positions of colons, leading to incorrect classification of label characters as SEMANTIC_COLON_PREFIX. The function assumed sequential token order, but the reordering disrupted character-to-token mapping. The root cause was the presence of separator newlines from segment boundaries, triggering the bug specifically for text segments after code blocks.

Solution

Whitespace Stripping with Offset Compensation

The tokenization logic was modified to strip leading whitespace before processing with Pygments while preserving the original character positions:

1. Strip Leading Whitespace: Leading whitespace was removed before tokenization to ensure consistent token ordering.

   stripped_text = seg_visible_text.lstrip()
   leading_ws_count = len(seg_visible_text) - len(stripped_text)
   tokens = tokenize_code(stripped_text, 'markdown')

2. Offset Mapping: Character-to-token alignment was maintained using offset mapping to ensure correct styling:

   for local_idx, (char_idx, char) in enumerate(chars):
       if local_idx < leading_ws_count:
           char_colors[char_idx] = markdown_text_color  # Default color for whitespace
       elif local_idx - leading_ws_count < len(tokens):
           _, token_type = tokens[local_idx - leading_ws_count]
           char_colors[char_idx] = get_token_color(token_type, token_colors, markdown_text_color)

Phantom Whitespace Fix

A secondary bug in Pygments, where trailing whitespace tokens caused index errors, was addressed by adding boundary checks:

if char_pos < len(code):  # Boundary check
    char_to_token.append((char_pos, token_type))

Result: All colon-prefixed labels now render consistently in turquoise, regardless of their position relative to code blocks, with no regression in whitespace handling. The solution maintains compatibility with existing Markdown processing while fixing the token reordering issue.