Resolved

Pygments MarkdownLexer fails to highlight inline code in Japanese text

Python
Markdown
Syntax Highlighting
Pygments
Localization
Text Processing
Lexer
Unicode

Inline code in backticks renders as plain text in Japanese Markdown segments due to Pygments' whitespace dependency | Solution implements manual backtick detection and full-width colon support

Issue

Inline code segments wrapped in backticks (e.g., `@mcp.tool`) failed to render with syntax highlighting in Japanese Markdown content. The issue specifically occurred when backticks were adjacent to Japanese characters without surrounding whitespace, causing inline code to appear as plain text instead of highlighted code. This resulted in visual inconsistencies where code in Japanese text rendered identically to regular text, while the same code in English text rendered correctly.

Culprit

The root cause was a dependency in the lexer used for Markdown parsing:

  1. Whitespace Requirement: The lexer required whitespace around backticks for inline code detection, which is absent in Japanese text due to its lack of inter-word spaces.
  2. Character-Specific Detection: The system only recognized ASCII colons (:), failing to detect full-width Japanese colons (：), which are commonly used in Japanese Markdown.

Solution

To address these issues, the following adjustments were made:

  1. Character-By-Character Scanning: Implemented a manual scanning mechanism to identify backtick pairs regardless of surrounding characters or whitespace.
  2. Token Splitting with Boundaries: Modified token processing to split text at backtick boundaries, applying syntax highlighting to the enclosed content while preserving the original formatting for non-code segments.
  3. Full-Width Character Support: Extended character detection to include full-width variants of punctuation marks, ensuring compatibility with Japanese Markdown conventions.

The updated implementation ensures inline code in Japanese Markdown is correctly highlighted, resolving the visual inconsistencies and improving readability for mixed-language documentation.