VSCode semantic highlighting differs from Pygments syntax coloring

Working with Python syntax highlighting to align Pygments-based tokenization with VSCode’s semantic highlighting, implemented a semantic analysis layer to correctly distinguish function parameters from loop variables and other assignment targets.

Issue

Python code highlighting using Pygments incorrectly rendered:

• Function parameters and loop variables in the same purple color (#dfbfff) instead of distinguishing them.
• For loop variables (e.g., for i in range(10)) as purple instead of gray (#cccac2).
• Built-in functions (e.g., enumerate(), print()) without consistent salmon coloring (#f28779).

The issue arose because Pygments relies solely on pattern-based syntax tokenization, while VSCode combines TextMate grammar (theme JSON) and semantic tokens (LSP) for context-aware classification.

Culprit

Key differences in highlighting approaches:

1. Pygments: Uses only syntax tokenization (e.g., all Token.Name for variables).
2. VSCode/Pylance: Combines TextMate grammar for basic syntax and semantic tokens for context-aware classification.

Pygments treated all variable-like tokens equally, while VSCode’s semantic layer applies:

• Parameter token type exclusively to formal function/method/lambda parameters.
• Variable token type for loop variables, comprehensions, and assignments.

Solution

Implemented a semantic analysis post-processor with four key components:

1. Semantic Token Color Mappings

Extended theme color mappings to include semantic-specific types:

• Parameters mapped to purple (#dfbfff).
• Function calls mapped to yellow.

2. Context-Aware Token Analysis

Created a semantic analysis layer that:

• Identified function parameters in definitions as semantic parameters.
• Distinguished function calls from other variable uses.
• Excluded specific cases like loop variables, comprehensions, and assignment targets.

3. Dual-Mode Color Resolution

Updated the color resolution logic to handle both:

• Semantic tokens (e.g., SEMANTIC_PARAMETER).
• Pygments tokens (e.g., Token.Keyword).

4. Integration Pipeline

Modified the tokenization pipeline to chain:

1. Pygments tokenization.
2. Semantic analysis.
3. Color mapping.

Verification Test Case:

• Loop variables and line numbers rendered in gray (correct).
• Built-in functions like print() rendered in salmon (correct).

Result: Achieved over 90% match with VSCode/Pylance for common Python patterns, with explicit handling of semantic distinctions between function parameters and loop variables.