Document similarity visualization shows meaningless semantic connections

Working with D3.js v7 and Python’s standard library to replace document similarity visualization (using Sentence Transformers embeddings) with a framework co-occurrence network. The solution processes 952 documents to extract 44 frameworks and 627 meaningful co-occurrence relationships, visualized as an interactive force-directed graph in Astro.

Issue

The document similarity visualization produced meaningless connections between logs based on semantic similarity scores (cosine similarity > 0.65) from full-text embeddings. Common infrastructure keywords (Docker, K3s, Redis) created false positives where unrelated problems appeared similar:

• “PostgreSQL connection timeout” showed 67% similarity with “Redis connection timeout”
• Semantic embeddings treated titles, problems, solutions, and code blocks equally
• No architectural insights emerged from the 2D UMAP projection

Culprit

The root cause was using 384-dimensional semantic embeddings (all-MiniLM-L6-v2) that:

1. Lacked structural awareness - Embedded entire documents without distinguishing sections
2. Captured surface similarity - Common terms (Python, Docker) dominated similarity scores
3. Used arbitrary thresholds - 0.65 cosine similarity had no logical interpretation
4. Required ML dependencies - Added complexity with numpy/sentence-transformers/umap

The visualization showed “documents are similar” rather than “technologies work together”.

Solution

Framework Extraction (Python)

Replaced embeddings with co-occurrence counting using regex-based framework extraction:

# Extract frameworks per document
doc_frameworks = {
    "log1.md": {"Python", "FastAPI", "PostgreSQL"},
    # ...
}

# Build co-occurrence matrix
for frameworks in doc_frameworks.values():
    for fw1, fw2 in combinations(frameworks, 2):
        co_occurrence[(fw1, fw2)] += 1

Force-Directed Visualization (D3.js)

Created interactive network with:

• Nodes: Frameworks sized by usage frequency (√count * 3 + 5)
• Edges: Co-occurrence strength (√weight * 0.8 thickness)
• Layout: Force simulation with charge (-300), link distance (30), and collision detection
• Interactions: Hover tooltips showing top connections and usage stats

Key Implementation Files

1. generate_similarity_data.py (new):

   • Processes 952 documents in ~2s
   • SHA256 deduplication removes 821 duplicates
   • Outputs similarity-data.json with 44 nodes/627 edges

2. SimilarityMap.astro (D3.js):

   // Force simulation setup
   const simulation = d3.forceSimulation(nodes)
     .force("charge", d3.forceManyBody().strength(-300))
     .force("link", d3.forceLink(edges).distance(30))
     .force("collide", d3.forceCollide(10));

Performance Metrics

• Processing: 952 docs → 44 frameworks in 2.1s (Python stdlib only)
• Visualization: 60fps rendering with 627 edges (D3.js v7)
• Data reduction: 12,264 co-occurrences → 627 meaningful edges (≥2 occurrences)

The solution reveals architecture patterns like:

• LLM stack: vLLM ←→ FastMCP ←→ Python ←→ K3s
• Backend stack: Python ←→ FastAPI ←→ PostgreSQL ←→ Redis