paper-clustering

Paper Clustering — Topic Map for a PDF Folder

Purpose

Generate a topic-level knowledge map for any folder of academic PDFs. Produces:

• Cluster structure (Leiden algorithm on SPECTER2 embeddings)
• Top-15 bridge papers (betweenness centrality)
• Interactive UMAP scatter plot (HTML)
• Markdown report (with optional NMI / ARI validation if ground truth provided)

When to Use

Invoke when the user asks to:

• Build a knowledge map for a paper folder
• Find cross-cutting / bridge papers in a literature collection
• Compare an algorithmic clustering vs the user's existing manual category scheme
• Pre-write literature-review structure exploration

Routing

Invoke the slash command /cluster-papers <folder> [options] defined in this plugin's commands/cluster-papers.md. The command file describes the full pipeline orchestration including when to use Haiku subagents for paper-type tagging and cluster naming.

When NOT to Use

• The user wants research-design classification (mediation vs moderation vs experimental — SPECTER2 cannot distinguish these reliably; Anita should warn the user).
• The folder has < 30 papers (clustering is unreliable below this threshold).
• The user wants single-paper analysis, abstract writing, or APA formatting.

Known Limitations

1. SPECTER2 captures topic similarity, not study design. A paper that mentions "mediation" in its abstract may be misclassified as a mediation study even if it isn't (Elena POC lesson).
2. Review papers tend to score high on betweenness ("fake bridges") because they cover multiple topics by design. The pipeline produces a separate "empirical-only" version that excludes reviews + meta-analyses.
3. The cluster labels generated by Haiku are approximate. The main user should verify them against cluster member lists in the report.