paper-clustering

paper-clustering

Cluster a folder of academic PDFs using SPECTER2 embeddings + Leiden algorithm, with betweenness-based bridge paper identification and UMAP visualization.

Features

• Topic clustering: SPECTER2 (Allen AI's academic paper embedding) + Leiden community detection, with resolution sweep
• Bridge papers: Top-15 papers by betweenness centrality (cross-cluster connectors)
• Interactive visualization: UMAP scatter plot in HTML (Plotly)
• Markdown report: Cluster summaries + sample members + bridge analysis
• Optional ground-truth validation:
  • SQLite db with topic_category column → NMI/ARI metrics
  • Markdown cross-reference file with ⤷ markers → bridge overlap analysis
• Two clustering versions: with all papers; excluding reviews/meta-analyses (avoids "fake bridges" from review papers)

Installation

macOS / Linux

cd ~/.claude/plugins/
git clone <repo-url> paper-clustering
# OR copy from another machine

cd paper-clustering/scripts
bash setup.sh

Windows

cd $env:USERPROFILE\.claude\plugins\
git clone <repo-url> paper-clustering

cd paper-clustering\scripts
powershell -ExecutionPolicy Bypass -File setup.ps1

Prerequisites

• Python 3.9+
• pdftotext (poppler-utils):
  • macOS: brew install poppler
  • Ubuntu/Debian: apt install poppler-utils
  • Windows: choco install poppler or scoop install poppler

What gets installed

• venv at ~/.cache/paper-clustering/venv/ (macOS/Linux) or %LOCALAPPDATA%\paper-clustering\venv\ (Windows)
• ~600MB of Python packages (CPU-only PyTorch + SPECTER2 stack)
• SPECTER2 model (~440MB) is downloaded by HuggingFace on first run

The plugin directory itself stays small (no models or venvs inside it, so it's safe to put the plugin in GDrive-synced locations).

Usage

From Claude Code:

/cluster-papers <pdf-folder>
/cluster-papers <pdf-folder> --db <ground-truth.db>
/cluster-papers <pdf-folder> --cross-refs <map.md> --cross-symbol "→"

Options

Option	Description	Default
(positional)	PDF folder (required, recursive scan)	—
--db <path>	SQLite ground-truth db	auto-detected
--cross-refs <path>	Markdown file with cross-category markers	auto-detected
--cross-symbol <sym>	Symbol marking cross-category papers	⤷
--output <path>	Output directory	<folder-name>_clustering/
--force	Re-run all stages even if intermediate files exist	off

Auto-detection rules

SQLite ground truth: Plugin scans the PDF folder and its parent for .db/.sqlite/.sqlite3 files. Uses schema-based detection: a table is considered a candidate if its columns include keywords from all 3 categories:

• Key columns: citation_key, bibtex_key, key, paper_id
• Topic columns: topic, category, tag, subject, class
• Content columns: title, abstract

If multiple candidates, the user is asked to choose.

Markdown cross-references: Plugin scans for .md files in the PDF folder and its parent. A file is a candidate if it contains ≥5 lines with the cross-symbol (default ⤷).

If neither auto-detection nor manual override yields a ground-truth source, the pipeline still runs and produces the main analysis without validation.

Outputs

In <output>/:

File	Description
report.md	Main markdown report
cluster_map_all.html	UMAP scatter, all papers
cluster_map_empirical_only.html	UMAP scatter, excluding reviews+meta
bridges_all.csv	Top-15 bridges (all papers)
bridges_empirical_only.csv	Top-15 bridges (empirical only)
data/	Intermediate JSONs (re-run cache)

Known Limitations

1. First run downloads ~440MB SPECTER2 model (then cached at ~/.cache/huggingface/).
2. Abstract extraction noise: ~5% of papers may include "Article history" or download-banner noise in the extracted abstract (does not significantly affect SPECTER2 vectors).
3. SPECTER2 captures topic similarity, not study design: A paper's abstract mentioning "mediation" can put it near other mediation work even if it isn't a mediation study itself. Do not use this tool for research-design classification.
4. NMI/ARI are imperfect metrics: A low NMI may reflect genuine concept overlap in the field rather than algorithm error.
5. Review papers tend to be flagged as bridges: This is structural — they span topics by design. The "empirical_only" version excludes them.

How it works