hallucite

hallucite

Finds fabricated ("hallucinated") references in academic paper PDF files. Each reference is checked against academic databases (offline DBLP, plus CrossRef, arXiv, Semantic Scholar, and other open bibliographic databases); references that no database can confirm are escalated to an interactive LLM triage step, which writes a report for human review.

Three stages: extract and verify use no LLM (verification queries the online databases unless --offline restricts it to the local ones); triage is the only step that uses an LLM, which can be a cloud or a local model. See PLAN.md for the design and architecture.

One repo serves as the runnable project (the mise tasks below) and one shared plugin tree for Claude Code and Codex CLI. The Claude metadata lives under .claude-plugin/; the Codex metadata lives under .codex-plugin/ and .agents/plugins/marketplace.json. The bundled skill in skills/hallucite/ drives the same scripts for both tools.

Setup (once)

Run from this directory. Requires mise and pdftotext from poppler, which the extractor shells out to (e.g. brew install poppler).

mise install          # provision Python 3.12 + uv (auto-venv)
mise run install      # uv pip install -r requirements.txt  (hallucinator)
mise run install-cli  # download the hallucinator CLI binary into .bin/ (checksum-verified)
mise run build-dblp   # build the offline DBLP database at ~/hallucite/dblp.db (~4.6 GB, ~20-30 min)

The offline DBLP database lives at ~/hallucite/dblp.db, outside this repo, which keeps the 2.5 GB file out of git. Set $HALLUCITE_DBLP to store it somewhere else.

Run the audit (Stages 1+2, no LLM)

mise run audit -- <pdf-file-or-dir>            # required: a PDF file, or a directory of PDF files
mise run audit -- <pdf-file-or-dir> [options]  # everything after the target is forwarded as-is

Writes out/<paper_id>.json (every reference plus per-database verification) and out/summary.json (status counts plus the DBLP build date). Options: --dblp PATH, --out DIR, --mailto EMAIL, --offline (no network; offline DBLP plus hallucinator's built-in Standards matcher, and a missing DBLP file disables DBLP rather than falling back to dblp.org), --disable-dbs LIST (comma-separated), --no-verify. The DBLP path defaults to $HALLUCITE_DBLP (else ~/hallucite/dblp.db) and the output dir to out. A reference needs triage when its db_verification.status is anything other than verified (not_found, mismatch, or unparsed). Re-running into the same --out is idempotent (triage_verdicts.json accumulates by paper_id:number).

Triage the residue (Stage 3, an interactive LLM agent)

mise exec -- python skills/hallucite/scripts/triage.py worklist --out out          # add --pending to skip done
mise exec -- python skills/hallucite/scripts/triage.py worklist --paper <id> --out out  # one paper's slice
mise exec -- python skills/hallucite/scripts/triage.py status --out out             # per-paper done / pending

Stage 3 reads the per-paper JSON the audit has already written, so it can run on finished papers while the audit is still processing the rest — no need to wait for the whole corpus. Verdicts accumulate, and worklist --pending surfaces only references not yet recorded. To fan triage out, hand each worker its own worklist --paper <id> slice (exact id match) instead of the shared worklist, so a worker can't grab the wrong paper (e.g. paper6 vs paper66); record locks the verdicts file, so concurrent workers don't lose each other's verdicts.

Hand the worklist to an interactive LLM agent such as Claude Code or Codex CLI ("triage the unverified references in out"), or use the installed plugin (below). The agent classifies each reference title-first: a partial-match is a real, locatable publication with the cited title but a slipped metadata field (a citation error); a title that matches no real publication is likely-hallucinated, not a partial-match — even when a different paper by the same authors exists. Categories: real-published, real-grey-literature, real-preprint-or-unpublished, partial-match, likely-hallucinated, unclear. The agent records verdicts with structured fabrication signals, then assembles the reports:

mise exec -- python skills/hallucite/scripts/triage.py record <paper_id> <number> <category> "<finding>" \
  --signals '{"title_match":"no","authors_match":"yes","venue_match":"no","doi_status":"none"}' --out out
mise exec -- python skills/hallucite/scripts/triage.py report --out out