scrape-conference

Scrape Conference Author Info

Produce ./conference-authors-data/<venue>_<year>/authors.csv from a venue name or listing URL. Zero setup: stdlib scripts + your own reading.

0. Locate the plugin scripts

The scripts/ and templates/ dirs live at the plugin root, exposed as ${CLAUDE_PLUGIN_ROOT}. Always run python from there so from scripts import ... resolves, e.g.: cd "${CLAUDE_PLUGIN_ROOT}" && python3 -c "from scripts import fetch, enumerate_listing; ...". Use absolute paths for the run folder, which lives in the user's working dir (a different place): <user-cwd>/conference-authors-data/<venue>_<year>/ with a cache/ subdir. Pass that absolute path as cache_dir=/output paths.

1. Identify the venue

• Match the URL host against templates/*.json (match list). Hit → use it.
• Miss → read references/venues.md "Detect a new venue" and derive a template: fetch the listing, look at a SAMPLE of anchors (scripts/html_extract.extract_anchors), find the regex that selects paper links, save a new templates/<host>.json.

2. Enumerate papers

fetch.get(listing_url) → enumerate_listing.parse_listing(html, page_url=listing_url, template). For paginated listings, repeat per page. Write all rows to papers.json. For venues where the detail page holds the PDF link, fetch each detail page and parse_detail to fill pdf_url/doi (cache makes this resumable).

3. Free enrichment first → invoke the enrich-authors skill

Run it over all papers. Collect authors; mark gaps.

4. Fill gaps with PDFs (only flagged papers)

• Count the gaps. If > 50, tell the user "~N papers need PDF reading, proceed?" and wait. (This is the only step that costs real time/tokens.)
• For arxiv-style venues with no direct PDF, use arxiv_resolve (title→pdf).
• Download each gap PDF (download_pdf.download) → invoke extract-paper-authors → merge affiliations into papers.json.

5. Export

Flatten papers.json authors → export_csv.write_authors(rows, '<run>/authors.csv'). Write summary.md: paper count, author count, % with affiliation/ORCID/career, metadata-vs-PDF split, and any fetch_failed/pdf_failed.

Rules

• Resumable: never re-fetch a cached URL; honor per-paper status in papers.json.
• Never fabricate data; unknown fields stay blank.
• Proxy is opt-in via env only; never write secrets to disk or git.