protocol-kri-extractor

Protocol KRI Extractor

Extracts monitoring rules (KRIs) from any clinical trial protocol PDF. Protocol-agnostic: no hardcoded section names, visit labels, or therapeutic areas.

Output schema

Every extracted KRI matches this structure:

{
  "kri_id": "SOA-V1-001",
  "kri_name": "V1- IMP administration",
  "description": "What this KRI monitors and why it matters",
  "category_id": "SOA",
  "category_label": "Schedule of Activities",
  "rule_for_llm": "V1- Verify that [specific actionable check with exact values]",
  "protocol_reference": "Section X.X, Page N: \"verbatim quote ≤30 words\"",
  "additional_footnotes": "Footnote N: verbatim text — or null"
}

Five categories (universal across all trials — from ICH GCP, not protocol-specific):

• SOA — Schedule of Activities
• ELIG — Eligibility (inclusion + exclusion)
• SAF — Safety & Toxicity
• END — Endpoints & Statistics
• OPS — Operations & Compliance

---

How to run

Step 0 — Setup (first time only)

pip install pdfplumber pymupdf camelot-py[cv] opencv-python-headless openpyxl --break-system-packages -q

Input required

• Protocol PDF file path
• Output directory (will be created)
• Optional: golden set JSON path (for step 4B comparison)

Full pipeline command

python /path/to/scripts/run_pipeline.py \
  --pdf /path/to/protocol.pdf \
  --out /path/to/output/ \
  [--golden /path/to/golden_set.json]

Or run steps individually — see Step-by-step below.

---

Step-by-step

Read references/steps.md for the detailed prompt templates and logic for each step.

Phase 1 — Discover

Step 1A — Manifest: Read cover pages + TOC. Map every section to SOA/ELIG/SAF/END/OPS.

Step 1B-Camelot — Table Extraction (PRIMARY): Use Camelot (lattice mode) via scripts/camelot_table_extractor.py to extract the SoA table into soa_table.csv and soa_table.json. This is the PRIMARY source of truth for the SoA procedure × visit grid. Camelot reads table line geometry from the PDF and gives ~99% structural accuracy — deterministic, reproducible, and not affected by LLM variance. Handles multi-page tables automatically. The CSV and JSON outputs become the canonical SoA data for all downstream steps.

Step 1B-Vision — Vision Fallback (SECONDARY): For cells where Camelot detects the lattice structure but the cell content contains footnote superscripts (e.g., "X¹⁰", "X¹³·¹⁴") that Camelot may read as empty, use Claude Vision at 450 DPI as fallback. Also use multi-pass vision (full + left 55% crop + right 55% crop at 450 DPI) with majority voting for wide tables (>12 columns) to recover any cells Camelot missed. Save conflicts in multipass_conflicts.json.

Step 1B-ColDetect — Column Boundary Detection: After extraction, run scripts/vision_table_extractor.py column detection to verify column boundaries match the Camelot-extracted structure. Flag any discrepancies.

Step 1B — Ontology: Build the SoA ontology from the Camelot-extracted table data (verified by vision fallback where needed). Run Footnote Cross-Validation as a safety net.

Phase 2 — Extract (6-Step SOA Process)

The SOA extraction follows a strict 6-step process:

1. Visit mapping: Pull all visits and their timing from the Camelot CSV. Establish naming conventions (V0, V1, V2... EDC_EOS). From this point forward, use ONLY these canonical names.
2. Table verification: Compare the Camelot CSV against the PDF image for verification. Flag any discrepancies. Save soa_table.csv and soa_table.json as the tracking artifacts.
3. Check-in KRIs: For each visit, create ONE check-in KRI verifying the subject attended within the protocol-specified timing window.
4. Procedure KRIs: For each visit, create ONE KRI listing ALL procedures required at that visit (in the format: V1 - procedure name). Also create one KRI per procedure × visit cell.
5. Footnote enrichment: After the table-based KRIs are complete, read all footnotes from the protocol and enrich each KRI with relevant footnote details. Also create cross-visit rule KRIs for protocol-wide rules (fasting requirements, dosing windows, 10-day lipid rule, IP administration sequence, missed visit escalation, EDC retention, safety follow-up periods, etc.).
6. Self-verification: Cross-check that every X cell in the Camelot CSV has a corresponding KRI. Report: N/N cells covered = 100%.

For non-SOA categories, extraction uses one LLM call per category producing raw_{CAT}.json:

• ELIG: one KRI per criterion/sub-criterion
• SAF: every reporting timeline, stopping rule, emergency protocol
• END: every endpoint, analysis set, statistical rule
• OPS: IMP handling, blinding, records, compliance

Phase 3 — Validate (3 independent passes + heuristics)

Step 3A — Completeness: Every Camelot CSV cell with "X" must have a KRI. Step 3A+ — Clinical Heuristics: 10 protocol-agnostic heuristics:

• H1-H7: Drug accountability at EOS, pre-questionnaire procedures, ICF at first contact, AE from first dose, concomitant meds at all visits, screening symmetry, vital signs dual measurement.
• H8 — Edge-Column Footnote Reconciliation: For procedures in the first/last column, cross-check against footnotes. If a footnote says "at EDC/EOS" but ontology shows V0 (or vice versa), correct it.
• H9 — Epoch Boundary Plausibility Check: If a procedure has marks in treatment epoch (V5-V20) but also a single isolated mark in screening (V0/V1), flag it for review. Single marks in a distant epoch are suspicious.
• H10 — Contiguous Coverage Gap Detection: For procedures with 5+ visits, check for suspicious holes (>15% gap ratio). Re-verify gaps against the Camelot CSV. Use detect_contiguous_gaps() from scripts/vision_table_extractor.py. Step 3B — Accuracy: 20 random KRIs re-verified against source pages. Threshold ≥85%. Step 3C — Consistency: Same procedure across visits must have consistent details.

Phase 4 — Assemble + Compare

Step 4A — Assembly: Run scripts/step4a_assemble.py to merge all category files → extracted_kris.json + Extracted_KRIs.xlsx (Excel workbook with 5 sheets: SOA, ELIGIBILITY, SAF&TOX, END&STAT, OPE&COM — blue headers, frozen row 1, wrapped text). Step 4B — Golden Set Prompt: After assembly, ask the user if a golden set is available. Step 4C — Golden Set Comparison: Category-by-category LLM comparison with protocol evidence for every difference. Step 4D — Comparison Verification: After Step 4C, run a reconciliation pass. For each "MISSING" verdict, search the extracted KRIs for any KRI whose rule_for_llm contains the same key terms (procedure name + visit prefix). If found, reclassify as EQUIVALENT or SUBSET (false negative in comparison). Also for each "DIVERGENT" verdict, re-read both rules and confirm they truly check different clinical requirements — not just differently worded versions of the same check. Output a comparison_verified.json with corrections.

---

Output artifacts

Each pipeline run produces these files in the output directory:

File	Description	Source
manifest.json	Protocol metadata and section map	Step 1A
soa_table.csv	Canonical SoA matrix (Camelot)	Step 1B-Camelot
soa_table.json	SoA matrix + visit-procedure mapping (Camelot)	Step 1B-Camelot
ontology.json	SoA ontology (visits, procedures, footnotes)	Step 1B
raw_SOA.json	SOA KRIs (check-in + procedure + cross-visit)	Step 2
raw_ELIG.json	Eligibility KRIs	Step 2
raw_SAF.json	Safety KRIs	Step 2
raw_END.json	Endpoint KRIs	Step 2
raw_OPS.json	Operations KRIs	Step 2
extracted_kris.json	All KRIs assembled	Step 4A
Extracted_KRIs.xlsx	Excel workbook (5 sheets)	Step 4A
gaps_report.json	Completeness + heuristic results	Step 3A/3A+
accuracy_report.json	20-KRI accuracy sample	Step 3B
consistency_report.json	Procedure family consistency	Step 3C
comparison_report.json	Golden set comparison (if provided)	Step 4C

---

Quality rules (apply to every KRI)

1. Faithfulness: Use exact drug names, doses, thresholds, timing windows from the protocol. Never generalize ("emergency treatment" → name the drugs).
2. Data source: Washout KRIs must say "by checking medication logs and visit timestamps".
3. Lab panels: Include all analytes from the protocol footnote — never just "biochemistry panel".
4. Vitals position: Use the exact position wording the protocol uses (e.g. "supine position").
5. Visit prefix: Every SOA rule_for_llm starts with visit code: V1-, S2-, All visits-.
6. Analysis sets: Use the protocol's exact definition — ITT ≠ mITT ≠ FAS.
7. No hallucination: Every KRI must cite a real section + page. If unsure, omit.
8. Measurement detail: Physical assessments must include units, positioning, and preparation when the protocol specifies them (e.g. "weight in kilograms, shoes removed").
9. Visit window check-in KRI (MANDATORY): Every single visit in the SoA table MUST have a dedicated "check-in / within-window" KRI as its FIRST KRI. This includes all screening visits, treatment visits, follow-up visits, and unscheduled visits. The KRI verifies the visit occurred AND fell within the protocol-specified timing window (day reference ± tolerance). Never skip any visit.
10. Table is truth: The SoA table (via Camelot CSV) is the single source of truth for which procedures occur at which visits. Footnotes can ADD context (enrichment) but cannot OVERRIDE the table's X marks. If a footnote says "annually" but the table shows X at V10/V13/V16/V19, use the table's visits, not the footnote's interpretation.

---

Comparing against a golden set

After Step 4A assembly completes, always ask the user:

"Do you have a golden set to compare against? You can provide a file path or upload it."

If the user provides a golden set, run Step 4C (see references/steps.md for full details).

How comparison works

Phase 1 — Matching: For each category (SOA, ELIG, SAF, END, OPS) separately, match extracted KRIs to golden KRIs using semantic similarity (not just ID matching). Handle 1:many and many:1 splits.

Phase 2 — Two-tier semantic judging: LLM evaluates each pair using two criteria:

Tier 1 — Semantic coverage (lenient): Does the extracted KRI check the SAME clinical requirement? If yes, differences in phrasing, sentence structure, verbosity, or explanatory context are IRRELEVANT. Focus on WHAT is verified, not HOW the sentence reads.

Tier 2 — Factual precision (strict): When either rule contains specific protocol facts — numeric thresholds, visit windows (±days), drug names & doses, named instruments/scales, washout durations, specific methods (e.g. "ultrasound-guided") — those facts MUST be accurate and present. Wording around them doesn't matter, but the facts themselves do.

Verdicts:

• EQUIVALENT — same clinical check (Tier 1 pass), factual details consistent or both absent (Tier 2 pass). Phrasing/wording differences are irrelevant.
• SUBSET — same check but extracted OMITS a specific factual detail that the golden includes.
• SUPERSET — same check but extracted ADDS factual detail from the protocol beyond what golden includes.
• DIVERGENT — different clinical requirement (Tier 1 fail), OR factual details contradict each other.

Phase 3 — Protocol evidence: For every non-EQUIVALENT pair, read the cited protocol pages and show a 3-column comparison.

Phase 4 — Coverage gaps: Identify golden KRIs with no extracted match and extracted KRIs with no golden match.

Comparison report structure

The final comparison_report.json contains:

• Score: (EQUIVALENT + SUPERSET + 0.5*SUBSET) / total_golden * 100

  • =80 → PASS | 60-79 → ITERATE | <60 → REWORK

• Per-category breakdown with counts per verdict
• Differences table: every non-EQUIVALENT pair with extracted rule, golden rule, protocol evidence
• Missing from extracted: golden KRIs with no match
• Extra in extracted: extracted KRIs with no golden match

---

Reference files

• references/steps.md — detailed LLM prompt templates for each step
• references/kri_examples.md — annotated KRI examples per category
• scripts/camelot_table_extractor.py — Camelot-based table extraction (PRIMARY for SoA)
• scripts/vision_table_extractor.py — Vision-based extraction + multi-pass + Heuristic 10
• scripts/step4a_assemble.py — Assembly + Excel generation (no LLM needed)