harness

harness-lab

A Claude Code plugin for running controlled experiments — the way a lab runs them.

When you want to know whether prompt phrasing A beats B, whether the cheaper model is good enough, or whether your v3 rewrite actually improved anything, you need a comparison, not a one-off run. harness gives you a repeatable structure for that: it files the experiment, runs each variant in isolation, judges each one against criteria you set up front, and aggregates the results into a single report.

The model: lab + research groups

The main Claude Code session acts as the lab lead (PI). It never runs experiments itself — it clarifies the question, files the lab, and dispatches a research group per variant. Each group is a short pipeline of subagents:

Lab lead (main session)
│
├── clarify the question  →  file the lab (INTENT / hypothesis / STATUS)
│
├── variant v1 ── group ──►  setup ──► script ──► run ──► verdict
├── variant v2 ── group ──►  setup ──► script ──► run ──► verdict
├── variant vN ── group ──►  setup ──► script ──► run ──► verdict
│
└── aggregate verdicts  →  report  →  REPORT.html  →  conclusion + recommendation

Agent	Model	Does
harness:setup	haiku	scaffolds vN/artifact/ — mock data, inputs, samples
harness:script	sonnet	writes the replayable run.sh (the experiment's execution logic)
harness:run	haiku	runs the script, captures the transcript, writes a PASS/FAIL verdict
harness:report	haiku	aggregates everything into a dark, navigable REPORT.html

Why it's structured this way

• Design / execution split. The main session owns the design (intent, hypothesis, decision criteria, final report); the subagents own execution (scaffolding, scripts, runs, verdicts). Each side stays in its lane.
• Model ROI. sonnet — the expensive model — appears in exactly one wave, writing the script. Everything else (scaffolding, running, reporting) runs on haiku. You pay for reasoning where it matters.
• Research-group isolation. A variant's agents never read another variant's files. No cross-contamination, and each group fits comfortably in context.
• Verdicts before opinions. harness:run judges each variant against the hard criteria in the hypothesis and grounds the verdict in transcript numbers — not vibes. The lab lead reads verdicts and decides.

Install

claude plugin marketplace add LH151732/harness-lab
claude install harness@harness-lab

That's it. The skill auto-triggers on experiment-shaped requests; the four agents come bundled.

Usage

Just describe a comparison. The skill triggers on phrasing like:

• "compare v1 v2 v3"
• "test this prompt across Sonnet and Haiku"
• "isolate this function from the project and benchmark two implementations"
• "v2 didn't work, let's try v3"
• or invoke it explicitly with /harness

Two modes

The lab lead always asks which mode fits before doing anything:

• Mode A — parallel. You know the directions to compare. It signs off v1..vN at once and dispatches them in concurrent waves, then produces one report.
• Mode B — iterate. You don't know the next move yet. It runs v1, shows you the verdict, and you decide what v2 should change — repeating until you're satisfied.

What you get

Every experiment lives under /tmp/harness-lab/<slug>-<timestamp>/:

<slug>-<ts>/
├── INTENT.md           what you're trying to find out + mode
├── hypothesis.md       the claim, the variants, the hard PASS/FAIL criteria
├── STATUS.md           append-only execution trace (UTC timestamps)
├── REPORT.html         the aggregated, navigable report
└── v1/, v2/, .../
    ├── README.md       this variant's spec
    ├── artifact/       scaffolding + run.sh
    ├── runs/transcript.log
    └── verdict.md      PASS / FAIL / CONCERNS + key numbers

The final message back to you is deliberately terse: a conclusion (winner / failure / inconclusive, numbers first) and a recommendation (1-3 concrete next steps), plus the path to REPORT.html for the full archive.

Requirements

• Claude Code with plugin support.
• The experiments run as local shell scripts in /tmp, so whatever your experiment needs (an interpreter, an API key, a CLI) must be available in your environment. harness orchestrates; it doesn't sandbox.

Customizing

The four agents and six templates are plain Markdown / HTML under agents/ and skills/harness/templates/. Fork and edit them — the REPORT.html template, in particular, is built to be retargeted: it ships with generic run.sh / artifacts / transcript slots per variant, and the harness:report agent fills them from whatever your experiment produced.

License

MIT — see LICENSE.