re-assess-harness

re-assess-harness

Every harness component encodes an assumption about what the current model can't do. Model releases close some of those gaps. Run this skill on every project that has a .harness/ whenever a model lands.

Rule of thumb: strip first, add later. The path from Opus 4.5 to 4.6 dropped sprint decomposition AND per-sprint evaluation. The path forward looks similar — fewer scaffolding components, not more.

When to invoke

• A new model has been released and you intend to use it.
• A provider routing or pricing change makes a previous role assignment uneconomical.
• Output quality has improved naturally and you suspect the harness is now over-engineered.

Strip checklist

For each component, ask: "Does the new model still need this?" If no, remove. If unsure, A/B test.

• Session strategy — reset vs continuous. Did context anxiety improve? Try continuous if you were resetting.
• Per-iteration evaluation cadence. Did the model get better at sustained coherence? Try end-of-run evaluation only.
• Sprint / chunk decomposition. Did the model start handling longer tasks coherently? Drop the chunking.
• Generator's self-assessment scaffolding. Did the model improve at honest self-critique? Lighten the scripted handoff.
• Pivot threshold. Did the model get better at hill-climbing? Raise the threshold (fewer forced restarts).
• Number of calibration examples. Did the model's taste improve? You might need fewer anchors.
• Role assignment per model. If a smaller model now matches a previous large-model performance for a role, swap it in.

Add checklist (smaller — only after stripping)

• New tool support unlocked? Multi-modal verification, longer context retrieval, programmatic tool use — any of these might enable a verification action you couldn't do before.
• Capabilities that previously required a separate role? If the planner is now strong enough to also negotiate contracts, you could merge — though usually you shouldn't (principle P1 still holds).

Procedure

1. Pick a representative project with the most run history.
2. Note current harness_version in config.json.
3. For each strip-checklist item, run an A/B: same prompt, harness with vs without the component. Compare verdicts and cost.
4. Strip what passed the A/B. Bump harness_version. Document each stripped component in progress.jsonl with the A/B evidence.
5. Propagate changes to other projects (after their owners review).
6. Commit the new version with a CHANGELOG entry listing what was stripped, what was added, and the cost delta.

What good looks like

A re-assessment that:

• Cites specific A/B numbers (cost, iteration count, verdict scores).
• Removes at least one component (if the model genuinely improved).
• Bumps harness_version cleanly.
• Has a rollback path — the previous config is recoverable from git.

Anti-patterns

• Skipping re-assessment because "if it ain't broke don't fix it". The cost of unnecessary scaffolding is real — tokens, latency, and brittleness. Strip first, then preserve what's load-bearing.
• Adding components without A/B. You'll add cost without lift.
• Re-assessing on a project with no run history. No baseline to compare against. Pick one with at least N runs.
• Stripping calibration examples. Almost never the right call. Models improve at execution faster than they improve at taste anchoring.