write-evaluator-prompt

write-evaluator-prompt

The evaluator's job is to operate the artifact and grade it against the agreed contract — skeptically, with calibration examples grounding its taste. This is the role most likely to drift; budget extra tuning time here.

When to invoke

• During instantiate-harness.
• During tune-harness, whenever a human spot-check disagrees with an evaluator verdict.
• After a re-assess-harness indicates the evaluator's cadence or scope should change.

Checklist

• Voice: skeptical, terse, looking for failure. Not "balanced". Not "constructive". Looking for what's wrong, by default.
• Verification is mandatory, not optional. The prompt cites the specific verification actions from the contract and requires the evaluator to run them before grading.
• Calibration examples are referenced by ID. When grading, the evaluator must say "this is closer to calibration/02 than calibration/04 because …". Anchors taste.
• Per-criterion pass/fail with citation. Every criterion gets a binary verdict + a quote/location from the artifact backing it. No "mostly passes".
• Forbidden hedge list. Explicit list of phrases the evaluator may not use: "looks fine", "appears to work", "should be good", "minor nit", "overall solid". Each one is a sycophancy tell.
• Granular over global. Criteria reference specific lines, paragraphs, rows, elements — never "the whole thing".
• Hard-threshold criteria explicit. If a hard-threshold criterion fails, the verdict is FAIL regardless of weighted score.
• Pivot recommendation in verdict. When the evaluator sees the same criterion failing repeatedly, it recommends a pivot in the verdict's next_action field.

Template slots

• {{DOMAIN}}
• {{VERIFICATION_ACTIONS}} — list from the contract
• {{CALIBRATION_PATH}} — relative path to .harness/calibration/
• {{RUBRIC_PATH}} — relative path to .harness/rubric.json
• {{FORBIDDEN_HEDGES}} — domain-tuned list (e.g., for outreach: "happy to chat", "would love to connect", "wanted to reach out")
• {{VERDICT_SCHEMA_PATH}} — points at runtime/schemas/verdict.schema.json

Calibration discipline

The single biggest lever for evaluator quality. Three to five examples, each with:

• The artifact itself (full content).
• A worked verdict — per-criterion pass/fail, with rationale.
• A label: excellent · slop · broken.

The evaluator's prompt references these by relative path. On every verdict, the evaluator must cite at least one calibration example as the comparison anchor.

Anti-patterns

• "Be fair" / "be balanced". Drifts sycophantic. Use "skeptical" or "looking for failure".
• No worked examples in the prompt. Without them the evaluator has no taste anchor.
• Grading from artifact description alone. Principle P5 forbids. The verification actions must be executed.
• Aggregate-only score, no per-criterion breakdown. Generator can't act on "you got a 7/10". It can act on "c2 failed because line 14 contradicts line 22".
• Evaluator suggests how to fix. No. Evaluator grades. Suggesting the fix muddies the next iteration. Generator owns the fix.