eval-before-ship

Eval Before Ship

The Law

AN LLM FEATURE IS NOT READY UNTIL NUMBERS EXIST.
"It looked good when I tested it" is not an eval.
"I ran a few examples and it worked" is not an eval.
A named suite, a defined metric, a pass rate, a failure analysis,
and a baseline comparison IS an eval. All five. Not four.

When to Use

Trigger before any of these:

• Merging a PR that adds or modifies a prompt
• Deploying an LLM feature to any environment users can reach
• Switching models or providers on an existing feature
• Changing retrieval logic, rerankers, or chunk strategy in a RAG pipeline
• Updating few-shot examples or system prompt structure

When NOT to Use

• Exploratory prototypes that will not reach users (note it: "eval required before production")
• Config-only changes that cannot affect model output (timeouts, logging, env vars)

What Counts as an Eval

An eval must have all five components:

Component	What It Means	What Does NOT Count
Named test suite	File with labelled examples in evals/	"I tested it manually"
Defined metric	Accuracy %, faithfulness score, task pass rate	"It seemed accurate"
Pass threshold	Explicit minimum (e.g., ≥ 85%)	No threshold = no standard
Failure analysis	≥ 5 failures examined and categorised	"There were a few errors"
Baseline comparison	This version vs. previous version or control	First release exempt; all subsequent require it

The Process

Step 1 — Define the Eval Before Writing the Prompt

Answer these before touching the prompt:

• What does a correct output look like for this task?
• What metric will you use to measure it?
• What is the minimum acceptable score for production?

If you cannot answer these before building, the task is not well enough specified to build.

Step 2 — Build a Representative Test Set

evals/
  <feature-name>/
    test-set.jsonl     ← labelled examples, one JSON object per line
    harness.py         ← eval runner
    results-<date>.md  ← documented results

test-set.jsonl format:

{"id": "001", "input": "...", "expected": "...", "tags": ["edge-case"]}

Label ground truth before running any model. Seeing outputs first contaminates labels.

Minimum 50 examples; 200+ for features used at volume. Include: edge cases, short inputs, long inputs, ambiguous inputs, adversarial rephrasing.

Step 3 — Run and Record

python evals/<feature-name>/harness.py --model <model-id> --seed 42

Record: pass rate (X/N = Y%), latency p50/p95, failure category breakdown.

Step 4 — Analyse Failures

Examine every failing case. Assign each to one category:

• Format violation — wrong output structure
• Factual error — wrong answer when correct answer was in context
• Hallucination — claim not grounded in any provided context
• Refusal — model declined a legitimate request
• Off-topic — response doesn't address the input

The most common failure category is your next fix.

Step 5 — Document

Suite: evals/email-classifier/test-set.jsonl (200 examples)
Model: claude-sonnet-4-6, temperature: 0.0, seed: 42

Pass rate: 178/200 = 89% (threshold: ≥ 85% ✓)

Failure breakdown:
  - Format violation: 12 (long emails > 2000 tokens)
  - Hallucination: 10 (invented labels not in schema)

Baseline: v1 (82%) → v2 (89%), delta: +7pp ✓
Results: evals/email-classifier/results-2026-06-07.md

Rationalization Red Flags

These thoughts mean you have NOT completed an eval — stop and build one:

• "I ran it manually and it looked good" — you tested a sample you chose; an eval tests a sample that represents the distribution
• "We'll add evals after we ship" — the regression will be a 2-week cycle: 3 days to notice, 2 days to diagnose with no baseline, 1 week to ship the fix; the eval would have taken a day
• "The model always does this correctly" — based on how many labelled examples?
• "It's only an internal tool" — internal tools fail the same ways, just with fewer headlines
• "This is a small prompt change" — adding a single instruction shifted one team's pass rate from 89% to 72%; "small" has no meaning in output distribution terms
• "We don't have time to build a test set" — you have time to diagnose production failures?

Completion Statement Format

When eval-before-ship is satisfied, state it like this:

Eval complete.
Suite: evals/<feature>/test-set.jsonl — N examples
Model: <model-id>, temperature: X, seed: Y
Pass rate: A/N = B% (threshold: ≥ C% ✓)
Top failure mode: <category> (N cases — <root cause>)
Failure analysis: evals/<feature>/failure-analysis-<date>.md ✓
Baseline: <previous> = X% → this version = Y%, delta: +Zpp ✓
  [First release: baseline field may be marked "N/A — initial version"]
Results stored: evals/<feature>/results-<date>.md ✓

The pass rate, failure analysis, and baseline comparison are not optional. On the first release, baseline may be marked "N/A — initial version"; for all subsequent releases it is required.

Why This Matters

LLM outputs are stochastic. A prompt that works on the examples you tried does not predict performance on the next 1000 inputs. Eval suites catch regressions before users do — and they make it possible to improve without guessing.