safety-evaluation

Goal

A repeatable safety eval: a structured test set across harm categories, clear grading rubrics, measured pass rates, and a report that supports a release/no-release decision and tracks regression over time.

What to evaluate (cover both failure directions)

• Under-refusal (unsafe) — does it produce disallowed/harmful content across the harm categories from harm-modeling? (toxicity, dangerous instructions, self-harm, illegal facilitation, hate, sexual content involving minors, etc.)
• Over-refusal (unhelpful) — does it wrongly refuse benign requests? Safety that destroys utility is its own failure; measure both.
• Robustness — does safety hold under paraphrase, role-play, multilingual, encoded, and multi-turn pressure? (Deeper adversarial work: safety-red-team.)
• Groundedness / truthfulness — for factual/high-stakes tasks, are answers correct and properly hedged/cited? (RAG groundedness: see rag-security.)
• Bias — disparate quality/refusal across groups (full analysis: bias-fairness-assessment).

Steps

1. Derive harm categories and high-stakes scenarios from harm-modeling.
2. Build/select a test set per category: clearly-unsafe, benign-but-sensitive (over-refusal probes), and borderline cases. Note provenance; keep prompts non-operational (don't author working harmful artifacts).
3. Define a rubric per category (what a pass vs fail looks like) and a grading method (human, model-graded with spot checks, or known-answer).
4. Run, score, and compute pass rates with thresholds per category.
5. Report results, failures with examples, trends vs prior runs, and gaps.

Output

A safety eval report: per-category pass rate vs threshold, notable failures (redacted), over- vs under-refusal balance, and prioritized fixes. Offer to wire it into CI via ai-safety-engineer. Use security-diagramming:infographic for a scorecard and security-reporting for the writeup.

Notes

Measure both safety and helpfulness — a model that refuses everything scores "safe" but is useless. Keep the eval set versioned so results are comparable across releases. Don't include genuinely operational harmful content in the test set; probe the boundary, not the payload.