From eval-guide
Generates concrete test cases grouped by quality signal from eval plans or agent descriptions. Outputs CSVs (Question + Expected response) and docx reports for Copilot Studio evaluations.
How this skill is triggered — by the user, by Claude, or both
Slash command
/eval-guide:eval-generatorThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
This skill produces the **Stage 2** artifact of the `/eval-guide` lifecycle: importable test cases for Copilot Studio's Evaluation tab plus a `.docx` test-case report for human review. It is the standalone form of `/eval-guide` Stage 2.
This skill produces the Stage 2 artifact of the /eval-guide lifecycle: importable test cases for Copilot Studio's Evaluation tab plus a .docx test-case report for human review. It is the standalone form of /eval-guide Stage 2.
Primary mode — the conversation already contains an /eval-suite-planner output (an eval plan with acceptance criteria on the Value × Risk matrix). Generate one set of cases per criterion, grouped by quality signal.
Fallback mode — no plan in conversation. Accept a plain-English agent description and generate test cases from scratch (6–8 cases minimum), using the same data model.
This skill covers Stage 2 (Set Baseline & Iterate) of the MS Learn 4-stage framework. After Stage 2, run the cases (Stage 3) and interpret results with /eval-result-interpreter (Stage 4).
Maturity callout — Pillar 2 (Build your eval sets): Stage 2 advances Pillar 2 from L100 Initial ("no established eval set") to L300 Systematic ("versioned eval set with coverage purposefully targeted"). The CSV files you generate ARE the Pillar 2 artifact.
When invoked as /eval-generator (with or without input):
Scan the conversation for a planner output (acceptance criteria with quadrants, methods, pass/fail conditions, quality dimensions).
/eval-suite-planner. Run /eval-suite-planner <description> first for the best results."Default to Single Response. ~80% of agents are single-response Q&A. Conversation mode only fits agents that do real multi-step workflows.
| Mode | Best for | Limits | Supported methods |
|---|---|---|---|
| Single response (default) | Factual Q&A, tool routing, specific answers, safety tests | Up to 100 cases per set | All 7 methods |
| Conversation (multi-turn) | Multi-step workflows, context retention, clarification flows | Up to 20 cases, max 12 messages (6 Q&A pairs) per case | General quality, Keyword match, Capability use, Custom (Classification) |
Switch to conversation mode only when:
If you switch to conversation mode, also recommend creating a complementary single-response set for criteria that need Compare meaning / Text similarity / Exact match (which conversation mode doesn't support).
This is the most important departure from older versions of this skill.
Methods live on the quality signal (the test set), not on the criterion. Every criterion in a signal inherits the same method set. A signal can have one method (most common) or several (e.g., a compliance signal that needs both Compare meaning for content correctness AND Keyword match for required disclaimers).
The internal data structure:
{
"agent_name": "...",
"test_sets": [
{
"quality_dimension": "Policy Accuracy",
"methods": ["Compare meaning", "Keyword match"],
"criteria": [
{
"criterion_id": 1,
"statement": "The agent should return the correct PTO days for the employee's office and tenure, citing the Time Off Policy.",
"quadrant": "critical",
"pass_condition": "Response contains the correct PTO number for the user's office/tenure AND cites the Time Off Policy.",
"fail_condition": "Incorrect number, missing citation, or cites the wrong policy.",
"custom_rubric": "",
"cases": [
{
"id": 1,
"question": "How many PTO days do LA employees get?",
"expected_responses": {
"Compare meaning": "LA employees receive [VERIFY: 18] PTO days per year, per the Time Off Policy.",
"Keyword match": "Time Off Policy, PTO, [VERIFY: 18]"
}
}
]
}
]
}
]
}
Rules:
methods: [] — the method set for the whole signal. Pick one when one fits; pick multiple only when the signal genuinely needs them.statement, quadrant, pass_condition, fail_condition, optional custom_rubric. No per-criterion method field.expected_responses: { method → value } — one entry per method in the signal's method set that needs a per-case reference. Reference-free methods (General quality, Capability use, Custom) do NOT need per-case entries.[VERIFY: ...] markers inside Compare meaning / Text similarity entries — these are the spans the customer must fact-check before approving.| Method | Per-case data | Where the grading rule lives |
|---|---|---|
| Compare meaning | expected_responses["Compare meaning"] = canonical answer (paraphrase OK; wrap facts in [VERIFY: …]) | LLM judge compares semantic equivalence of agent response vs. canonical |
| Text similarity | expected_responses["Text similarity"] = expected text | String similarity (0–1); default Pass ≥ 0.7 |
| Exact match | expected_responses["Exact match"] = exact string | Byte-equal (after normalization) |
| Keyword match | expected_responses["Keyword match"] = comma-separated keyword list ("escalate, manager, callback") | All keywords present (default) or any-keyword mode |
| General quality | none | LLM judge grades against criterion.pass_condition / fail_condition |
| Capability use | none | Pass if the agent invoked the right tool/topic (named in the criterion's pass condition) |
| Custom | none per case; criterion.custom_rubric carries the rubric | LLM judge follows the rubric verbatim |
For criteria with method = Custom in the signal's method set, draft a custom_rubric from the criterion's pass/fail conditions — e.g., "Rate the response Pass / Fail. Pass = [pass_condition]. Fail = [fail_condition]. Output PASS or FAIL with a one-sentence reason." Don't leave Custom criteria without a rubric.
From a plan: for each criterion, write 1+ cases. Match the criterion's quadrant to case count:
For each case:
question — a realistic input the agent would receive in production. Specific, not a placeholder. Include names, dates, IDs, context a real user would provide.expected_responses — one entry per reference-needing method in the signal's method set. Wrap factual content in [VERIFY: …].From scratch (no plan):
Use this only when Step 1 selected Conversation mode.
Conversation test set constraints:
General quality, Keyword match, Capability use, Custom (Classification).Compare meaning, Text similarity, Exact match.Format per case:
Conversation Test Case #N: [Scenario Name]
Turn 1 — User: [realistic user message]
Turn 1 — Agent (expected): [expected response or behavior description]
Turn 2 — User: [follow-up that depends on Turn 1 context]
Turn 2 — Agent (expected): [expected response maintaining context]
Turn 3 — User: [further follow-up]
Turn 3 — Agent (expected): [expected response]
Method: [General quality / Keyword match / Capability use / Custom]
Keywords (if Keyword match): [comma-separated list]
What this tests: [one sentence on the multi-turn capability being evaluated]
Critical turn: [which turn is most likely to fail and why]
Rules:
Conversation test sets cannot be CSV-imported. They must be created in Copilot Studio via Quick conversation set, Full conversation set, Test chat → test set, or Manual entry. The output of this skill in conversation mode serves as a planning blueprint the customer uses to drive manual entry — call this out explicitly.
The most common cause of false failures in eval results is wrong expected responses, not wrong agent answers. Defend against this with [VERIFY: …] markers — but only as a review aid, not as final output.
Compare meaning / Text similarity expected responses goes inside [VERIFY: ...] — e.g., "LA employees receive [VERIFY: 18] PTO days per year, per the [VERIFY: Time Off Policy v3.2].""Employees are eligible…") — only the facts you want the customer to verify.In Keyword match lists, you can wrap individual keywords in [VERIFY: …] if they're factual (e.g., URLs, version numbers, exact policy names).
At export time, strip every [VERIFY: …] wrapper. By the time the customer has clicked Approve, every span has been confirmed or edited — the brackets have served their purpose. Apply the regex \[VERIFY:\s*([^\]]*)\] → $1 to every value before writing it to the CSV or the customer-facing .docx test-case report. The internal stage-2-data.json may keep them for traceability if you re-launch the dashboard, but no customer-facing artifact should contain them.
.docx reportFor each test_set, write one CSV named eval-<signal-slug>-<YYYY-MM-DD>.csv. Exactly two columns:
"Question","Expected response"
No Testing method column. Copilot Studio's Evaluate tab assigns the method per row in its own UI after import — it is not pre-encoded in the CSV. The companion eval-setup-guide-<agent>-<date>.docx walks the customer through the manual method-assignment step.
Row generation rule. One row per active case per criterion (no case × method explosion). Per row:
Question = the case's question.Expected response = whichever of the case's expected_responses is most informational, picked by this priority order against the signal's method set:
Compare meaning → case.expected_responses["Compare meaning"].Text similarity → case.expected_responses["Text similarity"].Exact match → case.expected_responses["Exact match"].Keyword match → case.expected_responses["Keyword match"] (comma-separated keyword list).General quality / Custom / Capability use) → leave the cell empty.Strip every [VERIFY: …] marker from the cell value before writing the row. Replace [VERIFY: <content>] → <content>. The CSV is the customer's eval set; it must contain clean expected responses with no review-tooling syntax. See Step 6.
The customer can edit any cell before or after import — the CSV's pre-fills are starting points, not final values. The eval-setup-guide.docx tells them when to edit (e.g., switching a row's cell from canonical-answer to keyword-list when they decide the row should use Keyword match in the CPS UI).
A signal with 12 cases produces exactly 12 rows.
CSV format rules:
Question, Expected response."".Methods NOT available via CSV import:
.docx report into the CPS Custom configuration..docx test-case reportUse the /docx skill to generate eval-test-cases-<agent>-<date>.docx. Structure:
Agent Vision summary (5–6 lines from Stage 0/Stage 1 if available).
Value × Risk matrix summary — criteria grouped by quadrant with pass/fail conditions.
Test cases by quality dimension — for each quality signal:
custom_rubric if Custom is in the signal's methods.[VERIFY: …] markers — they were a review aid in the dashboard, but every span has now been confirmed or edited. The .docx ships clean.Method mapping summary — count of cases per method, with notes on which methods need manual addition (Custom, sometimes Capability use).
What these tests catch — 3–4 bullet points naming what the customer would have missed without these tests.
Next steps: "Import the CSVs into Copilot Studio's Evaluation tab. Add Custom cases manually using the rubrics below. Run the suite and pass the results to /eval-result-interpreter."
Maturity snapshot:
| Pillar | Baseline | After this kit | Next-session target |
|---|---|---|---|
| 1 — Define what "good" means | L300 ✓ (from Stage 1) | L300 ✓ | — |
| 2 — Build your eval sets | L100 Initial | L300 Systematic ✓ | — |
| 4 — Improve and iterate | L100 Initial | L100 Initial | L300 (after Stage 4 triage) |
Tell the customer: "One CSV per quality signal — paste directly into Copilot Studio's Evaluation tab. Each test case is repeated once per method in that signal's method set so the Testing method column matches the row's Expected response. Custom cases need to be added manually with their rubrics — those are listed in the .docx report."
Display before ending. Eval kits are useless without human validation.
| # | Checkpoint | What to verify |
|---|---|---|
| 1 | Questions are realistic | Every Question is a real production input — not a placeholder. Check for typos, abbreviations, ambiguity that real users would include. |
| 2 | Expected responses are correct | Verify every [VERIFY: …] span against the actual knowledge sources. #1 source of false failures. |
| 3 | Method choices match what you're testing | Compare meaning for paraphrasable answers, Keyword match for required phrases, Custom for nuanced rubrics. Wrong method = wrong signal. |
| 4 | Custom rubrics are precise | For Custom criteria, read the custom_rubric. Vague rubrics ("Is the response good?") behave like General quality with extra steps. Sharpen until the rubric forces a binary verdict. |
| 5 | Negative test coverage | For adversarial / Low Value · High Risk criteria, verify the expected behavior matches policy (refuse / redirect / escalate — pick the right one). |
| 6 | Coverage spans the full Vision | Every Vision capability and boundary has at least one case. Gaps surface here, not in production. |
| 7 | Conversation mode chosen for the right reasons (if applicable) | Multi-turn cases test capabilities users actually exercise. If the agent mostly handles standalone questions, single-response gives better signal. |
Mandatory reminder: "This test set was AI-generated. Before running it against your agent, a domain expert must review every Question, Expected response, and Custom rubric. Wrong expected responses cause correct agent answers to fail."
[VERIFY: …]. Always.Similarity test method (replace with Compare meaning or Keyword match)./eval-suite-planner I'm building an HR policy bot...
[planner outputs eval plan with criteria on Value × Risk matrix]
/eval-generator
<- generates from the plan, one CSV per quality signal (2 cols: Question + Expected response, one row per case)
<- produces .docx test-case report
/eval-generator I'm building a meeting-notes agent that takes a transcript and produces structured action items.
<- generates from scratch, 6-8 cases, single signal, single method (Compare meaning by default)
/eval-generator I'm building a travel-booking agent that handles multi-turn flight search, seat selection, purchase.
<- detects multi-turn behavior, generates 4-6 conversation test cases as a planning blueprint
<- recommends complementary single-response set for standalone queries
/eval-generator
<- no plan, no description provided — asks for input
/eval-suite-planner — Stage 1: produces the eval plan this skill consumes./eval-result-interpreter — Stage 4: takes the run results and produces a triage report./eval-faq — methodology Q&A grounded in Microsoft's eval ecosystem./eval-guide — the orchestrator. Wraps Stages 0–4 with an interactive dashboard checkpoint at each stage.Provides a checklist for code reviews covering functionality, security, performance, maintainability, tests, and quality. Use for pull requests, audits, team standards, and developer training.
npx claudepluginhub microsoft/eval-guide