From oxy-skills
Evaluate the output of one of the 4 oxy instance-building skills (semantic-layer, workflow-builder, etl-builder, app-builder) against a rubric and propose specific improvements to the skill's SKILL.md. Use when the user asks to evaluate a skill, score skill output, or improve a skill based on test results.
How this skill is triggered — by the user, by Claude, or both
Slash command
/oxy-skills:oxy-instance-skill-evaluatorThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
You evaluate the output of one of the 4 oxy instance-building skills against its rubric and propose specific,
You evaluate the output of one of the 4 oxy instance-building skills against its rubric and propose specific,
actionable edits to the skill's SKILL.md file.
Activate when the user says:
/oxy:eval-and-improve invokes you for a specific skillAsk the user (or read from context) which skill just ran and where the output files are. Expected outputs per skill:
semantics/views/*.view.yml, semantics/topics/*.topic.ymlworkflows/*.workflow.yml, agents/*.agent.yml, example_sql/*.sqletl/sources/<provider>/, etl/runners/<provider>_<entity>.pyapps/*.app.yml or *.app.ymlRead the rubric for the skill being evaluated:
eval/rubrics/semantic-layer.mdeval/rubrics/workflow-builder.mdeval/rubrics/etl-builder.mdeval/rubrics/app-builder.mdIf eval/rubrics/ is not in the current directory, look for it relative to the skills plugin
directory (where this SKILL.md lives: ../../../eval/rubrics/).
Read every generated file. For each rubric item, check the file content. Do NOT skip any must-pass items. Verify:
For YAML files: Use Read to inspect structure. Check field names exactly — a field
named query: is different from sql_query:.
For Python files: Use Bash to run python -m py_compile <file> and check exit code.
For runtime checks: Use Bash to run the verification commands listed in the rubric:
oxy validate --file=<file>
oxy run <workflow>.workflow.yml --dry-run
oxy build
For each rubric item, record:
Present results grouped by MUST-PASS first, then SHOULD-PASS:
## Skill: oxy-workflow-builder
### MUST-PASS (X/10 passed)
✅ M1 — tasks: array present at top level (confirmed in workflows/daily_report.workflow.yml:4)
❌ M2 — type: sql found instead of type: execute_sql (line 12)
...
### SHOULD-PASS (X/5 passed)
✅ Q1 — semantic layer checked before SQL (saw grep commands in Claude output)
❌ Q4 — {% set %} syntax used instead of variables: block defaults
For each failed item, identify which part of the skill's SKILL.md caused it. Common patterns:
To find root causes:
SKILL.mdFor each failed must-pass item, propose a specific edit to the skill's SKILL.md.
Format proposals as:
### Fix for M2 (type: execute_sql)
File: skills/oxy-workflow-builder/SKILL.md
Section: "Workflow File Structure"
Issue: Template shows correct structure but no explicit warning about wrong values.
Proposed addition after line ~185:
> ⚠️ CRITICAL: The task type must be `execute_sql` exactly.
> Do NOT use `type: sql`, `type: execute`, or any other variant.
> These will pass `oxy validate` but fail at runtime.
Be specific: include the file, section, and exact text to add or change.
After presenting all proposed fixes, ask the user: "Should I apply these changes to the SKILL.md files now?"
If yes:
Always end with a summary table:
## Summary: oxy-[skill-name]
| Category | Passed | Failed | Total |
|----------|--------|--------|-------|
| Must-Pass | X | Y | Z |
| Should-Pass | X | Y | Z |
Skill ready for production: YES / NO (must-pass all green)
Top 3 fixes needed:
1. [M#] — one line description
2. [M#] — one line description
3. [Q#] — one line description
The correct syntax for the retrieval tool in agent files (commonly missing):
tools:
- type: execute_sql
database: <db_name>
- type: retrieval
src:
- example_sql/*.sql
- workflows/*.workflow.yml
key_var: OPENAI_API_KEY # embedding model API key env var
Required fields: type, src
Optional: name (defaults to "retrieval"), key_var, embed_model, top_k, db_path
--dry-run only works for SQL files (oxy run query.sql --dry-run). For workflow files,
--dry-run is silently ignored. The only true verification for workflows is running them.
Wrong field names (like type: sql instead of type: execute_sql) only surface at runtime.oxy build or oxy run commands fail due to missing OXY_DATABASE_URL, note this
as an environment issue, not a skill failure. Use oxy ≥ 0.5.27 or set the env var.Guides creation, editing, and verification of skills for AI coding agents using test-driven development with subagent scenarios. Use when authoring or debugging skills.
npx claudepluginhub oxy-hq/skills --plugin oxy-skills