agent-quality-validator

Agent Quality Validator

A quality assurance tool for Claude Code agent .md files. This skill analyzes agent definitions against established best practices and returns a structured score with actionable feedback.

Capabilities

1. YAML Frontmatter Validation — Confirms required fields (name, description, tools, model) are present and well-formed.
2. Description Quality Check — Detects vague descriptions that lack action words or a clear specialization scope.
3. When-to-Invoke Coverage — Verifies the agent body includes at least one concrete invocation example.
4. Tool Restriction Audit — Flags overly broad tool grants (["*"] or "All tools") that bypass the principle of least privilege.
5. Output Format Documentation — Checks that the agent describes its expected output or return format.
6. Scored Report — Returns a 0–100 integer score, a passed boolean (threshold: 80), a per-criterion breakdown, and a prioritised issues list.

Input Requirements

Provide one of the following:

Input type	Details
File path	Absolute or relative path to an agent .md file
Content string	Raw Markdown string containing the agent definition

The validator accepts both via AgentQualityValidator — pass a file path to validate_file() or a raw string to validate_content().

Output Format

{
    "score": 87,                      # int 0-100
    "passed": True,                   # bool — True when score >= 80
    "breakdown": {
        "frontmatter_completeness": 28,   # max 30
        "description_quality": 20,         # max 25
        "when_to_invoke": 20,              # max 20
        "tool_restrictions": 15,           # max 15
        "output_format": 4,                # max 10
    },
    "issues": [
        "output_format: No output format or return description found in body."
    ]
}

Scripts

File	Purpose
validator.py	Main implementation — AgentQualityValidator class

Quick usage

from validator import AgentQualityValidator

v = AgentQualityValidator()

# From a file
result = v.validate_file("path/to/my-agent.md")

# From a string
result = v.validate_content(raw_markdown_string)

print(result["score"])    # e.g. 87
print(result["passed"])   # True / False
print(result["issues"])   # List of human-readable problems

Quality Criteria

Criterion	Max Points	What is checked
Frontmatter completeness	30	name, description, tools, model fields all present and non-empty
Description quality	25	Contains action verbs; not just a noun phrase; reflects specialisation
When-to-invoke examples	20	Body contains a "when to invoke" section with at least one example
Tool restrictions	15	tools is not ["*"], "*", or "All tools"; points deducted for wildcard grants
Output format	10	Body documents expected output, return values, or response format

Pass threshold: score >= 80

Review checklist

Before treating a validator score as authoritative:

• The agent file under review has a .md extension and YAML frontmatter
• The score breakdown lists every failing criterion with a specific fix
• Tool restrictions in the agent's frontmatter were inspected for minimum-privilege
• "When to invoke" examples were checked for concreteness, not just presence
• The final 0–100 score is consistent across re-runs on the same file

When NOT to Use

• Validating skills (.md with SKILL.md format) — this validator is specific to agent files.
• Checking prompt templates or slash commands — different schemas apply.
• Enforcing style preferences beyond the criteria above — use a linter for that.