pytorch-skills

Skills

AI coding skills for PyTorch development. Skills are structured prompts that teach AI agents (Claude Code, Cursor, Copilot) how to perform PyTorch-specific tasks like debugging torch.compile, authoring operators, profiling, and triaging issues.

Install

npx skills add meta-pytorch/skills

Dev Setup

uv sync --group dev
uv run prek run -a

Create A Skill

uv run python scripts/new_skill.py my-new-skill

This creates a minimal SKILL.md, evals/eval.yaml, evals/checkers.py, and evals/fixtures/ scaffold. New skills normally do not require a manual .claude-plugin edit because this plugin ships the whole ./skills tree for npx skills; scripts/validate_plugin.py now checks that every skill stays covered by the configured plugin roots.

Structure

skills/
├── .claude-plugin/
│   └── marketplace.json      # Skill registry metadata
├── skills/
│   └── my-skill/
│       ├── SKILL.md           # Required — skill definition
│       ├── evals/
│       │   ├── eval.yaml      # Required — benchmark contract
│       │   ├── fixtures/      # Optional — task inputs, repros, traces
│       │   └── checkers.py    # Optional — custom outcome checks
│       ├── references/        # Optional — detailed docs
│       ├── scripts/           # Optional — helper scripts
│       └── assets/            # Optional — templates, resources
├── skill_eval/                # Eval harness, scoring, reporting
├── CONTRIBUTING.md
├── LICENSE
└── README.md

SKILL.md Format

Each skill is a markdown file with YAML frontmatter following the Agent Skills spec:

---
name: my-skill                    # Required. Lowercase, hyphens, digits. Must match directory name.
description: >-                   # Required. What it does AND when to use it. Max 1024 chars.
  Debug torch.compile graph breaks
  and guard failures. Use when
  encountering recompilation issues.
license: BSD-3-Clause             # Optional.
metadata:                         # Optional.
  author: pytorch
  version: "1.0"
---

# My Skill

Instructions for the agent...

Contributing

See CONTRIBUTING.md.

Quality Bar

Every skill is expected to ship with evaluation coverage. Today that means:

• evals/eval.yaml lives next to SKILL.md
• The eval defines a benchmark contract, not just a prompt example
• Reviewers approve the eval's credibility; the harness computes the score
• The harness emits a score per model and preserves raw run evidence for later comparisons

Each test case can combine:

• Deterministic outcome checks like file contents, JSON values, or exact output
• Optional Python checkers for PyTorch-specific verification
• Optional LLM judgment for semantic quality that is hard to check deterministically

Eval Harness

Quickstart

# 1. Validate the eval contract (syntax, required fields)
uv run python -m skill_eval validate skills/my-skill

# 2. Run a single eval
uv run python -m skill_eval run skills/my-skill

# 3. Run an A/B eval (baseline without skill vs treatment with skill)
uv run python -m skill_eval run-ab skills/my-skill

# 4. Run all evals in the repo
uv run python -m skill_eval run-all

Verbose and Debug Output

Use --verbose to see per-test details inline:

uv run python -m skill_eval run-ab skills/my-skill --verbose

Verbose output includes:

• The agent's response text (first 500 chars)
• Which tools and skills were invoked (e.g. Skill(my-skill))
• Each deterministic check result with PASS/FAIL and details
• Checker and LLM judge results when present
• Final score per test case

Use --debug-dir to save full artifacts to disk for post-mortem inspection:

uv run python -m skill_eval run-ab skills/my-skill --debug-dir ./debug

This writes per-test-case directories under debug/<variant>/<model>/<test_case>/ containing:

• claude_command.txt — the exact CLI command that was run
• claude_stdout.txt — full stream-json output from the Claude CLI
• claude_stderr.txt — stderr from the Claude CLI
• runner_metadata.json — exit code, duration, variant, model

Use --output-json to write the machine-readable result bundle:

uv run python -m skill_eval run-ab skills/my-skill --output-json results.json

Writing eval.yaml

Each skill needs an evals/eval.yaml next to SKILL.md. Minimal example:

eval_version: "1"
skill_name: "my-skill"
skill_path: "skills/my-skill/SKILL.md"

models:
  - "claude-opus-4-6"

test_cases:
  - name: "basic_usage"
    description: "The skill produces the expected output."
    user_message: "Do the thing my skill does"
    deterministic_checks:
      - type: "output_contains_ci"
        value: "expected phrase"
      - type: "skill_invoked"
        skill: "my-skill"

Key fields: