autoresearch-skill

Autoresearch Skill Improvement Loop

Generate a self-contained shell script that autonomously improves a Claude Code skill through iterative cycles. Each iteration: improve the skill, actually execute it against test prompts, evaluate the execution output, keep or discard. Deterministic steps (git, scoring, cleanup) run as plain bash. Subjective steps (improving, executing, evaluating) run via claude -p calls with generated prompts.

Workflow

Step 1: Analyze the Target Skill

Read all files in the target skill directory. Build two things:

File inventory -- what files exist and can be modified:

Skill: mermaid-svg
Files:
  - SKILL.md (2,100 words)
  - references/layout-patterns.md (3,200 words)
  - references/style-guide.md (1,800 words)

Execution profile -- how the skill behaves when used:

• Purpose: What the skill does when triggered (e.g., "converts Mermaid diagram syntax into hand-crafted SVG markup")
• Required tools: What tools a claude -p call needs to execute the skill (e.g., Read,Write,Glob for a skill that reads input and writes files; Read,Edit,Write,Glob,Bash for a skill that runs scripts; Read,WebSearch, WebFetch for a research skill)
• Artifacts: What files or output the skill produces when executed (e.g., *.svg files, *.ts files, text-only output to stdout)
• Cleanup commands: Bash commands to remove artifacts between executions (e.g., rm -f *.svg, rm -rf output/). Empty if the skill produces only text output.

Step 2: Gather Criteria

If the user has not provided optimization criteria, ask for them. Criteria are specific, measurable aspects of the skill to improve. Examples:

• "SVG output should use logical coordinate-based layout, not magic numbers"
• "Research responses should include source quality evaluation"
• "TypeScript guidance should cover monorepo project patterns"

Also ask whether the user wants to provide test prompts -- concrete user queries that exercise the skill. If not provided, generate 3-5 from the criteria. Test prompts should be realistic requests a user would make that trigger the skill.

Step 3: Build the Evaluation Rubric

Convert each criterion into a 1-5 scoring rubric with observable level descriptions. See references/prompt-templates.md for rubric construction guidance. Each level describes concrete, observable qualities of the execution output (not the skill files themselves).

Step 4: Compose the Prompts

Build three system prompts embedded in the script:

1. Improver prompt -- Instructs claude -p to read the skill files, analyze past results, propose and apply ONE improvement. Includes: the rubric, improvement strategies (from references/improvement-strategies.md), file constraints, and required HYPOTHESIS: output line.

2. Executor prompt -- Instructs claude -p to act as Claude with the skill loaded. The skill content is injected dynamically at runtime (the script reads the current SKILL.md + references each iteration). The executor receives a test prompt and responds as Claude would, following the skill's guidance.

3. Evaluator prompt -- Instructs claude -p to judge the execution outputs against the rubric. Receives: the collected execution outputs from all test prompts, the rubric, and outputs a structured score with AGGREGATE: line.

See references/prompt-templates.md for the full templates.

Step 5: Generate the Script

Generate a bash script following references/script-structure.md. Sections:

1. Configuration -- skill dir, branch, TSV path, test prompts array, cleanup commands, all system prompts as heredocs
2. Helper functions -- read_skill (reads current skill content at runtime), execute_skill (runs all test prompts, collects outputs), cleanup_artifacts (removes execution artifacts), summary (exit trap)
3. Setup -- create branch, initialize TSV
4. Baseline -- execute skill as-is against test prompts, evaluate outputs, record baseline score
5. Main loop -- improve, commit, execute, evaluate, decide, cleanup, log
6. Summary -- print results table on exit

Key design rules:

• The executor claude -p call receives the current skill content (read at runtime via read_skill), not a fixed heredoc -- the skill changes each iteration
• The executor gets tools from the execution profile (Step 1)
• After each execution+evaluation cycle, cleanup_artifacts removes any files the skill created, so the next iteration starts clean
• The evaluator judges execution output quality, not skill file content
• Score extraction: grep -oP 'AGGREGATE:\s*\K[0-9.]+'
• Keep/discard: score > best_score via bc -l
• Plateau: 3 consecutive discards triggers exit
• Trap EXIT for summary on Ctrl+C

Step 6: Write and Confirm

Write the script to improve-<skill-name>.sh in the repo root. Make it executable. Show the user:

• The execution profile (tools, artifacts, cleanup)
• The generated rubric
• The test prompts
• How to run: bash improve-<skill-name>.sh
• How to stop: Ctrl+C (prints summary)
• How to review: cat improvements.tsv and git log skill-improve/<tag>

Constraints

• The generated script must be self-contained -- prompts baked in as heredocs, no external dependencies beyond claude, git, bc, grep
• The executor must load the current skill content each iteration (not a stale copy from generation time)
• Cleanup must remove all execution artifacts between iterations
• The evaluator prompt and test prompts are fixed for the run
• improvements.tsv stays untracked by git

Additional Resources

Reference Files

• references/prompt-templates.md -- System prompt templates for all three claude -p calls (improve, execute, evaluate), rubric guide
• references/script-structure.md -- Complete bash script structure with execute step, cleanup, and dynamic skill loading
• references/improvement-strategies.md -- Catalog of skill improvement strategies, embedded in the improver system prompt