autoresearch

autoresearch

Autonomous hill-climb. Read code → generate mutations → measure → keep-or-revert → repeat. The runner ships ready-made; the model fills in the project-specific bits and lets the loop run.

State machine

TRIGGERED
  ↓
GATE: is the metric numeric + automatable? (no → bail, suggest alternative skill)
  ↓
READ_CODEBASE         (package.json, entry file, build config — parallel reads)
  ↓
INFER + CONFIRM       (one line: metric / measureCmd / targetFiles. NOT a Q&A round.)
  ↓
SCAFFOLD              (cp templates/* into .autoresearch/)
  ↓
WRITE_CONFIG          (fill config.js from inference; set buildCmd if compiled)
  ↓
WRITE_MUTATIONS       (read target files, write 15-30 mutations whose patterns
                       you have GREPPED for in the actual file)
  ↓
RUN                   (node .autoresearch/server.js & node .autoresearch/runner.js)
  ↓
runner exits with one of:
  - "exhausted"         → REGENERATE_BATCH → RUN
  - "plateau"           → STOP, report
  - "target"            → STOP, report
  - "interrupted"       → STOP, report
  - "preflight-failed"  → re-read target files, REWRITE mutations, RUN
  - "error"             → DIAGNOSE → fix → RUN

What the templates do

~/.claude/skills/autoresearch/templates/:

file	purpose
runner.js	Hill-climb loop. Dedupes by content hash, not name. Snapshots bestState + score to .autoresearch/best/ after each keep. Resume restores bestScore from kept-only experiments and consecutiveDiscards from trailing tail. Pre-flight aborts if all dryable mutations are no-ops. Stops on plateau / target / SIGINT.
server.js	Static server. Default :3456.
dashboard.html	Live dashboard, polls every 500ms. Adapts to whatever metric is configured.
config.example.js	Schema. Required: metric, maximize, targetFiles, measureCmd. Recommended: minImprovement, samples, buildCmd (compiled), plateauLimit, targetScore.
mutations.example.js	Two shapes: { name, apply(state) } for JS regex, or { name, applyCmd: "shell..." } for Python/Rust/Go or any language-fragile target.

Procedure

TRIGGERED → GATE

If the user's request has no numeric automatable metric, bail. Examples that should NOT trigger this skill:

• "make this copy better" → no metric
• "make this component cleaner" → judgment, not measurement
• "improve error handling" → no metric (suggest manual review)
• "make the nav smaller" → if "smaller" is bytes, OK; if it's px, no metric

READ_CODEBASE

In parallel: read package.json, the main entry file, any obvious build config. Don't ask the user anything yet.

INFER + CONFIRM (one line, not a form)

Emit one line stating what you'll do. The user can correct or stay silent.

"Optimizing performance in index.html via npx lighthouse http://localhost:3000. Running baseline now."

If a project doesn't surface enough signal (bare directory, no entry file), then ask. Otherwise proceed.

SCAFFOLD

mkdir -p .autoresearch
cp ~/.claude/skills/autoresearch/templates/runner.js          .autoresearch/runner.js
cp ~/.claude/skills/autoresearch/templates/server.js          .autoresearch/server.js
cp ~/.claude/skills/autoresearch/templates/dashboard.html     .autoresearch/dashboard.html
cp ~/.claude/skills/autoresearch/templates/config.example.js  .autoresearch/config.js
cp ~/.claude/skills/autoresearch/templates/mutations.example.js .autoresearch/mutations.js
echo '/results.json' >> .autoresearch/.gitignore
echo '/best/' >> .autoresearch/.gitignore

WRITE_CONFIG

Fill .autoresearch/config.js from inference. Common shapes:

// Lighthouse (noisy — needs samples + minImprovement)
{ metric: 'performance', maximize: true,
  targetFiles: ['index.html'], samples: 3, minImprovement: 3, targetScore: 99,
  measureCmd: 'npx -y lighthouse $URL --output=json --quiet --only-categories=performance',
  parse: (s) => ({ score: Math.round(JSON.parse(s).categories.performance.score * 100) }) }

// Bundle size (deterministic — minImprovement: 0, samples: 1, buildCmd separate)
{ metric: 'bundle_kb', maximize: false,
  targetFiles: ['vite.config.ts', 'src/index.tsx'], minImprovement: 0,
  buildCmd: 'npm run build',
  measureCmd: 'du -sk dist | cut -f1' }

// Build time (slightly noisy — samples: 3, no buildCmd because measure IS the build)
{ metric: 'build_s', maximize: false,
  targetFiles: ['vite.config.ts', 'tsconfig.json'], samples: 3, minImprovement: 0.5,
  measureCmd: '/usr/bin/time -p npm run build 2>&1 | awk \'/real/ {print $2}\'' }

// API p99 (very noisy — samples: 5)
{ metric: 'p99_ms', maximize: false,
  targetFiles: ['src/server.ts'], samples: 5, minImprovement: 5,
  buildCmd: 'npm run build && pm2 restart api',
  measureCmd: 'hey -n 1000 -q 50 $URL | awk \'/99%/ {print $NF*1000}\'' }

// Python eval accuracy
{ metric: 'accuracy', maximize: true,
  targetFiles: ['prompts/system.md'], samples: 1, minImprovement: 0.01,
  measureCmd: 'python eval.py --json | jq .accuracy' }

Verify the measure command runs once by hand before the loop starts. If baseline fails, runner exits with status "error" immediately — debug the command, don't grind.

WRITE_MUTATIONS

Read every file in targetFiles. Write 15–30 mutations into .autoresearch/mutations.js.

Two shapes. Pick per mutation:

// JS apply — for HTML/CSS/JS/JSON/YAML, anything string-replaceable
{ name: 'lazy load all images',
  apply: (state) => {
    state['index.html'] = state['index.html'].replace(/<img(?![^>]*loading)/g, '<img loading="lazy"');
    return state;
  } }

// applyCmd — for Python, Rust, Go, AST transforms, formatters, patches
{ name: 'remove unused imports via ruff',
  applyCmd: 'ruff check --select F401 --fix src/' }

Rules (these are why generic checklists lose):

• One change per mutation. Compound diffs hide what worked.
• Grep the pattern before writing the mutation. A regex that matches nothing is a no-op skip — and the runner's pre-flight will abort the whole batch if every dryable mutation is a no-op.
• Specific to what you read, not what you imagine. The Lighthouse run kept "use system font stack only" because the HTML had a Google Fonts link. That mutation only existed because somebody read the file.
• Cheap before exotic. Solid ground state first.

Idea ladder (apply only entries that match the code you read):

domain	cheap	medium	expensive
performance	loading=lazy, fetchpriority, preload	remove animations, system fonts, simplify gradients	swap libs, drop deps
bundle	tree-shake imports, drop polyfills	code-split routes, dynamic imports	swap heavy deps
build time	skip type-check on watch, lighter sourcemaps	parallelize, swc/esbuild	cache layers, incremental
latency	cache headers, dedupe DB calls	batch queries, denormalize	move work async, CDN
accuracy	tighten instructions, examples	restructure prompt, CoT	swap model

RUN

Two background commands:

node .autoresearch/server.js &       # dashboard
node .autoresearch/runner.js         # the loop

Open http://localhost:3456/dashboard.html. Stream progress inline:

[1/24] lazy load all images
  performance: 94
  ✓ keep (86 → 94)

REGENERATE_BATCH (when runner exits "exhausted")

Don't ask the user. Read results.json, identify what kept and what near-missed, write a new mutations.js. The runner dedupes by content hash, so:

• A reworded name with identical code → skipped (correctly).
• An identical name with different code → runs (correctly — it's a new mutation).
• A different name with identical code → skipped (correctly).

The runner reloads bestState and bestScore from the .autoresearch/best/ snapshot, so you don't lose ground.

Stop regenerating when:

• runner exits "plateau" — 5 consecutive non-improvements
• runner exits "target" — targetScore hit
• you've genuinely run out of ideas after re-reading the code

Final report

=== autoresearch: <metric> (status: plateau|target|interrupted) ===
baseline:   77      final: 99      gain: +29%

kept (7):
  +8  lazy load all images
  +3  fetchpriority on hero
  +1  reduce image quality in URLs
  ...

discarded:  33    crashed: 0    skipped: 5

Point them at .autoresearch/dashboard.html and .autoresearch/results.json.

Hard rules

1. One mutation, one measurement. No compounding.
2. Set minImprovement to noise floor. Without it, jitter wins compound into a phony best.
3. Grep patterns before writing mutations. If the regex doesn't match the actual file, it's a no-op skip.
4. Run baseline first. If it fails, fix the measure command before generating mutations.
5. Don't track results.json or best/ in git. Working files.
6. Side-effect-free measurement. No prod, no billing, no destructive ops in measureCmd.
7. One-line confirmation, not a Q&A. The skill is triggered to act — read first, infer, state assumptions, run. Block only on genuine ambiguity.

When NOT to use

• No automatable numeric metric (use a regular task, not this skill).
• One measurement >5 min (loop becomes unusable — use /loop instead).
• Changes need design/UX/copy judgment (use /product-design, /pencil-dev).
• Side effects are dangerous (production, billing, external services).

Reference run

A representative Lighthouse optimization run improved from 77 to 99 over 45 experiments. The included templates use a noise floor, content-hash deduplication, a build-aware loop, an applyCmd escape hatch, and crash-safe resume.