pipeline-scaffold — emit an experiment codebase from a slug or spec

pipeline-scaffold — emit an experiment codebase from a slug or spec

Purpose

Given a task type and either a slug or an experiment-design spec, drop a runnable starter codebase into research/code/<slug>/. The user installs deps, edits the config (data path, model name), and runs.

When to invoke

Triggers:

• "Scaffold experiment code"
• "Set up training code for X"
• "Make me a starter codebase for this study"
• "I have a spec, now make the code"
• User runs /research-helper:pipeline-scaffold <task>

Do NOT invoke for:

• "Fix this bug" / "debug my training" — not scaffolding
• "What does this code do?" — read it directly, don't overwrite
• "Run my experiment" — scaffold emits code; doesn't execute it
• "Make me an inference server" — out of scope

Workflow

1. Pick the right --task

If the user says	Use --task
"fine-tune BERT", "train a classifier", "GLUE / SST-2"	classification
"summarization", "translation", "T5 / BART fine-tune", "causal LM training"	generation
"prompt comparison", "in-context learning", "few-shot", "RAG eval", "no training"	prompt-eval
Unclear	Ask the user

2. Look for an existing experiment-design spec

Check research/experiments/. If a matching spec exists, pass it via --from-spec — this:

• Reuses the spec's slug for the scaffold's directory
• Parses **N seeds:** and pre-fills config.yaml
• Parses **Primary** (...): and pre-fills the primary metric field
• Adds a relative link to the spec in the scaffold's README

If multiple specs match, list them and ask the user which.

3. Confirm the slug + output dir

Default output is research/code/<slug>/. The user can override via --output. Confirm before running so we don't surprise them.

4. Call the script

With a spec:

python skills/pipeline-scaffold/scripts/scaffold.py \
  --task <task> \
  --from-spec research/experiments/<slug>-<date>.md

Without:

python skills/pipeline-scaffold/scripts/scaffold.py --task <task> --slug <slug>

If research/code/<slug>/ exists, the script refuses to overwrite. Pass --force to wipe-and-replace (confirm with the user first).

5. Walk the user through the generated README

Open research/code/<slug>/README.md. Surface in this order:

1. Setup: python -m venv venv, activate, pip install -r requirements.txt. For prompt-eval with a gated model: HF_TOKEN env var.
2. Configure: the config.yaml has placeholders that MUST be set:
   • model_name (default is a small reasonable choice)
   • dataset (default is a small reasonable choice)
   • For classification: text_column / label_column must match the dataset
   • For generation: input_column / target_column
   • For prompt-eval: prompt_template_id and possibly extra templates in prompt.py
3. Run single seed: python main.py --seed 42 --config config.yaml
4. Run all seeds: bash run_all_seeds.sh

Don't dump the file tree into chat — point at the README.

6. Don't auto-run training

The user runs the code. Surface the commands; stop there. Only run if the user explicitly says "yes, run it now."

7. Pre-run sanity checks to mention

• Disk space for model + checkpoint + cache (10-50 GB for medium models)
• GPU available (script auto-detects via torch.cuda.is_available())
• For gated models (Llama, Mistral): huggingface-cli login or HF_TOKEN
• If --from-spec was used: spot-check that the seeds in config.yaml match what the spec said. If parse_seeds warned to stderr, the seeds may have defaulted.

8. Where it lives

research/code/<slug>/. Tell the user the path; don't paste the file contents unless asked.

Failure modes

Output dir exists — script refuses; pass --force (confirm with user). If iterating, prefer a new --slug over force-overwriting.

Spec not found — exit 1 with "spec not found: <path>". Ask the user to verify the path.

Spec filename doesn't match <slug>-YYYY-MM-DD.md — parsing still works (seeds, primary metric); only the slug can't be derived from the filename. Pass --slug explicitly.

Spec parse warnings — script prints pipeline-scaffold: <warning> to stderr but doesn't fail. Note in handoff to the user that defaults were used for whatever didn't parse.

--task invalid — argparse rejects.

Neither --slug nor --from-spec — argparse error with helpful message.

Setup notes

Stdlib only for the scaffold itself. The EMITTED code has per-task requirements.txt files the user installs after scaffolding.

Defaults summary

Knob	Default	When to change
--task	(required)	—
--output	research/code/<slug>/	Override if the user asks
--from-spec	(none)	Use when a matching spec exists in research/experiments/
--force	off	Turn on only when intentionally overwriting