experiment-layout

Experiment Layout

Overview

Experiments must be independently reproducible. Each experiment directory is a frozen snapshot whose layout, numbering, and read-only status exist to keep prior runs from being mutated by later work. Subsequent experiments live in new directories with new numbers — you never rename, delete, or edit a prior experiment's files, even if the prior experiment is empty, mis-named, or mid-revision.

Violating the letter of the rules is violating the spirit of the rules.

When to Use

• Creating a new experiment thread under experiments/
• Adding more cells to an in-progress experiment
• Asked "where should I put this experiment?"
• Anything that creates directories under experiments/
• Tempted to rename, delete, move, or edit a prior experiments/expNN_* directory

Terminology

• Root files: the canonical source files at the project root (e.g. train.py, simulate.py, benchmark.py, pipeline.py, etc.) that define the production / default pipeline.
• Experiment: a thread of related comparisons exploring one question (e.g. exp03_<topic>/).
• Cell: a single concrete run within an experiment (e03a/, e03b/, …). One cell = one variant of one knob.

The Rules

1. New experiment → new directory with the next unused number

Existing layout:

experiments/
├── exp00_<topic>/
├── exp01_<topic>/
└── exp02_<topic>/

A new experimental thread goes in experiments/exp03_<topic>/. If exp03 is already taken (even by a half-empty stub or "reserved" placeholder), the new work goes in exp04_<topic>/.

Numbers are reservations. Reserved ≠ unused. Reserved ≠ free for re-use.

2. Previous experiments are READ-ONLY

No mv exp03 exp04. No rm -rf exp03. No editing files under any prior expNN_*. Even reading-then-writing-back counts as a modification.

If a prior experiment dir is empty, wrong-name, or looks like a stub — don't touch it. Confirm with the user. Touching it risks losing reproducibility evidence the user is relying on.

3. Variants live as eNNa, eNNb, eNNc, … inside the experiment

experiments/exp04_<topic>/
├── e04a/         ← control / anchor
├── e04b/         ← variant b
├── e04c/         ← variant c
├── e04d/         ← variant d
└── e04e/         ← variant e

Each cell tests one variant of one knob. The letter prefix eNN matches the experiment number. Five-cell coverage is a reasonable default for "diverse coverage of one variable"; use fewer cells if compute-constrained or more if the variable warrants it.

Quick Reference

Layout-level shape only — see the cell-source-isolation skill for what goes inside each cell directory.

experiments/
├── exp00_<topic>/                  ← read-only after run
│   ├── e00a/                       ← control / anchor (cell contents: see cell-source-isolation)
│   ├── e00b/                       ← variant b
│   └── e00c/                       ← variant c
├── exp01_<topic>/                  ← read-only after run
├── exp02_<topic>/                  ← read-only after run
└── exp04_<topic>/                  ← new experiment (exp03 reserved → use 04)
    ├── e04a/                       ← control
    └── e04b/                       ← variant b

Numbering / reservation pattern:

• Pick the next unused NN — do not reuse a reserved number, even an empty one.
• Variants within expNN_* are lettered eNNa, eNNb, eNNc, … matching NN.
• Every expNN_* directory other than the one currently being created is read-only.

Common Mistakes & Rationalizations

Excuse / mistake	Reality
"I'll just rename exp03 → exp04"	Rename touches a prior dir. Create exp04 fresh.
"exp03 is empty so I'll repurpose the slot"	Empty ≠ unreserved. Use the next free number.
"I'll mv the experiment dir, the contents are mine anyway"	Even if the contents are yours, the act of mv damages reproducibility evidence. Stop, ask, propose a new dir.
"The new experiment is exp04 because exp03 is taken — but I already created exp03_*, so I'll just rename it"	NO. The wrong-named dir you created stays where it is until the user decides what to do with it. Create exp04 fresh. Don't rename.
"I'll delete my own wrong-named exp03 dir, since I created it this session"	Still touches a prior dir. The user has not authorized the delete. Ask.
"This isn't really ML, it's a simulation / benchmark / A-B test, so the rules don't apply"	They do. Any project where you run repeated parametric variants and compare results benefits from this isolation.

Red Flags — STOP and Reset

If you find yourself about to do any of these, STOP, surface the situation to the user, and propose a non-destructive alternative:

• mv experiments/expNN_* (rename of any prior experiment dir)
• rm -rf experiments/expNN_* (delete of any prior experiment dir)
• cp -r experiments/expAA experiments/expBB where expBB will overwrite
• Editing any file under a prior expNN_* dir for any reason

Decision Flow

User asks for experiment work
        │
        ▼
Tempted to rename, delete, or edit prior expNN/?
        │
        ├── yes ──▶ STOP. Never edit prior experiment.
        │
        └── no
              │
              ▼
       New experiment thread?
              │
              ├── yes ──▶ Find next unused number N
              │              │
              │              ▼
              │          Create experiments/expNN_<topic>/
              │
              └── no ──▶ Add variant to in-progress experiment?
                              │
                              └── yes ──▶ Use existing expNN_*/
                                              │
                                              ▼
                                          Create eNNx/ inside it
                                          (cell contents: see cell-source-isolation)

Downstream steps — what files go inside the cell, how the runner script works, and how to record results — belong to the sibling skills.

Why These Rules Exist

The rules look pedantic. They exist because every weakening creates a path to non-reproducible results.

• Read-only-prior rule: Once a cell has been run and its metrics recorded, the cell's source files are evidence. Editing them in place breaks the link between recorded numbers and the code that produced them.
• Rename-breaks-references rule: A "harmless" mv exp03 exp04 rewrites the file paths that experiments/NOTES.md, prior PR descriptions, log files, and reviewers' bookmarks were all relying on. The recorded numbers now point at a directory that no longer exists, and the git history shows a rename instead of a frozen prior run. Create the new dir fresh; leave the old one alone.

The rigidity is the point. It converts "an experiment broke and I don't know why" into "experiment N is fine, this new experiment M+1 is the one with the issue."

See also

• cell-source-isolation — what goes inside a cell once the directory exists
• lab-journal-discipline — what to record after a run