executing-experiment-plans

Executing Experiment Plans

Overview

Load plan, review critically, execute every task, report when complete.

Announce at start: "Using datapowers:executing-experiment-plans to implement the plan."

Prefer subagents: If your harness supports subagents, use datapowers:subagent-driven-experimentation instead — quality is significantly higher.

The Process

Step 1: Load and Review Plan

1. Read the plan file from docs/datapowers/plans/.
2. Review critically — identify questions, missing context, contradictions with the spec.
3. If concerns: raise them with your human partner before starting.
4. If no concerns: create a TodoWrite (one item per task) and proceed.

Step 2: Execute Tasks

For each task:

1. Mark as in_progress.
2. Follow each step in order — they are bite-sized for a reason.
3. Run every verification command as written.
4. Log every model run via datapowers:experiment-tracking.
5. Commit at the cadence the plan specifies.
6. Mark as completed only after the verification command's output confirms success.

Step 3: Gate Checks

After each phase (data, features, modeling, validation):

• Was data leakage prevented? Re-check datapowers:preventing-data-leakage.
• Does the run beat the baseline? If not, stop and document.
• Are metrics reported with confidence intervals?

Step 4: Complete

After all tasks pass:

• Run datapowers:reproducibility-verification end-to-end in a clean environment.
• Run datapowers:writing-model-cards.
• Announce: "Using datapowers:finishing-an-experiment to close out."
• REQUIRED SUB-SKILL: datapowers:finishing-an-experiment.

When to Stop and Ask

STOP immediately when:

• Missing dependency or unavailable data source.
• Test fails for unexpected reason.
• Plan instruction is ambiguous.
• A verification command returns a value outside expected range.
• You hit a contradiction between plan and spec.

Ask for clarification — don't guess. Guessing in DS quietly corrupts results.

When to Revisit Earlier Steps

If a later task reveals an earlier assumption was wrong (e.g., your "stratify" was wrong, your label has leakage), STOP. Go back. Re-run from the first invalidated step. Don't patch downstream.

Common Failures

Failure	What it usually means	Fix
Test passes immediately	Test doesn't test what you think	Re-write test, watch it fail first
Metric far above baseline	Probably leakage	Re-check splits & feature timing
Metric below baseline	Bug or feature is harmful	Inspect, don't tune to mask
Cannot reproduce locally	Untracked env or data	Pin both before continuing

Verification Checklist

Before claiming the plan complete:

• Every plan checkbox ticked.
• Every verification command rerun in this session with passing output.
• Reproducibility verification passed end-to-end.
• Model card committed.
• All commits pushed.

Cross-References

• PREV: datapowers:writing-experiment-plans
• PREFERRED ALTERNATIVE: datapowers:subagent-driven-experimentation
• REQUIRED AT FINISH: datapowers:reproducibility-verification, datapowers:writing-model-cards, datapowers:finishing-an-experiment