analytical-planning

Analytical Planning

The plan is the bridge between the question and the query. Choices made in the plan are choices that can be audited, challenged, and revisited. Choices made silently inside SQL cannot.

What this skill catches

• Silent segmentation drift. Picking segments after seeing the data, then presenting the segmentation as if it were obvious from the start.
• Window cherry-picking. Choosing a comparison window that maximizes lift without naming the alternatives that were available.
• Missing validation. Running the headline query first, finding a number that looks right, and never checking the row counts, NULL rates, and date coverage that would expose a join bug or a partial period.
• Confound surprise. "Discovering" a confound after writing the conclusion, when a five-minute up-front list would have surfaced it.

What the plan must contain

1. Population definition (in SQL-able terms) — exact filters that select the rows the analysis will operate on. Filters belong here, not in the head of the analyst.
2. Time window — start, end, calendar boundaries, timezone explicit, and a one-sentence rationale for choosing this window over plausible alternatives (full prior month vs. trailing 30 days; comparable-period-prior-year vs. immediately-prior-period).
3. Metric construction — exact formula, with each input column named. If the metric is a ratio, name numerator and denominator and how each handles NULLs.
4. Segmentation — which segments will be reported, and which were considered but excluded (and why). Pre-specified segments get audit protection; post-hoc segments do not.
5. Comparison structure — what is being compared to what. Name the baseline explicitly. ("Treated cohort vs. matched-pre-period self-baseline" is different from "treated cohort vs. concurrent untreated cohort.")
6. Validation queries — the sanity-check queries that must pass before the headline query is trusted. Common ones: row counts at each filter step, NULL % on key columns, sum totals against a known reference, distinct-key counts, date-coverage histogram. Write these before the headline query, not after.
7. Confounds named up front — list of things you already suspect could mess this up. This list seeds confound-audit. Better to write down "could be confounded by concurrent campaign X" before you query than to discover it after.
8. Pre-specified rules — anything you commit to before seeing data: outlier handling rule, missing-data treatment, what counts as "meaningful." These are protected from confirmation drift.

The plan is the contract

Once the plan is written, deviations from it get logged in multiverse-analysis as path decisions. Pre-specified choices are protected from confirmation drift; post-hoc choices get tagged and disclosed in the writeup.

Logging

Save the plan to .analytics/plans/YYYY-MM-DD-<slug>.md.

Log this skill's activation:

python3 .analytics/db.py log-skill \
    --skill analytical-planning \
    --question-slug <slug> \
    --completed

For each major pre-specified choice in the plan (segmentation, window, baseline, outlier rule, missing-data treatment), log a path decision flagged as pre-specified:

python3 .analytics/db.py log-path \
    --question-slug <slug> \
    --decision-type "<segmentation | window | baseline | outlier-rule | missing-data>" \
    --chosen "<the chosen value>" \
    --alternatives "<alt 1>" "<alt 2>" \
    --rationale "<one sentence>" \
    --pre-specified

The --pre-specified flag matters: choices made before data are protected; choices made after are subject to multiverse scrutiny.

Boundary cases

"I'll figure out the segmentation after I see the data." That's allowed, but those segments are post-hoc and must be logged that way. They cannot be presented in the writeup as if they were planned.

"The plan feels like overhead for a small analysis." Match plan depth to stakes. Low-stakes: a paragraph. Medium: the eight items above, briefly. High: the eight items above, with each rationale defended.

"What if the plan turns out to be wrong?" That's the point. A wrong plan is recoverable — you log the deviation, explain why, and the audit trail stays intact. A silent deviation is not recoverable; you can't tell, weeks later, whether the choice was principled or convenient.

Handoff

Plan saved at .analytics/plans/YYYY-MM-DD-<slug>.md. Hand off to predict-then-query/SKILL.md.