analytical-self-review

Analytical Self-Review

The last gate before findings reach a decision-maker. This is the pass that catches pitfalls the earlier skills don't.

What this skill catches

• Aggregation traps. A directionally-correct overall result that reverses when broken out by an obvious subgroup (Simpson's paradox).
• Context-stripped percentages. "Up 50%" without the base rate, where 0.2% → 0.3% looks identical to 20% → 30%.
• Outcome leakage. Predictor information sourced from the outcome window, common when cohort membership is defined using post-treatment events.
• Question drift. The analysis answered a question close to but not the same as the one originally framed, and the writeup is being drafted as if it answered the original.
• Falsifying conditions that triggered and got ignored. "What would change my mind" was written down at framing time; the actual results meet a falsifying condition; the writeup is drafting the conclusion anyway.

The checklist

Simpson's paradox

Does the aggregate result reverse when broken out by a relevant subgroup? Always check at least one obvious segmentation (cohort tenure, customer segment, geography, time period, prior level of activity).

Base-rate fallacy

Are you reporting a relative or absolute change without context for the underlying base rate? "Conversion up 50%" from 0.2% to 0.3% is different from 20% to 30%.

Leakage

Did information from the outcome window contaminate the predictor window? Most common in cohort analyses where membership is defined using post-treatment data.

Survivorship bias

Are you comparing units that stayed to units that churned? Are dropped accounts represented?

Ecological fallacy

Are you inferring individual behavior from aggregate data, or vice versa?

P-hacking / forking paths

Has the multiverse analysis flagged this as fragile? If so, the writeup must lead with the fragility, not bury it.

Compare against the falsifying conditions

Pull up the "what would change my mind" statement from the question doc. Walk through each falsifying condition. Did any of them trigger? If yes, the conclusion needs to update.

Falsification queries

For the headline claim, write and run at least one query that would disprove the claim if it were false. (E.g., for "the new feature drives more conversions": run a query that would show no conversion difference if the feature were inert. Did it return what the conclusion predicts?)

The drift check

Re-read the original question and the analytical plan. Has the analysis answered the question that was framed, or has the question silently changed during the work? If the question changed, that is allowed — but the writeup must say so explicitly. ("During analysis, we expanded scope from new-customers-acquired-in-May to all customers active in May, because [reason].") Silent question-drift is one of the most common ways analyses end up answering the wrong thing.

Logging

Log this skill's activation:

python3 .analytics/db.py log-skill \
    --skill analytical-self-review \
    --question-slug <slug> \
    --completed

If the drift check, Simpson check, or falsification query surfaces a finding that changes the conclusion, log it as a path decision so the writeup reflects it:

python3 .analytics/db.py log-path \
    --question-slug <slug> \
    --decision-type "self-review-finding" \
    --chosen "<what the conclusion now says>" \
    --rationale "<which check fired and what it found>" \
    --fragility-note "conclusion updated post-results"

Boundary cases

"All checks pass cleanly." Possible. But before signing off, ask: did I work through each check, or did I scan the list and self-assert? Run at least one falsification query — if you skipped it, the review wasn't done.

"A check failed but I have an explanation." Document the explanation. The conclusion may still hold, but the writeup names the check that failed and the explanation that resolved it. "Simpson reverses, but the subgroup driving the reversal is too small to matter" is fine; "Simpson reverses but I didn't think it was important" is not.

"The analysis is small / ad-hoc." Light review still applies — at minimum, run the drift check and the falsifying-condition comparison. Those are the two most common ways even small analyses go wrong.

Handoff

After self-review passes (with any updates logged), hand off to evidence-over-claims/SKILL.md.