confound-audit

Confound Audit

A number is not a finding. A finding is a number plus a defensible argument that the number means what you think it means. Confound audit is the work between the two.

What this skill catches

• Premature causal claims. Treating an observed difference as caused by the intervention without ruling out alternative explanations that could produce the same number.
• Confounds you knew about but waved off. A confound was suspected up front; the headline result was strong; the audit got skipped because "the number speaks for itself." Numbers don't speak for themselves.
• Confounds you didn't know about. Domain-specific gotchas that the audit surfaces — and once surfaced, get logged to the confound library so the next analysis has them as a seed.
• Asymmetric scrutiny. Applying confound checks rigorously when the result is unwelcome, and waving them off when the result is convenient. The audit is required either way.

What this skill enforces

For each confound in your analytical-planning confound list (and any new ones surfaced):

1. State the confound — what alternative explanation could produce the observed result?
2. Assess plausibility — is this confound likely or unlikely to be operative here, given what you know?
3. Test it where feasible — can you query, segment, or compare to rule it in or out?
4. Document the result — confound ruled out (with evidence), plausible but unaddressable (with caveat), or confirmed (rethink the conclusion).

Confounds to consider by default — extend this list domain-by-domain as you learn:

• Selection effects — which units ended up in this cohort, and was the selection mechanism related to the outcome?
• Calendar effects — week-of-month, month-of-year, day-of-week, holiday proximity, fiscal-period boundaries.
• Concurrent interventions — was another campaign, feature flag, pricing change, A/B test, or experiment running in this window?
• ETL / system changes — did the way the data is recorded change between baseline and treatment windows?
• Definition or rule changes — did metric definitions, qualification rules, segmentation thresholds, or business policies change mid-window?
• Cross-program interaction — was the cohort receiving differential treatment from a separate program (loyalty, discount, retention, support) during the comparison window?
• Frequency-vs-intensity — is the per-unit metric confounded by changes in observation frequency (more sessions per user vs. more action per session)?
• Survival / attrition bias — are you comparing units that stayed to units that churned? Are dropped accounts represented?
• Regression to the mean — was the cohort selected on extreme values that would naturally regress regardless of intervention?
• Novelty / Hawthorne effects — does the observed effect reflect the treatment itself, or the fact that something visibly changed?

The confound library

When a confound is identified, named, and tested in any analysis, log it to the methodology DB. The library accumulates a domain-specific confound vocabulary over time, and is queried at the start of future audits as a seed list. This is one of the framework's compounding assets — the longer the practice runs, the sharper the seed list gets.

Logging

Log this skill's activation:

python3 .analytics/db.py log-skill \
    --skill confound-audit \
    --question-slug <slug> \
    --completed

For each confound considered, log to the confound library:

python3 .analytics/db.py log-confound \
    --domain "<your domain — e.g., 'product-analytics', 'marketing-attribution', 'finance'>" \
    --name "<short name, e.g., 'concurrent-experiment'>" \
    --description "<one-paragraph description>" \
    --detection "<how to test for it>" \
    --mitigation "<how to handle it when present>"

If a confound is repeated across analyses, the CLI increments times_observed and updates last_observed. The same library row gets reinforced over time — that's the learning loop.

Boundary cases

"All confounds were ruled out cleanly." Possible, but rare. Walk through the list one more time and ask whether you ruled them out by evidence or by assertion. Audits where every confound dies on contact tend to be audits that didn't really happen.

"The confound is plausible but I can't test it." That's allowed — but the writeup names the unaddressable confound as a caveat on the conclusion. Don't let "can't test" become "didn't test."

"This is a small ad-hoc analysis." Ad-hoc analyses get a lighter audit (top three confounds, briefly), but they don't get zero. The confound list is what makes "directional" honest rather than just unverified.

Handoff

After audit completes and confounds are logged, hand off to multiverse-analysis/SKILL.md.