evidence-over-claims

Evidence Over Claims

The number doesn't exist until it's traceable. This skill is the firewall between analysis and communication.

What this skill catches

• LLM-narrated statistics. "Revenue rose around 12%" with no actual query backing — a plausible-sounding number that the model emitted without computation.
• Memory-laundered numbers. Numbers from earlier in the conversation that the LLM is now rounding or paraphrasing from memory rather than re-running the artifact.
• Lost-source numbers. Numbers from a query that was run, edited, re-run, and never saved — so the artifact behind the number no longer exists.
• Absorbed claims. Numbers a user mentioned in passing that have quietly been absorbed into the writeup as if they were findings the analysis produced.
• Fragility-stripped findings. Numbers that came out of the multiverse with a fragility flag, but the flag got dropped on the way to the writeup.

What this skill enforces

For every numerical claim that will appear in a writeup, summary, email, slide, dashboard, or message:

1. Identify the claim — the specific number and what it asserts.
2. Identify the artifact — the saved SQL file, the methodology DB row, the reconciled source — that produced it.
3. Verify it can be re-run — the artifact must produce the same number again, today, by anyone with appropriate access.
4. Cite it — even if informally, the writeup names where the number came from.

If you cannot identify the artifact, you cannot include the number. Find the artifact, or remove the claim.

Practical pattern

In the writeup, every number gets a small parenthetical or footnote pointing to the artifact: (see queries/2026-05-04-onboarding-cohort.sql) or (methodology DB prediction #847, reconciled to GL on 2026-05-03). For internal work, this is enough. For external work, the artifact gets bundled or linked.

For numbers produced by the methodology DB itself (predictions, reconciliations, calibrated ranges), the prediction-ID or reconciliation-ID is the most precise citation.

Logging

Log this skill's activation:

python3 .analytics/db.py log-skill \
    --skill evidence-over-claims \
    --question-slug <slug> \
    --completed

If a claim was about to be made and was removed because no artifact existed, that's worth a one-line note in the writeup notes — building the discipline of catching ghost-numbers before they leak is part of what this skill is for.

Boundary cases

"The number is universally known." No it isn't. Even "Q1 revenue was $X" should cite the source (the GL pull, the audited statement, the dashboard date). Universal-knowledge claims are the most likely to be slightly wrong, because no one ever re-checked them.

"The number is approximate — 'about 12%'." Approximations need an artifact too. The artifact behind "about 12%" is the precise query result; the writeup's job is to round, not to invent. If you can't trace the precise number, "about" is hiding a missing source.

"This is an exec summary; citations are noisy." Use footnotes or an appendix. The receipt doesn't have to interrupt the prose — it just has to exist somewhere a reader can follow it back to the artifact.

"The user wants the number now and the artifact will catch up." Don't. The number-without-artifact is exactly the failure mode this skill exists to prevent. Either save the artifact first, or hold the number.

Handoff

This is the last skill in the methodology flow. After it completes, the writeup is ready for human review and the methodology DB has a complete record of the analysis. Run python3 .analytics/db.py calibration periodically to see the calibration curve building up across resolved predictions.