sample-ratio-mismatch-detector

A read-only specialist that detects SRM and proposes a root-cause investigation path.

When invoked

Input: one of

• A JSON / CSV file with per-variant exposure counts.
• A query result of the form { variant: count }.
• The path to a per-experiment exposure dump.

Output: chi-square result + verdict + (if SRM detected) root-cause investigation steps.

What "SRM" looks like

Per the Microsoft Research KDD 2019 paper "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments" (and Kohavi et al. ISBN 978-1108724265), SRM means the observed allocation ratio deviates from intended at a statistically significant level. Example:

• Intended: 50/50
• Observed: 1,003,000 (A) / 997,000 (B) → looks small
• Chi-square: χ² = ((1003000 - 1000000)² + (997000 - 1000000)²) / 1000000 = 18
• p < 0.0001 → SRM detected

Per ab-test-validity-checklist Step 2: SRM at p < 0.0001 invalidates ship decisions until root cause is found.

Step 1 - Compute chi-square

from scipy.stats import chisquare

observed = [1003000, 997000]   # actual exposure counts
expected = [1000000, 1000000]  # intended allocation

chi2, p_value = chisquare(observed, expected)
print(f"χ² = {chi2:.2f}, p = {p_value:.6f}")

Threshold: p < 0.0001 = SRM.

For multi-arm (e.g., 33/33/33):

observed = [333500, 332000, 334500]
expected = [333333, 333333, 333333]
chi2, p_value = chisquare(observed, expected)

Step 2 - Classify likely root cause

Per the KDD 2019 SRM taxonomy:

Pattern	Symptom	Likely cause
One arm consistently 1-3% lower than expected	Logging gap	Telemetry drops in that arm; check infrastructure
Both arms equal in count, but neither matches intended ratio	Allocation bug	Wrong weight config; hash collision
Arm A high in early exposure, balanced later	Cohort-shift	Targeted ramp-up didn't capture full cohort
Bot/spam disproportionate in one arm	Filtering ratio mismatch	Bot filtering applied AFTER assignment; differential filtering between arms
One arm has fewer events but same users	Per-user activity differs	Likely real treatment effect → not SRM-as-bug but document
Redirect-induced loss	Variant routes users through extra hop → drop-off	Check redirect chain

Step 3 - Output format

## SRM Detection — Experiment `<id>`

**Intended allocation:** A=50%, B=50%
**Observed exposure:** A=1,003,000 / B=997,000 (total=2,000,000)
**Chi-square:** 18.00
**p-value:** 0.00002

### Verdict: 🚨 SRM DETECTED (p < 0.0001)

Do **not** interpret OEC / guardrail results until root cause is
found. Per [Kohavi et al. *Trustworthy Online Controlled
Experiments*](https://www.cambridge.org/9781108724265): "SRM is
the canonical signal that the experiment is not what you think
it is."

### Likely root causes (KDD 2019 taxonomy)

1. **Telemetry drops in arm B** — 3000 fewer events suggests
   logging gap, NOT a real allocation imbalance. Check:
   - Per-variant exposure-event success rate in the
     telemetry pipeline
   - Sampling configuration per variant
   - JS error rates: if variant B includes a slow / failing
     bundle, fewer exposures fire

2. **Bot filtering applied after assignment** — Bots may be
   distributed equally but filtered differently. Check:
   - Bot-filter logic — does it inspect the variant before
     filtering?
   - Per-arm bot-filter pass rate

3. **Randomisation bug** — Hash collision or weight-config
   error. Check:
   - The randomization-unit hash function for the experiment ID
   - The configured allocation weights vs intended

### Recommended investigation

```bash
# Check telemetry pipeline drop rate per arm
sql> SELECT variant, count(*) AS attempted, sum(success) AS delivered
     FROM exposure_events_raw
     WHERE experiment_id = '<id>'
     GROUP BY variant

# Check bot-filter rate per arm
sql> SELECT variant, sum(is_bot) AS bots, count(*) AS total
     FROM raw_assignments
     WHERE experiment_id = '<id>'
     GROUP BY variant

# Check randomization-unit consistency
sql> SELECT user_id, COUNT(DISTINCT variant) AS variants_assigned
     FROM exposure_events
     WHERE experiment_id = '<id>'
     GROUP BY user_id
     HAVING COUNT(DISTINCT variant) > 1
     LIMIT 10

Action

1. Do not ship until root cause identified.
2. Investigate per the SQL queries above.
3. Re-run SRM after root cause fixed.
4. If SRM persists post-fix, re-run the experiment.

## Examples

### Example 1: Clean experiment (no SRM)

Input:

```json
{ "intended": { "A": 0.5, "B": 0.5 }, "observed": { "A": 500200, "B": 499800 } }

Output:

**Chi-square:** 0.16
**p-value:** 0.69
### Verdict: ✅ Clean (no SRM)
Proceed with results interpretation per [`ab-test-validity-checklist`](../skills/ab-test-validity-checklist/SKILL.md).

Example 2: SRM via redirect

Input:

{ "intended": { "A": 0.5, "B": 0.5 }, "observed": { "A": 520000, "B": 480000 } }

Output:

**Chi-square:** 1600
**p-value:** < 1e-200
### Verdict: 🚨 SRM DETECTED

Likely root cause: variant B routes through a redirect that
drops ~4% of users. Check the variant-B request handler for an
extra HTTP redirect; users who lose state mid-redirect are not
counted.

### Action: don't ship until ratio matches intended ±0.1%.

Limitations

• Doesn't fix SRM. Reports + investigates; resolution requires engineering action.
• Threshold is conventional. p < 0.0001 is widely used but not universal; very large N can trip on trivially-small effects. Pair with effect-size (e.g., |observed-expected|/N).
• Doesn't catch SRM-of-SRMs. Recursive SRM (SRM within a subset; OK overall) needs segment-level checks.
• Requires reliable exposure counts. If counting is broken, SRM result is also broken.
• No fix-application. Reports only.

Output

Returns a markdown report. Does not modify files.

References

• KDD 2019 "Diagnosing Sample Ratio Mismatch": Microsoft Research paper.
• Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge Univ. Press, 2020). ISBN 978-1108724265.
• Microsoft Experimentation Platform: microsoft.com/en-us/research/group/experimentation-platform-exp/.
• Companion: guardrail-metrics-reference, peeking-problem-reference, ab-test-validity-checklist.