flake-triage

Flake Triage

Decide whether a spec is genuinely flaky, broken, or hit by a regression. Input $ARGUMENTS is the spec name (file path or test title).

Requires browserstack MCP to pull recent runs — see qa-mobile-pack/mcp/README.md. If unavailable, ask the user to paste the last 10 run summaries (status + failed step + timestamp).

Procedure

1. Fetch the last 10 runs of $ARGUMENTS via browserstack MCP, sorted newest-first. For each run capture: status (pass/fail), failed_step, timestamp, app_version or release tag.
2. Compute:
   • pass_rate = passes / 10
   • unique_failure_steps = count of distinct failed_step values across the failing runs
   • recent_pass_rate = passes in runs 1–5 (most recent)
   • older_pass_rate = passes in runs 6–10
3. Apply the decision tree in order — first match wins:
   • recent_pass_rate < 0.2 AND older_pass_rate > 0.9 → locator-broke (regression after a release)
   • pass_rate < 0.5 AND unique_failure_steps == 1 → deterministic-flake (same step every fail)
   • pass_rate < 0.5 AND unique_failure_steps >= 2 → intermittent (different steps each fail)
   • Otherwise → stable (do not classify as flake)
4. Pick the recommended action from this map:
   • locator-broke → fix locator; check the latest release diff for the failing screen
   • deterministic-flake → investigate timing or data setup at the failing step
   • intermittent → file a ticket with the run links; consider retry policy; do not skip yet
   • stable → no action

Output template

**Spec:** $ARGUMENTS
**Pass rate (last 10):** N/10  (recent 5: N/5, older 5: N/5)
**Unique failure steps:** N

**Classification:** `<intermittent | deterministic-flake | locator-broke | stable>`

**Recommended action:** <action>

**Evidence:**
- <run timestamp> — <status> — <failed_step or "-">
- ... (up to 10)