systematic-product-debugging

Systematic Product Debugging

Hard Gate

NO HYPOTHESES UNTIL FOUR QUESTIONS ANSWERED:

1. When did it start? (exact date or window)
2. Which user segment is affected?
3. Which funnel step broke?
4. What confounding events occurred?

Process

digraph debug {
    trigger [label="Trigger: 'метрика упала' / 'metric dropped'", shape=box];
    q1 [label="Q1: When did it start?", shape=diamond];
    q2 [label="Q2: Which segment?", shape=diamond];
    q3 [label="Q3: Which funnel step?", shape=diamond];
    q4 [label="Q4: Confounding events?", shape=diamond];
    block [label="BLOCK: answer this question first.\nNo hypotheses yet.", shape=box, style=filled, fillcolor="#ffcccc"];
    hyp [label="PASS → propose ranked hypotheses\n(most likely first)", shape=box, style=filled, fillcolor="#ccffcc"];

    trigger -> q1;
    q1 -> block [label="unknown / skipped"];
    q1 -> q2 [label="known"];
    q2 -> block [label="skipped"];
    q2 -> q3 [label="known"];
    q3 -> block [label="skipped"];
    q3 -> q4 [label="known"];
    q4 -> block [label="skipped"];
    q4 -> hyp [label="all answered"];
}

The Four Questions (must answer in order)

Q1: When did it start?

• Exact date if known, or window ("between April 10–15")
• If unknown: "Look at the time-series chart. When does the line bend?"
• Do not accept "recently" or "lately"

Q2: Which segment is affected?

• New users vs returning? Specific geography? Device type? Tariff? Acquisition channel?
• Do not accept "all users" without checking — the first instinct is almost always wrong
• Narrow the segment before hypothesizing

Q3: Which funnel step broke?

• Which specific step shows the drop?
• Do not accept "conversion dropped" — conversion of what specific step?
• If unknown: "Where in the user journey does the number go wrong?"

Q4: Confounding events

• Code deployments (yours)
• Marketing campaigns starting or stopping
• Seasonal patterns (holiday, month-end, etc.)
• Competitor actions (price change, new product)
• Infrastructure incidents, third-party outages

Only after all four: propose hypotheses, ranked most-likely first.

For each hypothesis, state:

• What data would confirm it?
• What data would refute it?

Checklist

• Q1 — When did it start: ___
• Q2 — Which segment: ___
• Q3 — Which funnel step: ___
• Q4 — Confounding events: ___
• → Ranked hypotheses (most likely first)
• → For each: confirming data / refuting data

Red Flags

Thought	Reality
"понятно что это конкуренты"	Первая гипотеза редко верна. Сначала 4 вопроса.
"скорее всего сезонность"	Сезонность — последняя гипотеза, не первая.
"надо срочно что-то делать"	Срочность приводит к лечению симптома. Диагностика быстрее.
"всегда так было"	"Всегда" не объясняет, почему сейчас хуже.
"это очевидно"	Если очевидно — 4 вопроса займут 5 минут.
"надо переделать X"	Решение без диагностики — трата ресурсов.

When to Skip

• P0 outage where immediate rollback is the only action — run debug after stabilization
• Explicitly: "skip systematic-product-debugging"