a-b-test-design-pm

a-b-test-design-pm

Measure product performance and customer behavior to drive data-informed decisions.

Context

You are defining metrics or running experiments. If you have product data, usage patterns, or business goals, use them. Your goal is clear measurement frameworks.

Domain Context

• Metrics hierarchy: Strategic (north star) → operational → diagnostic
• Leading vs. lagging indicators: Early signals vs. business results
• Causation vs. correlation: Be careful about assuming cause
• Cohort analysis: Track specific user groups over time
• Experimentation velocity: Fast learning beats big bets

Instructions

1. Define success metrics: What measures success?
2. Identify leading indicators: What predicts future success?
3. Set targets: What's good performance?
4. Establish baseline: What's performance today?
5. Monitor regularly: Daily/weekly reviews to detect changes
6. Investigate changes: When metrics move, understand why

Output Artifact

Metrics framework or experiment plan including:

• Key metrics and definitions
• Baseline and targets
• Data collection method
• Review cadence
• Success criteria

Anti-Patterns

• Vanity metrics: Tracking volume without quality/retention
• Too many metrics: Tracking 50+ metrics dilutes focus
• No leading indicators: Only looking at lagging metrics
• Not investigating changes: Metrics move without understanding why

Further Reading

• Lean Analytics (Croll & Yoskovitz) — metrics frameworks
• Inspired (Marty Cagan) — product metrics