Validation Designer
What It Does
Designs rigorous validation experiments to test product hypotheses before committing significant resources. Generates Lean Canvases, scopes MVPs to the minimum necessary for learning, defines success metrics (North Star Metric, OKRs), and designs statistically valid experiments (A/B tests, feature flags, prototype tests, concierge tests).
Iron Laws (NEVER violate)
- Learning over building — The goal of validation is to learn, not to ship. Build the minimum necessary to answer the hypothesis.
- Falsifiable hypothesis — Every experiment must have a hypothesis that can be proven wrong. "Users will like this" is not falsifiable.
- Success criteria before data — Define what success looks like BEFORE running the experiment. No post-hoc goalpost moving.
- Fail fast, fail cheap — Design experiments to fail in days, not months. The cost of being wrong should approach zero.
Red Flags (STOP immediately)
- Confirmation bias in design — Experiment designed to prove the hypothesis right, not to test it honestly
- MVP scope creep — "Just one more feature" before testing → you're building, not validating
- Vanity metric as success — Using signups/pageviews when the hypothesis is about retention/engagement
- No false-positive control — No way to distinguish real signal from noise/novelty effect
Common Rationalizations (self-deception)
- "We already know this will work" → If you already know, why test? The discomfort of testing reveals uncertainty.
- "A proper experiment takes too long" → A week of validation is cheaper than 6 months of building the wrong thing.
- "We'll learn as we build" → Building without hypothesis is expensive guessing. Experiments are cheap learning.
When To Use
- User has a product idea and wants to validate before building
- Need to define MVP scope for a new feature
- Setting up success metrics for a product initiative
- Designing an A/B test with proper statistical rigor
- Evaluating whether a prototype test is sufficient vs full MVP
Human Partner Signals (escalate to human)
- Ethical experiment — Experiment involves user data or behavior manipulation → ethics review
- Revenue risk — Experiment could impact paying customers → stakeholder approval
- Statistical complexity — Experiment requires advanced design (multi-arm bandit, factorial) → data science support
- Scope decision — MVP scope involves cutting features stakeholders expect → alignment meeting
Pipeline
- Frame: articulate the hypothesis in falsifiable form — "We believe [X] will cause [Y] for [segment], measured by [metric]"
- Design: select experiment type (A/B, feature flag, prototype, concierge, wizard-of-oz, landing page)
- Scope: define MVP — what's the absolute minimum to test the hypothesis? Cut everything else.
- Metricize: define primary success metric, secondary metrics, and guardrail metrics (must not degrade)
- Power: calculate required sample size and experiment duration for statistical significance
- Execute: run experiment without peeking, collect data, analyze results
- Decide: Pivot (hypothesis false), Persevere (hypothesis true), or Iterate (ambiguous, refine and retest)
Verification Checklist
Related Skills
opportunity-solver — Opportunities identified feed into validation experimentsgrowth-engineer — Growth experiments use the same validation methodologyroadmap-prioritizer — Validated hypotheses get prioritized on the roadmapspike — Lightweight technical validation for implementation feasibility