Why this matters
As a Product Analyst, you will often be asked: “How long will this A/B test run?” or “Can we detect a +2% lift?” Getting sample size and MDE right prevents underpowered tests, saves calendar time, and avoids misleading decisions.
- Estimate test duration given traffic and target effect.
- Negotiate realistic expectations with product managers and designers.
- Choose an MDE that matters to the business and is feasible with available traffic.
- Document assumptions (alpha, power, baseline, variance) so results are defensible.
Concept explained simply
Minimum Detectable Effect (MDE) is the smallest true improvement (or decline) you want the test to reliably detect. If the real effect is at least the MDE, your test should catch it with the chosen power.
- Alpha (α): probability of a false positive (commonly 0.05 for two-sided tests).
- Power (1−β): probability to detect a true effect of size MDE (commonly 0.8).
- Sample size: how many observations per variant you need.
Core mental model: the trade-off triangle
With alpha and power fixed, three things are tightly linked:
- MDE ↔ Sample size ↔ Duration
Make MDE smaller → required sample goes up → duration goes up. Increase traffic → duration goes down for the same sample.
Quick formulas (80% power, α=0.05, two-sided)
Binary metric (e.g., conversion). Let baseline rate ≈ p and absolute MDE = d (in proportion points, not percentage). Approximate per-variant sample:
n ≈ 16 × p(1−p) / d²
Continuous metric (mean with known or well-estimated σ). Per-variant sample:
n ≈ 2 × (1.96 + 0.84)² × σ² / d² = 2 × 7.84 × σ² / d²
Duration (days) ≈ n_per_variant / daily_traffic_per_variant.
Worked examples
Example 1 — Binary metric, relative ask turned into absolute MDE
- Goal: Detect +10% relative lift from 5.0% conversion.
- Baseline p = 0.05 → target p₁ = 0.05 × 1.10 = 0.055 → absolute MDE d = 0.055 − 0.05 = 0.005 (0.5 pp).
- Sample per variant: n ≈ 16 × 0.05 × 0.95 / 0.005² = 16 × 0.0475 / 0.000025 ≈ 30,400.
- If you have 10,000 daily users split 50/50 → 5,000 per variant/day → ≈ 30,400/5,000 ≈ 6.1 days → plan ~7 days to cover weekly cycles.
Example 2 — Continuous metric (AOV)
- Baseline AOV unknown; sample data suggests σ ≈ $40.
- Want to detect +$2 (d = 2).
- n ≈ 2 × 7.84 × 40² / 2² = 2 × 7.84 × 1600 / 4 = 2 × 7.84 × 400 ≈ 6,272 per variant.
Example 3 — MDE and duration trade-off
- Binary metric, p = 0.2. Option A: d = 0.02 (2 pp). n ≈ 16 × 0.2 × 0.8 / 0.02² = 16 × 0.16 / 0.0004 = 6,400 per variant.
- Option B: d = 0.01 (1 pp). n ≈ 16 × 0.16 / 0.0001 = 25,600 per variant.
- Halving MDE quadruples the sample and duration (if traffic is unchanged).
How to choose an MDE that matters
- Business relevance: Convert MDE into expected revenue or user impact. If a 0.2 pp lift is negligible, don’t optimize for it.
- Feasibility: Check duration. If too long, increase MDE or focus on higher-traffic surfaces.
- Risk appetite: Teams with higher risk tolerance may choose larger MDE to move faster.
Tip: Convert relative to absolute MDE
Absolute MDE d = p₀ × relative_lift. Example: 8% baseline, ask for +5% relative → d = 0.08 × 0.05 = 0.004 (0.4 pp).
Step-by-step planning (mini-cards)
- Define the metric: binary or continuous? What is the baseline rate or variance?
- Fix α and power: typically α=0.05 (two-sided), power=0.8.
- Pick MDE: start with business-relevant value; sanity-check duration.
- Compute sample: use quick formulas above.
- Estimate duration: divide by per-variant daily traffic; add buffer for seasonality and data quality filters.
- Document: write assumptions and decisions so teammates can review.
Exercises you can do now
Do these before running the quick test. Aim to get them right without a calculator first, then verify.
Exercise 1 (mirrors ex1)
Baseline signup = 12%. Detect +1.2 pp absolute (from 12% to 13.2%). α=0.05, power=0.8. Approximate per-variant sample and 50/50 duration if you have 12,000 total daily visitors.
- Show: n per variant
- Duration: days to complete
Exercise 2 (mirrors ex2)
You can run for 5 days with 8,000 total daily visitors (50/50 split). Baseline conversion p=6%. What absolute MDE (in pp) is feasible at α=0.05, power=0.8?
- Show: d (pp)
- Interpret what this means in plain language.
Self-check checklist
- Converted relative lift to absolute pp when needed.
- Used per-variant traffic for duration, not total traffic.
- Matched binary vs continuous formula to the metric type.
- Rounded days up; planned buffer for weekly cycles.
Common mistakes and how to self-check
- Using total traffic for duration instead of per-variant. Self-check: divide by half the traffic if 50/50 split.
- Confusing relative with absolute effects. Self-check: always compute pp difference.
- Changing α or power mid-test to “finish faster.” Self-check: finalize parameters before launch.
- Underestimating variance for continuous metrics. Self-check: use historical σ or a pilot sample.
- Stopping when results look significant before reaching sample size. Self-check: respect the planned sample unless using proper sequential methods.
Practical projects
- Build a simple calculator in a spreadsheet: inputs (p, d, α, power, traffic) → outputs (n, duration).
- Create a one-pager template that documents test assumptions and MDE rationale.
- Analyze three recent tests: back-calculate their implied MDE and discuss if it matched business value.
Who this is for
- Product Analysts planning or reviewing A/B tests.
- PMs and Designers wanting quick, defensible duration estimates.
- Data Scientists needing a fast, practical refresher.
Prerequisites
- Comfort with percentages, proportions, and basic variance concepts.
- Awareness of A/B test structure (control vs variant).
Learning path
- Review A/B test basics: variants, primary metric, success criteria.
- Learn MDE and sample size trade-offs (this lesson).
- Practice with binary vs continuous formulas using historical baselines.
- Add realism: filters, data latency, and weekly seasonality buffers.
- Advance further: variance reduction and sequential designs (later lessons).
Mini challenge
You have 20,000 total daily visitors, baseline conversion 3%. You need the test done within 7 days. What is a realistic absolute MDE at α=0.05, power=0.8? Show your calculation and a one-sentence business interpretation.
Next steps
- Apply these formulas to your next planned experiment; share the plan with your team.
- Try the quick test below to confirm understanding.
- Move on to variance reduction and sequential testing once you’re comfortable here.
Ready for the Quick Test?
The quick test is available to everyone. If you are logged in, your progress will be saved automatically.