Running Time Estimation

Why this matters

Knowing how long to run an A/B test prevents wasted time and misleading results. As a Data Analyst, you will be asked to:

• Estimate how many days/weeks are needed to detect a specific lift.
• Balance business urgency with statistical rigor.
• Plan traffic allocation across variants (A/B or A/B/n).
• Communicate realistic timelines to product, marketing, and engineering.

Concept explained simply

Runtime estimation answers two questions: how many samples do we need per variant (sample size), and how many calendar days will it take with our traffic?

• Baseline metric: your current rate or mean (e.g., 5% conversion).
• MDE (Minimum Detectable Effect): the smallest change you care to detect (absolute or relative).
• Alpha (significance): usually 0.05 (5%).
• Power: usually 0.8 (80%).
• Traffic inputs: daily eligible users, allocation (e.g., 50/50), and any filters (geo, device).

Mental model

Think of each variant as a bucket you must fill with enough users to confidently tell two nearby percentages apart. The rarer the event (low conversion) or the smaller the MDE, the more users you need in each bucket. Calendar time is just the rate at which users drip into each bucket.

Let p1 = baseline rate, p2 = target rate = p1 + delta_abs
Let p_bar = (p1 + p2) / 2
Let Z_alpha2 = 1.96  (for alpha = 0.05, two-sided)
Let Z_beta   = 0.84  (for power = 0.80)

Approximate sample size per variant:
  n_per_variant ≈ 2 * (Z_alpha2 + Z_beta)^2 * p_bar * (1 - p_bar) / (delta_abs)^2

Where delta_abs = p1 * relative_MDE if MDE is given as a % lift.

Let sigma = standard deviation, delta_abs = desired mean difference
n_per_variant ≈ 2 * (Z_alpha2 + Z_beta)^2 * sigma^2 / (delta_abs)^2

eligible_daily = daily_users * eligibility_fraction
per_variant_daily = eligible_daily * allocation_fraction
runtime_days = ceil(n_per_variant / per_variant_daily)

4-step process to estimate running time

1. Translate MDE: If relative, convert to absolute. Example: baseline 5%, +10% relative -> +0.5 pp absolute, so p2 = 5.5%.
2. Compute sample size: Use the approximate formula above to get n per variant.
3. Map to days: Divide by daily eligible traffic per variant; round up.
4. Add calendar safeguards: Run at least 1 full business cycle (often 1–2 weeks) to cover weekday/weekend and avoid novelty effects.

Worked examples

n_per_variant ≈ 2 * (2.8)^2 * 0.0525 * (1 - 0.0525) / (0.005)^2
              ≈ 31,152 users per variant (approx)

n_per_variant ≈ 22,800 (approx)
Exposure available per variant = 100,000
=> Enough in one send; runtime = one campaign send.

n_per_variant ≈ 2 * (2.8)^2 * 40^2 / 3^2
              ≈ 2,778 orders per variant (approx)
Per-variant daily orders = 800 * 0.5 = 400
Runtime = ceil(2,778 / 400) ≈ 7 days (minimum). Recommend 1–2 weeks.

Exercises (hands-on)

Tip: The quick test is available to everyone; only logged-in users will have their progress saved.

1. Exercise 1 (mirrors the task below):

   Estimate runtime for a signup conversion A/B test with: baseline 4%, MDE +15% relative, alpha 0.05, power 0.80; daily visitors 12,000; eligibility 60%; split 50/50. Compute: (a) absolute MDE, (b) sample size per variant, (c) minimum days, (d) recommended calendar run.

Self-check checklist

• I converted relative MDE to absolute percentage points.
• I used the correct Z-values for alpha and power (≈1.96 and ≈0.84).
• I accounted for eligibility and split when mapping to days.
• I added a full business cycle buffer (usually 1–2 weeks).

Common mistakes and how to self-check

• Using relative MDE directly as absolute. Fix: delta_abs = baseline * relative_MDE.
• Ignoring eligibility filters. Fix: multiply daily traffic by the fraction that actually enters the test.
• Stopping mid-week because sample size hit. Fix: cover full weekday/weekend cycles unless you pre-registered a sequential design.
• Changing split mid-test. Fix: keep allocations stable; re-estimate if you must change.
• Multiple variants without alpha control. Fix: plan comparisons and adjust alpha or use a valid multi-arm framework.
• Variance underestimation for continuous metrics. Fix: use recent data to estimate sigma conservatively.

Mini challenge

You have baseline conversion 3% and want to detect +12% relative. Daily visitors = 30,000, eligibility = 50%, split = 50/50. Alpha = 0.05, Power = 0.80.

• 1) Compute absolute MDE.
• 2) Approximate n per variant.
• 3) Minimum days and a reasonable recommended run.

Who this is for

• Data Analysts, Product Analysts, and Marketers planning A/B tests.
• Engineers and PMs who need quick, defensible test timelines.

Prerequisites

• Basic probability and understanding of conversion rates or means.
• Familiarity with alpha, power, and confidence intervals.

Learning path

1. Understand metrics (binary vs continuous).
2. Learn MDE, alpha, power, and sample size basics.
3. Practice runtime estimation with real traffic inputs.
4. Handle multi-variant and filtered traffic scenarios.
5. Apply calendar safeguards and communicate timelines.

Practical projects

• Build a simple spreadsheet that calculates n per variant and runtime given inputs.
• Create a one-pager template to share test timelines with stakeholders.
• Backtest: compare your estimates to historical experiments and explain discrepancies.

Next steps

• Estimate runtimes for your next 3 planned experiments with different MDEs.
• Standardize a minimum runtime rule (e.g., at least 14 days) for your team.
• Learn when sequential testing is appropriate for faster, controlled stopping.

Quick Test (progress note)

This quick test is available to everyone. Only logged-in users will have their progress saved.