Running Time And Sample Size Basics

Why this matters

As a Marketing Analyst, you will plan and judge A/B tests for landing pages, emails, pricing, creatives, and funnels. Choosing the right sample size and run time prevents two costly outcomes: stopping too early (false winners) and running too long (wasting traffic and time). Mastering these basics helps you ship reliable wins faster.

• Estimate how many users you need before starting.
• Plan how long to run a test based on traffic.
• Avoid peeking mistakes and mid-week bias.

Concept explained simply

You want to detect whether Variant B truly performs differently from Variant A. Two practical decisions control this:

• How small a change you care to detect (Minimum Detectable Effect, MDE).
• How confident you want to be (significance level, alpha) and how often you want to catch true effects (power).

Inputs you typically choose:

• Baseline metric: e.g., conversion rate (proportions) or average order value (means) and its variability.
• MDE: smallest effect size that matters (absolute or relative).
• Alpha: commonly 0.05 (two-sided) for marketing tests.
• Power: commonly 0.8 (80%) or 0.9 (90%).
• Allocation: often 50/50 for two variants to finish faster.
• Daily traffic or events to estimate calendar days.

Mental model: a resolution dial

Think of MDE as a resolution dial. The smaller the change you want to see, the more data you need. Turn the dial down (tiny effects): tests take longer. Turn it up (bigger effects): tests finish faster.

• Smaller MDE → larger sample → longer run.
• Higher traffic or more events/day → shorter calendar time.
• More variability (noisier metric) → larger sample.
• Two-sided tests are safer defaults and usually require a bit more data than one-sided.

Quick formulas you can use

These are standard approximations to plan before you test. They are close enough for scoping and prioritization. For precise planning, use a stats calculator with the same inputs.

Worked examples

Example 1: Landing page conversion uplift

Goal: Detect a 10% relative uplift, baseline 5% → target 5.5%.

• Inputs: p1 = 0.05, p2 = 0.055, δ = 0.005, p̄ = 0.0525
• α = 0.05 (two-sided), power = 0.8 → Zα/2 = 1.96, Zβ = 0.84

Runtime with 10,000 visitors/day, 50/50 split → 5,000 per variant/day → ~6–7 days. Good practice: run at least 1–2 full weeks to cover weekdays/weekends.

Example 2: Email open rate

Goal: Detect +2 percentage points, baseline 20% → 22%.

• Inputs: p1 = 0.20, p2 = 0.22, δ = 0.02, p̄ = 0.21
• α = 0.05 (two-sided), power = 0.9 → Zα/2 = 1.96, Zβ = 1.28

If your list has 100,000 recipients split 50/50, one send can be enough.

Example 3: Average order value (means)

Goal: Detect +$3 AOV, baseline SD ≈ $20.

• Inputs: σ = 20, δ = 3
• α = 0.05 (two-sided), power = 0.8 → Zα/2 = 1.96, Zβ = 0.84

With ~300 orders/day total (150 per variant), you need ~5 days of orders; run for at least one full week.

How to plan runtime safely

1. Always pre-set MDE, alpha, power, and guardrails.
2. Compute n per variant; estimate days from your traffic.
3. Run for whole business cycles (commonly ≥ 1–2 full weeks).
4. Avoid peeking to stop at first p < 0.05; wait until sample-size and minimum duration are both met.
5. Prefer 50/50 allocation for speed unless there are strong risk reasons.

Common mistakes and how to self-check

• Stopping early on a spike: Self-check by confirming both sample-size target and minimum calendar duration are met.
• Choosing an unrealistically tiny MDE: Sanity-check against expected business impact and traffic; if runtime is months, increase MDE or choose a higher-impact change.
• Using visitor counts when the unit is sessions or orders: Match sample to the unit your metric is computed on.
• Ignoring variance for means tests: You need an SD estimate; use historical data.
• Not covering full weeks: Seasonal bias can flip results. Include at least one full weekday-weekend cycle.

Exercises

Try these. Then compare with the solutions.

Checklist before you launch

• MDE chosen and meaningful for the business.
• Alpha and power set (e.g., 0.05 and 0.8).
• Baseline and, if needed, SD estimated from recent data.
• Per-variant sample size computed.
• Planned start and end dates cover at least one full weekly cycle.
• Allocation set (often 50/50) and tracking QA done.

Mini challenge

You have 12,000 daily sessions, baseline add-to-cart rate 8%. You care about a 1 percentage point absolute lift. Alpha 0.05, power 0.8. Roughly estimate per-variant n and how many days you would run at 50/50. Then list two guardrails you would monitor. Compare to the worked examples to sanity-check your answer.

Who this is for

• Marketing Analysts planning or reviewing A/B tests.
• Growth, CRM, and Product Marketers who need reliable experiment timelines.
• Designers/PMs collaborating on test roadmaps.

Prerequisites

• Basic statistics: proportions, means, standard deviation.
• Comfort with metrics you test (e.g., conversion rate, AOV, CTR).
• Access to recent baseline data for your metric.

Learning path

1. Start here: MDE, alpha, power, baseline, and time-to-sample.
2. Next: Test design choices (allocation, tail direction, guardrails).
3. Then: Interpreting results (confidence intervals, lift, uncertainty).
4. Advanced: Sequential testing and multiple comparisons control.

Practical projects

1. Build a sample-size sheet: Create a spreadsheet with inputs (baseline, MDE, alpha, power, SD) and outputs (per-variant n, runtime). Include both proportions and means.
2. Backtest a past experiment: Using historical data, recompute the needed sample size and check if the original run met it. Summarize risks if it did not.
3. Traffic budgeting: For your next quarter’s test ideas, estimate n and calendar days for each. Prioritize by impact and feasibility.

Next steps

• Apply these formulas to your next planned test and compare with a calculator.
• Add minimum duration rules to your team playbook (e.g., ≥ 1–2 full weeks).
• Track a small set of guardrails (traffic balance, bot filtering, tracking health).

Take the quick test

Available to everyone for free. Progress is saved only for logged-in users.