Experiment Design

Why Experiment Design matters for Data Scientists

Experiment design turns business questions into measurable, causal evidence. As a Data Scientist, you will plan A/B tests, choose metrics that reflect value, ensure fair randomization, estimate the required sample size, and run trustworthy analyses that lead to decisions (ship, iterate, or stop). Doing this well accelerates product learning, reduces risk, and builds credibility with stakeholders.

Who this is for

• Data Scientists and Analysts shipping product changes via A/B tests.
• ML/AI practitioners validating model or policy impact (e.g., ranking, pricing).
• Product Managers and Engineers who partner on experiment decisions.

Prerequisites

• Comfort with descriptive statistics, confidence intervals, and p-values.
• Basic SQL for cohorting and metric calculation.
• Python or R for power analysis and statistical tests (optional but helpful).

Learning path

Worked examples

1) Hash-based random assignment in SQL

Ensure stable and reproducible assignment at the user level.

-- Pseudocode; adapt hash function to your warehouse
WITH base AS (
  SELECT user_id,
         ABS(MOD(HASH(user_id), 100)) AS bucket
  FROM users_daily
)
SELECT user_id,
       CASE WHEN bucket < 50 THEN 'treatment' ELSE 'control' END AS group
FROM base;

2) Sample size for a proportion lift (Python)

Compute required users per group to detect a +5% relative lift from 10% baseline.

# pip install statsmodels
from statsmodels.stats.power import NormalIndPower

power = NormalIndPower()
p1 = 0.10
lift = 0.05  # 5% relative lift
p2 = p1 * (1 + lift)
effect = p2 - p1
alpha = 0.05
power_target = 0.80

n_per_group = power.solve_power(effect_size=power.proportion_effectsize(p1, p2),
                                power=power_target, alpha=alpha, ratio=1.0,
                                alternative='two-sided')
print(round(n_per_group))

3) CUPED-style variance reduction (Python)

Use a pre-experiment covariate X (e.g., last week’s spend) to reduce variance on outcome Y.

import numpy as np
# y: outcome during experiment, x: pre-period covariate
# beta = cov(y, x) / var(x)
def cuped_adjust(y, x):
    x_centered = x - np.mean(x)
    beta = np.cov(y, x, ddof=1)[0,1] / np.var(x, ddof=1)
    y_cuped = y - beta * x_centered
    return y_cuped

# After adjustment, analyze group means of y_cuped as usual.

4) Guardrail threshold with auto-stop rule

Suppose page latency must not increase more than +2%. Define an alert:

IF (Latency_Treatment / Latency_Control) - 1 > 0.02 THEN
  FLAG = 'Stop and investigate';
END IF;

Document this rule in the analysis plan to avoid ad-hoc decisions under pressure.

5) Readout: effect, CI, decision

For a primary conversion metric:

# Example: difference in proportions with CI (Python)
import numpy as np
from statsmodels.stats.proportion import proportions_ztest, proportion_confint

succ = np.array([520, 480])  # [treatment, control]
n = np.array([5000, 5000])
stat, p = proportions_ztest(succ, n)
ci_low, ci_high = proportion_confint(succ[0], n[0], alpha=0.05, method='wilson')
print(p, ci_low, ci_high)
# Decision: if CI on uplift is above 0 and business threshold met, consider ship.

Drills and exercises

• Write H0/H1 for: “New search ranking increases add-to-cart rate by 3% relative.”
• Pick a primary, two secondary, and one guardrail metric for a mobile onboarding test.
• Choose the randomization unit for a courier-driver incentive change. Justify.
• Estimate MDE given traffic limits: what uplift can you detect in 14 days at 80% power?
• Create a seasonality-aware schedule for a checkout flow experiment.
• List two potential interference risks in a social feed experiment and mitigations.
• Design a stratified randomization by country; name strata and allocation.
• Draft an analysis plan: invariant checks, exclusions, test type, reporting template.
• Simulate peeking: explain why stopping early at a p-value dip is risky.
• Write a one-slide readout: effect, CI, risks, recommendation.

Common mistakes and debugging tips

• Too many primary metrics: Pick one. Others are secondary or guardrails.
• Unit mismatch: Randomize at the user level if effects spill across sessions.
• Underpowered tests: If traffic is low, increase duration, target a larger effect, or reduce variance (CUPED, stratification).
• Peeking: Sequential looks inflate Type I error; use fixed-horizon plans or proper sequential methods.
• Ignoring seasonality/novelty: Run for full cycles (e.g., weekly) and monitor time trends.
• Interference: Cluster randomize (e.g., by geography) or exclude spillover edges.
• Untracked changes: Freeze parallel launches on the tested surface; maintain an experiment calendar.
• Post-hoc metric fishing: Pre-register and mark exploratory analyses clearly.

Mini project: From idea to decision

Scenario: Your team proposes a new recommendation widget on product pages to increase purchases.

1. Hypothesis: Write H0/H1 with a relative MDE and a 95% confidence decision rule.
2. Metrics: Define primary (e.g., purchase conversion), two secondary (CTR, AOV), guardrails (latency, refund rate).
3. Unit & Randomization: Choose user-level assignment with hash-based bucketing; propose country stratification.
4. Power: Estimate sample size given baseline conversion 8%, target +4% relative, power 80%.
5. Risks: Identify interference (shared devices), novelty effects, overlapping homepage test; propose mitigations.
6. Analysis plan: List invariant checks, exclusion rules (bots, staff), test type, and reporting template.
7. Mock readout: Create a one-paragraph decision with effect, CI, and a recommended rollout plan.

Subskills

• Hypothesis Framing: Turn product ideas into testable statements with decision rules.
• Defining Primary And Guardrail Metrics: Pick one decision-driving metric and safety thresholds.
• Randomization And Unit Selection: Choose the level and method that avoid bias and spillover.
• Power And Sample Size Basics: Compute how big and how long your test needs to run.
• Experiment Duration And Seasonality: Plan around cycles and novelty effects.
• Handling Interference And Spillover: Detect and mitigate cross-unit effects.
• Multiple Experiments And Interaction Risks: Manage overlap and interaction effects.
• Analysis Plan And Readout: Pre-register, analyze, and communicate results clearly.
• Interpreting Results For Decisions: Translate stats to product choices responsibly.

Next steps

• Pick a real product surface and draft a one-page experiment brief using this guide.
• Practice sample size calculations for three different baselines and MDEs.
• Shadow or conduct a live readout; ask stakeholders to challenge your decision rule.
• Then explore advanced topics: sequential testing, cluster designs, and heterogeneity analysis.