Why this matters
As a Data Analyst, you will routinely present experiment results, translate them into business impact, and recommend whether to ship, iterate, or stop. Stakeholders expect a crisp, trustworthy readout, not just p-values. Strong readouts help teams avoid shipping harmful changes, catch false wins, and move faster with confidence.
- Decide if the variant is better, worse, or inconclusive.
- Explain impact (e.g., added orders, revenue) and risks (guardrail breaches).
- Recommend next steps supported by data and clear reasoning.
Concept explained simply
An experiment readout is your structured summary of what changed, by how much, how sure you are, and what to do next. Decision making balances statistical evidence, business impact, and risk controls.
Mental model
- Question: Did the variant move the primary metric in the desired direction?
- Magnitude: By how much (absolute and relative)?
- Confidence: Is the effect statistically and practically significant?
- Safety: Did guardrails stay within limits?
- Decision: Ship, iterate, or hold for more data.
Key terms you will use
- Absolute difference: variant - control (e.g., +0.8 percentage points).
- Relative lift: (variant/control - 1) (e.g., +8%).
- Confidence interval (CI): range of plausible effects; if it excludes zero (for improvements), that supports a meaningful change.
- p-value: probability of seeing data this extreme if there is truly no effect. Lower means stronger evidence of a real effect.
- Guardrail metrics: metrics you do not want to worsen (e.g., latency, error rate, cancellations).
Choosing metrics and guardrails
- Primary metric: the single metric used to make the final call (e.g., conversion rate).
- Secondary metrics: support insight and check side-effects (e.g., AOV, retention).
- Guardrails: must-not-worsen metrics with clear limits (e.g., page load time +1% max; error rate +0.1pp max).
What if metrics disagree?
Start with the primary metric. If it improves but a guardrail is breached, your default is do not ship. If secondary metrics move in mixed ways without guardrail breaches, consider business trade-offs, practical significance, and follow-up experiments.
Decision rules
- Statistical: Primary metric CI excludes zero in the desired direction (or p-value below your threshold).
- Practical: The expected gain exceeds your minimum meaningful effect (e.g., +0.3pp conversion or $0.50 ARPU).
- Safety: No guardrail breaches; no major data quality issues (e.g., sample ratio mismatch).
- Operational: Stable results across time segments (no novelty spikes only), reasonable device/geo consistency.
Stopping and peeking
Avoid changing the plan mid-flight. Pre-define duration/power. If you must peek, use adjusted methods or pre-set sequential rules. Otherwise, risk false positives.
Worked examples
Example 1: Conversion rate
Setup: Control p=10.0% (n=50,000), Variant p=10.8% (n=50,000). Difference d=+0.8pp; relative lift=+8%.
Approx SE for difference: sqrt[p1(1-p1)/n1 + p2(1-p2)/n2] = sqrt(0.1*0.9/50000 + 0.108*0.892/50000) ≈ 0.001931.
95% CI: 0.008 ± 1.96*0.001931 ≈ [0.0042, 0.0118]. CI excludes 0 → significant improvement.
Decision: If guardrails are OK, recommend ship. Communicate expected impact and any caveats (e.g., check by device).
Example 2: Average order value (AOV)
Setup: Control mean=$50, Variant mean=$51.2; SD≈$30 each; n=40,000 per group.
SE difference ≈ sqrt(30^2/40000 + 30^2/40000) = sqrt(900/40000*2) = sqrt(0.045) ≈ 0.212.
Difference = $1.20; 95% CI ≈ 1.20 ± 1.96*0.212 = [0.78, 1.62]. Positive and meaningful.
Decision: Ship if primary metric is revenue per session and guardrails hold. Note: Revenue data can be skewed; consider robust checks or longer run if heavy-tail concerns exist.
Example 3: Mixed movement
Variant raises CTR 5.0% → 6.0% (+20%), but revenue per session drops -3% due to cannibalization. Guardrail: refund rate worsens +0.2pp (limit +0.5pp) → within guardrail but primary metric (revenue/session) is negative.
Decision: Do not ship. Recommend iterate to reduce cannibalization (e.g., target high-intent segments) and re-test.
How to run a great readout
- Restate the goal and primary metric.
- Show experiment quality: traffic balance, duration, seasonality, data freshness.
- Report primary metric: absolute, relative, CI. One slide/text block.
- Report key secondary metrics and guardrails with clear green/yellow/red status.
- Translate to business impact (e.g., extra orders/day, revenue/week).
- State risks and unknowns (e.g., novelty, segment variance).
- Recommendation: Ship / Iterate / Inconclusive with rationale and next action items.
Business impact quick math
- Extra orders/day ≈ traffic_per_day × delta_conversion.
- Revenue lift/day ≈ extra_orders/day × AOV (or traffic × delta_ARPU).
- For confidence-aware ranges, compute using the CI bounds of delta metric.
Common mistakes and self-check
- Peeking and stopping on a temporary spike. Self-check: Plot metric by day; look for stabilization.
- Ignoring guardrails. Self-check: Explicitly mark pass/fail thresholds in your readout.
- Cherry-picking segments. Self-check: Start with all-users; treat segment wins as hypotheses for follow-up.
- Confusing relative vs absolute changes. Self-check: Always show both.
- Not translating to business impact. Self-check: Add orders/day or revenue/week estimate and a CI-based range.
- Over-interpreting tiny but significant effects. Self-check: Compare to a pre-defined minimum meaningful effect.
Sanity checks before deciding
- Sample ratio roughly balanced as planned.
- No major tracking outages or deploy incidents.
- Results consistent across key devices/regions unless you have a theory.
- Run covered at least one full business cycle (e.g., weekday/weekend).
Exercises (practice)
Do these before the Quick Test. Note: The Quick Test is available to everyone; only logged-in users get saved progress.
- Exercise 1: Compute the CI and decision for a conversion uplift with given counts.
- Exercise 2: Translate a conversion uplift into expected revenue per day and per month.
- Checklist before finalizing your answer:
- Reported absolute and relative change
- Included a 95% CI (or equivalent)
- Checked guardrails
- Stated a clear recommendation
Practical projects
- Create a one-page readout template with sections for objective, metrics, CI, guardrails, impact, decision, and next steps. Re-use it on future tests.
- Backtest: Pick 3 past experiments, re-calc absolute/relative lifts, add CI, and see if any decisions would change given your new framework.
- Segment stress test: For a completed experiment, compute the primary metric by device and country, note any large deviations, and propose a follow-up.
Who this is for
- Data Analysts who need to present clear experiment results and recommendations.
- PMs and Marketers collaborating with analysts on A/B tests.
Prerequisites
- Basic probability and confidence intervals
- Understanding of metrics (conversion, revenue/session, AOV)
- Basic spreadsheet skills (formulas for differences and CIs)
Learning path
- Start: Experiment design and metrics
- Now: Readout and decision making
- Next: Segmentation, heterogeneity analysis, and follow-up testing
Next steps
- Complete the exercises and take the Quick Test.
- Adopt the readout template for your next live test.
- Share a dry-run readout with a peer for feedback before stakeholder meetings.
Mini challenge
You see a +0.2pp lift (CI: [-0.05pp, +0.45pp]) in conversion, and a -0.3pp increase in refund rate (limit is +0.5pp). What do you recommend and why? Write 3 sentences: result summary, risk assessment, decision + next step.