Interpreting Results For Decisions

Why this matters

Data Scientists are trusted to turn experimental results into action. Stakeholders need clear, defensible decisions: ship or not, rollout to whom, what risks remain, and expected impact. You’ll often answer questions like:

• Is the lift real or noise? How big is it in business terms?
• Do guardrail metrics show hidden harm?
• What’s the expected value versus the cost and risk?
• Should we roll out to all users or a segment?

Concept explained simply

Interpreting results is about combining statistical evidence (how sure we are) with business context (what it’s worth) to make a clear decision.

Mental model: Traffic lights with a speedometer

• Green: Evidence strong and value high. Ship.
• Yellow: Evidence or value uncertain. Iterate, collect more data, or segment.
• Red: Evidence weak or harm exceeds guardrails. Stop.

The speedometer is effect size: even a statistically significant result may be too small to matter.

Key terms you’ll use

• Effect size: How big the change is (absolute and relative).
• Confidence interval (CI): Range of plausible effects. If it crosses 0, the signal is uncertain.
• p-value: How surprising the data is if there was no real effect. Not the chance your hypothesis is true.
• Practical significance: Is the effect big enough to be worth doing?
• Guardrails: Metrics that must not worsen beyond a threshold (e.g., retention, error rate).
• SRM (Sample Ratio Mismatch): Traffic split off from expected (e.g., 50/50 test gets 60/40). Do not trust results.

How to decide: a 5-step flow

1. Validate the experiment
   • Check SRM, allocation, data quality, and exposure logic.
   • Confirm pre-registered metrics and time window.
   Mini task

   Look at your variant counts. If they deviate more than a few percentage points from the intended split without a known reason, pause interpretation.

2. Quantify the effect
   • Compute absolute and relative lift.
   • Show the 95% CI for the effect.
   Mini task

   Write one sentence: “The main metric changed by X (CI [L, U]).” Keep units clear (percentage points vs percent).

3. Translate to business value
   • Convert lift into weekly or monthly revenue, cost saved, or risk reduced.
   • Compare to a decision threshold agreed with stakeholders.
   Mini task

   Multiply effect size by expected volume (e.g., sessions or orders) and value per unit to estimate expected value (EV).

4. Check guardrails and segments
   • Ensure key guardrails are within bounds.
   • Scan major segments for obvious harm. Treat segment hits as exploratory unless pre-registered.
   Mini task

   State: “Guardrails OK/Not OK. Largest segment risk is X with CI [L, U].”

5. Decide and document
   • Make a clear recommendation: Ship, Iterate, or Stop.
   • Include assumptions, EV, and remaining risks.
   One-slide template

   Decision: [Ship/Iterate/Stop]. Effect: [size, CI]. Value: [EV vs threshold]. Guardrails: [pass/fail]. Risks: [top 1–2]. Next step: [action + owner + timeframe].

Worked examples

Example 1: CTR lift that’s real but too small

Variant A: 50,000 users, 2,500 clicks (5.0%). Variant B: 50,000 users, 2,700 clicks (5.4%).

• Effect: +0.4 percentage points (pp), +8% relative.
• Approx 95% CI for difference: 0.125 pp to 0.675 pp (does not cross 0).
• Business: 1,000,000 weekly sessions, $0.50 per click → EV ≈ 0.004 × 1,000,000 × 0.50 = $2,000/week.
• Threshold to ship: $10,000/week. Decision: Don’t ship; iterate on a bigger improvement.

Example 2: AOV lift that clears value threshold

AOV baseline: $50. Variant B: $50.8. n=10,000 per arm, sd=20 each.

• Effect: +$0.80. 95% CI ≈ [+$0.246, +$1.354].
• Orders/week: 40,000. EV ≈ $32,000/week; conservative EV at CI lower bound ≈ $9,840/week.
• Implementation cost: $5,000/week. Decision: Ship (positive even at conservative bound).

Example 3: Guardrail violation blocks launch

Main metric: pageviews +2% (significant). Guardrail: 7-day retention −0.3 pp, CI [−0.6, 0.0]. Guardrail limit: no worse than −0.2 pp.

• Decision: Stop. Harm exceeds guardrail. Explore ideas to keep pageviews without hurting retention.

Exercises you can do now

Use the prompts below and record your answers. Then compare with the solutions.

• Checklist:
  • Validated the experiment setup (no SRM, correct exposure)
  • Computed absolute and relative lift
  • Provided a 95% CI
  • Converted effect to business value (EV)
  • Checked guardrails
  • Wrote a one-sentence decision with rationale

Exercise 1: CTR lift — should we ship?

Variant A: 50,000 users, 2,500 clicks. Variant B: 50,000 users, 2,700 clicks. Weekly sessions: 1,000,000. Revenue/click: $0.50. Ship if EV ≥ $10,000/week.

Expected output: Decision (Ship/Don’t ship) and a short rationale including effect size, 95% CI, EV vs threshold.

Hints

• Compute CTRs and the difference in percentage points.
• SE for difference in proportions: sqrt(pA(1−pA)/nA + pB(1−pB)/nB).
• EV = weekly_sessions × delta_CTR × revenue_per_click.

Show solution

CTRs: A=5.0%, B=5.4%. Difference=+0.4 pp (8% relative).

95% CI ≈ [0.125 pp, 0.675 pp] → statistically significant.

EV = 1,000,000 × 0.004 × $0.50 = $2,000/week.

Decision: Don’t ship. Rationale: Effect is real but below the $10k/week threshold; iterate for larger impact.

Exercise 2: AOV lift vs implementation cost

Baseline AOV: $50. Variant B AOV: $50.8. n=10,000 per arm, sd=20 each. Orders/week: 40,000. Implementation cost: $5,000/week.

Expected output: Decision and rationale with CI and conservative EV.

Hints

• SE of difference = sqrt(sd^2/n + sd^2/n).
• CI = diff ± 1.96 × SE.
• Conservative EV uses CI lower bound.

Show solution

Effect: +$0.8. SE ≈ sqrt(400/10000 × 2) ≈ 0.283. 95% CI ≈ $0.8 ± $0.554 → [$0.246, $1.354].

EV ≈ 40,000 × $0.8 = $32,000/week; conservative EV ≈ 40,000 × $0.246 = $9,840/week.

Decision: Ship. Rationale: Positive even at CI lower bound; exceeds cost by ≈ $4,840/week.

Exercise 3: Guardrails vs primary metric

Main metric: Signup rate +3% relative (significant). Guardrail: Customer support contact rate +0.25 pp, CI [+0.10, +0.40]. Guardrail limit: +0.15 pp.

Expected output: Decision and short risk note.

Hints

• Guardrails are hard limits even when primary improves.
• Consider mitigations: qualify rollouts, fix root cause, or redesign.

Show solution

Decision: Stop (or iterate). Rationale: Guardrail exceeds +0.15 pp (lower bound +0.10 pp is near limit, point estimate +0.25 pp beyond). Address causes before launch.

Common mistakes and self-check

• Misreading p-values: It’s not the probability the null is true. Self-check: Can you explain p-value without saying “probability the hypothesis is true”?
• Ignoring practical significance: A tiny, significant effect may not pay for itself. Self-check: Do you compare EV to a threshold?
• Multiple comparisons: Segment hunting inflates false positives. Self-check: Did you adjust or mark as exploratory?
• Peeking early: Stopping rules matter. Self-check: Was the analysis window pre-specified or sequential method used?
• Skipping guardrails: Wins that hurt retention or quality cost later. Self-check: Are guardrail CIs within bounds?
• Confusing pp vs %: 0.4 pp is not 0.4%. Self-check: State both clearly.
• SRM tolerance: Interpreting skewed splits. Self-check: Did you test and resolve SRM before analyzing?

Who this is for

• Data Scientists and Analysts running A/B tests or quasi-experiments
• Product Managers seeking evidence-based launch decisions
• Engineers contributing to experiment rollouts

Prerequisites

• Basic probability and statistics (proportions, means, CIs)
• Familiarity with A/B testing workflows
• Comfort with a spreadsheet or notebook for quick calcs

Learning path

1. Refresh stats: proportions vs means, CIs, effect sizes.
2. Decision economics: EV, thresholds, implementation cost, risk limits.
3. Guardrails: choose and justify; define limits.
4. Heterogeneity: segment checks and multiple-testing caution.
5. Documentation: one-slide decision memos; reproducible calcs.

Practical projects

• Decision memo: Take a past experiment, compute CI and EV, write a Ship/Iterate/Stop memo.
• EV calculator: Build a small spreadsheet that converts lift + volume + value into weekly EV with CI bounds.
• Guardrail dashboard: Show primary and guardrail metrics with thresholds and traffic lights.

Next steps

• Sample size and power analysis to plan tests
• Sequential testing or Bayesian approaches for faster, safer decisions
• Causal inference techniques (e.g., CUPED/regression adjustment) to reduce variance

Mini challenge

Write a one-sentence decision for a hypothetical test: “Variant B increases conversion by 1.2 pp (CI [0.3, 2.1]), EV $15k/week vs $8k threshold; guardrails pass; Ship to all users.” Keep it crisp and complete.

Quick Test

Take the quick test below to check your understanding. It’s available for everyone; only logged-in users will have their progress saved.