Who this is for
- Current or aspiring Data Analysts who run A/B tests, summarize experiments, or communicate the practical significance of results.
- Anyone who wants to move beyond p-values to quantify how big a finding is.
Prerequisites
- Basic descriptive statistics: mean, standard deviation, proportions.
- Comfort with square roots and simple algebra.
- Familiarity with hypothesis testing is helpful, not required.
Why this matters
Effect sizes answer the question: how big is the difference or relationship? In real analyst tasks, this shows up when you:
- Prioritize A/B test wins: A 1% lift might be statistically significant, but is it meaningful?
- Report on campaigns: Explain whether a correlation between spend and leads is small or substantial.
- Compare product variants: Communicate the magnitude of change in time-on-site or conversion rates.
Real-world example scenarios
- Product: New onboarding reduces time-to-first-value by 0.4 SD — a small but worthwhile improvement.
- Marketing: r = 0.35 between content volume and MQLs — about 12% of variance explained.
- Growth: New pricing reduces churn odds by ~30% versus old pricing.
Concept explained simply
Effect size is a standardized measure of how big a difference or relationship is. It complements p-values (which tell you if an effect is likely real) by telling you how large it is.
- Difference between two means: use Cohen's d (or Hedges' g for small samples).
- Difference between two proportions: use Cohen's h (or odds ratio for interpretability).
- Linear relationship: use Pearson's r and r².
- Association between categorical variables: use Cramer's V (from chi-square).
Mental model: signal vs. noise
Think of signal (the difference or association) divided by noise (variability). Standardizing by variability lets you compare effects across different units (seconds, dollars, scores).
How to choose an effect size (quick checklist)
- Two independent means (e.g., time on page): use Cohen's d.
- Two independent proportions (e.g., conversion rate): use Cohen's h or Odds Ratio.
- Continuous X and Y (linear): use r and r².
- Two categorical variables (any sizes): use Cramer's V.
- Multiple groups means (ANOVA): use η² or Cohen's f.
Useful formulas (plain text)
- Cohen's d = (Mean1 − Mean2) / s_pooled
- s_pooled = sqrt( ((n1−1)sd1² + (n2−1)sd2²) / (n1 + n2 − 2) )
- Hedges' g ≈ J × d, where J = 1 − 3/(4df − 1), df = n1 + n2 − 2
- Cohen's h = 2·arcsin(√p1) − 2·arcsin(√p2)
- Odds ratio (OR) = (a/b) / (c/d) = (a·d)/(b·c) for 2×2 table
- r² = proportion of variance explained by a linear relationship
- Cramer's V = √(χ² / (n·min(r−1, c−1)))
Rule-of-thumb thresholds (use with context)
- Cohen's d or h: ~0.2 small, ~0.5 medium, ~0.8 large
- r: ~0.1 small, ~0.3 medium, ~0.5 large
- Cramer's V: ~0.1 small, ~0.3 medium, ~0.5 large (for df=1; adjust with table size)
These are rough guides. Always consider business impact and domain.
Worked examples
Example 1 — Cohen's d for two means (A/B time on site)
- Data: A (n=200, mean=5.2, sd=1.8), B (n=210, mean=4.7, sd=1.6)
- Compute s_pooled:
sd1²=3.24, sd2²=2.56
Numerator=(199×3.24)+(209×2.56)=644.76+535.04=1179.80
Denominator=200+210−2=408
s_pooled=sqrt(1179.80/408)=sqrt(2.8936)=1.701 - d=(5.2−4.7)/1.701=0.294 (small)
- Interpretation: Variant A increases time on site by ~0.29 SD — small but may be meaningful at scale.
Example 2 — r and r² (ad spend vs. leads)
- Suppose r=0.35 from a scatterplot/regression.
- r²=0.35²=0.1225 → ~12.3% of variance in leads explained by spend.
- Interpretation: Moderate linear relationship; other factors still matter.
Example 3 — Odds ratio for conversion (old vs. new page)
- Data: Old: 58 converted, 342 not; New: 85 converted, 315 not.
- Odds (Old)=58/342=0.1696; Odds (New)=85/315=0.2698
- OR=(New/Old)=0.2698/0.1696=1.59
- Interpretation: New page has ~59% higher odds of conversion than old. Practical and clear to stakeholders.
Practice exercises
Do these on paper or a notepad. Then compare with the solutions. Everyone can take the quick test; only logged-in users will have progress saved.
- Exercise 1 (ex1): Compute Cohen's d for two means.
- Exercise 2 (ex2): Compute Cohen's h for two proportions.
- Exercise 3 (ex3): Compute an odds ratio and interpret.
Checklist before you compute
- Have you matched the effect size to your data type?
- Are your units consistent (means vs. proportions)?
- Did you use standard deviation (not standard error) for d?
- Did you round at the end, not at each step?
Common mistakes and self-check
- Mistake: Using standard error instead of standard deviation in Cohen's d. Self-check: Are you using the raw SD of each group?
- Mistake: Reporting p-value but not effect size. Self-check: Can a reader tell how big the effect is?
- Mistake: Treating thresholds as universal truth. Self-check: Add domain context or baseline conversion rates.
- Mistake: Mixing up odds ratio with risk ratio. Self-check: OR uses odds (events/non-events), not probabilities.
- Mistake: Rounding too early. Self-check: Keep 3–4 decimals until the final step.
Practical projects
- Analyze a past A/B test: compute Cohen's d for a key metric, and add a short interpretation line for stakeholders.
- Marketing funnel review: compute r and r² between ad spend and signups across 12 weeks; add a caveat about seasonality.
- Churn analysis by plan: compute odds ratio of churn between two plans and write a one-sentence insight for a product manager.
Learning path
- Review variability: SD vs. SE.
- Learn the right mapping: data type → effect size.
- Practice: compute d, h, r, OR on small examples.
- Communicate: add one sentence of business context for each effect size.
- Stretch: compute Cramer's V for a 2×3 table.
Next steps
- Apply these measures to your latest report; include both statistical significance and effect size.
- Move on to confidence intervals for effect sizes to express uncertainty, not just point estimates.
Mini challenge
Pick and compute
You test a new onboarding flow.
- Group A (old): n=500, mean time-to-value=3.8 days, sd=1.4
- Group B (new): n=520, mean=3.5 days, sd=1.3
Task: Choose the right effect size and compute it. Then write a one-line interpretation for a PM.
Note: Salary or impact of changes varies by country/company; treat any ranges you estimate as rough ranges.