Who this is for
Data Analysts who need to turn business ideas into testable, decision-ready A/B experiments. Also useful for Product Managers and Marketers collaborating on experiments.
Prerequisites
- Basic understanding of A/B testing concepts (control, variant, primary metric)
- Comfort with proportions and averages
- Familiarity with your product funnel and key metrics
Learning path
- Learn what makes a good hypothesis
- Practice writing clear, testable hypotheses
- Choose primary and guardrail metrics
- Define scope, audience, effect size, and decision rules
- Validate assumptions and common pitfalls
Why this matters in a Data Analyst role
Strong hypothesis framing prevents wasted tests and ambiguous results. Real tasks you will face:
- Translating a stakeholder idea into a testable statement
- Selecting the right primary metric and guardrails
- Scoping audience and duration so the test has enough power
- Setting decision rules to avoid p-hacking and premature stopping
Quick reminder: what a good hypothesis includes
- Change: what will be changed (the intervention)
- Audience: who is affected (segment)
- Direction: what you expect (increase/decrease/no worse than)
- Primary metric: a single success metric
- Guardrails: metrics that must not degrade
- Magnitude/time: expected effect size and test duration/decision rule
Concept explained simply
A hypothesis is your test plan in one sentence. It names the change, who sees it, what should happen, how you will measure success, and what must stay safe.
Mental model: Signal with guardrails
Think of your hypothesis as a car trip. The destination is your primary metric improvement (signal). Guardrails keep you on the road (no harm to critical metrics). The route defines audience, duration, and decision rules so you actually reach the destination without detours.
Worked examples
Example 1 — Checkout button copy
Business idea: Change the button copy from "Buy Now" to "Checkout Securely" to boost conversions.
- Change: Button copy on checkout page
- Audience: New visitors on desktop-only
- Direction: Expect an increase
- Primary metric: Completed purchases per session (conversion rate)
- Guardrails: Checkout error rate, average order value (must not decrease by more than 1%)
- Magnitude/time: Minimum detectable effect (MDE) 3%; run for 2 full weeks
- Decision rule: Ship if p-value < 0.05 and guardrails within limits
Null (H0): The copy change does not change purchase conversion for new desktop visitors.
Alternative (H1): The copy change increases purchase conversion for new desktop visitors.
Example 2 — Trial length
Business idea: Extend free trial from 7 to 14 days.
- Change: Trial length from 7 to 14 days
- Audience: New SaaS signups in North America
- Direction: Expect an increase
- Primary metric: Activation rate within 21 days
- Guardrails: Support tickets per signup (must not increase by >10%); Fraud rate
- Magnitude/time: MDE 5%; minimum sample 20k signups; analyze after 3 full weeks
- Decision rule: Launch if uplift >= 5% and guardrails acceptable
H0: No difference in activation rate. H1: 14-day trial increases activation rate.
Example 3 — Email subject personalization
Business idea: Personalize subject with first name.
- Change: Add first name token to subject
- Audience: Subscribers active in last 90 days
- Direction: Expect an increase
- Primary metric: Open rate
- Guardrails: Unsubscribe rate (must not increase by >0.2 pp); Spam complaint rate
- Magnitude/time: MDE 2 pp; send across 3 campaign drops
- Decision rule: Roll out if open rate increases and guardrails are stable
H0: No difference in open rate. H1: Personalization increases open rate.
Steps to write a strong A/B hypothesis
Step 1 — Name the change
Be specific: which element, where, how it changes.
Step 2 — Define the audience
Which users see it? Segment by device, geography, lifecycle stage if needed.
Step 3 — Predict direction
State increase/decrease/no worse than. Avoid vague words like "optimize".
Step 4 — Choose one primary metric
Pick a single success metric tied to the outcome. Avoid stacking multiple primaries.
Step 5 — Add guardrails
List metrics that must not degrade (e.g., error rate, refunds, unsubscribes).
Step 6 — Set effect size and power plan
Choose an MDE or practical significance threshold and ensure sample size/time can detect it.
Step 7 — Duration and stopping rule
Run for full business cycles (e.g., 1–2 weeks). Set a clear stop decision (e.g., p<0.05 with practical uplift).
Step 8 — Document assumptions/risks
Note dependencies (tracking, rollout feasibility, seasonal noise).
Step 9 — Write the full hypothesis
Combine into one statement plus bullet details for metrics and rules.
Template you can reuse
If we change [element] for [audience], then [directional effect] on [primary metric], because [reason]. We will run for [duration] targeting [MDE]. Ship if [stat rule] and guardrails [limits].
Exercises (do these now)
These mirror the graded tasks below. Your progress in the quick test is available to everyone; only logged-in users will have saved progress.
- Exercise 1 — Frame a hypothesis
Draft a complete hypothesis for reducing signup fields from 7 to 5 for mobile users. Include change, audience, direction, primary metric, guardrails, MDE, duration, and decision rule.
Self-check checklist
- Change is specific and scoped
- Audience segment is clear
- Single primary metric defined
- At least two guardrails set
- MDE or practical threshold stated
- Duration includes full cycles
- Decision rule prevents peeking/p-hacking
- Null and alternative can be stated
Common mistakes and how to self-check
- Vague goals: "Improve engagement" without a defined metric. Fix: pick one primary metric.
- Multiple primaries: Makes decisions messy. Fix: choose one; others are secondary.
- No guardrails: Wins that hurt retention. Fix: add 1–3 safety metrics.
- Undefined audience: Mixed effects cancel out. Fix: segment thoughtfully.
- No MDE or duration: Underpowered tests. Fix: plan sample size/time.
- Changing rules mid-test: Leads to false positives. Fix: pre-commit to decision rules.
Quick self-audit
Read your hypothesis and highlight: change (blue), audience (green), primary metric (yellow), guardrails (red), decision rules (purple). If any color is missing, revise.
Practical projects
Project 1 — Onboarding CTA
- Pick one onboarding screen and define a single CTA change
- Write the hypothesis using the template
- Select primary metric (e.g., next-step completion)
- Set guardrails (e.g., error rate, time-to-complete)
- Define MDE and duration
- Present a one-pager with your decision rule
Project 2 — Pricing page layout
- Identify a layout adjustment (plan card order)
- Draft hypothesis with audience (new visitors)
- Primary metric: plan selection rate to paid
- Guardrails: support chat rate, refund rate
- MDE and sample plan
- Mock the metric dashboard you would monitor
Project 3 — Email lifecycle
- Choose a lifecycle email (welcome)
- Hypothesis for content change
- Primary: click-through rate; Guardrails: unsubscribe/complaints
- Pre-register decision rule and duration
Mini challenge
Scenario: Reducing homepage hero image size to improve page load on mobile. Write a full hypothesis.
Show a possible answer
If we reduce hero image weight by 60% for first-time mobile visitors, then conversion to product page will increase, because faster load reduces bounce. Primary metric: product page views per mobile session. Guardrails: bounce rate (must not increase), time-on-site (must not drop by >5%). MDE: +4% product page views. Duration: 2 weeks. Decision: ship if p<0.05 and guardrails within limits.
Next steps
- Refine your Exercise 1 hypothesis using the checklist
- Share with a peer for critique on clarity and metrics
- Move on to experiment design and sample size estimation
Ready to test yourself?
Take the quick test below. Your results are available to everyone; only logged-in users will have saved progress.