Who this is for
Business Analysts, Product Analysts, and anyone framing hypotheses for experiments, pilots, or before/after changes.
Prerequisites
- Basic understanding of metrics (conversion, churn, revenue, NPS)
- Intro knowledge of experiments or before/after analysis
- Comfort with simple percentages and time windows
Why this matters
As a Business Analyst, you translate ideas into testable, decision-ready hypotheses. Defining the expected direction of change (increase/decrease or no change) focuses analysis, aligns teams, and prevents data dredging. It also determines whether you need a one-tailed or two-tailed evaluation, which affects power and sample size.
- Prioritization: If you expect only upside, you can run one-tailed tests and allocate fewer samples.
- Risk management: If a downside is risky (e.g., drop in activation), set guardrails and use two-tailed decisions.
- Clear decision rules: Pre-committing direction and thresholds avoids fishing for significance after the fact.
Concept explained simply
Direction of change answers: Do we expect the metric to go up, go down, or could it go either way? You also specify roughly how much change matters and over what time.
Mental model
Think of a hypothesis as a signpost plus guardrails:
- Signpost (direction): The sign (+/-) you expect on the primary metric for a defined segment.
- Magnitude threshold: The smallest change that matters (effect size) within a time window.
- Guardrails: Other metrics that must not worsen beyond limits.
One-tailed vs two-tailed — simple rule of thumb
- Use one-tailed when only improvement matters and a small decline is acceptable risk (e.g., minor UI tweak to speed).
- Use two-tailed when either direction is meaningful or risky (e.g., pricing, algorithm changes affecting trust/quality).
Choosing effect size and time window
- Effect size: The minimal detectable effect (MDE) that would change a decision, not just what’s statistically convenient.
- Time window: The period when the effect should show (e.g., 7 days for activation, 30 days for churn).
- These choices influence sample size and power.
Guardrail examples
- Conversion test: Guardrail = refund rate, support tickets
- Engagement push: Guardrail = opt-out/uninstall rate
- Pricing change: Guardrail = NPS/complaints, churn
5-step method to define direction
- Define the primary metric and segment. Example: Activation rate among new sign-ups in US.
- Write the business logic. One sentence on why the change should push the metric up or down.
- Pick the direction and test type. Increase/decrease; one-tailed or two-tailed.
- Set effect size and time window. State the minimum meaningful change and by when.
- Add guardrails and a decision rule. Specify “ship if … ; stop/rollback if …”.
Directional statement template
If we [change], we expect [metric] to [increase/decrease] by [at least/no more than] X within Y [time window] for [segment], while [guardrail metric(s)] stay within [limit]. Decision rule: [ship/rollback criteria].
Worked examples
Example 1 — Onboarding tooltip
Logic: A clearer tooltip reduces confusion at a key step.
Statement: We expect activation rate to increase by at least +2 percentage points within 7 days among new web sign-ups in US, while support tickets per 1k sign-ups do not increase by more than +5%.
Decision rule: Ship if activation +2pp or more and tickets ≤ +5%; rollback otherwise.
Example 2 — Price increase 5%
Logic: Higher unit price may reduce conversion but can raise revenue per visitor.
Statement: We expect revenue per visitor to increase by at least +3% within 14 days overall; two-tailed guardrail on conversion rate not to drop by more than −2%.
Decision rule: Ship if RPV ≥ +3% and conversion ≥ −2%; rollback if conversion < −2%.
Example 3 — Churn outreach
Logic: Proactive emails to at‑risk users reduce cancellations.
Statement: We expect 30‑day churn to decrease by at least −1.5pp among predicted high‑risk users, with NPS not decreasing by more than −1 point.
Decision rule: Scale if churn ≤ −1.5pp and NPS ≥ −1; reassess copy if NPS < −1.
Example 4 — Showing shipping cost earlier
Logic: Early transparency reduces late‑stage surprises, lowering cart abandonment.
Statement: We expect checkout completion rate to increase by at least +1pp within 14 days for mobile users, with average order value not decreasing by more than −1%.
Decision rule: Ship if completion +1pp and AOV ≥ −1%.
Definition checklist
- Primary metric named and measurable
- Segment defined (who is included)
- Direction chosen (+/− or two‑tailed)
- Effect size threshold and time window stated
- Guardrails and limits set
- Clear decision rule written
Common mistakes and self-check
- Vague direction: “Improve engagement” → Replace with “Increase 7‑day retention by ≥ +1pp”.
- No time window: Always include when the effect should be visible.
- Ignoring guardrails: Add at least one to prevent harmful trade‑offs.
- Post‑hoc direction flip: Do not change direction after seeing data; pre‑commit.
- Effect size too small: Don’t set thresholds below what you can detect reliably.
Self-check: Could a neutral reader decide “ship/rollback” from your statement alone? If not, add specifics.
Exercises
Do the exercise below (mirrors Exercise ex1). Write your answers. Then compare with the sample solution.
- Free trial email reminder: Define direction, threshold, time window, guardrails.
- Search relevance tweak: Define direction for CTR and a guardrail for complaint rate.
- Annual plan discount: Define direction for revenue per user and guardrail for churn.
Before you start — quick checklist
- Primary metric chosen?
- Segment defined?
- Threshold and time window set?
- Guardrails added?
- Decision rule stated?
Practical projects
- Pick two recent product ideas and write full directional hypothesis statements including decision rules. Present to a teammate for critique.
- Audit three past experiments and rewrite their hypotheses with explicit direction, MDE, time window, and guardrails. Note how decisions might change.
- Create a team-ready hypothesis template for your org using the statement pattern above.
Learning path
- Now: Direction and decision rules
- Next: Choosing metrics and guardrails with event/segment definitions
- Then: Power, sample size, and MDE basics
- Later: Interpreting results and handling heterogeneity
Quick test
Available to everyone. Only logged-in users will have their progress saved.
Mini challenge
You are proposing an in‑app nudge to complete profiles. Write one sentence with direction, threshold, time window, segment, guardrails, and a decision rule. Keep it under 35 words.
Next steps
- Use the Definition checklist whenever you write a hypothesis.
- Share your template with stakeholders and agree on guardrail defaults.
- Apply the method to your next experiment or rollout plan today.