Why this matters
Great experiments start with a crisp business problem. As a Product Analyst, you will routinely:
- Turn a vague request ("Let’s test a new banner") into a clear decision ("Should we highlight annual billing to lift paid conversion by 5% without hurting refunds?").
- Align stakeholders on the goal, decision, metrics, and constraints before any design or build starts.
- Prevent wasted tests by defining what success looks like and how you will decide based on evidence.
Real tasks you’ll face
- Clarify the decision behind a request (build or not build, change X vs Y, roll out vs roll back).
- Choose a primary metric and guardrails that match the business goal.
- Specify target audience, timeframe, and minimum detectable effect to plan sample needs.
- Map the lever (what you change) to the mechanism (why it should work) to the metric (how you’ll know).
Concept explained simply
Business Problem Framing is turning a fuzzy idea into a decision-ready statement with goal, options, metrics, and constraints. It answers: What decision will we make, for whom, by when, using which evidence?
Mental model: GDMAC+L
- Goal: What outcome matters (business and user)?
- Decision: The binary or comparative choice the experiment will inform.
- Metric: Primary success metric + guardrails (quality, cost, risk).
- Audience: Who is included (segment, platform, country)?
- Constraints: Limits like deadlines, traffic, regulations, brand.
- Levers: Which change(s) will drive the outcome and why (mechanism)?
Framing template (copy/paste)
Context: [why now / what changed] Goal: [business outcome + user value] Decision: [If metric improves by ≥X% with guardrails in range Y, we will Z] Primary metric: [one clear metric] Guardrails: [2–4 must-not-worsen metrics] Audience & scope: [segment, platform, geo, exposure] Lever(s) & hypothesis: [what you will change + mechanism] Constraints: [time, tech, legal, seasonality] Timeline: [test window, readout, decision date]
Worked examples
Example 1: Checkout drop-off
Context: Drop-off at payment step increased.
Goal: Recover revenue by improving checkout completion.
Decision: Roll out a free-shipping threshold banner in cart if it lifts checkout completion ≥3% without increasing returns rate >0.2 pp.
Primary metric: Checkout completion rate.
Guardrails: Returns rate, average order value (AOV), support contacts per order.
Audience: US web cart sessions.
Lever & hypothesis: Banner reduces surprise shipping cost, increasing completion via expectation setting.
Constraints: Season ends in 3 weeks; banner must use existing component.
Example 2: Notification opt-in
Context: Low opt-in reduces re-engagement.
Goal: Lift opt-in while preserving unsubscribe rate.
Decision: Change copy vs change default toggle? Choose approach that lifts opt-in ≥10% without raising weekly unsubscribes >0.1 pp.
Primary metric: First-session opt-in rate.
Guardrails: Weekly unsubscribe rate, push notification CTR.
Audience: New app users, Android only.
Levers & hypothesis: Value-prop copy highlights benefits; default toggle ON may be intrusive. Start with copy to avoid trust risk.
Example 3: Pricing page redesign
Context: Stakeholders want a full redesign.
Goal: Increase paid conversion and revenue per visitor.
Decision: Highlight annual plan with 10% discount if paid conversion + ARPU improve jointly (composite metric).
Primary metric: Paid conversion; Composite: paid conversion × ARPU.
Guardrails: Refund rate, support chat volume, page load time.
Audience: New visitors on web.
Lever & hypothesis: Prominence and clarity of annual plan reduces choice friction and increases commitment.
Constraint: No price changes; only layout/copy allowed.
How to apply it quickly
- State the decision: "We will [do X] if [metric] improves by ≥[threshold] with guardrails in range."
- Choose one primary metric that matches the goal. Add 2–4 guardrails.
- Specify audience and timeframe so your result generalizes.
- Name the lever and why it should work (mechanism).
- Write a one-page brief and share for alignment before design work.
Mini prompts to unblock you
- What business decision will this enable next week if the result is positive? Negative?
- Which single metric would convince a skeptical stakeholder?
- What could go wrong if this wins? Make it a guardrail.
- Who is excluded and why? Document it.
Exercises
Do these now. Then compare with the solutions below.
Exercise 1 — Rewrite the request (ex1)
Vague request: "Our trial conversions are down. Can we test a new pricing banner ASAP?"
Tasks:
- Rewrite as a decision statement.
- Define primary metric and at least 2 guardrails.
- Specify audience, timeframe, minimum effect to act.
- List 2 lever options and their mechanisms.
When done, check the solution below.
Exercise 2 — Metric tree and decision rule (ex2)
Problem: "Feature adoption is low for Saved Filters."
Tasks:
- Draft a simple metric tree from business outcome to leading metric.
- Propose a lever and testable hypothesis.
- Write a clear decision rule using a numeric threshold.
When done, check the solution below.
Quick checklist before you test
- Do we have exactly one primary metric?
- Is there a written decision rule with a numeric threshold?
- Are guardrails covering quality, cost, and user trust?
- Is the audience scoped so the result is actionable?
- Is the lever connected to a plausible mechanism?
Common mistakes and self-check
- Solution-first framing: Jumping to "Test a banner" before defining the decision. Fix: Start with the decision sentence.
- Metric mismatch: Using clicks when the goal is revenue. Fix: Pick the metric that reflects the decision.
- Too many goals: 3+ primary metrics. Fix: One primary, guardrails for safety.
- Missing threshold: "We’ll see" at readout. Fix: Pre-commit to a minimum detectable effect and decision rule.
- Ignoring constraints: No traffic/time to learn. Fix: Adjust scope, segment, or lever.
Self-audit mini task
Take your latest experiment brief and highlight:
- Blue: Decision sentence.
- Green: Primary metric and threshold.
- Yellow: Guardrails.
- Red: Any open assumptions. Close the reds before build.
Practical projects
- One-pager library: Create a 1-page framing brief for 3 common problems in your product (activation, retention, monetization). Aim for one day each.
- Metric tree workshop: Map outcome → drivers → levers for a critical metric. Validate with a PM and Designer.
- Guardrail catalog: Define default guardrails for your team (e.g., support contacts, refund rate, page speed, unsubscribe rate).
Who this is for
Product Analysts, Data/BI Analysts partnering with Product, and PMs who need to run reliable experiments and make clear go/no-go decisions.
Prerequisites
- Basic knowledge of product metrics (conversion, retention, ARPU).
- Comfort with A/B testing concepts (variants, power, MDE).
- Ability to pull metrics from your analytics stack.
Learning path
- Practice framing with the template on past experiments.
- Run a review with stakeholders; refine metrics and guardrails.
- Execute a small scoped test using your best-framed problem.
- Build a repeatable checklist your team uses before every test.
Next steps
- Complete the exercises above and compare with solutions.
- Take the quick test to validate your understanding.
- Apply the template to an upcoming experiment and share it for feedback.
Note: The quick test is available to everyone; only logged-in users get saved progress.
Mini challenge
An executive says: "Churn looks bad. Ship a reactivation email." In 5 minutes, write a decision-ready framing:
- Decision sentence with threshold.
- Primary metric + 2 guardrails.
- Audience and timeframe.
- Lever + mechanism.
Hint if you’re stuck
Start: "We will roll out a reactivation email to lapsed users if reactivation rate within 14 days improves by ≥X% with unsubscribe rate ≤Y% and support tickets ≤Z%."