Primary Metric Selection

Why this matters

In A/B tests, the primary metric is the single measure you optimize and judge success by. Choose it well, and decisions are clear. Choose it poorly, and you risk shipping changes that look good but don’t move the business.

• Product tasks: Decide if a new signup flow ships (primary metric: sign-up conversion per visitor).
• Marketing tasks: Evaluate landing page changes (primary metric: qualified lead rate).
• Growth tasks: Test referral prompts (primary metric: invites sent per user or new activated users per user).

Concept explained simply

The primary metric is your experiment’s “north star.” It answers: Did the change achieve the main goal?

• Primary metric: the one number that decides ship/rollback.
• Guardrail metrics: safety checks (e.g., crash rate, latency, unsubscribe rate) you don’t want to worsen materially.
• Secondary metrics: helpful diagnostics, not for the final decision.

Mental model: North Star + Seatbelts

Think of your test like a car trip. The destination is your primary metric. The seatbelts are guardrails. You need both: a clear destination and a safe ride.

How to choose a primary metric (step-by-step)

1. Define the goal: What outcome are you trying to improve? (e.g., more signups, higher bookings, increased activation)
2. Map the causal path: Draw the user journey. Mark where your change intervenes and which outcome it plausibly changes.
3. Pick the unit of analysis: Usually per user/visitor, not per session/pageview, to avoid over-counting heavy users.
4. Define the formula: Be precise. Example: “Signup conversion per visitor = (unique visitors who signed up) / (unique visitors exposed).”
5. Choose aggregation: Usually average per user over the experiment, or proportion of users achieving an event.
6. Set directionality: Higher is better or lower is better (e.g., “lower churn rate is better”).
7. Check sensitivity and variance: Will this metric move if the change works? Is variance manageable for a practical sample size?
8. Add guardrails: List metrics that must not degrade beyond a small threshold (e.g., latency, error rates, spam reports).
9. Pre-register thresholds: Define success criteria (effect size + significance), analysis plan, and exposure limits before launch.
10. Document: Write the metric spec in your experiment brief.

Worked examples

How to specify your primary metric

Use this template before you start the experiment:

• Name: [Clear, specific metric name]
• Intent: [What decision it supports]
• Unit of analysis: [user/visitor/recipient/account...]
• Population: [all exposed users or a subset (e.g., eligible to buy)]
• Formula: [e.g., unique users with event X / unique exposed users]
• Aggregation: [proportion, mean per user, median per account]
• Window: [e.g., within 7/14/28 days of first exposure]
• Direction: [higher is better / lower is better]
• Success threshold: [minimum uplift and significance level]
• Guardrails: [list + acceptable deltas]

Exercises

Try these now. Write your answers in a doc or notes. Then compare with the solutions.

1. Exercise 1 (mirrors EX1 below): Select a primary metric for an onboarding tooltip experiment, define unit, window, and guardrails.
2. Exercise 2 (mirrors EX2 below): Choose aggregation and window for revenue-adjacent metrics on a checkout button redesign.

• Self-check checklist:
  • Is the metric tied directly to the stated goal?
  • Is the unit of analysis user-centric and non-duplicative?
  • Is the window long enough to capture the effect but short enough to keep noise manageable?
  • Are guardrails covering user experience, reliability, and risk?

Common mistakes (and quick self-checks)

• Choosing a vanity metric (e.g., clicks) instead of an outcome metric. Self-check: Does this metric reflect value created for the user or business?
• Wrong unit of analysis (per session/pageview). Self-check: Would a heavy user contribute many times? If yes, switch to per user.
• Too noisy (e.g., revenue per user in a tiny test). Self-check: Can we detect the target effect with available sample size?
• Undefined window. Self-check: When exactly do we measure? From first exposure for N days?
• No guardrails. Self-check: What safety metrics must not worsen?
• Goal-metric mismatch. Self-check: If the change works perfectly, will this metric move?

Practical projects

• Spec a real experiment: Pick a recent product change idea. Write a metric spec using the template, including guardrails and thresholds.
• Metric audit: Take 3 past experiments. For each, critique the primary metric and propose an improved version.
• Power sanity check: Estimate the minimal detectable effect for your chosen primary metric using historical variance and typical traffic (simple spreadsheet).

Who this is for

• Aspiring and junior Data Analysts
• Growth and Product Analysts
• PMs who run or consume A/B tests

Prerequisites

• Basic probability and proportions (conversion rate, averages)
• Familiarity with product funnels and events
• Intro A/B testing concepts (control vs. treatment)

Learning path

• Start: A/B Testing Basics
• This subskill: Primary Metric Selection
• Next: Guardrails and Secondary Metrics; Sample Size and Power; Experiment Analysis and Interpretation

Next steps

• Use the template to draft your next experiment’s primary metric.
• Run the quick test below to check understanding.
• Apply in a small internal demo test (feature flag or staging data) to build muscle memory.

Mini challenge

You’re testing a new search results layout for a travel site. Choose the best primary metric, unit, and window. List two guardrails. Explain why your metric beats simple CTR.