Why this matters
As a Product Analyst, you convert experiment ideas into reliable results. The setup you do in the testing platform determines whether your A/B test is valid, measurable, and safe for users and revenue.
- Launch controlled rollouts without breaking core funnels.
- Target the right users and ensure consistent assignment (sticky bucketing).
- Track primary metrics and guardrails for trustworthy decisions.
- Plan traffic, duration, and stop rules to avoid false wins.
Real tasks you’ll handle
- Create variants and allocate traffic (e.g., 50/50, 90/10 ramp).
- Define exposure, targeting, and exclusions (e.g., only new users, exclude employees).
- Wire up events: conversions, clicks, feature usage, revenue.
- Configure guardrails (crash rate, latency, unsubscribe rate).
- QA in a preview environment and run a soft launch.
Concept explained simply
An experiment platform is a switchboard: you define who sees what, how much traffic to send, what success means, and when to stop. If the switchboard is wired wrong, your results are noisy or misleading.
Mental model: The 5 levers
- Units and Randomization: How users are bucketed (user_id, device_id).
- Exposure and Targeting: Who is eligible and when they enter.
- Allocation and Ramp: How much traffic each variant gets over time.
- Metrics and Guardrails: What you measure to decide and to keep users safe.
- Stop Rules and Duration: When you call the test and how you avoid peeking.
Step-by-step setup
- Define the hypothesis: What change, for whom, with what expected outcome?
- Pick unit of randomization: Usually user_id (web/app), occasionally device_id or session for anonymous flows.
- Targeting and exclusions: Countries, platforms, new vs. returning users, exclude internal IPs and bots.
- Create variants: Control and at least one variant. Name clearly (e.g., control_v1, paywall_variant_a).
- Traffic allocation and ramp plan: Start small (e.g., 10%) then ramp to 50% if guardrails hold.
- Metrics: Primary (decision metric), secondary, guardrails (e.g., error rate, latency, refund rate).
- Sample size and duration: Use baseline rate, desired effect size, power 80%, alpha 5%. Pre-commit minimum duration (e.g., 2–3 weeks) to cover cycles.
- Sticky assignment: Ensure users stay in their variant across sessions/devices where feasible.
- QA: Preview modes, test events firing, ensure no double counting or broken UI.
- Stop rules: Only stop if duration met and metrics meet pre-defined thresholds. Avoid peeking and early stopping unless for harm.
Pre-launch checklist
- Clear hypothesis and success criteria written.
- Targeting segment defined and exclusions added.
- Variants named and verified in preview.
- Primary/secondary/guardrail metrics selected and events firing.
- Allocation and ramp plan documented.
- Sample size and minimum duration estimated.
- Sticky assignment confirmed.
- QA sign-off captured.
Worked examples
Example 1: Web CTA color test (simple UI change)
- Hypothesis: Changing the signup button color increases click-through by 5% for new desktop users.
- Unit: user_id; fallback to device_id if anonymous.
- Targeting: Desktop, new visitors only, exclude employees and bots.
- Allocation: Start 10/90 (variant/control) for 24 hours, then 50/50.
- Metrics: Primary—Signup button CTR; Secondary—Account creation; Guardrails—Bounce rate, error rate.
- QA: Verify event signup_cta_click fires once; confirm sticky assignment across pages.
- Stop rule: Minimum 14 days, at least 1 baseline cycle.
Example 2: Mobile paywall variation via feature flag
- Hypothesis: New layout increases trial start rate by 7% among new iOS users.
- Unit: user_id (app install id as backup).
- Targeting: iOS, app version ≥ 8.2, new users; exclude purchasers.
- Allocation: 5% holdout (control), 45% variant A, 50% control remainder (or 50/50 after ramp).
- Metrics: Primary—Trial start; Secondary—Subscription at D7; Guardrails—Crash rate, session length drop.
- QA: Toggle flag in staging; verify paywall renders and events trial_start, paywall_view fire.
- Stop rule: Minimum 21 days or 10k exposures, whichever later.
Example 3: Pricing page copy test across locales
- Hypothesis: Clearer value message improves checkout start by 3% globally.
- Unit: user_id, namespace “pricing”.
- Targeting: All locales except those with legal copy constraints; exclude active subscribers.
- Allocation: 50/50, simultaneous across locales with stratified analysis.
- Metrics: Primary—Checkout start; Secondary—Payment conversion; Guardrails—Refund rate, NPS drop threshold.
- QA: Locale switch testing, currency formatting, translation completeness.
- Stop rule: 3 weeks, ensure weekend and payday coverage.
Exercises
Do these to practice the exact clicks and fields you’ll configure in a platform.
Exercise 1: Configure a mock onboarding experiment
Scenario: You test a shorter signup form for new web users. Baseline signup completion is 40%. You expect +5% relative uplift. Minimum detectable effect: 5% relative. Power 80%, alpha 5%.
- Choose unit and targeting.
- Define variants and naming.
- Set allocation and ramp plan.
- Pick primary, secondary, guardrail metrics.
- Estimate minimum duration and sample size (roughly; show assumptions).
- Write stop rules.
Self-check checklist
- Unit is stable across sessions (not session-scoped for signup completion).
- Employees and bots excluded.
- Events mapped to metrics and tested in preview.
- Guardrails include error rate and latency.
- Ramp documented and safe.
- Stop rules include time, power, and harm override.
Common mistakes and how to self-check
- Peeking early: Decide only after pre-committed duration and sample size.
- Wrong unit: Session-level for user outcomes inflates variance. Prefer user_id.
- Leaky targeting: Employees/bots skew results—add exclusions.
- Event mismatch: Names differ between code and platform—QA in preview and live sample.
- Cross-test contamination: Overlapping experiments in same namespace—use namespaces or mutual exclusions.
- Non-sticky assignment: Users flip variants—enable sticky bucketing.
- Underpowered tests: Too little traffic—adjust MDE or runtime.
Quick self-audit before launch
- Can a single user enter multiple arms? If yes, fix stickiness.
- Are guardrails visible on the test dashboard?
- Is there a documented rollback plan?
- Do you have a holdout or control clearly defined?
Practical projects
- Instrument a demo site with two events and write a mock platform config for a 50/50 test.
- Create a ramp plan template with criteria to move from 10% to 50% traffic.
- Draft a QA script: steps, expected events, screenshots, and pass/fail notes.
Who this is for
- Product Analysts running or supporting experiments.
- PMs and Data Scientists collaborating on A/B tests.
- Engineers setting up flags and events with analysts.
Prerequisites
- Basic probability and statistics for A/B testing.
- Understanding of product funnels and key metrics.
- Ability to validate event logging in dev tools or logs.
Learning path
- Start: Hypotheses and metrics selection.
- Then: Experiment setup in the platform (this lesson).
- Next: QA, monitoring, and interpreting results.
- Finally: Decision-making and documenting learnings.
Next steps
- Complete the exercise and run the quick test below.
- Create a reusable experiment template for your team.
- Shadow an existing experiment from setup to decision.
Progress saving note: The quick test is available to everyone. If you’re logged in, your progress will be saved.
Mini challenge
You have 20% of traffic available this week for a sensitive checkout test. Outline a 3-step ramp plan, guardrails you’ll monitor live, and your harm-stop threshold. Keep it to 6–8 bullet points.