Experiment Population Definition

Why this matters in the job

Defining the experiment population is the foundation of trustworthy A/B tests. As a Product Analyst, you will be asked to:

• Scope who is eligible for a test so results reflect the intended users.
• Prevent contamination (users seeing both variants or treatment outside the window).
• Make results reproducible and explainable to product, engineering, and leadership.
• Protect fairness and comply with legal constraints (e.g., age, region restrictions).
• Power the experiment correctly by estimating the reachable sample size.

Who this is for

• Product Analysts and Data Scientists designing or reviewing experiments.
• Product Managers who need to interpret experiment results.
• Engineers implementing experiment assignment.

Prerequisites

• Basic understanding of A/B testing concepts (control vs. treatment, randomization).
• Comfort with event data (users, sessions, events, attributes).
• Basic statistics (sampling, independence, measuring outcomes).

Concept explained simply

Experiment population definition answers: “Who exactly can be part of this experiment, when, and under what conditions?” It is more than just “all users.” It sets the rules for eligibility, assignment unit, timing, and exclusions so your results reflect the intended target users and are free of avoidable bias.

Core definitions you’ll use

• Unit of randomization: The entity randomized (user, device, session, account, store, region).
• Inclusion criteria: Conditions that must be true to be eligible (platform, geography, lifecycle stage, baseline activity).
• Exclusion criteria: Conditions that disqualify users (prior exposure, employees, bots, legal restrictions).
• Exposure window: When assignment and first exposure can happen.
• Attribution window: How long you measure outcomes after exposure.
• De-duplication rule: Ensure a unit is assigned once and stays in that variant.
• Interference: One unit’s treatment affecting another’s outcomes (e.g., shared devices or teams).

Define your experiment population in 7 steps

1. State the decision question. What business decision will the test inform? Mini task: Write one sentence: “We need to know if X improves Y for Z.”
2. Define the target population. Platform, region, language, lifecycle stage. Mini task: List exact attributes (e.g., iOS, US, new users <= 7 days).
3. Choose the unit of randomization. Prefer the smallest independent unit. If spillover likely, move up (e.g., team or account). Mini task: Write “Unit = ____ because ____.”
4. Write inclusion criteria. Clear, measurable rules; specify data fields and time windows. Mini task: Draft 3–6 bullet rules.
5. Write exclusion criteria. Prior exposure, employees, bots, legal restrictions, conflicting tests. Mini task: Draft 3+ bullet rules.
6. Set timing windows. Assignment start/end, exposure window, attribution window. Mini task: Specify dates or relative windows (e.g., 14-day attribution).
7. Define de-duplication and handling edge cases. Sticky assignment by unit ID, one-sided holdout if needed, cross-device policy. Mini task: Write the exact ID used and the cross-device rule.

Worked examples

Pre-randomization checklist

• Decision question and target population are written in one sentence each.
• Unit of randomization chosen with a stated reason.
• Inclusion and exclusion criteria are measurable from available fields.
• Assignment, exposure, and attribution windows are explicit.
• Sticky assignment and cross-device policy defined.
• Conflicts with other live tests checked; guardrails set.
• Sample size and expected traffic validated against the target population.

Common mistakes and how to self-check

• Mistake: Mixing new and returning users when outcome is onboarding-related. Self-check: Confirm inclusion is “first-time only.”
• Mistake: Randomizing by session for long-lived effects. Self-check: Ask “Does treatment persist?” If yes, use user/account.
• Mistake: Prior exposure to similar features. Self-check: Add an exclusion on historical exposure flags.
• Mistake: Region/legal mismatch. Self-check: Verify geo and consent filters are enforceable.
• Mistake: Overlapping experiments. Self-check: Query if eligible users are enrolled in other tests with same surface.
• Mistake: Inconsistent IDs across platforms. Self-check: Define exact ID precedence (device_id → temp_id → user_id) and reconciliation.

Practical projects

• Draft a population spec for a homepage layout test and review it with a pretend PM. Include all seven steps and a data dictionary of used fields.
• Audit a past experiment (real or hypothetical) and list 5 improvements to eligibility, timing, or unit choice. Estimate the bias reduced.
• Create a “population template” document your team can reuse. Include fill-in sections and example phrasing.

Hands-on exercises

Do these before the test. They mirror the graded exercises below.

Mini challenge

Pick any recent feature idea and in 5 minutes draft the three rings (target, eligible, assigned) plus the unit of randomization. If you can’t state each in one sentence, your scope is probably unclear.

Quick Test

Available to everyone. Only logged-in users have their progress saved.

When ready, take the quick test below.

Next steps

• Apply the seven-step template to your next experiment plan.
• Peer-review a teammate’s population definition; check for the common mistakes above.
• Move on to sampling, power, and minimal detectable effect to ensure your defined population can reach sign-off timelines.

Learning path

• Before: Experiment goals and metrics; Baseline analysis.
• Now: Experiment population definition (this lesson).
• Next: Randomization strategy; Sample size and power; Exposure and attribution design; Guardrail metrics and monitoring.