Who this is for
Business Analysts and aspiring analysts who need to turn a hypothesis into a clear, ethical, and feasible plan for gathering data to make a decision.
Prerequisites
- You can state a problem and a hypothesis in plain language.
- Basic understanding of metrics, events, and segments.
- Comfort reading simple tables or dashboards.
Progress note: The quick test is available to everyone. If you are logged in, your progress will be saved automatically.
Why this matters
In real projects, you rarely “get the data later.” Decisions and deadlines force you to plan what to collect now. Good planning prevents wasted engineering time, missing fields, and inconclusive results.
- Product: Specify events, properties, and IDs so you can measure conversion after a feature launch.
- Operations: Define how to sample support tickets to validate a root-cause hypothesis.
- Marketing: Plan UTM and campaign metadata to attribute leads correctly.
Concept explained simply
Planning data collection is deciding exactly what evidence you need to test your hypothesis and how you will gather it—before you start. You pick the metric, the units, the fields, the sample, and the timeline so the analysis is possible and trustworthy.
Mental model
Decision → Hypothesis → Measurement → Data → Analysis → Decision. If any arrow is fuzzy, planning must sharpen it before you collect.
Quick check: is your hypothesis measurable?
- Can you name the outcome metric?
- Can you say who/what is measured (unit of analysis)?
- Can you name the time window?
- Do you know where the data will come from?
Step-by-step: Plan data collection
- Anchor to decision
What decision will this data inform?Tips
- Phrase it as: “We will decide to X if Y.”
- Define the threshold for action (e.g., +5% conversion).
- Define variables
- Outcome (dependent): the result you want to change.
- Drivers (independent): the factors you believe cause change.
- Controls: factors you hold constant or account for.
- Operational definitions
Make each metric unambiguous.- Exact formula, filters, time window, and attribution rule.
- Example: “Activation = user completes onboarding checklist within 7 days of signup.”
- Unit of analysis
One row equals what?- User, session, order, ticket, company, day, etc.
- State the level of aggregation (e.g., per user per week).
- Data sources
- Primary: survey, interview, usability test, experiment.
- Secondary: product logs, CRM, billing, support tools.
- Third-party: market datasets (only if needed and allowed).
- Sampling strategy
- Population and frame (who is eligible?).
- Method: random, stratified, systematic, convenience (state trade-offs).
- Size: rough estimate for signal detection; prefer larger, balanced samples when uncertain.
- Field list (schema)
- IDs: user_id, session_id, order_id, experiment_variant.
- Timestamps: event_time, created_at, window_start/end.
- Dimensions: device, channel, country, segment flags.
- Outcome: binary or numeric measurement.
- Instrumentation plan
- Event names and triggers.
- Where collected (app, web, backend).
- Who owns implementation and review.
- Quality and bias checks
- Completeness, accuracy, timeliness, duplication.
- Coverage of key segments; nonresponse risk if surveys.
- Ethics & privacy
- Minimize collection; avoid sensitive data unless essential.
- Consent, retention window, access controls, de-identification.
- Pilot test
- Collect a small sample; verify fields, logic, and joinability.
- Fix issues before full rollout.
- Field execution
- Timeline, responsibilities, checkpoints.
- How issues are reported and resolved.
- Documentation
- One-page plan + data dictionary.
- Versioning and change log.
Worked examples
Example 1: Free trial length and activation
Hypothesis: Extending trial to 21 days increases activation by 10% among SMB signups.
- Decision: Roll out 21-day trial if activation improves ≥10% within 30 days of signup.
- Outcome: Activation within 7 days (operational definition as above).
- Unit: User.
- Drivers: Trial length (14 vs 21 days, via experiment flag).
- Controls: Signup channel, country, device type.
- Sources: Product logs; experiment platform; CRM for segment (SMB).
- Sampling: A/B, random assignment; all eligible SMB signups during 4 weeks.
- Fields: user_id, signup_time, variant, activated_flag, activation_time, channel, country, device.
- Quality: Ensure variant recorded on every event; verify activation event deduplication.
- Ethics: No PII beyond necessary IDs; 90-day retention for raw logs.
- Pilot: 2 days; confirm variant and activation joins.
Example 2: Live chat and checkout abandonment
Hypothesis: Adding live chat on cart page reduces abandonment by 5%.
- Decision: Keep live chat if abandonment rate decreases ≥5% at 95% confidence.
- Outcome: Abandonment (no purchase within 24h of cart add).
- Unit: Session.
- Drivers: Chat availability (on/off), chat engagement (clicked or message sent).
- Controls: Traffic source, device, time of day.
- Sources: Web events; chat tool logs; orders database.
- Sampling: Stratified by device (mobile/desktop) to balance.
- Fields: session_id, user_id (if available), device, source, chat_shown, chat_clicked, cart_add_time, purchase_time, abandon_flag.
- Quality: Ensure time zones consistent; handle multiple sessions per user.
- Pilot: 1 week small rollout (10%).
Example 3: Payment method and fraud
Hypothesis: Fraud is higher for payment method X compared to others.
- Decision: Add extra verification for method X if fraud rate is ≥2x baseline.
- Outcome: Confirmed fraud label within 14 days.
- Unit: Order.
- Drivers: payment_method.
- Controls: order_amount, country, new_vs_returning, device.
- Sources: Orders DB; risk review outcomes.
- Sampling: All orders for past 3 months.
- Fields: order_id, user_id, order_time, amount, country, device, payment_method, fraud_flag, fraud_decision_time.
- Quality: Lag in fraud labeling; use cutoff to avoid right-censoring.
- Ethics: Mask PAN/PII; store only tokens.
Planning checklist
- Decision and threshold are written.
- Outcome, drivers, controls defined and operationalized.
- Unit of analysis and time window set.
- Sources and ownership identified.
- Sampling method and size rationale noted.
- Field list includes IDs, timestamps, dimensions, outcome.
- Instrumentation responsibilities and review process set.
- Quality checks and known risks listed.
- Ethics, consent, and retention addressed.
- Pilot test planned with success criteria.
- Documentation and versioning ready.
Exercises
Do these to cement the skill. Your answers can be brief bullet lists.
Exercise 1: Draft a lean plan
Scenario: You suspect a smaller price increase (+3%) will not harm monthly conversions for the Basic plan.
Task: Draft a concise data collection plan (10–12 bullets) that would let you test this hypothesis in 4 weeks.
- Include: decision rule, outcome metric with operational definition, unit, sources, sampling, key fields, controls, and a short pilot plan.
Mirror your work with the format in the “Worked examples.”
Exercise 2: Fix this flawed plan
Flawed plan snippet: “We will check revenue after the price change. Data comes from billing. We will look at users. If revenue goes up, we keep it.”
Your job: List at least 5 issues and propose concrete fixes for each.
- Think about missing operational definitions, confounders, sampling, time window, and ethics.
Self-check before you move on
- Can someone else implement your plan without asking you basic questions?
- Does every field have a purpose linked to your hypothesis?
- Will you be able to join all data sources reliably (IDs, timestamps)?
Common mistakes and how to self-correct
- Vague outcomes: Fix by writing a one-sentence operational definition.
- Wrong unit of analysis: Ensure it matches how outcomes occur (e.g., orders not users).
- No controls or segmentation: Include key dimensions known to affect outcomes.
- Data not joinable: Add stable IDs and synchronized timestamps.
- No pilot: Always run a short pilot to catch instrumentation gaps.
- Collecting excess PII: Minimize; keep what is necessary and document retention.
Fast audit template you can reuse
- Decision and threshold documented? Yes/No
- Outcome defined operationally? Yes/No
- Unit, window, and sources clear? Yes/No
- Sampling method justified? Yes/No
- Field list complete with IDs/timestamps? Yes/No
- Ethics/consent/retention noted? Yes/No
- Pilot steps specified? Yes/No
Practical projects
- Event spec: Draft an event schema for a checkout funnel (3–5 events, fields, triggers). Pilot it on a staging environment.
- Survey plan: Create a 7-question usability survey with sampling rules and a data dictionary.
- Attribution tags: Define a UTM taxonomy and a plan to validate it with a one-week pilot.
Mini challenge
In one paragraph, write a decision rule, outcome definition, and field list for testing whether showing product ratings on category pages increases product page views per session. Keep it under 120 words.
Learning path
- Hypothesis Framing: turn business questions into testable statements.
- Planning Data Collection: this lesson—make the test measurable and feasible.
- Experiment Design & Analysis: analyze the collected data and decide.
- Communication: summarize the plan and results for stakeholders.
Next steps
- Complete the exercises above.
- Take the Quick Test below to check your understanding.
- Apply the checklist to an active project at work.