Dataset Design And Sampling

Why this matters

Great models start with great datasets. As an Applied Scientist, you will decide what data to collect, how much is enough, how to split it, and how to keep it representative, fair, and leak-free. Real tasks include:

• Designing training/validation/test splits that reflect production traffic.
• Handling rare but critical cases (fraud, safety, long-tail intents).
• Avoiding leakage when data has time, users, or groups.
• Balancing cost of labeling with performance needs.

Good dataset design reduces iteration cycles, prevents misleading metrics, and increases model reliability.

Concept explained simply

Dataset design and sampling is choosing which examples to include and how to split them so your model learns patterns it will see in the real world.

Mental model

Think of your dataset as a map of reality. If parts of the map are missing or exaggerated, your model will get lost. Your job: sketch the map at the right scale, keep the landmarks (rare cases), and check that the map matches the terrain over time.

Key principles

• Define target distribution first: Which users, time windows, regions, and tasks does the model serve?
• Stratify by critical factors: Class labels, time, geography, device, user groups, or any attribute that shifts outcomes.
• Split by groups to avoid leakage: Ensure the same user, session, or item family does not appear in both train and test.
• Use time-aware splits when there is drift: Train on past, validate slightly later, and test on the most recent period.
• Size for power: Ensure enough positive examples per slice to detect performance differences (e.g., ≥300 positives per important slice if feasible).
• Handle imbalance thoughtfully: Combine class-weighting/oversampling with evaluation on the true distribution.
• De-duplicate and near-de-duplicate: Prevent the same or nearly identical records from inflating performance.
• Create evaluation slices: Always report metrics by key segments, not only overall.

Worked examples

Example 1: Credit default prediction (rare positive class)

Goal: Predict default within 90 days. Positive rate ≈ 3%.

• Target distribution: Match recent 6 months of applications.
• Split: Time-based. Train on months 1–3, validate on month 4, test on months 5–6.
• Sampling: Stratify by region and income band. Ensure ≥300 defaults per split; need about 10,000 examples per split at 3% rate.
• Leakage guard: Group by customer; a customer appears in only one split.
• Evaluation: AUC-ROC overall; PR-AUC for positives; slices by region and income band.

Example 2: Product image classification (long tail)

Goal: Identify product category among 200 classes; many rare categories.

• Target distribution: Production skew with long tails.
• Training: Oversample rare classes 2–5x or use class-balanced sampling.
• Validation/Test: True distribution (no oversampling) to avoid inflated metrics.
• Deduplication: Remove near-duplicates by product ID or perceptual hash grouping before splitting.
• Evaluation: Macro-F1 and per-class F1 to ensure long-tail quality.

Example 3: Search ranking (session-based)

Goal: Rank documents given a query; labels from clicks.

• Group split: Split by session or user to avoid leakage of behavior patterns.
• Temporal holdout: Last 2 weeks as test to measure robustness to recent changes.
• Sampling: Downsample head queries for training to diversify tail; keep true distribution for test.
• Bias check: Clicks are biased by position; include counterfactual or debiasing weights in training; report slice metrics by device and locale.

Example 4: Time-series anomaly detection

• Windowed data: Use sliding windows; ensure windows from the same incident do not span train and test.
• Split: Train on earlier windows, validate on middle, test on latest.
• Imbalance: Anomalies are rare; oversample anomalies for training; evaluate using precision/recall at anomaly level, not point level only.

Step-by-step workflow

1. Define scope: Users, timeframe, locales, and critical slices you must cover.
2. Inventory signals: Labels, features, and any grouping keys (user, item, session).
3. Choose split strategy: Random, group-aware, or time-aware (often a combination).
4. Plan sample sizes: Ensure enough positives per important slice; compute totals from base rates.
5. Design training sampling: Balance classes via weights or oversampling; keep evaluation at true distribution.
6. Deduplicate: Exact and near-duplicate checks.
7. Create evaluation slices: By class, region, device, time, and risk categories.
8. Document decisions: Why this distribution, split logic, and known trade-offs.

Exercises

These mirror the exercises below. Do them in order. Tip: The quick test is available to everyone. Only logged-in users have their progress saved.

Exercise 1: Compute sample sizes for rare events

You are building a safety classifier where the positive rate is ~2%. You want 300 positive examples in validation and 300 in test. Calculate how many total examples you need in each split and outline a training sampling plan.

Exercise 2: Design a leakage-safe split

You have transactions with fields: transaction_id, user_id, timestamp. You predict chargebacks within 30 days. Propose train/val/test split logic that avoids leakage and reflects production.

Exercise 3: Build evaluation slices

For a multilingual intent classifier (languages: EN 70%, ES 20%, FR 10%), propose evaluation slices and how you would balance training while keeping test representative.

• Checklist before you submit your answers:
  • Did you justify your split choice?
  • Did you quantify sample sizes?
  • Did you specify training vs. evaluation distributions?
  • Did you include leakage guards?

Common mistakes and self-check

• Leakage from time or users: Self-check by searching for identical user_ids or near-duplicate timestamps across splits.
• Evaluating on oversampled data: Self-check by verifying test set uses the true production distribution.
• Ignoring long-tail classes: Self-check with macro-F1 and per-class metrics, not only overall accuracy.
• Too few positives per slice: Self-check by counting positives per slice; target ≥300 if possible.
• No deduplication: Self-check by hashing key fields and checking overlaps across splits.

Practical projects

• Project A: Create a dataset card for a binary classifier. Include target distribution, split design, leakage risks, and slice metrics plan.
• Project B: Simulate class imbalance. Train a simple model with and without class-balanced sampling; compare per-class F1 changes.
• Project C: Build a time-based holdout. Show how metrics drift month-to-month; document when to refresh training data.

Mini challenge

Your model underperforms on new users. Without collecting more labels, propose a dataset redesign that could improve generalization. Consider group-aware splits, augmentation, and reweighting.

Who this is for

• Applied Scientists, ML Engineers, and Data Scientists designing training/evaluation datasets.
• Team leads defining labeling budgets and quality standards.

Prerequisites

• Basic probability and classification metrics (precision/recall, ROC/PR AUC).
• Comfort with data manipulation (e.g., Python/Pandas) and handling timestamps/IDs.

Learning path

1. Understand target distribution and slices.
2. Master split strategies (random, group-aware, time-aware).
3. Handle imbalance (weights, oversampling) and keep evaluation realistic.
4. Implement deduplication and leakage checks.
5. Create slice reporting and monitor drift over time.

Next steps

• Complete the exercises and take the quick test.
• Apply the checklist to a current project and document your dataset decisions.
• Review results with your team and iterate on slices and sampling.