Building Training Datasets And Labels

Who this is for

You build or support ML systems and need reliable datasets and labels: ML engineers, data scientists, data/ML ops, analysts transitioning to ML.

Prerequisites

• Basic Python/SQL comfort
• Familiarity with common ML tasks (classification/regression)
• Understanding of train/validation/test concepts

Why this matters

In real projects, model performance is capped by data quality. You will be asked to: define label taxonomies, sample and split data correctly, avoid leakage, write labeling guidelines, measure labeler agreement, and version datasets so teams can reproduce results.

• Product task: Launch a new abuse classifier. You must create an initial labeled set, balance classes, and define a clear “abuse” rubric.
• Ops task: Retrain a model monthly. You must version datasets, maintain consistent splits, and track drift.
• Research task: Compare two labeling strategies. You must measure label reliability and compute inter-annotator agreement.

Concept explained simply

A training dataset is a carefully chosen set of examples plus labels that represent the real problem you want the model to learn. Good datasets are representative, clean, well-split, and consistently labeled with a clear definition of what each label means.

Mental model

Think of the dataset as a spec for the model’s behavior. The model will learn to mimic the labelers’ decisions on the sampled examples. If the labels are inconsistent or the examples don’t reflect reality, the model will copy those flaws.

Key decisions you must make

• Task framing: classification, regression, ranking, sequence tagging, detection
• Label taxonomy: names, definitions, edge cases, and examples
• Sampling strategy: random, stratified, time-aware, importance sampling for rare cases
• Splits: train/validation/test by entity/time; ensure no leakage or duplicates across splits
• Labeling method: manual annotation, programmatic (heuristics/weak supervision), distant supervision, or hybrid
• Quality control: guidelines, gold examples, audits, consensus rules, agreement metrics
• Balance: handle class imbalance with sampling or cost-sensitive learning
• Versioning: immutable dataset releases, split hashes, and change logs
• Documentation: dataset card with purpose, sources, licenses, known limitations
• Privacy & safety: remove PII where not needed, follow least-privilege access

Worked examples

Example 1: Image classification (defects vs. no-defect)

1. Task & labels: Binary classification {defect, clean}. Edge cases: hairline scratches; if scratch length > 5mm → defect.
2. Sampling: Stratified by product line and camera, include different lighting conditions.
3. Splits: By production date (month). Train: older months; Val: recent week; Test: most recent week to simulate deployment.
4. Guidelines: Include visual references; specify when glare is not a defect.
5. Quality: Two annotators per image; disagreements adjudicated by lead. Track per-annotator accuracy on gold images.
6. Versioning: Release v0.1 with 20k images; store split hashes and a changelog note: “Added line-3 camera.”

Example 2: Text sentiment (positive/neutral/negative)

1. Labels: {pos, neu, neg}. Rules: sarcasm defaults to neg unless explicit praise.
2. Sampling: Stratified by language variant and channel (email, chat). Oversample short messages where labelers often disagree.
3. Splits: By customer ID to avoid seeing the same customer in train and test.
4. Labeling: Start with weak heuristics (presence of strong sentiment words) to pre-label; human review corrects them.
5. Quality: Compute class-wise precision on a gold set weekly; require ≥0.8 on minority class before expanding.
6. Imbalance: Upsample minority (pos) in training only; keep validation/test natural.

Example 3: Fraud detection (time-series)

1. Labels: fraud = chargeback within 90 days; non-fraud = no chargeback within 90 days.
2. Sampling: Include weekends/holidays where behavior differs.
3. Splits: Time-based. Train: months 1–6, Val: month 7, Test: months 8–9. Prevent future features at training time (no post-transaction signals).
4. Leakage watch: Exclude any features derived after the prediction moment (e.g., chargeback outcome flags).
5. Evaluation: PR-AUC due to strong class imbalance; calibrate thresholds per business cost.

Step-by-step: Build a small training set now

1. Define labels
   Pick a binary task (e.g., spam vs. not-spam for short messages). Write rules for ambiguous cases (e.g., promotions vs. transactional notifications).
2. Sample 200 raw examples
   Stratify by source and length. Deduplicate exact and near-duplicates (e.g., case-insensitive match).
3. Split
   Train: 140, Val: 30, Test: 30. If users exist, split by user to avoid leakage.
4. Labeling
   Have 2 people label independently. Use 10 gold examples to calibrate.
5. Quality
   Compute agreement (simple percent agreement). Review disagreements and update rules.
6. Version & document
   Save v0.1 with counts per split and final label rules.

Quality control and label reliability

• Agreement metrics: start with percent agreement; for subjective tasks, use Cohen’s kappa (2 labelers) or Fleiss’ kappa (≥3) to account for chance agreement.
• Consensus policies: majority vote; or weighted by labeler accuracy on gold items.
• Audits: sample 5–10% of items weekly for expert review; track recurring failure modes.

Common mistakes and how to self-check

• Leakage via entity overlap: The same user/product appears in train and test. Fix: split by entity or time where needed.
• Moving target labels: Guidelines change mid-project. Fix: freeze rules per version; re-label a subset to assess drift.
• Imbalanced metrics: Only reporting accuracy when positive rate is low. Fix: use PR-AUC, F1, class-wise metrics.
• Hidden duplicates: Near-duplicate texts in train and test. Fix: fuzzy matching or hash-based deduplication before split.
• Ignoring edge cases: No policy for borderline items. Fix: add an “uncertain” bucket during drafting, then resolve into final rules.

Exercises

Do these in your notebook or doc, then compare with the solutions revealed below.

• Exercise 1: Define labels and splits for a support ticket urgency classifier.
• Exercise 2: Spot leakage and propose a time-aware split for a subscription churn dataset.

Practical projects

• Moderation triage: Build a 1k-example dataset for harmful content vs. safe, with clear sarcasm rules; measure kappa ≥ 0.7.
• Pricing anomalies: Create a regression dataset predicting typical price and a binary outlier label for anomalies; ensure time-based splits.
• Support topic tagging: Multi-label dataset with 10 topics, consensus via 3 labelers; document prevalence per topic.

Learning path

1. Start small: 200–500 examples with tight rules
2. Set up splits and versioning from day one
3. Measure agreement and fix guidelines
4. Scale labeling with programmatic priors + human review
5. Monitor drift and update versions on a schedule

Mini challenge

Draft a 1-page labeling guideline for a 3-class text classifier (toxic, borderline, safe) with at least 5 edge cases and example phrases. Include a plan to compute agreement and resolve disagreements.

Next steps

• Integrate your dataset into the training pipeline with immutable version IDs
• Add continuous labeling for new edge cases
• Automate data validation checks (schema, duplicates, distribution shifts)