Reproducible Training Sets

Why this matters

As an MLOps Engineer, you will be asked to train and re-train models reliably. Stakeholders expect that a model trained last month can be reproduced today with the same data and splits. Reproducible training sets make experiments comparable, speed up debugging, and protect against silent data leakage.

• Audit and compliance: Show the exact data used for a specific model version.
• Debugging: Re-run a training job and get identical inputs to isolate code issues.
• Fair evaluation: Ensure train/validation/test splits never leak data across boundaries.
• Collaborative science: Teammates can re-create your dataset outputs from the same recipe.

Who this is for

Engineers and data scientists who manage datasets and training pipelines and need deterministic, audit-ready data inputs.

Prerequisites

• Basic Python knowledge.
• Familiarity with CSV/Parquet files and directory structures.
• Understanding of train/validation/test concepts.

Concept explained simply

Think of a reproducible training set as a recipe card. If you follow the same steps with the same ingredients, you get the same dish every time. In ML, those ingredients and steps are recorded in a dataset manifest.

Mental model: The Recipe Card

• Ingredients: raw data snapshot (files/tables), labels, and augmentation rules.
• Measurements: checksums, row counts, schema, timezones.
• Instructions: deterministic split logic and random seeds.
• Result: a manifest that fully specifies how the training set was constructed.

Core workflow (step-by-step)

1. Pin the sources: Use immutable references (file checksums, snapshot IDs, or exact query + warehouse snapshot time).
2. Extract and freeze: Pull data into a snapshot directory and compute checksums and counts.
3. Deterministic split: Use a stable method (e.g., hash of a primary key + seed) or time-based cutoff for temporal data.
4. Record everything: Write a manifest.json with sources, transforms, seeds, and outputs.
5. Train: Your training job reads the manifest to load data, not ad-hoc paths.
6. Evaluate and register: Log metrics together with the manifest reference and model version.
7. Archive: Keep the snapshot and manifest to reproduce in the future.

{
  "dataset_name": "churn_v1",
  "created_at": "2026-01-04T00:00:00Z",
  "seed": 42,
  "sources": [{"path": "data/raw/customers_2025-12-31.parquet", "sha256": "...", "rows": 120345}],
  "transforms": ["filter country in [US,CA]", "join subscriptions by customer_id"],
  "split": {"method": "keyed_hash", "key": "customer_id", "ratios": {"train": 0.8, "val": 0.1, "test": 0.1}},
  "outputs": {
    "train": {"path": "data/processed/churn_v1/train.parquet", "sha256": "...", "rows": 96276},
    "val":   {"path": "data/processed/churn_v1/val.parquet",   "sha256": "...", "rows": 12034},
    "test":  {"path": "data/processed/churn_v1/test.parquet",  "sha256": "...", "rows": 12035}
  },
  "env": {"python": "3.11", "pandas": "2.2.0"}
}

Worked examples

Example 1: Tabular dataset with keyed-hash split

Goal: deterministically split a CSV by hashing a stable key (e.g., customer_id) with a seed so every rerun yields identical splits.

import hashlib, json, csv
seed = 42
salt = str(seed).encode()

def bucket_for_id(x):
    h = hashlib.sha256(salt + str(x).encode()).hexdigest()
    # map to [0, 9999]
    return int(h[:4], 16) % 10000

with open("customers.csv", newline="") as f:
    rows = list(csv.DictReader(f))

train, val, test = [], [], []
for r in rows:
    b = bucket_for_id(r["customer_id"])
    if b < 8000: train.append(r)
    elif b < 9000: val.append(r)
    else: test.append(r)

# Write outputs and a simple manifest
# (Compute file checksums as well in a real pipeline.)

Example 2: Image classification with augmentations

Goal: list every image with checksum, label from folder name, and record augmentation policy and seeds.

import hashlib, os, json
seed = 123

def sha256_file(p):
    h = hashlib.sha256()
    with open(p, 'rb') as f:
        for chunk in iter(lambda: f.read(1<<20), b''):
            h.update(chunk)
    return h.hexdigest()

items = []
for root, _, files in os.walk("images"):
    for fn in files:
        if fn.lower().endswith((".jpg", ".png")):
            p = os.path.join(root, fn)
            label = os.path.basename(root)
            items.append({"path": os.path.relpath(p, "images"), "label": label, "sha256": sha256_file(p)})

manifest = {
    "seed": seed,
    "augmentations": {"flip": True, "color_jitter": {"brightness": 0.2}, "seed": seed},
    "items": items
}
print(json.dumps(manifest)[:2000])

Example 3: Time-series with time-based split

Goal: split by cutoff date to avoid leakage. Record timezone and resampling rules.

Checklist before you train

• Data sources are pinned (checksums or snapshot timestamp recorded).
• Split logic is deterministic (keyed hash seed or time cutoff).
• All seeds (numpy, torch, tf, python) are set and logged.
• Transformations and filters are listed in the manifest.
• Output files and their checksums are recorded.
• Label mappings and class counts are stored.
• Environment versions affecting IO are captured.

Exercises

Do these hands-on tasks. Full instructions and solutions are in the exercise blocks below on this page.

• Exercise 1: Create a reproducible train/val/test split for a tabular CSV and emit a manifest with checksums.
• Exercise 2: Build an image dataset manifest that logs file hashes, labels, split, and augmentation parameters.

Common mistakes and self-checks

• Random split without a fixed method: If you create splits by shuffling rows each run, you will not reproduce. Use a keyed-hash split or fix a seed and persist the split indices.
• Forgetting time semantics: In time-series, random splits leak future into past. Use cutoffs by timestamp.
• Not logging label sources: If labels were recomputed, you must record the label code version/date or commit hash.
• Only recording filenames: Filenames can change; checksums give identity. Self-check: compute a checksum over concatenated file hashes to verify the set.
• Ignoring environment versions: Different readers (CSV vs Parquet versions) can change parsing. Record versions that affect IO.

Practical projects

• Tabular churn project: from a raw customer table, build a reproducible dataset with keyed-hash split and a manifest including schema and label mapping.
• Image quality classifier: scan an image folder, compute checksums, generate deterministic splits, and record augmentation parameters in the manifest.
• Sensor forecasting: implement time-based splits with explicit cutoffs and timezone; log resampling and window features in the manifest.

Mini challenge

Take an existing project and freeze its dataset into a versioned folder (e.g., datasets/project_X/v2). Produce a manifest that, when handed to a teammate, lets them reconstruct train/val/test exactly. Acceptance criteria: row counts match, class distributions match to the unit, and file-level checksums match.

Quick test & progress

Take the quick test below to check your understanding. The test is available to everyone. Only logged-in users will have their progress saved automatically.

Learning path

• Start here: reproducible training sets and manifests.
• Next: data lineage and tracking transformations.
• Then: model versioning tied to dataset manifests.
• Later: automation in CI/CD for data pipelines.

Next steps

• Adopt a manifest template and use it on every new dataset.
• Automate checksum computation and split generation in your pipeline.
• Attach the dataset manifest ID to every trained model artifact.