Reproducibility And Artifact Management

Why this matters

In real Machine Learning projects, you must be able to rerun an experiment months later and get the same result, explain exactly which data and code created a model, and confidently promote artifacts to production. Reproducibility and artifact management are how you keep promises to teammates, auditors, and customers.

• Investigations: Re-run a failed training job with identical environment and data.
• Compliance: Show lineage from model back to data snapshot and code commit.
• Operations: Promote the correct model artifact to staging/production without ambiguity.
• Teamwork: Share experiments and compare runs apples-to-apples.

Concept explained simply

Reproducibility means: if someone uses the same code, data, configuration, and environment, they should get the same results. Artifact management means: everything your ML workflow produces (datasets, features, models, metrics, plots, configs, logs) is versioned, named clearly, stored safely, and traceable.

Mental model

Imagine two things: a lab notebook and a warehouse.

• Lab notebook (tracking): records your run settings, seeds, code commit, dataset version, metrics, and outputs.
• Warehouse (artifact store): safely holds versioned artifacts with labels and checksums so you can fetch exactly the right item later.

Key components

• Code versioning: Commit IDs for training and inference code.
• Environment pinning: Exact versions of dependencies, OS/base image, and hardware notes if relevant.
• Data versioning: Immutable dataset snapshots or references with checksums.
• Configuration: Structured config (YAML/JSON) stored with the run.
• Randomness control: Seed all frameworks (NumPy, PyTorch, TensorFlow, sklearn) and note nondeterministic ops.
• Experiment tracking: Run IDs, metrics, params, artifacts, lineage.
• Artifact registry: Consistent naming, versions (semantic or numeric), promotion stages (dev, staging, prod), checksums.

Worked examples

Example 1: Pin the environment

Goal: Freeze dependencies and base image so anyone can rebuild the same environment.

# Generate pinned dependencies
pip freeze > requirements.txt

# Optional: capture Python and OS info
python -V  # e.g., Python 3.10.13

# Minimal Dockerfile example (if you containerize)
# Dockerfile
# FROM python:3.10.13-slim
# WORKDIR /app
# COPY requirements.txt ./
# RUN pip install --no-cache-dir -r requirements.txt
# COPY . .
# CMD ["python", "train.py", "--config", "configs/run.yaml"]

Record the exact Python version, requirements file checksum, and Docker image tag if used.

Example 2: Version the data

Goal: Ensure the training data snapshot is immutable and traceable.

# Create a manifest describing the data snapshot
# data_manifest.json
{
  "dataset_name": "transactions_v3",
  "storage_uri": "s3://ml-data/transactions/2024-07-31/",
  "record_count": 10439821,
  "schema_hash": "4cc3...e1",
  "content_hash": "sha256:9fbc...77",
  "created_at": "2024-07-31T23:59:59Z"
}

Use the manifest in your run record so training always points to the same snapshot.

Example 3: Model artifact naming and promotion

Goal: Store models with predictable names and promote them safely.

# Example artifact layout
models/
  fraud-detector/
    1.2.0/
      model.pkl
      metrics.json
      run.json
      data_manifest.json
      checksum.txt  # sha256 of model.pkl
    1.2.1/
      ...
    staging -> 1.2.1   # alias/symlink to current staging
    prod    -> 1.2.0   # alias/symlink to current production

Hands-on: exercises

Do these to internalize the concepts. You can complete them locally. Keep your outputs in a single folder called "repro_lab".

Exercise 1: Make a run reproducible (ex1)

1. Create a small script that trains any simple model (e.g., logistic regression) on a toy dataset.
2. Set seeds for Python, NumPy, and your ML framework.
3. Generate a pinned requirements.txt and record Python version.
4. Create a run.yaml recording: timestamp, git_commit (placeholder if not using git), data_manifest reference, config used, seeds, and output artifact paths.

Exercise 2: Design an artifact registry (ex2)

1. Define a naming scheme: <project>/<model-name>/<version>/
2. Create a sample directory for version 0.1.0 with model.bin, metrics.json, and checksum.txt (sha256 hash of model.bin).
3. Create aliases: staging and prod pointing to versions.
4. Write a short promotion checklist (conditions to move from staging to prod).

Exercise checklist

• [ ] Seeds set and logged
• [ ] Pinned environment captured
• [ ] Data manifest present
• [ ] Run config saved
• [ ] Artifacts named and checksummed
• [ ] Promotion policy written

Common mistakes and self-check

Frequent pitfalls

• Not pinning dependencies, leading to silent version drift.
• Using live/mutable data instead of a snapshot.
• Forgetting to set or log random seeds.
• Storing large binary artifacts directly in plain Git instead of an artifact store.
• No checksums, so corrupted or wrong files go unnoticed.
• Config edited manually without saving the exact used version.
• Mixing train/test during preprocessing and not saving the exact transforms.

Self-check

Try to reproduce your last run on a fresh machine or container. You pass if:

• Hash of your model file is identical.
• Key metrics match within tolerance (or exactly for deterministic pipelines).
• Run record resolves to concrete code commit, data snapshot, and config.
• Anyone can locate the correct artifact via name + version alone.

Practical projects

• Reproduce a baseline: Build a deterministic baseline model for a small dataset, store artifacts, and write a one-page reproducibility report.
• Artifact lifecycle demo: Train two model versions, promote one to staging, switch prod alias, and document rollback.
• Data drift rehearsal: Save two data snapshots, retrain model, compare metrics, and log lineage differences.

Learning path

1. Start with basic versioning: Git for code, requirements.txt for deps.
2. Add data manifests and snapshot references.
3. Introduce experiment tracking for runs and metrics.
4. Set up an artifact registry layout and promotion flow.
5. Automate in CI to validate reproducibility on fresh environments.

Who this is for

• ML Engineers making models production-ready.
• Data Scientists collaborating in teams.
• MLOps practitioners standardizing pipelines.

Prerequisites

• Basic Python and a ML framework (sklearn, PyTorch, or TensorFlow).
• Familiarity with Git and virtual environments or containers.

Next steps

• Automate reproducibility checks in CI.
• Add model evaluation and data validation gates before promotion.
• Standardize run templates and naming conventions across the team.

Mini challenge

Take an old experiment you ran. Without changing code, rebuild the environment from scratch, fetch the exact data snapshot, and attempt to reproduce the model file hash and metrics. Note any gaps you had to patch and update your process to prevent them next time.

Quick Test

Everyone can take the test for free. If you are logged in, your progress is saved.