Label Versioning And Corrections

Who this is for

MLOps engineers, data engineers, and ML practitioners who maintain datasets and need reliable, auditable ways to update labels without breaking reproducibility.

Why this matters

• You will fix mislabels, update class names, and merge/split classes as projects evolve.
• Models trained on corrected labels must be reproducible and comparable to previous versions.
• Auditors and teammates will ask: what changed, why, and can we roll it back?

Concept explained simply

Label versioning tracks how labels change over time, just like code versioning. A label correction is a small, documented edit to the truth you train on. Together, these let you reproduce any past model and safely improve labels.

Mental model

Think of labels as a layered cake: base labels at the bottom, small correction patches stacked on top. Each layer is recorded. You can rebuild the cake to any layer (version), compare slices (metrics), and keep the recipe (metadata) for each change.

A safe label-correction workflow

1. Freeze a base: choose a label version (e.g., labels-v1.2) to patch.
2. Propose a patch: collect issues/mislabels, prepare a small change set.
3. Review: a second person checks changes; document rationale.
4. Apply and version: apply patch, run checks, tag new version (labels-v1.3).
5. Retrain and compare: train with v1.3, compare metrics to v1.2.
6. Decide: keep or roll back; update changelog.

Worked examples

Directory and files (one practical structure)

dataset/
  images/...
  labels/
    labels-v1.2.jsonl
    labels-v1.3.jsonl
    patches/
      edits-v1.2-to-v1.3.jsonl
    schema/
      schema-v1.yaml
      schema-v2.yaml
      mappings/
        rename-automobile-to-car.json
        split-dog-small-large.json
  splits/
    train.txt
    val.txt
    test.txt
  meta/
    changelog.md
    versions.csv

version: labels-v1.3
base: labels-v1.2
date: 2026-01-04
author: Sam Reviewer: Alex
reason: Corrected 12 dog->cat errors found by triage query
artifacts:
  patch: patches/edits-v1.2-to-v1.3.jsonl
  guideline: G-2024-07
checks:
  changed_items: 12
  class_counts_delta: {"dog": -12, "cat": +12}

Handling schema changes safely

• Renames: store a rename map; metrics comparable 1:1.
• Merges (A+B -> C): keep a mapping file and note that historic metrics may only be comparable after remapping.
• Splits (A -> A1, A2): define deterministic rules; record them; maintain an evaluation mapping to aggregate A1+A2 to old A when comparing to historic models.

{
  "merge": {
    "sedan": "car",
    "hatchback": "car"
  }
}

Quality checks and safeguards

• Frozen splits: do not silently move items across train/val/test when changing labels. If you must correct test labels, create a new test version (test-v2) and never compare v1 to v2 without noting it.
• Sanity checks: ensure no orphan classes, no empty polygons, valid bbox coordinates, and class distribution deltas make sense.
• Inter-annotator agreement: sample 50 items; double-annotate and compute agreement (e.g., percent agreement, Cohen's kappa) to validate corrections.
• Lineage stamp: for each model, record raw data hash, label version, split version, and training config hash.

Common mistakes

• Silent schema change: renaming classes without a recorded mapping.
• Overwriting labels in place: losing the ability to reproduce past models.
• Mixing data and label changes: not isolating label-only versions, making debugging harder.
• Correcting test labels but comparing metrics to the old test set.
• Non-deterministic patching: unordered or ambiguous patch rules.

Exercises

Do these hands-on tasks. The quick test at the end is available to everyone; sign in to save your progress.

Exercise 1 — Build and apply a small label patch

Goal: create a patch that flips 5 mislabeled items and outputs a new label version with a mini changelog.

1) base.jsonl (snippet)
{"id":"img01.jpg","label":"dog"}
{"id":"img02.jpg","label":"dog"}
...
2) patch.jsonl
{"id":"img02.jpg","from":"dog","to":"cat","reason":"ear shape"}
...
3) Apply:
- Read base into dict by id
- For each patch: assert dict[id].label == from; then set to
- Write out labels-v1.1.jsonl sorted by id
4) Changelog:
version: labels-v1.1
base: labels-v1.0
changed_items: 5
class_counts_delta: {"dog": -5, "cat": +5}

Exercise 2 — Rename and split with mappings

Goal: perform a rename (automobile -> car) and a split (dog -> dog_small, dog_large) using rules and produce comparable metrics plan.

Order:
1) Rename
2) Split with area rule
Artifacts:
- schema/mappings/rename-automobile-to-car.json
- schema/mappings/split-dog-small-large.json
- evaluation_remap.json:
  {"aggregate_for_old": {"dog": ["dog_small","dog_large"]}}

Practical projects

• Build a label patch CLI that validates "from" before applying and outputs a delta report.
• Create a schema migration tool that supports rename, merge, and split with a dry-run mode and a class distribution preview.
• Automate a label QA pipeline that runs sanity checks and produces a versioned HTML report per label release.

Learning path

1. Start with deterministic patch application and changelogs.
2. Add schema migration (rename/merge/split) with mapping files.
3. Introduce automated checks and lineage stamps for each model training run.
4. Scale to larger datasets with storage-efficient diffs and CI checks.

Prerequisites

• Basic understanding of dataset splits (train/val/test).
• Comfort with JSON/CSV and simple scripting.
• Familiarity with version control concepts (commits, tags).

Next steps

• Integrate label version tags into your training pipelines.
• Add a mandatory review step before publishing new label versions.
• Track evaluation remaps for fair historical comparisons.

Mini challenge

Given labels-v1.5 and a patch that fixes 20 items only in validation, produce labels-v1.6 and a one-page QA summary including: changed count per split, class deltas, and a note on whether comparisons to past validation results remain fair. Keep it deterministic and reproducible.