Data Lineage And Documentation

Why this matters for Machine Learning Engineers

Models fail quietly when data changes loudly. Data lineage shows where data comes from, how it changes, and where it goes. Good documentation tells people how to use it safely. Together they enable:

• Root-cause analysis when model metrics drop after a data change
• Impact analysis before a schema or transformation change
• Compliance proofs (PII removal, retention policies)
• Reproducibility of training and consistent backfills
• Onboarding and handoffs without tribal knowledge

Concept explained simply

Data lineage is the map of your data’s journey: each stop (node) is a dataset or step, each arrow (edge) is a transformation. Documentation is the travel guide that explains rules, risks, and how to operate the route.

Mental model

Imagine a passport that gets stamped at every hop:

• Source stamp: where data came from and when
• Code stamp: which code/version transformed it
• Check stamp: what validations passed or failed
• Owner stamp: who is responsible
• Downstream stamp: who depends on this output

A minimal lineage + docs spec you can apply today

Record this for every pipeline output (table, feature view, model dataset):

• Identity: dataset name, environment, owner, contact
• Provenance: input datasets, job name, run id, run time window
• Code: repo, commit/tag, job version, environment image
• Schema: column list, types, units, nullable flags, schema hash
• Transformation summary: key steps, filters, joins, aggregations
• Quality checks: rules and latest results
• PII & compliance: fields with sensitivity labels and policies
• Downstream deps: jobs, models, dashboards consuming this dataset
• Change log: what changed, when, why, and migration notes

{
  "dataset": "features.user_churn_daily",
  "env": "prod",
  "owner": "ml-platform@company",
  "inputs": [
    "raw.users", "raw.events", "ref.exchange_rates"
  ],
  "job": {
    "name": "build_user_churn_features",
    "run_id": "2026-01-01T02:00Z",
    "window": "[2025-12-31, 2026-01-01)"
  },
  "code": {
    "repo": "git://company/ml-features",
    "commit": "a1b2c3d",
    "image": "features:py3.11-1.4"
  },
  "schema": {
    "hash": "sha256:...",
    "columns": [
      {"name": "user_id", "type": "string", "nullable": false},
      {"name": "sessions_7d", "type": "int", "nullable": false, "unit": "count"},
      {"name": "avg_session_minutes", "type": "float", "unit": "minutes"},
      {"name": "is_premium", "type": "bool"}
    ]
  },
  "transform_summary": "Events joined to users on user_id; 7-day window aggregations; timezone normalized to UTC; outliers winsorized at p99.",
  "quality": [
    {"rule": "null_rate(avg_session_minutes) < 0.05", "status": "pass"},
    {"rule": "sessions_7d >= 0", "status": "pass"}
  ],
  "pii": {"contains_pii": false, "notes": "user_id is hashed."},
  "downstream": ["model.user_churn_v3"],
  "changelog": [
    {"date": "2025-12-20", "change": "Added avg_session_minutes", "impact": "Non-breaking; nullable"}
  ]
}

Change log entry template
- Date:
- Change type: [schema | logic | schedule | infra]
- Description:
- Reason:
- Backward compatibility: [breaking | non-breaking]
- Migration/Backfill plan:
- Blast radius (downstreams affected):
- Owner + approval:

Worked examples

Who this is for

• Machine Learning Engineers building and maintaining feature pipelines
• Data Engineers supporting ML training and inference data
• Analysts who consume ML outputs and need trustable datasets

Prerequisites

• Basic SQL and a data orchestration tool familiarity
• Understanding of schemas, partitions, and job scheduling
• Git basics for versioning

Learning path

1. Capture minimal lineage fields for one existing pipeline
2. Add quality checks and record their results
3. Write a concise runbook and data dictionary
4. Pilot a change log process for schema or logic changes
5. Automate lineage capture from your orchestrator and storage

Exercises

Note: The quick test is available to everyone; only logged-in users will have saved progress.

Exercise checklist

• Lineage record has all minimal fields
• Schema or logic changes are captured in change log
• Backfill or migration plan is present where needed
• Quality checks are specific and measurable

Common mistakes and how to self-check

• Mistake: Only documenting tables, not transformations. Self-check: Does your summary list filters, joins, and aggregation levels?
• Mistake: No code version in lineage. Self-check: Can you reproduce last Tuesday’s run with the same commit and image?
• Mistake: Missing units and null semantics. Self-check: For every numeric column, is unit and null policy written?
• Mistake: Treating change logs as optional. Self-check: Can someone reconstruct why a metric shifted on a specific date?
• Mistake: Not tracking downstream consumers. Self-check: Before a breaking change, do you get a concrete list of affected jobs/models?

Practical projects

• Add minimal lineage to one production feature pipeline and export a one-page PDF report for stakeholders
• Create a data dictionary for a core feature view, including units, null handling, and calculation rules
• Design a change management workflow: change log file, review checklist, and notification template
• Automate recording of run metadata (run id, window, commit) into table properties or a sidecar audit table

Mini challenge

Pick a model with degraded performance last month. Using lineage and change logs, find the earliest upstream change that could explain the drop. Draft a rollback or backfill plan in 5 bullet points.

Next steps

• Roll out the minimal lineage spec across top 3 pipelines
• Add automated collection from your scheduler and storage
• Introduce service-level objectives (freshness, completeness) and alerting
• Schedule a quarterly lineage audit and documentation refresh