Handling Late Data And Reprocessing

Why this matters

In real systems, data arrives late, out-of-order, or corrected after the fact. If you ignore it, your aggregates drift, labels become wrong, and models degrade. Handling late data and reprocessing lets you reconcile truth over time without breaking dashboards or model lineage.

• You will re-run backfills after a bug fix or partner resend.
• You will update training sets and models when labels are corrected.
• You must keep results reproducible and auditable across versions.

Who this is for

• MLOps Engineers building batch or streaming pipelines.
• Data/ML Engineers responsible for reliable metrics and training data.
• Anyone who needs consistent versions across data, features, and models.

Prerequisites

• Basic understanding of data pipelines (batch or streaming).
• Familiarity with dataset versioning and immutable artifacts.
• Knowing event time vs processing time is helpful but not required.

Concept explained simply

Late data are records that belong to a past time but arrive now. Reprocessing is the controlled re-run that corrects past results using late or fixed data. To do this safely, we use:

• Event time windows with a lateness allowance (watermark).
• Idempotent, re-runnable jobs with deterministic inputs.
• Data and model versioning so each correction is traceable.

Mental model

Think of your data as a ledger. New facts can append late, and sometimes you discover an entry was wrong and must be amended. You never erase history; you add a new, corrected entry and publish a new versioned report/model that references the exact data snapshot used.

Core patterns you will use

• Watermarking windows: keep windows open for a grace period (e.g., 3 days) to absorb common delays before finalizing.
• Incremental recomputation: recompute only affected windows/partitions to save time and cost.
• Deduplication and ordering: stable keys and event-time ordering prevent double counting.
• Versioned truth: every correction becomes a new dataset version; models reference exact dataset versions.
• Immutable artifacts: never overwrite in place; publish a new version/tag and deprecate the old.
• Time-travel capable storage or snapshots: allow point-in-time reads when rebuilding training sets.
• Safe rollouts: publish new versions behind a flag; enable after validation; keep rollback path.

Worked examples

Example 1 — Late events in daily revenue

Situation: Payment events can arrive up to 48 hours late. You publish daily revenue.

1. Define watermark: event_time + 72h grace (48h typical + buffer).
2. Daily job computes revenue for day D but marks it provisional until watermark passes.
3. If late events for day D arrive within 72h, recompute only partition D.
4. Publish dataset version: revenue_v2024-11-07.r2 (r2 = recomputation count).
5. Downstream dashboards read the latest version by alias: revenue_latest.

Example 2 — Partner resend requires backfill

Situation: A partner re-sends one week of corrected events (Nov 1–7).

1. Freeze a new input snapshot: partner_feed_v1.3_correction_2024-11-10.
2. Backfill compute for partitions Nov 1–7 only.
3. Write to new output dataset version: metrics_v2.1.
4. Run validation: totals and record counts vs. prior version; diff report stored.
5. Promote alias metrics_latest to v2.1 upon validation.

Example 3 — Corrected labels require model retrain

Situation: Labels for fraud were wrong for October; corrections arrive.

1. Create corrected label dataset: labels_v3.0.
2. Rebuild training set snapshot: training_data_v5.0 referencing labels_v3.0.
3. Retrain model to produce model_v2.4 with metadata: inputs=[training_data_v5.0], code_commit=abc123.
4. Run backtests on October–November; compare to model_v2.3.
5. Canary deploy model_v2.4; promote on success. Keep rollback to v2.3.

Example 4 — Change data capture (CDC) updates

Situation: Upserts arrive for customer addresses; historical aggregates must reflect the latest address at event time.

1. Ingest CDC with valid_from and valid_to periods (slowly changing dimension Type 2 style).
2. For each fact, join on address valid at event_time.
3. Reprocess only affected partitions when CDC updates close/open intervals.
4. Publish new dataset version with change manifest describing impacted partitions.

How to implement reliably

1. Decide lateness SLAs per source (e.g., 24–72h grace) and encode as watermarks.
2. Partition by event date; store deterministic inputs and outputs with versions.
3. Make jobs idempotent: deterministic grouping, stable keys, overwrite-only-target-partitions.
4. Track lineage: inputs, code commit, parameters, run ID, and output version.
5. Validate: compare aggregates, counts, and sampled diffs before promotion.
6. Communicate: publish release notes for each new version and deprecation timeline.

Common mistakes and how to self-check

• Finalizing too early: choose watermarks based on evidence of lateness; monitor late arrival rate.
• Overwriting in place: always produce a new version; keep immutability for audit and rollback.
• Global reprocess: re-run only impacted partitions/windows; keep a manifest of changes.
• Ignoring dedup: ensure stable primary keys and idempotent merges.
• Model-data mismatch: model must reference exact dataset version used; store lineage.
• No validation: define acceptance thresholds; block promotion if exceeded.

Exercises

Do these to cement the concepts. The quick test is available to everyone; only logged-in users get saved progress.

Practical projects

• Build a mini pipeline that computes daily metrics with a 48h watermark and publishes provisional and final outputs with version tags.
• Create a backfill command that accepts a date range and generates a change manifest of impacted partitions.
• Implement a training pipeline that reads a versioned snapshot and writes model artifacts with complete lineage metadata.

Learning path

• Start: Event time vs processing time and watermarks.
• Next: Idempotent writes, partitioning, and dedup keys.
• Then: Versioned datasets and artifact lineage.
• Finally: Automating backfills and safe promotions.

Mini challenge

Create a simulation where 10% of events arrive 24–72 hours late. Produce a daily metric that first publishes provisional results, then finalizes after a watermark. Log every promotion with input and output versions, and generate a diff report comparing provisional vs final totals.

Next steps

• Add monitoring for late arrival rates and automatically adapt watermarks within safe bounds.
• Introduce change manifests to target reprocessing to affected partitions only.
• Standardize version naming and release notes across data and models.