Backfills And Recomputes

Why this matters

Backfills and recomputes keep your feature store trustworthy. In real MLOps, you will: fix feature bugs and recompute history, add new lookback windows for models, recover from source data gaps, and rebuild online features without downtime. Doing this safely avoids data leakage, outages, and expensive mistakes.

Concept explained simply

Backfill means computing features for past time ranges that are missing or outdated. Recompute means rebuilding features for ranges you already computed, usually after a bug fix, logic change, or source correction.

Key concepts and decisions

• Scope: full (all partitions) vs partial (specific dates/entities).
• Stores: offline (batch) vs online (serving). Usually backfill offline first, validate, then materialize online.
• Point-in-time correctness: join using event/cutoff time to prevent label leakage.
• Idempotency: design upserts with deterministic keys so re-runs don't duplicate records.
• Versioning: write to a new feature version, validate, then switch readers.
• Orchestration: chunk runs by partition (e.g., day) with retries and bookmarks/watermarks.
• Cost control: target only impacted partitions; parallelize within limits; pause low-value windows.
• Data quality: row counts, completeness, distribution drift, and join rate checks before rollout.
• Online updates: prefer shadow/backfill to a new version, then gradually cut traffic. Use TTL to retire stale entries.

Worked examples

INSERT OVERWRITE TABLE features.user_30d_purchase_count_v2 PARTITION (dt)
SELECT
  u.user_id,
  t.event_time AS feature_time,
  COUNT(*) FILTER (WHERE e.event_time BETWEEN t.event_time - INTERVAL '30' DAY AND t.event_time) AS cnt,
  t.dt
FROM time_grid t
JOIN events e ON e.user_id = t.user_id AND e.event_time <= t.event_time
JOIN users u ON u.user_id = t.user_id
WHERE t.dt BETWEEN '2025-11-01' AND '2026-01-01'
GROUP BY u.user_id, t.event_time, t.dt;

MERGE INTO fs.churn_score_180d_v1 AS tgt
USING (
  SELECT customer_id, cutoff_time,
         some_score_fn(events, cutoff_time) AS churn_score
  FROM feature_builder
  WHERE dt = :day
) AS src
ON tgt.customer_id = src.customer_id AND tgt.feature_time = src.cutoff_time AND tgt.version = 'v1'
WHEN MATCHED THEN UPDATE SET churn_score = src.churn_score
WHEN NOT MATCHED THEN INSERT (customer_id, feature_time, version, churn_score)
VALUES (src.customer_id, src.cutoff_time, 'v1', src.churn_score);

Checklists and safe steps

1. Define scope
   • What feature(s) and version?
   • Time window(s) and entity filters?
   • Offline only or also online?
2. Design for safety
   • Point-in-time joins; no future lookups.
   • Idempotent MERGE/UPSERT keys: entity_id + feature_time + feature_name + version.
   • Write to a new version; avoid in-place edits.
3. Run efficiently
   • Partition by date; parallelize within resource limits.
   • Use bookmarks/watermarks to resume on failure.
   • Estimate cost and runtime; throttle if needed.
4. Validate
   • Row counts and null rates.
   • Join completeness and unique key checks.
   • Distribution comparison (KS test or simple percentiles).
5. Roll out
   • Materialize to online as new version.
   • Shadow or canary release before full switch.
   • Monitor feature freshness and serving errors.

Exercises

Do the exercise below. Everyone can do it for free; saved progress is available to logged-in users.

Common mistakes and self-check

• Leakage: joining features using processing time instead of event/cutoff time. Self-check: verify no feature record has feature_time after label_time.
• Non-idempotent writes: INSERT only without keys. Self-check: rerun a partition and confirm row count does not grow.
• Unscoped recompute: full rebuild when a small partition would do. Self-check: list exactly which partitions/entities are truly impacted.
• Skipping validation: promoting without row-count and distribution checks. Self-check: write down pass/fail thresholds before running.
• Risky online edits: overwriting the active version. Self-check: ensure a shadow version exists and a rollback switch is ready.

Practical projects

• Build a backfill job that recomputes a 7-day rolling metric for the last 90 days, with MERGE-based idempotency and daily partition retries.
• Implement blue/green feature versioning: write v2 offline, validate, materialize to online as v2, and add a flag to switch 10%→100%.
• Create a late-data patcher: detect affected partitions from a source delta table and trigger targeted recomputes only for those days.

Who this is for

• MLOps engineers running feature stores in production.
• Data engineers supporting ML training/serving pipelines.
• ML engineers needing reliable historical features.

Prerequisites

• Comfort with batch data processing (SQL or Spark).
• Understanding of feature stores (offline vs online) and point-in-time joins.
• Basic orchestration knowledge (e.g., cron, Airflow-like DAG concepts).

Learning path

• Start with feature definitions and point-in-time joins.
• Learn materialization to offline and online stores.
• Then practice targeted backfills, recomputes, and versioned rollouts.
• Finally, implement monitoring for freshness, completeness, and drift.

Next steps

• Draft a backfill SOP template for your team, including scope, safety, validation, and rollout.
• Automate a canary rollout for online features with easy rollback.
• Set alerts for late-arriving data and automatic partial recomputes.

Mini challenge

Scenario: Yesterday’s events for region APAC were delayed by 12 hours. The model trains weekly and serves features online.

• Decide: partial or full recompute?
• Which store(s) to update and in what order?
• How will you validate and roll back if needed?

Quick Test

This quick test is available to everyone. Only logged-in users will have their progress saved.