Point In Time Correctness

Why this matters

Point-in-time (PIT) correctness prevents label leakage in machine learning. When you compute features for training or serving, you must use only information that was available at the prediction moment. This is critical for:

• Credit risk approvals: using only data known before the decision time.
• Fraud detection: computing features at swipe time, not after chargebacks arrive.
• Recommendations/ads: joining user metrics as-of impression time, not after engagement.
• Demand forecasting: using inventory and price snapshots valid at forecast cut-off.

If you skip PIT correctness, offline metrics look great but models fail in production. Getting this right is a core responsibility of a Machine Learning Engineer using feature stores.

Concept explained simply

At any prediction time t, your model can only see features that were both measured before t and actually available to the system by t. Two timestamps matter for each feature row:

• event_time (when the fact happened or is effective)
• available_at (when the fact became available to the ML system after processing/ETL)

PIT correctness rule: for a prediction at time t, include a feature row only if event_time ≤ t and available_at ≤ t.

Key definitions

• Label leakage: Using future information when creating features, causing overly optimistic offline performance.
• As-of join: A time-aware join that picks the latest feature snapshot where event_time ≤ t and available_at ≤ t.
• Late-arriving data: Facts that occurred earlier but arrived in storage later. They must be guarded by available_at to avoid leakage.
• Training-serving skew: Differences between how features are generated offline vs online. PIT correctness reduces this.

Worked examples

How to implement PIT correctness (practical steps)

1. Store timestamps per feature row
   • event_time: when the feature is effective.
   • available_at (or write_time): when the feature row became queryable.
2. Build as-of joins for training
   • For each training example with prediction_time t, select the latest feature row where event_time ≤ t AND available_at ≤ t.
3. Serving consistency
   • Online store updates should maintain the same available_at semantics.
   • Optionally cache a read snapshot time per request to keep multi-feature consistency.
4. Test for leakage
   • Unit test: No joined feature row has available_at > prediction_time.
   • Data audit: Sample joined datasets and validate monotonicity constraints.

-- For each training row r(prediction_time t, entity_id e)
SELECT r.id, f.value
FROM training r
LEFT JOIN (
  SELECT * FROM feature_store
) f
ON f.entity_id = r.entity_id
AND f.event_time <= r.prediction_time
AND f.available_at <= r.prediction_time
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY r.id ORDER BY f.event_time DESC, f.available_at DESC
) = 1;

Exercises

These mirror the graded tasks below. Try here first, then open the Quick Test afterwards. Note: the Quick Test is available to everyone; only logged-in users get saved progress.

Rows:
1) last_login_count: event_time=2023-07-09 22:55, available_at=2023-07-09 23:05
2) support_ticket_count: event_time=2023-07-10 11:55, available_at=2023-07-10 12:05
3) payment_failure_flag: event_time=2023-07-08 10:00, available_at=2023-07-12 09:00 (late-arriving)
4) profile_created_at: event_time=2021-02-01 00:00, available_at=2021-02-01 00:00

A: user_id=42, t=2023-03-05 10:15
B: user_id=42, t=2023-03-06 09:00
C: user_id=7,  t=2023-03-05 22:00

user=42: value=10, event_time=2023-03-05 09:00, available_at=2023-03-05 09:05
user=42: value=12, event_time=2023-03-06 08:00, available_at=2023-03-06 10:00
user=7:  value=3,  event_time=2023-03-05 20:00, available_at=2023-03-05 20:10

• Checklist before you move on:
  • You compare both event_time and available_at to prediction time.
  • You pick the latest qualifying row when multiple rows match.
  • You avoid filling gaps with future data.

Common mistakes and self-check

• Mistake: Using only event_time in joins.
  Self-check: Count how many joined rows have available_at > prediction_time. This must be zero.
• Mistake: Recomputing historical features with today’s logic but tagging them as if known back then.
  Self-check: Preserve available_at reflecting when the recomputed value would have been visible (often the batch completion time), not the original event_time.
• Mistake: Mixing sources with different latencies without a unified available_at.
  Self-check: For each source, document latency and set available_at accordingly; test the intersection constraint.
• Mistake: Timezone drift and daylight saving errors.
  Self-check: Normalize to UTC for event_time, available_at, and prediction_time; store offsets if needed.
• Mistake: Training-serving skew due to different aggregations.
  Self-check: Snapshot an offline feature row and compare with an online read at the same cutoff; values should match within tolerance.

Practical projects

• Implement an as-of joiner: Given training events and a features table with event_time and available_at, build a utility that returns the latest valid row per entity and time. Include unit tests for late-arriving data.
• Backfill with availability: Recompute a 7-day rolling metric historically, and write available_at equal to each day’s batch completion time. Validate no leakage.
• PIT validator: Write a small checker that scans a training set and reports violations where available_at > prediction_time, with counts by feature source.

Learning path

• Foundations: Timestamps, timezones, and windowed aggregations.
• Feature store concepts: entities, feature views, event_time, available_at/write_time.
• PIT joins: offline training data creation and time-travel semantics.
• Online serving: freshness SLAs, TTL, and consistency checkpoints.
• Monitoring: data audits, latency tracking, and leakage guards.

Who this is for

• Machine Learning Engineers building training datasets and online features.
• Data Engineers implementing feature pipelines.
• Applied scientists validating offline vs online performance.

Prerequisites

• Basic SQL or data frame joins.
• Understanding of entity keys and timestamps.
• Familiarity with batch vs streaming ingestion.

Mini challenge

You maintain a daily customer_spend_30d feature recomputed at 02:00 UTC. Write acceptance criteria for PIT correctness when joining to transactions:

• State the rule using event_time and available_at.
• Describe a unit test that must fail if a transaction at t uses the 02:00 update from the next day.
• Describe a metric you’d monitor in production to catch violations.

Next steps

• Finish the exercises above, then take the Quick Test to confirm understanding.
• Apply the as-of join to one of your datasets and run the PIT validator.
• Document event_time and available_at for every feature you own.