Who this is for
- Machine Learning Engineers shipping models to production.
- Data Scientists preparing training datasets that must match serving-time behavior.
- Data/Platform Engineers building or integrating a feature store.
Prerequisites
- Comfort with batch and streaming data processing.
- Understanding of model training, validation, and inference flows.
- Basic SQL and familiarity with concepts like joins, timestamps, and windows.
Why this matters
Real-world ML systems fail when training features don\u2019t match serving-time features. Getting offline vs online features right prevents data leakage, reduces prediction drift, and improves latency and reliability. Typical tasks you\u2019ll do:
- Create point-in-time correct training datasets.
- Decide which features must be served with low latency vs computed offline.
- Set freshness SLAs/TTLs and backfill strategies.
- Design pipelines so offline and online transformations stay aligned.
Concept explained simply
Offline features are computed in bulk (batch) for analysis and training. They prioritize completeness, historization, and cost efficiency. Online features are computed or retrieved at request-time or near-real-time for inference. They prioritize low latency and freshness.
Mental model
Think of a restaurant: the pantry (offline) holds prepared ingredients made in batches earlier; the line station (online) keeps the few hot, perishable items that must be ready instantly. Your dish (prediction) needs both: most prep was done earlier, but a few crucial items are made fresh right before serving.
Key differences and trade-offs
- Latency: Offline (minutes-hours), Online (milliseconds).
- Freshness: Offline (as of last batch), Online (up-to-the-second or cached for short TTL).
- Cost: Offline cheaper per row at scale; Online more expensive per query.
- Consistency: You must align logic so training matches serving (feature parity).
- Time-travel: Offline supports historical, point-in-time joins; Online prioritizes current values.
Point-in-time correctness checklist
- Every training row uses feature values available strictly before the label timestamp.
- Use event_time and created_at (ingestion time) to avoid late-arrival leakage.
- Prefer left join on keys plus a \"last observed before t\" lookup.
- Freeze feature definitions per training run for reproducibility.
Feature parity checklist
- Single source of truth for transformations (shared code or spec).
- Identical default values, null handling, and window definitions in both stores.
- Automated offline-online consistency tests on a sampled time range.
- Version features; deploy online updates only after offline backfill is ready.
Latency budgets and TTL tips
- Set a per-feature freshness SLO (e.g., p99 age < 60s).
- Use short TTL caches for hot features; back them with a streaming update if needed.
- Avoid recomputing heavy aggregates online; pre-aggregate offline and increment online with deltas.
Worked examples
Example 1: Fraud detection on card transactions
- Offline features: 7-day merchant spend, user velocity over 30 days, chargeback rate per merchant last quarter.
- Online features: Count of attempts by card in the last 5 minutes, last transaction amount, device fingerprint risk score from the last session.
- Offline batch computes rolling windows daily; backfill for training with point-in-time joins.
- Online store maintains short-window counters via streaming; TTL 10 minutes for rate features.
- At inference: combine precomputed merchant aggregates (fetched quickly) with fresh session counters.
Trade-off: You avoid heavy window computation online by pre-aggregating offline; you maintain only fast, short windows online.
Example 2: CTR prediction for ads
- Offline: User long-term interest embeddings (updated daily), ad quality score (weekly), historical CTR by segment (daily).
- Online: User recent session clicks (last 10 minutes), current page context, time-of-day bucket.
Latency budget: < 50 ms. Solution: embeddings in online key-value store; session clicks kept in a small, fast store with TTL 15 minutes.
Example 3: Customer churn model
- Offline: 90-day product usage aggregates, billing history, NPS score history.
- Online: Current session length, last action timestamp.
Training uses a snapshot at t with only values before t; serving uses the latest online signals to capture recency effects without retraining daily.
Common mistakes and self-check
- Leakage: Using features at training that weren\u2019t known at the prediction time. Self-check: Verify every feature value timestamp <= label time.
- Parity drift: Slightly different logic offline vs online (e.g., null fills). Self-check: Run an offline \u2192 online replay test on a sampled day; compare distributions.
- Overloading online store: Computing heavy windows per request. Self-check: Profile p95 latency; move heavy aggregates offline.
- Stale features: No freshness SLOs or alerts. Self-check: Track feature age; alert when age > SLO.
Exercises
These mirror the graded exercises below. Try here first, then submit your answers in the exercise panel.
Exercise 1: Classify and serve features for real-time fraud
Scenario: You score each card swipe in real time. Features: (A) user_total_spend_30d, (B) attempts_last_3m, (C) merchant_chargeback_rate_90d, (D) device_recent_failures_10m, (E) cardholder_age_years.
- Task: For each feature, decide Offline, Online, or Both. Propose TTL or batch frequency and a brief reason.
Show tips
- Short windows (minutes) often belong online; long windows (days) often offline.
- Low-volatility attributes can be offline-only but may be cached online for latency.
Exercise 2: Point-in-time training join
You have events (transactions) with event_time and label fraud_flag. You also have feature snapshots keyed by user_id with valid_from and valid_to. Write a point-in-time correct query or step plan to join snapshots so each transaction uses the last feature value before event_time.
Show tips
- Use a range condition valid_from <= event_time and (valid_to > event_time or valid_to IS NULL).
- Prefer window functions if your SQL dialect supports them.
Exercise 3: Freshness SLOs and alerting
For a CTR model, you serve features: user_embedding_daily, session_clicks_10m, page_context. Propose a freshness SLO and TTL for each and define an alert rule when the feature age exceeds the SLO.
Show tips
- Embeddings tolerate hours; session signals need minutes.
- TTLs should be slightly larger than the update period to mask minor jitter.
- Before you submit, checklist:
- Each feature has a store decision and a freshness plan.
- Your joins never use future data.
- Your alert rules specify a clear threshold and action.
Practical projects
- Build a tiny feature store mock: a batch job to compute 7-day aggregates and a simple key-value service to serve a 10-minute counter. Validate parity on a sampled day.
- Create a point-in-time dataset generator: given events and snapshot tables, output training rows with leakage-proof joins.
- Implement freshness monitoring: track feature age, expose a daily report, and simulate an alert when age exceeds SLO.
Learning path
- Next: Point-in-time joins and time-travel data modeling.
- Then: Feature definitions, versioning, and a shared transformation layer.
- Then: Streaming updates, TTLs, and online store scaling patterns.
- Finally: Monitoring parity, freshness, and feature drift in production.
Next steps
- Do the exercises and compare with the provided solutions.
- Take the quick test to check understanding. Anyone can take it; only logged-in users get saved progress.
- Apply the concepts by refactoring one of your model\u2019s features into offline vs online parts.
Mini challenge
Pick one of your current model features and redesign it into two parts: (1) a heavy offline aggregate updated daily; (2) a light online delta updated within minutes. Write down the exact transformation, update cadence, TTL, and how you will test parity.