Feature Generation Steps

Why this matters

MLOps Engineers make features reliable, reproducible, and fast. Good features are the backbone of training pipelines and batch scoring. Your daily tasks include building point-in-time correct aggregates, handling late data, preventing leakage, and materializing features for both training and inference with identical logic.

• Own feature definitions as code and version them.
• Backfill features for historical training windows safely.
• Detect and fix training–serving skew.
• Monitor freshness, drift, and null spikes.

Concept explained simply

A feature is a computed column that summarizes raw data into a machine-usable signal. In batch pipelines, features are reproducible transformations with a time boundary.

Mental model: a kitchen with clear recipes. Each feature has a recipe card (contract):

• Name and business meaning
• Entities/keys (e.g., user_id, card_id)
• Time semantics (event_time, windows, timezone)
• Definition (SQL/pseudocode), null handling, units
• Validation checks and owner

Core steps for feature generation

Worked examples

Tables:
  orders(user_id, order_id, amount, order_ts)
  training_points(user_id, reference_ts)

Goal:
  For each training point, count orders in (reference_ts - 7d, reference_ts].

SQL (conceptual):
SELECT t.user_id,
       t.reference_ts,
       COUNT(o.order_id) AS user_7d_orders_count
FROM training_points t
LEFT JOIN orders o
  ON o.user_id = t.user_id
 AND o.order_ts > t.reference_ts - INTERVAL '7 day'
 AND o.order_ts <= t.reference_ts
GROUP BY t.user_id, t.reference_ts;

Tables:
  transactions(card_id, amount, event_ts)

Approach:
  1) Bucket by hour: bucket_ts = date_trunc('hour', event_ts)
  2) For each card_id and hour, compute rolling 24h sum at the end of the hour.

SQL (conceptual):
WITH hourly AS (
  SELECT card_id,
         date_trunc('hour', event_ts) AS hour_ts,
         SUM(amount) AS amt_hour
  FROM transactions
  GROUP BY 1,2
)
SELECT h1.card_id,
       h1.hour_ts AS reference_ts,
       (
         SELECT SUM(h2.amt_hour)
         FROM hourly h2
         WHERE h2.card_id = h1.card_id
           AND h2.hour_ts > h1.hour_ts - INTERVAL '24 hours'
           AND h2.hour_ts <= h1.hour_ts
       ) AS card_24h_amount_sum
FROM hourly h1;

Input:
  items(item_id, title_text, created_ts)

Outputs (per item_id, created_ts):
  - title_len_chars
  - title_alpha_ratio = letters / total_chars
  - title_embed_16 (placeholder: 16-dim avg of character code buckets)

Pseudocode:
for row in items:
  t = normalize_unicode(row.title_text)
  L = len(t)
  alpha = count_regex(t, '[A-Za-z]')
  alpha_ratio = safe_div(alpha, L)
  embed = simple_avg_bucket_embedding(t, dim=16)
  emit(item_id=row.item_id, ts=row.created_ts,
       title_len_chars=L,
       title_alpha_ratio=alpha_ratio,
       title_embed_16=embed)

Design decisions that matter

• Granularity: per-event, per-entity-per-time bucket (hour/day), or snapshot. Finer granularity increases storage but reduces leakage risk.
• Window size: shorter windows capture recency, longer windows improve stability. Consider multiple windows (7d, 30d).
• Imputation: constant vs model-based. Keep it stable between training and inference.
• Materialization cadence: balance latency (freshness) vs cost.

Time alignment and leakage

Rule: Every feature must only use data whose event_time <= reference_time of the row being scored/trained.

• Late-arriving data: either reprocess with watermarking or accept eventual consistency.
• Backfills: always recompute using historical snapshots or event logs, not today’s corrected state.
• Labels: compute labels with a lookahead window that starts strictly after reference_time.

Feature quality checks

• Schema: types, ranges, allowed null rates.
• Statistical: distribution drift vs baseline, min/max sanity, cardinality caps.
• Freshness: max(ts_now - latest_materialized_ts) per feature.
• Skew: training vs inference distribution differences.
• Join coverage: rate of missing joins by entity and time.

Implementation patterns (batch + serving)

• Single source-of-truth transformation: share the same logic between training backfills and batch inference.
• Windowed aggregates: precompute per entity/time-bucket for speed, then roll up at scoring.
• Idempotent writes: deterministic keys (entity, reference_ts, version) allow safe retries.

Monitoring features in production

• Freshness SLOs (e.g., 95% of hourly aggregates ready by T+20m).
• Null/zero spikes detection and alerting.
• Distribution drift: compare daily histograms to a moving baseline.
• Latency and cost per job run; detect regressions after definition changes.

Exercises

Do these to lock in the concepts. Anyone can do them; if you log in, your progress will be saved.

1. Exercise 1 — Point-in-time SQL feature
   Design a query for a 14-day rolling unique_items_count per user at a given reference_ts. See full instructions below in Exercises section.
2. Exercise 2 — Feature spec + validation
   Write a feature definition (YAML-like) and create validation rules for nulls, ranges, and freshness.

Common mistakes and how to self-check

• Using now() in training features. Fix: reference per-row timestamp only.
• Joining on latest snapshot without time filter. Fix: add event_time <= reference_time in join condition.
• Forgetting timezone normalization. Fix: convert all timestamps to UTC and document it.
• Leaky imputation (using global stats from full history). Fix: compute stats on training window only and version them.
• Unbounded cardinality features (e.g., IDs one-hot). Fix: hash, bucket, or top-K with OOV bucket.

Practical projects

• Retail cohort features: 7/30/90-day spend and visit counts per user with a backfilled training set.
• Fraud hourly rollups: 1h/6h/24h card activity features with lateness watermark and monitoring.
• Content features: title/body text lengths, ratios, and simple embeddings materialized daily.

Who this is for

• MLOps Engineers implementing training and batch inference pipelines.
• Data Engineers collaborating on feature stores and ETL.
• ML Engineers needing reproducible feature definitions.

Prerequisites

• Comfort with SQL windowing and joins
• Basic Python or similar for transforms
• Familiarity with batch scheduling and data partitioning

Learning path

1. Point-in-time correctness basics
2. Windowed aggregation patterns
3. Imputation and scaling consistency
4. Backfilling and versioning
5. Validation and monitoring

Next steps

• Complete the two exercises and run the Quick Test below.
• Add monitoring checks to a feature you already built.
• Plan a backfill and a safe rollout of a new feature version.

Mini challenge

You must compute user_30d_avg_order_value for batch scoring at 02:00 UTC daily. Yesterday, late orders arrived at 03:00. What do you do?

• Answer hint: materialize at 02:00 with a 2-hour watermark (exclude 02:00–04:00 late data), then run a correction backfill for the affected window later and version the output.