Feature Store Operations

What this skill covers

Feature Store Operations is the day‑to‑day practice of making ML features reliable, reproducible, and fast to serve. As an MLOps Engineer, you make sure data scientists can develop features once and use them safely in training and online inference without leakage, staleness, or performance surprises.

Why it matters for MLOps Engineers

• Cuts model time‑to‑production by making features reusable.
• Reduces online incidents by monitoring freshness and latency.
• Improves offline/online consistency, boosting model accuracy.
• Enables transparent ownership and governance across teams.

Roadmap to proficiency

1. Master offline vs online separation
   Goal: Keep batch analytics isolated from low‑latency serving while using shared definitions.
   Mini task: Identify which features truly need online materialization.
2. Guarantee point‑in‑time correctness
   Goal: Train with only information available at the event time.
   Mini task: Implement an as‑of join and verify no future data leaks.
3. Define features and ownership
   Goal: Versioned, discoverable feature specs with clear owners and SLAs.
   Mini task: Write one feature spec including owner, source, TTL, and tests.
4. Freshness SLAs and monitoring
   Goal: Track max allowed staleness; alert before it impacts models.
   Mini task: Add a freshness probe and dashboard for 3 key features.
5. Backfills and recomputes
   Goal: Rebuild historical features after logic changes without downtime.
   Mini task: Run a backfill for the last 90 days in a shadow table, then promote.
6. Prevent training/serving skew
   Goal: Align code paths, transforms, and encodings across environments.
   Mini task: Compare one feature’s offline vs online distribution and fix deviations.
7. Online serving performance
   Goal: Meet p95 latency and availability targets.
   Mini task: Add request sampling and measure p50/p95/p99 latency per feature group.

Worked examples

-- Events table: transactions (user_id, event_ts, label)
-- Feature source: balances_snapshots (user_id, snapshot_ts, balance)
-- Goal: for each transaction, use the most recent balance BEFORE event_ts

SELECT t.user_id,
       t.event_ts,
       t.label,
       b.balance AS balance_pti
FROM transactions t
LEFT JOIN LATERAL (
  SELECT balance
  FROM balances_snapshots b
  WHERE b.user_id = t.user_id
    AND b.snapshot_ts <= t.event_ts
  ORDER BY b.snapshot_ts DESC
  LIMIT 1
) b ON TRUE;

# Pseudo client for a feature store
from datetime import timedelta

order_amount_7d = {
  'name': 'order_amount_7d',
  'owner': 'growth-ml@company',
  'entity': 'user_id',
  'source': 'warehouse.orders',
  'aggregation': {
    'type': 'sum',
    'column': 'amount',
    'window': '7d'
  },
  'ttl': '14d',
  'freshness_sla': '24h',
  'online': True,
  'version': 'v1'
}

fs.register_feature(order_amount_7d)
fs.materialize_offline(feature='order_amount_7d', start='2025-01-01', end='2025-12-31')
fs.materialize_online(feature='order_amount_7d', since='2025-12-01')

from datetime import date

# New business rule: exclude refunds from order amount

def compute_order_amount_7d(dt):
    # Read from raw orders for date dt
    df = read_orders(dt)
    df = df[df['type'] != 'refund']
    agg = df.groupby('user_id')['amount'].sum()
    write_feature_partition('order_amount_7d', dt, agg)

# Backfill last 180 days into a shadow location
for dt in daterange(date(2025, 7, 1), date(2025, 12, 31)):
    compute_order_amount_7d(dt)

# Validate before promote
validate_shadow_vs_prod('order_amount_7d', checks=['count','null_rate','p50','p95'])
promote_shadow('order_amount_7d')

import time
from datetime import datetime

FEATURES = ['order_amount_7d', 'balance_pti', 'days_since_signup']
SLA_SEC = {
  'order_amount_7d': 24*3600,
  'balance_pti':  3600,
  'days_since_signup': 24*3600
}

def check_freshness():
    stale = []
    now = int(time.time())
    for f in FEATURES:
        last_ts = fs.get_last_update_epoch(f)
        age = now - last_ts
        if age > SLA_SEC[f] * 0.8:  # warn before breach
            stale.append((f, age, SLA_SEC[f]))
    return stale

issues = check_freshness()
for f, age, sla in issues:
    notify(f"Feature {f} nearing staleness: age={age}s SLA={sla}s")

# Shared transform used offline and online

def country_to_id(country: str) -> int:
    lut = {
        'US': 1, 'CA': 2, 'GB': 3, 'DE': 4
    }
    return lut.get(country, 0)

# Offline
train_df['country_id'] = train_df['country'].apply(country_to_id)

# Online request
req['country_id'] = country_to_id(req['country'])

# Skew check
assert distribution_diff(
    sample_online('country_id'),
    sample_offline('country_id')
) < 0.05

Skill drills

• Write an as‑of join for two tables with different time grains and verify no future timestamps are used.
• Add TTL to three online features and confirm eviction works as expected.
• Create a freshness dashboard that tracks age, last write, and upstream job status.
• Run a 30‑day backfill in a shadow table and perform schema + distribution checks before promotion.
• Measure p50/p95/p99 read latency for a hot feature set under load and write a 3‑line summary.

Common mistakes and debugging tips

Mini project: Real‑time risk scoring features

Scenario: You need features for a transaction risk model with a p95 latency target of 20 ms online and 24h freshness SLA for aggregates.

1. Define three features: last_transaction_amount (online), txn_count_7d (online), avg_ticket_30d (offline only).
2. Implement point‑in‑time historical retrieval for training data over 90 days.
3. Materialize online features with TTLs (e.g., 48h) and a background refresher.
4. Add freshness probes and an alert at 80% of SLA.
5. Create a backfill plan to recompute 90 days if logic changes.

Who this is for

• MLOps Engineers building and operating ML platforms.
• Data Engineers supporting real‑time inference.
• ML Engineers taking models from notebooks to production.

Prerequisites

• Comfort with SQL window functions and joins.
• Python for data pipelines and basic testing.
• Familiarity with streaming/batch concepts and key‑value stores.
• Basic monitoring concepts (SLIs/SLOs/SLAs).

Learning path

1. Study point‑in‑time joins and leakage prevention; implement an as‑of join.
2. Write feature specs with ownership, TTL, and freshness SLAs; add simple tests.
3. Build offline pipelines; then materialize a minimal online set.
4. Add freshness monitoring and alerts; create a small on‑call runbook.
5. Run a backfill safely using shadow tables; validate and promote.
6. Benchmark online reads; optimize keys, batching, and caching.

Practical projects

• Feature catalog cleanup: add owners, descriptions, and SLAs to 20 existing features.
• Latency hardening: achieve p99 ≤ 50 ms by batching and key‑coalescing without changing model code.
• Skew guardrail: build a nightly job that compares offline vs online distributions for top 10 features and posts a summary.

Subskills

Master these to operate feature stores confidently:

• Offline Online Feature Separation — Identify which features require low‑latency serving vs batch‑only use.
• Point In Time Correctness — Guarantee no data leakage using as‑of joins and event times.
• Feature Definitions And Ownership — Versioned specs with owners, SLAs, and validation tests.
• Freshness SLAs And Monitoring — Track staleness and alert before breaches.
• Backfills And Recomputes — Safe, idempotent recomputes with shadow promotion.
• Preventing Training Serving Skew — Shared transforms, schema contracts, and drift checks.
• Online Serving Performance Basics — Keys, TTLs, caching, batching, and latency budgets.

Next steps

• Complete the drills and the mini project.
• Review each subskill section below and schedule focused practice.
• When ready, take the skill exam to validate your understanding.