Serving Features With Low Latency

Why this matters

Low-latency feature serving keeps online ML predictions fast and reliable. It directly affects user experience, conversion rates, and model ROI.

• Fraud checks during payment must complete before authorization.
• Recommendations and ads need user and item features in a few milliseconds.
• Search ranking and personalization rely on fresh counters and embeddings.

Who this is for and prerequisites

Who this is for

• Machine Learning Engineers implementing online inference.
• Data/Platform Engineers building feature stores and serving APIs.
• MLOps practitioners responsible for reliability and SLOs.

Prerequisites

• Basics of feature stores (offline vs online, materialization).
• Key-value and cache concepts (e.g., Redis-like stores).
• Streaming basics (windows, event time) are helpful but not required.

Concept explained simply

Serving features with low latency means precomputing or quickly computing the values your model needs and storing them in a place optimized for fast reads. During a live request, you fetch by a key (like user_id) and return the features quickly, every time.

Mental model

Think of a well-organized convenience store:

• The shelf (online store) holds items (features) you need right now.
• Restockers (batch/stream jobs) keep shelves fresh.
• Expiry labels (TTL) ensure stale items are removed.
• Fast checkout (API + cache) keeps the line moving.

Core building blocks for low-latency serving

• Online store: A low-latency key-value database (e.g., in-memory or SSD-optimized) for reads by feature key.
• Materialization: Batch (periodic) or streaming (continuous) jobs write features into the online store.
• Schema and keys: Stable entity keys (user_id, item_id) and predictable feature names.
• Caching tiers: In-process request cache, local node cache, and the online store itself.
• TTL and freshness: Time-to-live enforces maximum staleness; also store feature timestamp to check freshness on read.
• Read path: Minimize network hops, reduce payload size, and use efficient serialization.

Design toolkit

• Precompute vs on-demand: Prefer precompute for heavy joins/window aggregations; compute on-demand only if cheap and cacheable.
• Denormalize hot features: Store ready-to-serve values to avoid online joins.
• Small payloads: Ship only features the model needs; avoid large blobs unless essential.
• Streaming freshness: Use tumbling/sliding windows; update counters continuously.
• Idempotent upserts: Make writes safe to retry; deduplicate by event_id + window.
• Point-in-time correctness: Ensure features reflect only data available before the prediction timestamp.
• Fallbacks: Define defaults when a key is missing or data is stale to avoid request failures.

Worked examples

1) Fraud: last_5m_txn_count

• Source: Transaction stream keyed by card_id.
• Compute: Sliding 5-minute count updated per event.
• Materialize: Stream job upserts to online store with TTL 10 minutes.
• Read path: GET by card_id; expect p95 < 5 ms from store, payload < 1 KB.
• Fallback: If missing, use 0 and flag feature_missing=true.

2) Recommendations: user_profile + item_popularity

• user_profile: Precomputed daily vector (small, e.g., 64 dims) stored per user_id.
• item_popularity: Hourly counts per item_id from stream aggregation.
• Read path: Multi-get (user_id + up to 50 item_ids). Use pipelined or batch reads.
• Caching: Local LRU for user_profile (TTL 10 min). Online store TTL 26 hours.
• Fallback: If item missing, default popularity to low percentile.

3) Search ranking: merchant_service_score

• Source: Service tickets + delivery delays; computed hourly.
• Materialize: Batch hourly job writes score per merchant_id with timestamp.
• SLA: p95 read < 8 ms; batch job SLAs to write within 10 minutes of the hour.
• Staleness guard: Reject reads older than 3 hours; else use last known and set stale=true.

Blueprint: from event to millisecond read

1. Define latency SLO (p95) and freshness target (max age).
2. List features needed by the model and their size.
3. Choose materialization mode: streaming for sub-hour freshness; batch if hourly+ is fine.
4. Design keys and schema; plan idempotent upserts.
5. Set TTL and per-feature freshness validation on read.
6. Add caching: in-process short TTL (e.g., 30–120 s) for hot keys.
7. Measure: load test p50/p95/p99 at expected QPS.
8. Monitor: latency, error rate, staleness age, key miss rate, and skew vs offline.

Consistency and correctness

• Point-in-time reads: Store feature event_time; reject or fallback if it exceeds staleness limits.
• Offline-online skew: Compare batch-scored features vs online reads over the same entity/time. Alert on drift beyond a threshold.
• Exactly-once illusion: Aim for idempotent-at-least-once. Use deterministic keys to overwrite duplicates.

Caching and TTL strategy

• Per-request cache: Share reads for the same key within one request.
• In-process LRU: 30–120 s TTL for hot keys; size by memory budget.
• Online store TTL: Based on freshness requirements (e.g., 2x window size for counters).
• Negative caching: Cache misses briefly (e.g., 5–30 s) to reduce repeated lookups.

Monitoring and SLOs

• Latency: p50/p95/p99 for feature fetch and total request.
• Staleness: Feature age distribution; alert when age > freshness target.
• Correctness: Shadow reads vs offline ground truth samples.
• Availability: Error rate and timeouts; circuit-breakers with safe defaults.

Common mistakes and self-check

• Online joins at request time increase latency unpredictably.
• Oversized payloads (unused features) slow deserialization.
• No TTL or timestamps leads to silently stale data.
• Ignoring p95/p99; optimizing only p50 hides tail issues.
• No fallback strategy turns a cache miss into a user-visible failure.

Exercises

Practice these before taking the quick test. Your progress is saved only if you are logged in; everyone can take the test.

1. Latency budget (maps to Exercise 1): Break down a 60 ms SLA between network, feature fetch, and inference.
2. Windowed counter design (maps to Exercise 2): Pick materialization mode and TTL for a last-10m counter.
3. Fallback plan (maps to Exercise 3): Define defaults and stale-read behavior.

• [ ] I can compute a latency budget and defend trade-offs
• [ ] I selected materialization and TTL per feature
• [ ] I defined fallbacks and staleness guards
• [ ] I planned monitoring for latency, staleness, and skew

Mini challenge

You run a personalization API with p95 = 80 ms. Current feature fetch takes 35 ms p95. You must add two new features: a 1-hour popularity score (batch hourly) and a last-2m click count (streamed). Propose a design that keeps total p95 under 80 ms, including how you will cache, TTLs, and fallback values.

Practical projects

• Build a sliding-window counter feature (5m, 1h) from a mock clickstream and serve via a lightweight API with p95 < 20 ms under 500 RPS.
• Create a denormalized user profile feature set (10–20 fields), materialize hourly, add in-process cache, and measure tail latencies.
• Implement staleness monitoring: emit feature_age_ms on read and alert when > target for 5 consecutive minutes.

Learning path

• Before: Feature Store Basics, Offline vs Online stores
• Now: Serving Features With Low Latency
• Next: Online consistency, Backfills and replays, Canary rollouts for features

Next steps

• Finish the exercises above.
• Take the quick test below to check your understanding.
• Pick one practical project and implement it end-to-end.

Try the Quick Test

Anyone can take the test. Only logged-in users will have their progress saved.