Preprocessing At Inference Time

Who this is for

Machine Learning Engineers and backend practitioners who ship models behind APIs and need predictable, low-latency feature preparation at inference time.

Prerequisites

• Comfort with Python or similar for data transforms
• Basic understanding of your model’s expected inputs
• Familiarity with REST/HTTP or gRPC APIs
• Knowing the training-time preprocessing steps used for your model

Why this matters

In production, raw requests are messy. You must convert them into model-ready features quickly and safely. Good inference-time preprocessing:

• Prevents training–serving skew and accuracy drops
• Meets latency SLOs and throughput goals
• Protects your service with validation and limits
• Enables observability and faster incident response

Concept explained simply

Inference-time preprocessing is the set of transformations that turn a live request into the exact feature tensor(s) your model expects. It must be deterministic (same input, same features), fast, and safe.

Mental model

Think of a funnel:

• Gate: validate and limit payloads
• Shape: clean and normalize to a strict schema
• Enrich: add derived fields or fetch features (with timeouts)
• Pack: produce model-ready tensors with pinned parameters

Everything is a contract: schema in, features out, with strict budgets.

Core principles

• Determinism and parity: use the exact same parameters from training (means, stds, vocab, tokenizers, image sizes, category maps)
• Latency budget: define a P95/P99 budget for preprocessing and measure it
• Stateless handlers: avoid per-request global mutations; allow safe concurrency
• Validation and limits: schema checks, size limits, timeouts, and safe defaults
• Observability: log decision points, count fallbacks, tag out-of-distribution events
• Graceful degradation: if enrichment fails, fallback to a simpler path

Typical workflow

1. Load pinned artifacts at startup (vocab.json, scaler.pkl, image mean/std). Do not compute them on live traffic.
2. Validate input (types, required fields, max sizes). Reject early with clear errors.
3. Normalize (whitespace, casing, timezones, image channels, units).
4. Derive features (hash buckets, day-of-week, length, bigram indicators).
5. Optional feature fetch from online store with strict timeouts and fallbacks.
6. Pack tensors in the exact order/shape expected by the model.
7. Profile and monitor latencies and skew counters.

Worked examples

Example 1: Text sentiment API

Goal: Convert user text into token IDs for a model that expects lowercase, max length 128, and a fixed vocabulary with an UNK token.

Request:
{"text": "Amazing product!!! LOVE it."}

Features:
{
  "input_ids": [101, 5872, 4031, 999, 999, 999, 102, ...],
  "attention_mask": [1,1,1,1,1,1,1,...]
}

Example 2: Tabular fraud score

Goal: Standardize numeric fields and one-hot encode categories using training-time stats.

means = {"amount": 53.2, "age_days": 430.0}
stds  = {"amount": 20.5, "age_days": 120.0}
country_map = {"US":0, "CA":1, "GB":2, "OTHER":3}

Input: {"amount": 80, "age_days": 365, "country": "DE", "timestamp": "2025-04-05T10:22:00+02:00"}
Output: [1.31, -0.54, 3, 8, 0.0]

Example 3: Image classifier

Goal: Produce a 3x224x224 tensor normalized by channel with pinned mean/std.

Implementation patterns

• Ship transforms with the model artifact (same versioning)
• Load artifacts at process start; avoid disk I/O per request
• Use strict schema checks and descriptive error messages
• Apply timeouts around remote calls; keep a no-remote fallback path
• Cache small lookups (e.g., vocab maps) in-memory; avoid global mutation
• Record feature stats counters (nulls replaced, OOV rate, truncation rate)

Common mistakes and how to self-check

• Recomputing stats on live traffic → Self-check: confirm means/stds are loaded from artifact
• Forgetting timezone normalization → Self-check: assert all timestamps are UTC before deriving features
• Changing tokenization between training and serving → Self-check: verify vocab checksum matches training version
• Unlimited payload sizes → Self-check: add explicit size/length guards and tests
• Silent feature reordering → Self-check: unit test feature order against a golden sample
• Unbounded remote fetches → Self-check: enforce timeouts and track fallback rates

Before shipping checklist

• Artifacts pinned and versioned with the model
• Schema validation with clear errors
• Latency budget documented and profiled at P50/P95/P99
• Guards: size limits, timeouts, retries capped
• Fallback logic defined and measured
• Golden test cases for feature parity

Exercises

Do these, then compare with the solutions below.

Mini challenge

Your tabular model expects standardized numeric features and a 10k-category hash bucket for product_id. Live traffic includes new product_ids daily. Outline how you will handle unseen ids, enforce limits, and keep latency under 20ms for preprocessing. Keep it to 4–6 bullet points and focus on determinism and safety.

Learning path

1. Map training transforms → write their serving equivalents
2. Create schema and validators → add unit tests with golden cases
3. Pin artifacts → load at startup and add checksums
4. Add latency timers → set budgets and alerting
5. Hardening → limits, timeouts, fallbacks
6. Canary release → compare feature distributions and error rates

Practical projects

• Build a small REST service that preprocesses text into model tensors with pinned vocab, including OOV handling and truncation
• Implement a tabular pipeline that standardizes features and falls back when the online feature fetch times out; log fallback metrics
• Create an image preprocessor with strict size/type validation and CHW normalization; run a load test to verify P95 latency

Next steps

• Add observability: counters for null replacements, OOVs, truncations, and fetch timeouts
• Automate parity tests in CI with golden samples from training
• Document the schema contract and latency budget for your endpoint

Quick Test

Available to everyone. Only logged-in users will have their progress saved.