Why this matters
As an MLOps Engineer, you keep models reliable in the real world. Monitoring input (features) and output (predictions) distributions helps you catch data drift, detect broken pipelines, and understand when retraining or reconfiguration is needed. Real tasks you will face:
- Detect when a feature's range shifts (e.g., new country codes or changed units) before accuracy drops.
- Spot unusual prediction rates (e.g., approvals suddenly spike) and prevent downstream risk.
- Decide when to retrain by tracking meaningful changes instead of reacting to noise.
Concept explained simply
Distributions describe how values are spread. If the distribution of inputs or outputs changes too much from a healthy baseline, your model may be operating on different data than it learned from.
Mental model
Imagine your model as a musical instrument tuned to a certain key (the baseline data). When the audience (incoming data) changes key, the music sounds off. Distribution monitoring tells you when the key changed and by how much.
Key signals to monitor
- Numeric features: shifts in mean/median, variance, quantiles, histogram differences, KS statistic, PSI, Hellinger/Jensen–Shannon distances.
- Categorical features: category frequency changes, new/rare categories, Chi-square test, entropy changes.
- Outputs (predictions): predicted class rate, score/probability histogram, uncertainty/confidence distribution, calibration drift (e.g., Brier, ECE), threshold crossing rates.
- Quality context: missingness rates, out-of-range counts, type/schema violations, deduplication anomalies, sudden seasonality changes.
Setting baselines and windows
- Pick a reference window that reflects healthy behavior (often a stable recent window; training data can be a backup, but production is better).
- Choose comparison windows (e.g., hourly/daily/weekly) based on traffic volume and seasonality.
- Use rolling metrics and confidence bounds; small windows need caution due to variance.
- Segment by key dimensions (e.g., country, device, product line) to catch local drifts.
Tip: handling cold starts and seasonality
- Cold start: bootstrap a baseline using training data + short production burn-in.
- Seasonality: compare like-for-like (e.g., this Monday vs previous Mondays).
Worked examples
Example 1 — Numeric feature drift with PSI
Feature: transaction_amount. Reference bin shares: [0–10)=0.40, [10–50)=0.35, [50–100)=0.15, [100+)=0.10. Current: 0.25, 0.30, 0.25, 0.20. PSI = Σ (c−r) * ln(c/r) ≈ 0.20. Interpretation: moderate drift; investigate. If alert threshold is 0.20, trigger an alert.
Example 2 — Categorical feature drift with Chi-square
Feature: country. Reference counts: A=500, B=300, C=200. Current: A=360, B=310, C=330. Chi-square test yields p < 0.001. At α=0.01, reject stability → drift detected. Actions: check upstream mapping, confirm business changes, update baseline if intentional.
Example 3 — Output distribution drift
Model: credit approval probability. Reference: mean score=0.42, approvals=28%. Current: mean=0.55, approvals=45%. Also, more scores near 0.9. Even without labels, this suggests behavior change. Check input drift, threshold logic, and calibration; review recent marketing or user acquisition that could change population.
How to compute popular drift metrics (step-by-step)
PSI (Population Stability Index)
- Bin the feature using fixed edges from the reference window (avoid re-binning on current).
- Compute reference share r_i and current share c_i per bin (replace zeros with a small ε like 1e-6).
- PSI = Σ (c_i − r_i) * ln(c_i / r_i).
- Heuristics: <0.1 small, 0.1–0.25 moderate, >0.25 large drift.
KS test (numeric, non-parametric)
- Compare empirical CDFs of reference vs current.
- KS statistic is the max vertical distance between CDFs.
- P-value depends on sample sizes; with very large n, tiny shifts can be significant. Track effect size too.
Chi-square (categorical)
- Use reference frequencies as expected counts, current as observed.
- Compute χ² and p-value; ensure expected counts per category are not too small (combine rare categories if needed).
- Also monitor new/unknown categories rate.
Jensen–Shannon / Hellinger (distribution distance)
- Symmetric distances suitable for both numeric (binned) and categorical distributions.
- Stable thresholds must be calibrated per feature due to scale differences.
Sane thresholds and alerts
- Start conservative: PSI ≥ 0.2 or KS ≥ 0.1 on key features; adjust after observing false positives.
- Require persistence: trigger if breach persists for N windows (e.g., 2 of last 3) to reduce noise.
- Alert on directionally critical changes (e.g., missingness spike > 2x, new category rate > 1%).
- For outputs, alert if predicted positive rate changes > 30% relative to baseline unless explained by business seasonality.
Common mistakes and how to self-check
- Using training data only as baseline. Self-check: does your baseline reflect recent healthy production?
- Re-binning current data differently than reference for PSI. Self-check: are bin edges fixed from reference?
- Chasing p-values with huge sample sizes. Self-check: do you also track effect sizes (PSI/KS magnitude)?
- Ignoring segments. Self-check: do key slices (e.g., region) show larger drift than global?
- Not handling missing/unknowns. Self-check: are missing and unknown categories explicit bins?
Exercises (practice now)
Mirror of the exercises section below. Try first, then open the solutions.
Exercise 1 — Compute PSI and decide alert
Reference bin shares: [0–10)=0.40, [10–50)=0.35, [50–100)=0.15, [100+)=0.10. Current shares: 0.25, 0.30, 0.25, 0.20. Threshold: alert if PSI ≥ 0.20.
- Bins are same as reference
- Replace any zero with ε if needed
- Compute PSI and compare to threshold
Show hint
PSI = Σ (c−r) * ln(c/r). Expect around 0.20.
Exercise 2 — Interpret Chi-square result
Reference categorical counts: A=500, B=300, C=200. Current: A=360, B=310, C=330. Chi-square test returns p = 0.0004 at α = 0.01. What do you do?
- Confirm if a business change explains this
- Inspect upstream mapping for categories
- Decide whether to update baseline vs. retrain
Show hint
p < α indicates drift; next, diagnose cause before retraining.
Practical projects
- Project 1: Build a notebook that computes PSI/KS for top 10 features and an output-score histogram, comparing last day vs last 14 days.
- Project 2: Deploy a weekly job that flags features with moderate drift for 2 of the last 3 windows and posts a summary to your team channel.
- Project 3: Create a segment-aware dashboard (e.g., per country) with separate thresholds and missingness tracking.
Who this is for
- MLOps Engineers and ML Engineers responsible for production reliability.
- Data Scientists shipping models to production and maintaining them.
Prerequisites
- Basic statistics (distributions, p-values, percentiles).
- Understanding of your model's inputs, outputs, and business KPIs.
- Ability to fetch reference and current data windows.
Learning path
- Learn core drift metrics (PSI, KS, Chi-square, JSD/Hellinger).
- Define reference windows and segments that mirror your business.
- Implement calculations with fixed bins and stable sampling.
- Set thresholds and persistence rules; test on historical data.
- Automate computation and alerting; add diagnostics playbooks.
Next steps
- Instrument your pipeline to log feature histograms and prediction histograms.
- Calibrate thresholds with past incidents to balance sensitivity and noise.
- Integrate drift alerts with retraining criteria and business approval steps.
Mini challenge
Pick one numeric feature and one categorical feature from a production model. Define reference windows, compute PSI (numeric) and Chi-square (categorical) for the last 7 days, and write a 3-line summary: drift magnitude, likely cause, and action (ignore/monitor/alert/retrain).
Before you take the quick test
The quick test below is available to everyone for free. Only logged-in users will have their progress saved.