ML Specific Monitoring

What is ML Specific Monitoring?

ML-specific monitoring tracks model health beyond generic app metrics. As an MLOps Engineer, you watch the data and predictions themselves: feature distributions, drift, calibration, fairness, feedback loops, and when to retrain. This turns opaque models into observable systems you can trust in production.

Why it matters for an MLOps Engineer

• Prevents silent failures when data or behavior changes.
• Shortens incident time-to-detect and time-to-recover.
• Supports responsible AI via bias/fairness guardrails.
• Enables safe automation of retraining and rollouts.

Who this is for

• MLOps Engineers deploying and operating ML services.
• Data/ML Engineers adding monitoring to existing pipelines.
• Team leads needing reliable, auditable ML in production.

Prerequisites

• Python basics; ability to read pandas/numpy code.
• Familiarity with model metrics (AUC, F1, MAE/MAPE).
• Comfort with batch or streaming data pipelines and logging.

Learning path (Roadmap)

1. Define objectives and risks: List what failure looks like (e.g., drift, stale features, bias, over/under-confidence).
2. Instrument logging: Log feature vectors (or summaries), predictions, model version, timestamp, and optional ground truth when available.
3. Establish baselines: Store training/validation distribution summaries and baseline performance.
4. Monitor input/output: Track feature and prediction distributions, missing values, ranges, and rates.
5. Drift + performance: Add PSI/JS/KS for drift; rolling AUC/F1/MAE where labels arrive.
6. Calibration + fairness: Compute Brier score, reliability curves; add group metrics like demographic parity ratio.
7. Alerts + SLOs: Thresholds with sample-size guards and consecutive-breach rules.
8. Feedback loops: Collect labels; compare predictions vs truth; close the loop.
9. Retraining triggers: Automate safely using sustained drift/performance signals and data-availability checks.
10. Review and iterate: Weekly reviews, post-incident learnings, threshold tuning.

Worked examples

1) Feature drift with PSI

Population Stability Index (PSI) flags shifts between baseline and production distributions. Rule of thumb: 0.1–0.25 moderate, >0.25 significant.

import numpy as np

def psi(expected, actual):
    eps = 1e-12
    expected = np.array(expected, dtype=float)
    actual = np.array(actual, dtype=float)
    return np.sum((actual - expected) * np.log((actual + eps) / (expected + eps)))

# three-bin example (sums to 1)
expected = [0.2, 0.3, 0.5]
actual   = [0.1, 0.4, 0.5]
print(round(psi(expected, actual), 3))  # ~0.098

2) Concept drift via rolling performance

When labels arrive with delay, compute rolling performance once labels are in. Use shorter windows for faster detection; longer for stability.

import numpy as np

def rolling_metric(y_true, y_pred, window=7):
    # Simple rolling accuracy for illustration
    acc = []
    for i in range(len(y_true)):
        s = max(0, i - window + 1)
        yt, yp = np.array(y_true[s:i+1]), np.array(y_pred[s:i+1])
        acc.append(np.mean((yt == (yp >= 0.5).astype(int))))
    return acc

# simulate delayed labels by computing later; code here just shows rolling calc

3) Calibration monitoring (Brier score + reliability)

import numpy as np

def brier_score(y_true, y_prob):
    y_true = np.array(y_true)
    y_prob = np.array(y_prob)
    return float(np.mean((y_prob - y_true)**2))

# Reliability curve (10 bins)
def reliability(y_true, y_prob, bins=10):
    idx = np.minimum((y_prob * bins).astype(int), bins-1)
    bin_stats = []
    for b in range(bins):
        mask = (idx == b)
        if np.sum(mask) < 1:
            bin_stats.append(( (b+0.5)/bins, None, 0))
        else:
            conf = np.mean(y_prob[mask])
            obs = np.mean(y_true[mask])
            bin_stats.append((conf, obs, np.sum(mask)))
    return bin_stats

# Example
y_true = [0,1,1,0,1,0,1,0,1,0]
y_prob = [0.2,0.7,0.8,0.3,0.9,0.4,0.6,0.2,0.8,0.3]
print("Brier:", round(brier_score(y_true, y_prob), 3))
for c,o,n in reliability(y_true, y_prob):
    if o is not None:
        print("bin mean p=", round(c,2), "observed=", round(o,2), "n=", n)

Under-confidence means observed rates > predicted; over-confidence is the opposite. Track both the Brier score and per-bin gaps.

4) Group fairness guardrail

import numpy as np

def demographic_parity_ratio(y_pred, sensitive):
    # y_pred: binary decisions (0/1), sensitive: group labels A/B
    g = np.array(sensitive)
    p = np.array(y_pred)
    rate_A = np.mean(p[g == 'A'])
    rate_B = np.mean(p[g == 'B'])
    return min(rate_A, rate_B) / max(rate_A, rate_B)

# Example
pred = [1,1,0,0,1,0,1,0]
sensitive = ['A','A','A','A','B','B','B','B']
print(round(demographic_parity_ratio(pred, sensitive), 2))

5) Monitoring input and output distributions

import numpy as np

def js_divergence(p, q, eps=1e-12):
    p, q = np.array(p, float), np.array(q, float)
    p = p / (p.sum() + eps)
    q = q / (q.sum() + eps)
    m = 0.5*(p+q)
    def kl(a, b):
        return np.sum(a * np.log((a+eps)/(b+eps)))
    return 0.5*kl(p,m) + 0.5*kl(q,m)

# Example: prediction score histogram drift vs baseline
baseline_hist = [100, 300, 600]
prod_hist     = [200, 400, 400]
print(round(js_divergence(baseline_hist, prod_hist), 3))

Track both feature drift and prediction drift. A jump in high-confidence predictions without label changes can indicate calibration issues or upstream data problems.

Drills and exercises

• Compute PSI for three features; propose thresholds with sample-size floors.
• Create a 7-day rolling F1 chart; add an alert on 10% drop versus prior 14-day mean.
• Generate a reliability curve; label bins as under- or over-confident.
• Calculate demographic parity ratio for two groups; propose an action plan if < 0.8.
• Design an alert that triggers only after two consecutive breaches and min 500 samples.
• Write a pseudocode policy that combines drift + new labels to schedule retraining.

Common mistakes and troubleshooting

• Only monitoring accuracy: Add drift, calibration, and fairness to catch earlier signals.
• Alert fatigue: Require consecutive breaches and sample-size minimums; group alerts by feature or service.
• Ignoring label delay: Use input/output drift as leading indicators while waiting for performance metrics.
• Missing context: Log model_version, data snapshot IDs, and feature schema hash to speed up root-cause analysis.
• Static thresholds: Use baselined z-scores or percent deltas to adapt to seasonality.
• Unsafe retraining: Gate automation with data-quality checks, canary evaluations, and rollback plans.

Mini project: Production-grade monitoring starter

Goal: Build a daily batch monitoring job for one model covering drift, performance (delayed labels), calibration, and fairness, with alerts and a retraining trigger.

Practical projects

• Batch monitor MVP: Daily job computing drift, calibration, and fairness; exports a compact HTML/JSON report.
• Micro-batch stream: Update distribution summaries every 15 minutes; alert only on 2+ successive breaches.
• Calibration improvement: Measure miscalibration, apply Platt scaling or isotonic on new data, compare Brier score before/after.

Subskills

• Data Drift Feature Drift — Detect and quantify shifts in feature distributions using PSI/KS/JS with sane thresholds.
• Concept Drift And Performance Monitoring — Track rolling metrics with label delay and interpret concept shift.
• Prediction Quality Feedback Loops — Build label collection and closed-loop evaluation.
• Calibration Monitoring Basics — Monitor Brier score and reliability; detect under/over-confidence.
• Bias And Fairness Checks Basics — Compute parity/odds metrics with sample-size floors.
• Monitoring Input Output Distributions — Track feature and prediction distributions, nulls, ranges.
• Automated Retraining Triggers Basics — Design safe, multi-signal trigger policies.

Next steps

• Implement logging and one drift check this week; expand gradually.
• Socialize thresholds and fairness metrics with stakeholders.
• Pilot a canary retrain using your trigger policy and document learnings.