Monitoring Quality Over Time

Who this is for

• AI Product Managers who own live ML/LLM products and need to keep quality stable and improving.
• Data Science and MLOps collaborators aligning on metrics, alerts, and runbooks.
• Leaders wanting reliable, measurable product outcomes from AI features.

Prerequisites

• Basic understanding of offline metrics (e.g., accuracy, F1, NDCG) and online metrics (e.g., CTR, conversion).
• Familiarity with experiments (A/B, holdouts) and dashboards.
• Comfort discussing sampling, cohorts, and data drift at a high level.

Why this matters

Shipping an AI feature is step one. Keeping it useful, safe, and cost-effective every day is the real job. Monitoring quality over time helps you:

• Catch regressions early (before users churn or incidents escalate).
• Understand seasonality vs. true model decay.
• Detect data drift and evaluate when to retrain or recalibrate.
• Balance quality with latency, cost, and safety guardrails.

Concept explained simply

Monitoring quality over time means measuring the most important signals of user value, safety, and reliability on a routine basis and reacting quickly when they move out of healthy ranges.

Mental model

• Inputs: data distribution, prompts, features, traffic mix.
• System: model + policies + infrastructure.
• Outputs: predictions/responses, latency, cost, and user outcomes.
• Feedback: human labels, user interactions, golden datasets.

You choose a small set of leading indicators (drift, safety flags, calibration) and lagging indicators (conversion, retention, CSAT). You define baselines and acceptable bounds, then watch trends and cohorts.

What to monitor (the essentials)

• Product outcomes: conversion, CTR, satisfaction, task success.
• Model quality: F1/AUC, NDCG/MRR, ROUGE/BLEU, response ratings, calibration.
• Safety/Policy: toxicity rate, PII leaks, hallucination rate, prompt injection success rate.
• Reliability: latency percentiles, error rate, timeout rate, uptime.
• Cost/Throughput: cost per 1k requests, token usage, compute per prediction.
• Drift: feature distribution drift (e.g., PSI), label drift, prompt/query mix changes.

Designing metrics and SLOs

Pick a small, meaningful set (3–7) that covers value, safety, and reliability. Example:

• Value SLO: “Weekly NDCG@10 ≥ 0.78 (7-day rolling).”
• Safety SLO: “Hallucination rate ≤ 2% daily, ≤ 1% weekly.”
• Reliability SLO: “p95 latency ≤ 800 ms, 99.9% of requests.”
• Cost Guardrail: “Cost per 1k requests ≤ $X; alert at +20% week-over-week.”

Setting baselines and thresholds

1. Step 1: Establish a baseline window (e.g., last 28 days) with stable performance.
2. Step 2: Use rolling windows (7-day, 14-day) to reduce noise.
3. Step 3: Add seasonality awareness (compare vs. prior week or same weekday).
4. Step 4: Define alert tiers: warn at small deviations, page at large or sustained deviations.
5. Step 5: Include cohort and segment views (new vs. returning, locale, platform).

Detecting drift

• Input drift: Track feature distributions and prompt/query mix. Use simple indices like PSI (Population Stability Index) or distance on embeddings.
• Label drift: Target prevalence changes (e.g., fraud rate shifts).
• Concept drift: Performance declines while inputs look stable → model no longer fits reality.

Instrumentation checklist

• Define 3–7 top-line metrics + 3 guardrails (safety, latency, cost).
• Set SLOs with rolling windows and alert tiers.
• Create a golden dataset and review cadence.
• Log enough context to slice by cohort.
• Track drift on key inputs and outputs.
• Document a clear runbook and ownership.

Worked examples

1) Fraud classification (binary)

• Value: Recall@Precision≥0.9; weekly AUCPR ≥ 0.84.
• Reliability: p95 latency ≤ 120 ms.
• Cost: CPU per 1k requests ≤ baseline +15%.
• Drift: PSI on transaction_amount, country, device_type; alert at PSI ≥ 0.25.
• Runbook snippet: If recall drops 3 days in a row and PSI high on country, check data ingestion for that region; if confirmed, hotfix rules and schedule retrain.

2) Search ranking (NDCG)

• Value: NDCG@10 (logged) ≥ 0.80 (7-day rolling).
• Seasonality: Weekend traffic differs; compare Fri–Sun vs prior Fri–Sun.
• Drift: Query category mix; if “new releases” spikes, expect temporary NDCG dip.
• Action: Auto-expand golden set with high-traffic new-release queries; run targeted evaluation weekly during launches.

3) LLM assistant (hallucination risk)

• Safety: Hallucination rate ≤ 2% daily; toxic content ≤ 0.3%.
• Value: User rating ≥ 4.3/5 weekly; task success ≥ 70%.
• Guardrails: Prompt-injection success ≤ 0.5% on red-team prompts.
• Runbook: If hallucination > 2%, enable stricter policies, expand citations requirement, increase human review sampling until back within bounds.

Practical projects

• Build a golden dataset: 150 cases covering typical + edge scenarios. Define scoring rubric and review monthly.
• Author a runbook: One page with triggers, checks, actions, owners, and timelines.
• Create a synthetic drift simulator: Vary one feature weekly and observe metric and alert behavior.

Exercises

These mirror the exercises below so you can complete them here and submit in the exercise section.

Exercise 1: Draft a monitoring plan

Pick an AI recommender for a home page. Specify 3–5 core metrics with SLOs, 2 guardrails, drift checks, alert thresholds, and a weekly review cadence. Use this template:

• Value metrics + SLOs:
• Safety metrics + SLOs:
• Reliability metrics + SLOs:
• Cost guardrails:
• Drift checks:
• Alert thresholds (warn/page):
• Review cadence + owners:

Exercise 2: Diagnose a drop

Scenario: NDCG@10 fell from 0.81 to 0.74 over 3 days. p95 latency unchanged. Cost +10%. Query mix shows more long-tail queries. PSI: title_length 0.05, category 0.28, user_locale 0.07. What is your likely cause and first 3 actions?

Common mistakes and self-check

• Too many metrics → nobody watches. Self-check: Can you explain the top 5 without notes?
• No cohorts → issues hide. Self-check: Can you slice by locale/device/new users?
• Ignoring seasonality → false alarms. Self-check: Do you compare to the right prior periods?
• No runbook → slow responses. Self-check: Do you have owners and actions defined?
• Only offline metrics → misses user value. Self-check: Do you track outcomes like conversion or task success?

Learning path

• Start with defining product outcomes that matter.
• Map outcomes to a compact metric set + SLOs.
• Add guardrails (safety, latency, cost) and drift detection.
• Build a golden dataset and a review ritual.
• Create dashboard, alerts, and a runbook. Iterate monthly.

Next steps

• Finish the exercises below and compare with the solutions.
• Take the quick test. Note: Anyone can take it; logged-in learners get saved progress.
• Apply the monitoring plan to one real feature within a week.

Mini challenge

In 10 minutes, write a one-sentence SLO that balances value and safety for your current AI feature. Share it with your team and ask, “What would break this?”