Why this matters
When an ML system misbehaves, every minute counts. Root Cause Analysis (RCA) helps you move from vague alarms to precise fixes. In real roles, you will:
- Diagnose sudden metric drops after a new model deploy.
- Explain business KPI changes tied to data or model drift.
- Isolate whether incidents come from data, model, code, infra, or configuration.
- Decide safe rollbacks and short-term mitigations while planning durable fixes.
Who this is for
- Machine Learning Engineers and Data Scientists operating models in production.
- ML Ops/Platform Engineers building monitoring, alerting, and incident workflows.
Prerequisites
- Basic understanding of training/serving pipelines, features, and model metrics (e.g., F1/AUC).
- Familiarity with service metrics (latency, error rate) and data quality metrics (freshness, null rate).
- Ability to read logs/dashboards and run small ad-hoc queries or notebooks.
Concept explained simply
Think of your ML system like a kitchen. When a dish tastes wrong (incident), you check ingredients (data), recipe (code/model), tools (infrastructure), and timing (schedules). RCA is the recipe to figure out exactly what went wrong and why.
Mental model: 7S loop
- Signal: What alert fired? Metric changed? User complaint?
- Symptom: What is observably wrong? When did it start?
- Scope: Which segments/buckets/paths are affected?
- Suspects: Data, Model, Code, Infra, Config, Upstream, External.
- Samples: Pull examples; compare good vs bad.
- Simulate: Reproduce locally or in staging; A/B diff.
- Settle: Fix, verify, add guardrails to prevent recurrence.
Common incident signals
- Model metrics: AUC/F1/MAE shift; calibration off.
- Data quality: Freshness lag, null/invalid spikes, schema changes.
- Drift: PSI/KL spikes on features or prediction distribution.
- Service health: P95 latency, error rate, timeouts, throttling.
- Business KPIs: CTR/conversion/retention drops.
Step-by-step Root Cause Analysis
- Stabilize and triage
Open
- Assess severity: user impact, blast radius, trend.
- Quick mitigation: rollback model/version, switch to fallback rules, or disable experimental flag if safe.
- Capture the snapshot: timestamps, versions, configs, recent changes.
- Frame the problem
Open
- Symptom statement: what changed, by how much, since when.
- Scope analysis: segment by country, device, model version, traffic source, feature availability.
- Diff before/after the change boundary.
- Generate suspects (Ishikawa)
Open
- Data: freshness, nulls, range, distribution drift, label issues.
- Model: new weights, thresholds, feature importance shift, calibration.
- Code: preprocessing parity (train vs serve), serialization/versioning.
- Infra: resource saturation, timeouts, autoscaling, GPU/CPU contention.
- Config/Experiment: flag routing, bucketing, thresholds, feature store keys/TTL.
- Upstream/External: third-party APIs, schema changes, outages.
- Run discriminating tests
Open
- Golden set replay: run known labeled samples through current pipeline and prior good version.
- Shadow compare: route a small slice to previous model for live A/B diff.
- Data checks: null rate, freshness, PSI per feature; schema diff of offline vs online features.
- Infra checks: latency by stage, error type histogram, container restarts, CPU/GPU utilization.
- Confirm cause
Open
- 5 Whys: keep asking why until you reach a process root, not just a symptom.
- Reproduce in staging with controlled inputs; verify the issue disappears when the suspect is fixed.
- Fix and prevent
Open
- Implement the least risky mitigation first (rollback, hotfix, toggle).
- Add guardrails: schema contracts, canary checks, offline/online parity tests, version pins.
- Write a brief incident note with timeline, root cause, impact, and action items.
Worked examples
Example 1: AUC dropped after deploy, data nulls spiked
Signal: AUC 0.86 → 0.78 within 30 minutes of deploying model v3. PSI for feature "device_type" = 0.42; live null rate for "device_type" 18% (was 0.5%).
RCA walkthrough
- Scope: Affects Android traffic mostly.
- Suspects: Online feature retrieval.
- Tests: Compare offline vs online preprocessing; inspect feature store reads. Find key format changed to include app_version, but client still writes using old key. Cache misses → nulls.
- Fix: Hotfix client key format; temporary default for null device_type to "unknown" with calibrated fallback threshold.
- Prevention: Contract tests on feature keys; canary read checks; alert on null rate > 2% for top features.
Example 2: Latency spike, timeouts but model metrics steady
Signal: P95 latency 120ms → 900ms, error rate 0.7% → 4%. AUC steady on shadow traffic.
RCA walkthrough
- Scope: Only requests using new text-embedding feature.
- Suspects: Infra and preprocessing.
- Tests: Resource graphs show CPU at 95%. Code diff reveals new on-the-fly sentence transformer added to live path without batching. Cold-start downloads model weights on first request.
- Fix: Preload weights on startup; enable micro-batching; scale replicas; add timeout budget for embedding stage.
- Prevention: Performance gate in CI; load-test rules; warmup hooks.
Example 3: Drift alert but business KPI stable
Signal: PSI on "country" = 0.3, but CTR unchanged and model metrics stable.
RCA walkthrough
- Scope: All traffic.
- Suspects: Monitoring baseline configuration.
- Tests: Baseline window fixed to last quarter vs current post-campaign traffic. Country mix legitimately changed due to marketing push; labels/metrics stayed fine.
- Fix: Use rolling baseline; per-feature alert with business-aware thresholds.
- Prevention: Document feature with expected seasonality and campaigns.
Checklists
Incident triage checklist
- Define start time and first bad version.
- Roll back or toggle safely if user impact is high.
- Capture configs, flags, model/feature versions.
- Note business impact and segments affected.
Data and model parity checklist
- Nulls, freshness, out-of-range, category cardinality changes.
- Offline vs online preprocessing parity verified.
- Feature store key/schema/TTL parity checked.
- Thresholds/calibration consistent with training.
Infra and routing checklist
- P95/P99 latency by stage; error histogram by type.
- Autoscaling events; restarts; resource saturation.
- Experiment flags and bucket assignments correct.
- Fallbacks functioning and measurable.
Hands-on exercises
Practice here, then submit answers in the exercises section below. Everyone can use the exercises; sign in to save progress.
Exercise 1 (ex1): Metric drop with feature nulls
Given: After v3 deploy, F1 -10%, PSI(device_type)=0.42, null rate(device_type)=18%, latency steady, error rate <1%. Training handled null device_type as "unknown"; online logs show many cache misses on feature store reads introduced at deploy time.
Task: Identify the most probable root cause and propose a low-risk mitigation and a prevention step.
Exercise 2 (ex2): False-positive drift
Given: Weekly drift job reports high PSI on hour_of_day and country; business KPI and validation AUC unchanged; a seasonal sale started yesterday.
Task: Explain why this can be a false-positive and how to adjust monitoring to reduce noise while keeping safety.
Exercise 3 (ex3): Latency and timeouts on embedding stage
Given: P95 latency spikes only for requests with text features. CPU ~95%, GPU idle. New library loads model weights (~500MB) on first request, no warmup. Batch size = 1, no caching.
Task: Propose three specific changes to restore SLOs without sacrificing quality.
Common mistakes and self-checks
- Stopping at the first symptom: Always ask "why" until you reach a process or control that failed.
- Ignoring segment scope: Always break down by version, region, device, feature availability.
- Assuming drift = bad: Validate business impact and model metrics; contextualize with campaigns/seasonality.
- Overfitting fixes: Prefer reversible mitigations first; test in canary before full rollout.
- Poor documentation: Log versions/configs; keep a short incident report to accelerate future RCA.
Self-check: Did you really find the root cause?
- Can you reproduce the issue and make it disappear with one targeted change?
- Does the fix hold across segments and over time?
- Do you have a guardrail to prevent recurrence?
Practical projects
- Build a canary checker that runs a golden set through current and previous model, logging per-feature nulls and PSI.
- Create an offline/online parity test for your top 5 features with schema and key contract validation.
- Instrument per-stage latency tracing and add alerting on the slowest stage.
Learning path
- Monitoring foundations: metrics, SLOs, dashboards.
- Data quality checks: freshness, nulls, schema, drift.
- Deployment safety: canaries, shadowing, rollbacks.
- Incident response: runbooks, on-call, postmortems.
Mini challenge
A new threshold change improves precision but hurts recall; business KPI drops only for new users. What is your top suspect and the first discriminating test you would run? Write your answer in two sentences.
Take the quick test
The quick test is available to everyone; sign in to save your progress.