Why this matters
In MLOps, incidents hit both reliability and business outcomes. A failing feature store, a drifting model, or a misconfigured rollout can silently degrade predictions, impacting users and revenue. Effective triage restores service quickly; strong postmortems prevent repeats and improve on-call life.
- Real tasks you will do: assess severity and blast radius; execute rollback or failover; coordinate across data, platform, and product; write blameless postmortems with clear actions.
- Use this lesson to standardize your response, reduce time-to-detect, and shorten time-to-recover.
Note: The Quick Test at the end is available to everyone; only logged-in users will have saved progress.
Concept explained simply
Incident triage is the rapid process of understanding what is broken, how bad it is, and what to do right now to protect users and the business. Postmortems are structured write-ups after recovery to learn, fix, and improve.
Key terms
- Severity (SEV): how bad the impact is (users/business). Guides urgency and coordination.
- SLI/SLO: metrics and targets that define acceptable reliability and quality.
- Triage: stabilize, isolate cause, choose mitigation (rollback, bypass, rate-limit, failover).
- Runbook: step-by-step guide to diagnose and mitigate a known class of issues.
- RCA (Root Cause Analysis): structured analysis (5 Whys, fishbone) to find contributing factors.
- Blameless postmortem: focuses on systems and processes, not people.
Mental model
Think of two loops:
- Fast loop: Detect → Triage → Mitigate → Communicate → Recover.
- Slow loop: Analyze → Decide improvements → Implement → Verify in future incidents.
Optimize handoffs between loops. During the fast loop, speed and safety matter more than perfection.
The triage flow (7 steps)
- Acknowledge and declare: Assign a DRI (directly responsible individual). Create an incident channel and short summary (what/impact/status).
- Assess severity: Use SLO breaches and business impact to set SEV level.
- Stabilize safety: Stop the bleeding (rollback model, failover to cached features, disable risky experiments).
- Gather signals: Logs, metrics, traces, model quality monitors (drift, data integrity, feature freshness), and recent changes.
- Form a hypothesis: Narrow scope. Prefer reversible, low-risk tests.
- Mitigate: Execute runbook steps. Document actions and timestamps.
- Communicate: Update stakeholders with concise status, ETA, and user guidance. Close with outcomes and follow-up plan.
Quick role guide
- DRI: coordinates, decides on mitigation, posts updates.
- Investigator(s): deep dive on data/model/system signals.
- Scribe: keeps the timeline and decisions.
- Stakeholders: product, support, leadership, affected teams.
What to watch (ML-specific signals)
- Prediction quality: online proxy metrics, shadow labels, delayed ground truth, canary vs baseline.
- Drift: PSI/JS distance on features/targets, population stability vs training.
- Data integrity: nulls, schema mismatches, value ranges, categorical cardinality.
- Feature freshness: staleness lag, backfill completeness.
- Latency and throughput: model server p50/p95, feature store read times.
- Infra health: CPU/GPU, memory, container restarts, autoscaling events.
- Downstream errors: 4xx/5xx ratios, timeouts, queue backlogs.
- Business KPIs: conversion, fraud catch rate, false positive rate (when available).
Worked examples
Example 1: Data drift causing higher false declines
- Alert: PSI 0.35 on merchant_category; support tickets spike; model precision down ~15% (delayed labels).
- Triage: SEV-2; rollback to previous model; enable fallback rules for top merchants; throttle new rules experiment.
- Signals: spike in nulls for merchant_category (18% vs 2% baseline); feature store backfill job failed.
- Mitigation: hotfix to impute category; rebuild backfill; reprocess last 24h features.
- Postmortem actions: add null-rate SLO with alert; dependency health check before training; runbook: “category-null spike.”
Example 2: Latency spike from feature store region outage
- Alert: p95 latency from 120ms → 1.8s; error rate 6%.
- Triage: SEV-1; switch traffic to secondary region; reduce batch size; enable cached features path.
- Signals: read timeouts to primary store; no model CPU saturation; upstream network incident.
- Mitigation: force regional failover; warm caches; raise autoscaling min replicas.
- Postmortem actions: automated health-check failover; chaos drill quarterly; cache hit-rate SLO.
Example 3: Bias alert in recommendation system
- Alert: subgroup A click-through drops 30% vs others after rollout.
- Triage: SEV-2 (equity impact); partial rollback for subgroup A; enable baseline model for them.
- Signals: new embedding quantization; A/B canary shows disproportionate effect.
- Mitigation: disable quantization for A path; collect additional telemetry.
- Postmortem actions: fairness checks in pre-prod; subgroup canary gates; update rollout checklist.
Runbooks and on-call checklist
Runbook skeleton (copy/paste)
- Title: [Runbook name]
- Scope: Systems and models covered
- Preconditions: Access, tools, permissions
- Quick triage: Severity rubric and rollback toggle
- Diagnosis steps: Ordered checks with expected signals
- Mitigations: Safe, reversible actions + rollback
- Validation: Metrics to confirm recovery
- Communication: Status update template
- Escalation: Who and when
- [ ] Can I rollback/disable quickly and safely?
- [ ] Are SLOs breached or just noisy metrics?
- [ ] Did anything change recently (code, data, config)?
- [ ] Is the blast radius understood?
- [ ] Are stakeholders updated with plain language and ETA?
Postmortem template
Open template
- Summary: What happened and impact
- Timeline: Key events with timestamps
- Detection: How it was detected; time to detect
- Response: Actions taken; time to mitigate/recover
- Root causes and contributing factors: Systems/process focus
- What went well/poorly: Tools, runbooks, comms
- Action items: Specific, owner, due date, status
- Prevention: Tests, alerts, guardrails to add
Common mistakes and self-check
- Chasing perfect root cause before stabilizing. Self-check: Did we execute a safe mitigation within minutes?
- Over-alerting. Self-check: Are alerts tied to SLOs and deduplicated?
- Blame-focused write-ups. Self-check: Does the postmortem emphasize systems and safeguards, not individuals?
- Vague actions. Self-check: Do all actions have owners and deadlines?
- No rehearsal. Self-check: When was the last drill of this runbook?
Practical projects
- Build a severity rubric and incident runbook for “feature freshness lag.” Test it with a dummy incident.
- Implement a drift alert that auto-attaches top 5 suspect features and recent change logs to an incident ticket.
- Run a tabletop exercise: simulate a model rollback and complete a postmortem using the template.
Who this is for, prerequisites, learning path
Who this is for
- MLOps engineers, data platform engineers, and ML engineers who participate in on-call and reliability.
Prerequisites
- Basic monitoring/observability (metrics, logs, traces).
- Familiarity with your deployment and feature pipelines.
- Comfort with incident communication basics.
Learning path
- Before: Model monitoring fundamentals; Alert design and SLOs.
- Now: Incident triage and postmortems (this lesson).
- Next: Automated rollback, canary strategies, resilience testing and chaos drills.
Exercises
Do these now. Then try the Quick Test to check your understanding.
Exercise 1: Create a triage plan for a drift incident
Alert context: precision down ~15%; PSI on merchant_category = 0.35; latency normal; support tickets rising; last deployment 2 days ago; nulls in merchant_category 18% (baseline 2%). Produce: (1) severity, (2) first safe mitigation, (3) top 5 diagnostic checks, (4) a short decision tree for rollback vs hotfix, (5) a 3-line stakeholder update.
Exercise 2: Write a concise postmortem
Scenario: Feature store outage caused 502 errors for 45 minutes; mitigation was failover to cached features; root cause: expired TLS cert on primary store. Produce: (1) a 6-part postmortem using the template, (2) three action items with owners and due dates, (3) one metric to alert earlier next time.
- [ ] I stated severity with a clear rationale.
- [ ] I chose a mitigation that is reversible and safe.
- [ ] I listed concrete signals to check and why.
- [ ] My postmortem actions include owner + deadline.
- [ ] I proposed a metric or guardrail to prevent recurrence.
Mini challenge
You receive: p95 latency normal; prediction volume down 35%; drift low; error rate up to 5% (timeouts); deployment happened 4 hours ago with new client SDK. In 5 lines, decide severity, immediate mitigation, the most likely subsystem to check first, a rollback choice, and a one-sentence update for stakeholders.
Next steps
- Automate attaching key diagnostics (drift summary, top null features, last 10 config changes) to incidents.
- Add pre-deploy checks: schema diff, feature availability, shadow evaluation.
- Schedule quarterly incident drills using your highest-risk runbooks.