Alerting On Failures And Delays

Why this matters

In MLOps, delays and failures silently erode trust: stale features cause bad decisions, missed batch predictions hurt SLAs, and slow or failing online models break user journeys. Effective alerting tells the right person, at the right time, with the right context—so incidents are resolved fast and wasteful noise is avoided.

• Data pipelines: detect late or missing data before downstream jobs amplify the issue.
• Training jobs: fail fast when jobs crash or exceed time budgets.
• Batch predictions: alert when outputs are late, incomplete, or erroring.
• Online inference: page when error rate or latency spikes; ticket when degradation is minor.

Concept explained simply

Alerting is the guardrail for your ML production system. You define signals (metrics, logs, events) and thresholds that reflect user impact. When a signal crosses a threshold for long enough, you notify on-call with a clear, actionable message and a runbook.

Mental model

• Smoke detector: Detect symptoms (errors, latency, staleness) before the fire spreads.
• Severity ladder: Page for user impact now; create tickets for maintenance issues.
• Actionable alerts only: Every alert must answer “what broke, why it matters, what to do next.”

Core signals and practical thresholds

• Job success rate: alert if success == 0 for a schedule window (e.g., no hourly runs succeed for 2 hours).
• Data freshness: alert if latest partition or event is older than X (e.g., 2x the schedule).
• Error rate: page if error_rate > 5% for 5 minutes (tune to your baseline).
• Latency (p95/p99): alert if p95 > budget (e.g., 300 ms) for 10 minutes.
• Backlog/lag: alert if queue lag or records waiting > threshold, and rising.
• Heartbeat: alert if a heartbeat metric or table row hasn’t updated within 2x interval.

Start strict, then tune to minimize false positives while keeping time-to-detect low.

Worked examples

Example 1: Hourly batch prediction job fails

Signal: job_success_count over a rolling 2h window.

{"alert": "BatchPredictionsFailed",
"expr": "sum_over_time(job_success{job=\"batch_predict\",env=\"prod\"}[2h]) == 0",
"for": "10m",
"labels": {"severity": "page"},
"annotations": {"summary": "No successful batch predictions in 2h (prod)",
"action": "Check scheduler logs, job container logs, and input dataset availability. If input missing, pause downstream consumers and start backfill per runbook BP-01."}}

Why it works: avoids paging for a single missed run; catches prolonged outage.

Example 2: Ingestion delay (data freshness)

Signal: freshness = now() - max(event_time) for topic/table.

-- Pseudo-SQL freshness check
SELECT TIMESTAMPDIFF(MINUTE, MAX(event_time), NOW()) AS freshness_min
FROM events_prod;
-- Alert if freshness_min > 30 for 15 minutes

{"alert": "DataFreshnessExceeded",
"expr": "freshness_min{source=\"events_prod\"} > 30",
"for": "15m",
"labels": {"severity": "ticket"},
"annotations": {"summary": "Events data freshness > 30 min",
"action": "Inspect ingestion connectors, broker lag, and dead-letter queues. If lag rising, scale consumers and replay DLQ per runbook DF-02."}}

Why it works: focuses on user-visible freshness rather than internal component metrics.

Example 3: Online inference latency and errors

Signals: p95_latency_ms and error_rate.

{"alert": "InferenceLatencyHigh",
"expr": "histogram_quantile(0.95, sum(rate(latency_bucket{svc=\"inference\",env=\"prod\"}[5m])) by (le)) > 300",
"for": "10m",
"labels": {"severity": "page"},
"annotations": {"summary": "p95 latency > 300ms for 10m",
"action": "Check recent deploys, autoscaling, and upstream feature store latency. Mitigate by scaling out replicas or rolling back."}}

{"alert": "InferenceErrorsSpike",
"expr": "rate(errors_total{svc=\"inference\",env=\"prod\"}[5m]) / rate(requests_total{svc=\"inference\",env=\"prod\"}[5m]) > 0.05",
"for": "5m",
"labels": {"severity": "page"},
"annotations": {"summary": "Error rate > 5%",
"action": "Check model canary, feature availability, and dependency status. Fallback to previous model if needed."}}

Why it works: two complementary alerts capture both slowness and correctness issues.

Design your alerts — step-by-step

1. Define user-impacting SLOs: e.g., 99% hourly batch success; p95 < 300 ms; freshness < 30 min.
2. Pick SLIs: metrics/logs that map to those SLOs.
3. Choose thresholds and durations: balance sensitivity vs. noise.
4. Routing and severity: page for immediate impact; ticket for non-urgent issues.
5. Add runbook steps: concrete next actions and owners.

Exercises

Do these in a sandbox or by drafting rules in a safe environment.

Exercise 1: Hourly batch inference delays

Design an alert that triggers when an hourly batch inference pipeline has no successful runs in 2 hours OR the message queue backlog exceeds 1000 for 15 minutes. Include severity, annotations, and actions.

Exercise 2: Error-budget burn alert

Your online model has a 99.5% success SLO (0.5% error budget). Create a two-window burn-rate alert: fast window 5m, slow window 1h. Page when both windows indicate budget burn >= 4x.

Common mistakes and self-check

• Alerting on causes, not symptoms: prefer freshness over “connector CPU high.”
• One-off spikes causing pages: add duration and multi-window logic.
• Noisy duplicate alerts: group by service/job and route once.
• Missing runbooks: every alert should suggest first 3 diagnostic steps.
• No ownership: define team, on-call rotation, and escalation.
• Static thresholds in dynamic traffic: consider percentiles and rates.

Practical projects

• Implement end-to-end alerting for a toy pipeline (ingest → feature store → batch predict). Include heartbeats, freshness, and success rate.
• Add online inference alerts: error rate, p95 latency, and model-specific fallbacks.
• Create a runbook library with copy-pastable commands and dashboards to check.

Learning path

• Start: understand SLI/SLO basics and severity/routing.
• Build: implement batch and online alerts with durations and grouping.
• Refine: add multi-window burn-rate and maintenance silence windows.
• Scale: measure alert quality (noise, MTTA, MTTR) and iterate.

Who this is for

• MLOps engineers responsible for pipelines, model services, and reliability.
• Data engineers integrating ML workloads with orchestration systems.
• ML engineers deploying models to production.

Prerequisites

• Basic metrics concepts (counters, gauges, histograms) and logs.
• Familiarity with your orchestrator/scheduler and service metrics.
• Ability to write simple query/threshold expressions.

Next steps

• Draft alerts for one batch pipeline and one online service.
• Run a game day: simulate failure and latency spikes; tune thresholds.
• Add runbooks and define on-call rotation and escalation policy.

Mini challenge

You deploy a new model version. 15 minutes later, error rate is 3% (baseline 0.3%), p95 latency is stable. Draft a single alert that would have caught this quickly without being noisy. Include severity, duration, and the first three debugging steps.

Quick Test

Take the quick test below to check your understanding. Everyone can take it for free; only logged-in users will have their progress saved.