SLA Monitoring And Alerting

Why this matters

As a Data Platform Engineer, you run time-sensitive pipelines that power dashboards, ML features, and operational systems. When data arrives late or fails silently, leaders decide with outdated numbers and services degrade. SLA monitoring and alerting turns these risks into known, actionable signals.

• Guarantee delivery: For example, “Daily revenue table complete by 06:30 UTC.”
• Detect issues early: Catch slow tasks, missing runs, or stale datasets before stakeholders do.
• Reduce on-call noise: Route the right alert to the right person with the right urgency.

Concept explained simply

Simple definition

An SLA (Service Level Agreement) states a promise about your data system’s behavior, like “Dataset X is updated by 07:00 UTC on weekdays.” Monitoring tracks signals that reflect whether you meet that promise. Alerting notifies humans when you’re off track or at risk.

Mental model

Think of an SLA as a train timetable. Monitoring is the station clock and sensors that check speed and delays. Alerts are the station announcements to staff when a train will be late so they can take action before passengers are stranded.

Key terms

• SLA: The promise (deadline/quality target). Often external-facing.
• SLO: Internal target that supports the SLA (e.g., 99% of runs by 06:30 UTC).
• SLI: Measurable indicator (e.g., minutes since last successful load, runtime p95).
• Hard vs. Soft SLA: Hard must be met; soft is preferred but not always critical.
• Run-level vs. Dataset-level: Run-level checks task/job execution; dataset-level checks freshness/row counts/quality.

Designing SLAs in orchestration platforms

1. List critical consumers: Dashboards, ML models, downstream jobs depending on timely data.
2. Define the promise: Deadline, frequency, and tolerance. Example: “Complete by 06:30 UTC, Mon–Sat, 99% of days.”
3. Choose SLIs: Completion time, data freshness (lag minutes), run success ratio, data completeness (row counts), anomaly scores.
4. Set thresholds: Baseline from last 30 days of healthy runs. Use p95/p99 duration and typical window for freshness.
5. Alert routing: Severity levels (warning, critical) mapped to the on-call schedule or channel.
6. Runbooks: Clear steps per alert type, with decision trees and rollback plans.
7. Feedback loop: Track alert rate, false positive ratio, mean time to acknowledge (MTTA), and mean time to recover (MTTR). Tune regularly.

Worked examples

Example 1: Daily sales DAG by 06:30 UTC

• SLA: Sales fact table is complete by 06:30 UTC, Mon–Sat, 99% of days.
• SLIs: DAG completion time, p95 task duration, row count vs. 7-day median, freshness lag from source.
• Alerts:
  • Warning at 06:15 if DAG is still running and p95 runtime predicts lateness.
  • Critical at 06:31 if not successful.
  • Warning if row count < 90% of 7-day median.
• Runbook: Check cluster capacity, queue length, slowest task logs; consider skipping non-critical tasks to meet deadline.

Example 2: Streaming pipeline freshness

• SLA: Derived stream is ≤ 5 minutes behind source during business hours.
• SLIs: End-to-end lag minutes, consumer offsets, error rate per minute.
• Alerts:
  • Warning at 3 minutes lag for 5 consecutive checks.
  • Critical at 6 minutes lag or sustained error spikes.
• Runbook: Auto-scale consumers; if offsets stall, restart consumers; validate source throughput; if source outage, set maintenance status.

Example 3: Multi-hop data mart

• SLA: Marketing mart updated by 08:00 UTC.
• Dependencies: Raw → Staging → Core → Mart.
• Approach: Track SLIs per hop and at the mart. Predict lateness when any upstream p95 exceeds budget.
• Alerting: Deduplicate into a single incident (root cause upstream), link all impacted downstream SLAs.

Alert design that reduces noise

• Severity: Warning for predicted lateness or soft thresholds; Critical when SLA is breached or data risk is high.
• Deduplication: One incident per root cause; suppress duplicates from downstream jobs.
• Escalation: If not acknowledged in X minutes, escalate to secondary on-call.
• Silence windows: Maintenance windows or known noisy backfills. Document them and auto-disable after a timer.
• Clear messages: Include pipeline, SLA name, deadline, current SLI, suspected root cause, and first action.

Alert message template
- Pipeline: {pipeline_id}
- SLA: {sla_name}
- Status: {warning|critical}
- Evidence: {sli_name}={value} vs threshold {threshold}
- Deadline: {deadline_utc}, Predicted: {eta_utc}
- First action: {runbook_step}
- Link to runbook: {runbook_id}

Runbooks and auto-remediation

1. Immediate triage: Identify failing task, last healthy run, recent changes.
2. Containment: Pause downstream consumers or notify stakeholders of delay.
3. Fix: Retry with increased resources, clear stuck locks, reprocess a missing partition.
4. Recovery: Backfill affected partitions; verify row counts and quality checks.
5. Prevention: Add regression checks, tune thresholds, or optimize the slow step.

Exercises

Complete these tasks, then check your answers. Everyone can take the test; only logged-in users have saved progress.

Submission checklist

• I stated clear, measurable SLAs.
• My SLIs map to those SLAs and have realistic thresholds.
• Alerts include evidence and a first action.
• Escalation rules prevent paging loops.
• Silence windows are temporary and auditable.

Common mistakes and how to self-check

• Too many alerts: If weekly alert count is high and MTTA is low-impact, consolidate or raise thresholds.
• Downstream noise: Suppress child alerts; page on the root cause.
• Static thresholds: Recompute from rolling baselines monthly or when workloads change.
• Missing runbooks: If an alert doesn’t suggest a first action, it’s not production-ready.
• Ignoring freshness: Completion success without data freshness checks gives false confidence.

Practical projects

• Top-5 SLA rollout: Pick five business-critical pipelines, define SLAs/SLIs, add alerts and runbooks, and review after two weeks.
• Freshness dashboard: Build a page that shows freshness lag and predicted breach times for key datasets.
• Auto-remediation pilot: Implement safe auto-retry and scale-out for a noisy pipeline; measure reduced pages.

Learning path

1. Baseline: Instrument run durations, success rates, and data freshness for your pipelines.
2. Define: Draft SLAs for the top consumers with clear deadlines and tolerances.
3. Alert: Configure warning/critical rules and routing with escalation.
4. Operate: Add runbooks, on-call rotation, and weekly alert reviews.
5. Improve: Add anomaly detection and predictive alerts based on historical durations.

Who this is for

• Data Platform Engineers building and operating schedulers and pipelines.

Prerequisites

• Basic orchestration knowledge (DAGs/tasks/runs).
• Familiarity with metrics and logs.
• Understanding of business deadlines and data dependencies.

Next steps

• Implement SLAs for one pipeline this week.
• Run a tabletop incident using your new runbooks.
• Take the quick test below to validate your understanding.

Mini challenge

You have a pipeline that usually completes by 05:50 UTC. Today at 05:35 UTC, it’s only 40% complete, predicted finish 06:40 UTC. Draft a single warning alert message that includes the SLI value, predicted finish, and one concrete action that could help meet the SLA.