Why orchestration and scheduling matter for Data Engineers
Modern data platforms rely on hundreds of tasks running in the right order, at the right time, with solid recovery when things go wrong. Orchestration and scheduling give you control: you define dependencies, time windows, SLAs, retries, backfills, and alerts so data arrives fresh and trustworthy. Mastering this skill unlocks reliable pipelines, predictable delivery to stakeholders, and faster incident recovery.
What you’ll learn
- Design Directed Acyclic Graphs (DAGs) that are maintainable, observable, and idempotent.
- Choose schedules (cron/event-driven) and define SLAs that match business needs.
- Configure retries, timeouts, and actionable alerts that reduce noise.
- Run safe backfills and targeted re-runs without double-processing data.
- Parameterize pipelines for partitions (e.g., date, region) and reuse.
- Separate dev/stage/prod with promotion rules and minimal risk.
- Monitor pipeline health and respond with clear incident runbooks.
Who this is for and prerequisites
- Who this is for: Data Engineers, Analytics Engineers, Platform Engineers, and SREs supporting data platforms.
- Prerequisites: Basic Python or SQL; comfort with batch/ETL concepts; familiarity with a scheduler/orchestrator (e.g., Airflow/Prefect/Luigi) is helpful but not required.
Learning path
-
1) DAG basics and dependencies
Model pipelines as DAGs, enforce acyclic flow, and design idempotent tasks. Use clear naming and small, single-purpose tasks.
-
2) Scheduling and SLAs
Pick cron or event triggers, define SLAs tied to business deadlines, and ensure concurrency settings support them.
-
3) Retries, timeouts, and alerts
Set retry counts/backoff, per-task timeouts, and actionable alerts with run context and clear next steps.
-
4) Backfills and re-runs
Backfill missing partitions safely, avoid duplication, and use data checks to validate before/after.
-
5) Parameterization
Template dates/regions, drive runs from config, and keep logic DRY while preserving observability.
-
6) Environment separation
Promote from dev → stage → prod with versioned code, immutable artifacts, and isolated configs/queues.
-
7) Monitoring and health
Track SLAs, durations, success rates, and data validation metrics. Alert on symptoms, not just failures.
-
8) Incident runbooks
Write step-by-step recovery guides including rollback points, backfill commands, and communication templates.
What happens when your DAG fails at 3 a.m.?
Your orchestrator should retry with sensible backoff, notify the on-call person with context (task, run_id, last logs), and point to the runbook. If the issue persists, the pipeline should fail fast, protect data quality with idempotency, and make backfill steps clear.
Worked examples
Example 1 — Daily batch DAG with SLA and dependencies (Airflow-style)
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import PythonOperator
def extract(ds, **kwargs):
# idempotent: read partition for ds, write to staging/ds
pass
def transform(ds, **kwargs):
# idempotent: transform staging/ds -> warehouse/ds
pass
def validate(ds, **kwargs):
# data checks: row counts, freshness thresholds
pass
def load(ds, **kwargs):
# publish to serving layer
pass
def on_failure(context):
# minimal alert: task_id, dag_id, run_id, log_url
print(f"ALERT: {context['task_instance']}")
with DAG(
dag_id="daily_sales_pipeline",
schedule="30 2 * * *", # 02:30 daily
start_date=datetime(2023, 1, 1),
catchup=False,
default_args={
"retries": 3,
"retry_delay": timedelta(minutes=5),
"on_failure_callback": on_failure,
},
sla=timedelta(hours=1, minutes=0), # pipeline SLA of 1h
max_active_runs=1,
) as dag:
start = EmptyOperator(task_id="start")
t_extract = PythonOperator(task_id="extract", python_callable=extract, op_kwargs={})
t_transform = PythonOperator(task_id="transform", python_callable=transform)
t_validate = PythonOperator(task_id="validate", python_callable=validate)
t_load = PythonOperator(task_id="load", python_callable=load, execution_timeout=timedelta(minutes=20))
end = EmptyOperator(task_id="end")
start >> t_extract >> t_transform >> t_validate >> t_load >> end
Notes: SLA tracks overall duration; per-task timeouts prevent silent hangs; idempotent task design enables safe retries.
Example 2 — Parameterized pipeline by date and region
from datetime import date
def run_partition(run_date: str, region: str):
# treat (run_date, region) as a unique partition key
# read raw/run_date/region -> stage/ -> warehouse/
pass
# Example invocations by orchestrator
for region in ["us", "eu", "apac"]:
run_partition(run_date=str(date.today()), region=region)
Notes: Store run parameters in logs/metadata. Keep outputs partitioned by the same keys to ensure idempotency and easy backfills.
Example 3 — Safe backfill for the last 7 days
from datetime import date, timedelta
def backfill(days: int = 7):
today = date.today()
for i in range(1, days + 1):
ds = str(today - timedelta(days=i))
# enqueue run for ds with catchup semantics
print(f"Enqueue backfill for {ds}")
# Before running, ensure:
# - outputs are partitioned by date
# - writes are overwrite/upsert-safe
# - SLA alerts may be muted for backfills if needed
Notes: Run backfills off-peak with limited concurrency; verify counts before and after; avoid mixing backfills with prod SLAs unless necessary.
Example 4 — Event-driven trigger on file arrival (sensor pattern)
def wait_for_file(path, timeout_minutes=30):
# poll storage until file appears; abort if timeout
pass
# Upstream
# wait_for_file("raw/invoices/{{ ds }}/drop.done")
# then extract -> transform -> load
Notes: For event-driven flows, add a safety timeout and alerts that include which file/partition was missing.
Example 5 — Dev/Stage/Prod separation with configuration
import os
ENV = os.getenv("ENV", "dev")
CONFIG = {
"dev": {"concurrency": 2, "warehouse": "dev_wh", "alerts": "dev-alerts"},
"stage": {"concurrency": 4, "warehouse": "stage_wh","alerts": "stage-alerts"},
"prod": {"concurrency": 10, "warehouse": "prod_wh", "alerts": "prod-oncall"},
}[ENV]
print(f"Running in {ENV} with {CONFIG}")
Notes: Use the same code artifact across environments; switch only configuration, credentials, and queues. Promote versions forward, never edit in place.
Drills and exercises
- Create a 3-task DAG (extract, transform, load) with explicit dependencies and a 45-minute SLA.
- Set retries=3 with exponential backoff and a 15-minute execution timeout on the slowest task.
- Parameterize the pipeline by date and run it for yesterday’s partition.
- Simulate a failure and confirm logs, alert content, and retry behavior are useful and idempotent.
- Run a backfill for the last 3 days and verify row counts do not double.
- Split deployment into dev and stage; prove stage uses separate queues and credentials.
Common mistakes and debugging tips
Coupling tasks too tightly
Symptom: Small changes ripple through the entire DAG. Fix: Make tasks single-purpose; pass partition keys not massive payloads; persist intermediate outputs.
Non-idempotent writes
Symptom: Retries create duplicates. Fix: Write by partition; use overwrite/upsert; include run_id or checksum guards.
Unbounded retries
Symptom: Hours of silent looping. Fix: Cap retries, add exponential backoff, and alert after the first failure with context.
Overlapping runs
Symptom: Two runs write to the same partition. Fix: Use max_active_runs=1 (or equivalent) and resource-level locks for shared sinks.
Backfills during business hours
Symptom: SLA breaches for current-day jobs. Fix: Schedule backfills off-peak and throttle concurrency; temporarily mute non-critical alerts.
Mini project: Sales daily pipeline with SLAs and backfills
Build a pipeline that ingests daily sales CSVs, transforms them into a star schema, validates totals, and publishes to a serving table.
- Requirements:
- Schedule: 02:00 daily; SLA: complete by 03:00.
- Idempotent tasks; per-task timeout 20 minutes; retries=3 with backoff.
- Parameters: date and region; run today for all regions.
- Backfill: last 7 days, throttled to avoid SLA impact.
- Monitoring: duration trend, success rate, row-count validation, alert with context.
- Runbook: steps to re-run a failed date, verify counts, and communicate status.
Subskills
- DAG Design And Dependencies — Model tasks cleanly, ensure acyclic flow, and design for idempotency.
- Scheduling And SLAs — Choose cron/event triggers and define deadlines that reflect business needs.
- Retries Timeouts And Alerts — Recover automatically, avoid hangs, and notify with actionable context.
- Backfills And Reruns — Repair missing data safely without duplicates or gaps.
- Parameterized Pipelines — Reuse logic across partitions, regions, and configs.
- Environment Separation Dev Stage Prod — Promote safely with isolated configs and resources.
- Monitoring Pipeline Health — Track durations, success, data checks, and alert on symptoms.
- Incident Runbooks — Step-by-step guides for fast, consistent recovery.
Practical projects
- Data freshness watchdog: a tiny DAG that scans key tables, checks max partition timestamps, and alerts when stale.
- Partitioned backfill tool: script that enqueues backfills for a date range and reports row deltas before/after.
- Environment promotion demo: same code artifact deployed to dev→stage→prod with different concurrency and alert channels.
Next steps
- Harden your alerting with run context and quick links to logs and runbooks.
- Add data quality checks (row counts, unique keys) as first-class tasks, not afterthoughts.
- Automate backfill workflows with guardrails and reporting.
Skill exam
Ready to test yourself? Start the exam below. Everyone can take it for free. If you log in, your progress and results will be saved.