DAG Concepts And Task Design

Who this is for

Analytics Engineers and BI Developers who need to design reliable, maintainable data workflows in orchestrators (e.g., Airflow, Prefect, Dagster). If you schedule dbt models, refresh dashboards, or move data between systems, this is for you.

Prerequisites

• Basic SQL and data warehouse concepts
• Familiarity with ELT workflows (extract, load, transform)
• Comfort with scheduled jobs and logs

Progress note: You can take the Quick Test at the end for everyone; only logged-in users will have their progress saved.

Why this matters

In real teams, you will:

• Run daily ELT pipelines that must not fail silently.
• Backfill historical data after fixing a bug or adding a new model.
• Coordinate dependencies across extracts, quality checks, dbt transforms, and dashboard refreshes.
• Design tasks that are safe to retry and easy to monitor.

Concept explained simply

A DAG (Directed Acyclic Graph) is a set of tasks connected by one-way dependencies. No cycles means you can always find a top-to-bottom order to run tasks.

Mental model

Imagine a daily checklist where some items depend on others. You cannot publish metrics before transformations, and you cannot transform before raw data arrives. Draw boxes (tasks) and arrows (dependencies) from prerequisites to dependents.

Core terms (open to learn)

• Task/Node: A single unit of work (e.g., load table, run dbt model).
• Edge/Dependency: Upstream must succeed before downstream runs.
• Upstream/Downstream: Inputs vs. outputs in the dependency chain.
• Idempotency: Safe to run multiple times without bad side effects.
• Determinism: Same inputs and code yield same outputs.
• Atomicity: Each task does one coherent thing; success or failure is clear.
• Schedule/Trigger: When to start a run (cron, event, manual).
• SLA/Timeout: Expectations and limits on run duration.
• Inter-task communication: Passing small signals or references between tasks; prefer external storage for large data.
• Task groups/Mapping: Group or fan-out tasks for parallel execution.

Task design principles

• Make tasks small and focused (atomic), but not so tiny that orchestration overhead dominates.
• Prefer idempotent operations (e.g., upserts, overwrite partitions) to allow safe retries.
• Be explicit about inputs/outputs: tables, partitions, files, or markers.
• Build clear dependencies via data contracts, not timing guesses.
• Define retries with jitter and timeouts appropriate to the system you call.
• Design for backfills: parameterize by date/partition and avoid hard-coded "today" logic.
• Validate with data quality checks before publishing to downstream consumers.
• Control concurrency and rate limits when calling external APIs.

Worked examples

Example 1 — Daily ELT with transforms

Goal: Load yesterday's orders, transform, publish marts.

• Tasks: extract_orders → load_raw_orders → transform_dbt → quality_checks → publish_marts
• Partition: execution_date (e.g., 2025-01-05) parameter passed to each task.
• Idempotency: load overwrites or upserts the date partition; transforms are pure queries.
• Retries: extract has 3 retries with backoff; transforms 1 retry; publish 0–1 retry.
• Timeouts: extract 10m, transforms 30m, publish 5m.

Example 2 — File arrival sensor

Goal: Wait for a daily file, then load and transform.

• Tasks: wait_for_file → load_to_stage → transform_dbt → validate_row_counts
• Sensor: checks storage for file with pattern prefix/YYYY-MM-DD.csv
• Failure mode: Sensor timeout if file never arrives. Mitigation: set max wait and alert.
• Idempotency: load overwrites staging partition; transforms are deterministic.

Example 3 — Backfill a month with parallelism

Goal: Re-run last month's partitions safely.

• Tasks: fan-out per date: extract[d] → load[d] → transform[d]
• Concurrency: limit to 5 dates at a time to respect API rate limits.
• Idempotency/backfill: tasks overwrite the date partition; no duplicate rows.
• Quality: per-date validation before marking success.

Step-by-step: design a small DAG

1. Define the outcome: e.g., "Publish clean daily revenue table for date D".
2. List data inputs/outputs: raw.orders[D] → staging.orders[D] → mart.revenue_by_day[D].
3. Cut into atomic tasks: extract, load, transform, validate, publish.
4. Wire dependencies: extract → load → transform → validate → publish.
5. Set parameters: use D from the scheduled run; avoid now() inside SQL.
6. Add reliability: retries, timeouts, alerts, SLAs.
7. Plan backfills: allow D to be a range; cap concurrency.

Common mistakes and self-check

• Mistake: One giant task does everything. Fix: split into extract/load/transform/validate/publish.
• Mistake: Non-idempotent loads that append duplicates. Fix: overwrite or upsert partitions.
• Mistake: Using timestamps in code without parameters. Fix: pass execution date to queries.
• Mistake: Guessing timing instead of dependencies. Fix: wait for upstream markers or checks.
• Mistake: No limits on parallel backfills. Fix: set concurrency pools and rate limits.

Self-check: Can you safely rerun yesterday and a full backfill without manual cleanup? If yes, design is likely robust.

Exercises

Complete the exercises below. The Quick Test is available to everyone; only logged-in users will have results saved.

Exercise 1 — Design a daily sales ELT DAG

Design a DAG to ingest daily sales from an API, transform with dbt-like models, run data quality checks, and publish a dashboard table.

• Include: task list, dependencies, retries/timeouts, partitioning by date, and idempotency plan.
• Acceptance checklist:

• Tasks are atomic and clearly named
• Dependencies prevent transforms before data arrival
• Idempotent loads and transforms
• Retries/timeouts declared
• Quality checks before publish

Exercise 2 — Refactor a monolithic task

You inherited a single task "process_all_things" that pulls files, loads them, transforms, and emails results. Refactor into a maintainable DAG.

• Deliver: proposed tasks, inputs/outputs per task, dependencies, and concurrency limits.
• Acceptance checklist:

• No task mixes IO, transforms, and notifications together
• Clear upstream/downstream mapping
• Safe to retry any task without duplicates
• Email/notification only after validations pass

Mini challenge

Design for failure: Your transform sometimes times out due to warehouse load. Update your plan to add a fallback (e.g., increase timeout, reduce parallelism, or split by partition), while keeping idempotency and SLAs intact.

Hint

• Fan-out large transforms by partition and cap concurrency.
• Use smaller batch windows during peak hours.

Practical projects

• Build a daily product analytics pipeline: extract app events, load to warehouse, transform sessions, validate metrics, and publish a weekly KPI table.
• Implement a file-driven ingestion flow that waits for partner CSVs, loads them idempotently, and backfills a month of history.
• Create a backfill tool: parameterize your DAG by date range with safe concurrency and robust logging.

Learning path

• Start: DAG foundations and task design (this lesson).
• Next: Scheduling patterns, sensors/events, and SLAs.
• Then: Data quality automation and alerting strategies.
• Finally: Backfills, reprocessing, and cost-aware orchestration.

Next steps

• Refine your current pipelines using the checklists above.
• Practice a one-week backfill in a sandbox and verify no duplicates.
• Take the Quick Test below to lock in concepts.