Why this matters
Data platforms must deliver reliable pipelines. Failure handling standards make your orchestrator predictable during outages, schema changes, flaky networks, and bad data. You will use these standards to protect SLAs/SLOs, enable safe backfills, reduce pager noise, and speed up recovery.
- Keep business dashboards fresh even when one upstream system is down.
- Rerun failed loads safely without duplicating data.
- Alert the right owners with the right severity and context.
- Automate retries and fallbacks so humans focus on true incidents.
Concept explained simply
Failure handling standards are a small set of rules every job follows when something goes wrong. They define how to classify failures, when to retry, when to stop, how to alert, and how to safely rerun.
Mental model
- Detect: Know something failed (timeouts, heartbeats, checks).
- Decide: Classify (transient vs persistent; data vs code; upstream vs downstream).
- Act: Retry, skip, fallback, quarantine, or stop.
- Recover: Rerun idempotently and verify.
- Learn: Log context and create a clear runbook entry.
Standard components of failure handling
Failure classes
- Transient infra: network hiccups, 429/503, brief storage unavailability.
- Persistent config/code: bad credentials, schema breaking change, code bug.
- Data issues: nulls, type mismatch, unexpected volume, duplicates.
- External dependency: upstream API outage, third-party quota.
Policies you will standardize
- Retries: exponential backoff with jitter; cap max attempts and total delay.
- Timeouts and heartbeats: fail tasks that hang; prefer smaller timeouts with retries over very long timeouts.
- Idempotency: upserts/merge, checkpointing, deduping to support safe reruns.
- Alerts: severity by impact (SLO breach risk); include owner, run id, error, quick triage steps.
- Fallbacks: cached data, reduced mode, circuit breaker if dependency is unstable.
- Quarantine: dead-letter queue for bad records; continue pipeline if safe.
- Rerun/backfill rules: how to reprocess dates, partitions, or assets.
- Ownership: every task has an owner and criticality tag.
Severity mapping (suggested)
- Info: auto-recovered transient retry that succeeded.
- Warning: non-critical task failed; downstream skipped; no SLO risk.
- High: critical task failure with SLO risk; page owner during business hours.
- Critical: materialized data outage or repeated failures across runs; 24/7 page.
Worked examples
Example 1: Safe retries for a warehouse load
Scenario: Task loads a partition to the warehouse. Sometimes the warehouse returns 500 errors for 1–2 minutes.
- Policy: retry 6 times with exponential backoff (30s, 60s, 120s, 240s, 240s, 240s) and jitter +/-20%.
- Idempotency: use MERGE/UPSERT with a unique key on (date, record_id) and a load_id for traceability.
- Timeout: 10 minutes per attempt; fail fast if connection drops.
- Alerting: only alert if all retries fail; include partition and warehouse error code.
Minimal pseudo-config
{
"retries": 6,
"retry_strategy": "exponential_backoff_jitter",
"first_delay_seconds": 30,
"max_delay_seconds": 240,
"timeout_seconds": 600,
"idempotent": true,
"on_failure_alert": {
"severity": "High",
"include": ["task_owner", "run_id", "partition", "error_summary"]
}
}Example 2: Handling upstream API rate limits
Scenario: Task calls an API that returns 429 during bursts. Calls are billable.
- Policy: set concurrency limit 1–2 per worker; add client-side rate limiting.
- Retries: 5 attempts with backoff honoring Retry-After header; jitter to avoid herd.
- Fallback: serve cache if data is <= 2 hours old; mark as degraded if older.
- Circuit breaker: open if 50% of attempts fail over 5 minutes; switch to cached mode and alert.
Minimal pseudo-config
{
"concurrency_limit": 2,
"retry_on": [429, 503, "timeout"],
"respect_retry_after": true,
"max_attempts": 5,
"circuit_breaker": {"failure_rate_threshold": 0.5, "window_minutes": 5},
"fallback": {"use_cache_if_fresh_minutes": 120, "degraded_if_older": true},
"alert_on_circuit_open": true
}Example 3: Data quality guardrail with quarantine
Scenario: Transform step finds 3% malformed records (threshold 1%).
- Policy: move bad records to a dead-letter table with error reasons.
- Continue: proceed with good records; mark dataset as partial but usable.
- Alert: Warning severity with sample errors and counts; create ticket automatically if persists 3 runs.
- Backfill: provide a rerun command to reprocess quarantined records after fix.
Minimal pseudo-config
{
"dq_check": {"bad_rate_threshold": 0.01},
"on_exceed": {
"quarantine": "table:raw_bad_records",
"continue_with_good": true,
"alert": {"severity": "Warning", "include_counts": true},
"auto_ticket_after_runs": 3
},
"rerun_quarantined_command": "process_quarantine --since 7d"
}Patterns and decisions
- Fail-fast vs keep-going: Fail-fast on contract/schema violations. Keep-going on partial data if downstream can tolerate and is tagged non-critical.
- Retries vs longer timeouts: Prefer short timeouts with retries for flaky networks.
- Global vs task-level standards: Define org-wide defaults; allow overrides with justification.
- Backfills: Require idempotency and partition-scoped reruns; record run lineage.
- Recovery verification: Post-recovery data checks (row counts, freshness, aggregates).
Exercises
Exercise 1: Draft a failure-handling standard for a 3-step ETL DAG
DAG: extract → transform → load. Current state: no retries, no timeout, non-idempotent load, on-failure always alert critical.
Task: Write a concise policy (10–15 lines) that sets defaults for retries, backoff, timeouts, idempotency, alerting, and reruns/backfills.
- Deliverable: a short JSON-like block or bullet list of rules.
- Scope: partitioned daily runs.
Hints
- Use exponential backoff + jitter and cap total retry window.
- Make load idempotent (MERGE/UPSERT) and store a run_id.
- Alert only after exhausting retries; differentiate severity by task criticality.
Exercise 2: Design a plan for a rate-limited API task
Scenario: API often returns 429 for bursts; also returns occasional 500. You pay per call.
Task: Propose concurrency limits, retry counts/delays, cache/fallback, and circuit breaker rules. Limit cost and reduce pager noise.
- Deliverable: a short plan (bullets) with concrete numbers.
Hints
- Honor Retry-After headers when present.
- Use small concurrency and jittered backoff.
- Define when to switch to cache and how to alert.
Self-check checklist
- Retries include exponential backoff and jitter.
- Timeouts are set per attempt.
- Idempotent writes or deduplication are specified.
- Alerting severity distinguishes transient vs persistent failures.
- Backfill/rerun rules are clear and safe.
- Owners and criticality tags are defined.
Common mistakes and how to self-check
- Infinite retries or very long timeouts: set a maximum attempts and total cap.
- No jitter: synchronized retries cause thundering herd. Add randomization.
- Non-idempotent loads: duplicates after reruns. Use keys, MERGE, or de-dup staging.
- Alerting on every retry: pager fatigue. Alert only after final failure, keep logs for intermediate retries.
- Backfills that overwrite live data: enforce partition-scoped reruns and validation steps.
- Missing owners: incidents linger. Require task_owner metadata.
Quick self-audit
- Pick a critical DAG. Can you rerun yesterday only, safely, without duplicates?
- Open a failed run. Do logs show error, attempt number, and retry delay?
- Is there a runbook link and owner in the alert payload?
Practical projects
- Harden one production-like DAG: Add retries with jitter, timeouts, and idempotent load. Measure MTTD/MTTR before vs after.
- Implement quarantine: Route bad records to a table and create a simple rerun command to process them.
- Build a severity policy: Tag tasks with criticality and implement alert routing rules. Simulate failures to test.
Project acceptance criteria
- Reruns do not produce duplicates.
- Alerts fire only after final failure and include owner, run id, and next steps.
- Transient faults recover automatically without human intervention >= 90% of the time.
Who this is for
- Data Platform Engineers implementing orchestration standards.
- Data Engineers responsible for reliable pipelines.
- Analytics Engineers who own downstream models and need safe reruns.
Prerequisites
- Basic understanding of DAGs, tasks, and scheduling.
- Familiarity with your warehouse/streaming sink and how to implement idempotent writes.
- Comfort reading logs and interpreting HTTP/DB errors.
Learning path
- Define org-wide defaults (retries, timeouts, alerting).
- Implement idempotent write patterns for your sinks.
- Add DQ checks and quarantine flows.
- Test backfills and reruns on a sandbox dataset.
- Roll out to top 5 critical DAGs; measure reliability.
Next steps
- Document your standards in a shareable template.
- Automate linting to enforce required settings (retries, owners, timeouts).
- Schedule a game day: simulate outages and evaluate recovery.
Mini challenge
A nightly DAG fails at transform due to a schema change upstream (new non-nullable column). Describe in 6–8 lines how your standards classify, alert, mitigate, and recover, including backfill.
Quick Test
Anyone can take the quick test for free. If you log in, your progress and results will be saved to your learning profile.