Retries And Alerts

Why this matters

In ETL and orchestration, things fail: APIs rate-limit, databases lock, networks hiccup. Smart retries and clear alerts keep data fresh, SLAs met, and on-call humans sane.

• Meet SLAs: Recover automatically from transient errors without waking someone up at 2 AM.
• Reduce noise: Alert only when human action is needed; suppress flapping alerts.
• Protect downstream: Stop bad data from propagating by failing fast on permanent errors.

Concept explained simply

Think of retries as a thermostat for failures. When heat (errors) spikes briefly, the thermostat (retries with backoff) stabilizes the temperature (pipeline health) without human intervention. If the spike persists, raise an alert so a human can fix the root cause.

Key pieces

• Transient vs permanent failure: Transient (e.g., 429, timeouts) often succeed if retried; permanent (e.g., 400 bad request, schema mismatch) need a fix, not retries.
• Backoff and jitter: Wait longer between retries (backoff) and add randomness (jitter) to avoid thundering herd.
• Idempotency: Safe to run multiple times without breaking data. Design tasks to be idempotent if they may retry.
• Timeouts and Circuit breakers: Cap how long a single try runs; stop retrying after a sensible limit.
• Alerts and escalation: Notify the right person with context; escalate only if unresolved.

Design defaults that work 80% of the time

1. Classify errors: Retry on timeouts, connection errors, 429, 5xx. Do not retry on 4xx (except 429).
2. Retry policy: 5 attempts, exponential backoff starting at 2 minutes, max delay 15 minutes, add ±20% jitter.
3. Timeouts: Per-try timeout 10–20 minutes; total task budget within SLA.
4. Alerting: Alert only on final failure or after X consecutive runs fail. Include run ID, task name, impact, last error, and next steps.
5. Escalation: Pager only when data freshness or revenue is at risk; otherwise send channel/email and create a ticket.

Worked examples

# Pseudocode-like parameters (conceptual)
default_args = {
  'retries': 5,
  'retry_exponential_backoff': True,
  'retry_delay': '2m', # starting delay
  'max_retry_delay': '15m',
  'execution_timeout': '20m',
  'email_on_failure': False  # avoid noisy per-try alerts
}

# Task-level handling
# - Inspect HTTP status: if 429/5xx -> raise RetryableError
# - If 4xx (not 429) -> raise NonRetryableError to fail fast

# Alerting (conceptual):
# On final failure: send message with DAG id, run id, task id, impact, and last 50 lines of logs.

# Conceptual Prefect-style config
@task(retries=5, retry_delay_seconds=120, retry_jitter_factor=0.2, timeout_seconds=1200)
def load_to_warehouse():
    # raise RetryError for connection/timeouts
    # raise Fail for schema mismatch or validation errors

@flow
def pipeline():
    load_to_warehouse()
    # On final failure: post to on-call channel and open ticket.

#!/usr/bin/env bash
set -euo pipefail

attempts=5
base_delay=120   # seconds
max_delay=900

for i in $(seq 1 $attempts); do
  if ./run_job.sh; then
    echo "Success"; exit 0
  fi
  if [[ $i -eq $attempts ]]; then
    echo "Final failure: alerting" >&2
    # send alert with last logs snippet and run metadata
    exit 1
  fi
  # exponential backoff with jitter
  delay=$(( base_delay * 2**(i-1) ))
  if (( delay > max_delay )); then delay=$max_delay; fi
  jitter=$(( RANDOM % (delay / 5 + 1) ))  # ~20% jitter
  sleep $(( delay + jitter ))
done

Checklist before you ship

• Retries only on transient errors; permanent errors fail fast.
• Exponential backoff with jitter configured.
• Per-try timeout and max total time respect the SLA.
• Task is idempotent or has compensating actions.
• Alerts include owner, impact, last error, and next steps.
• Escalation path and quiet-hours policy agreed with stakeholders.

Who this is for

ETL Developers, Data Engineers, and Analytics Engineers who schedule and operate pipelines and want reliable, low-noise operations.

Prerequisites

• Basic knowledge of your orchestrator (Airflow, Prefect, Dagster, dbt Cloud, or cron).
• Understanding of HTTP status codes, database errors, and SLAs.
• Ability to read logs and identify error patterns.

Learning path

1. Identify transient vs permanent failures in your pipelines.
2. Add backoff, jitter, and timeouts to retryable tasks.
3. Make tasks idempotent; add guards to writes.
4. Define alert content and escalation thresholds.
5. Run a failure drill and tune noise down.

Common mistakes and how to self-check

• Retrying non-retryable errors: If an error is 4xx (not 429) or a schema mismatch, retries waste time. Self-check: sample last failures; if repeats unchanged, stop retrying.
• No jitter: Simultaneous retries can overwhelm services. Self-check: Did many tasks retry at the exact same second?
• No timeouts: A hung task blocks the queue. Self-check: Any task running longer than historical P95 without timing out?
• Alert fatigue: Per-try alerts spam channels. Self-check: Alert volume vs incidents—ratio should trend down over time.
• Lack of context: Alerts without links or run IDs prolong MTTR. Self-check: Can a new on-call person act within 5 minutes using the alert alone?

Practical projects

• Retrofit one flaky pipeline with a robust retry policy, then compare success rates week-over-week.
• Build an alert template that includes run metadata, impact, and runbook steps; adopt it across two pipelines.
• Implement idempotent loads using MERGE/UPSERT; verify duplicates do not occur after forced retries.

Exercises

Mini challenge

Pick one pipeline with more than 3 failures last month. Classify top 2 failure types as transient or permanent, implement one policy improvement (retry tuning or fast-fail), and measure the next two weeks: success rate, average duration, and alert count.

Quick Test

Available to everyone. If you log in, we’ll save your progress and results.

Next steps

• Apply the 80/20 defaults to one real task this week.
• Schedule a failure drill with your team: force a timeout and review alert clarity.
• Document your retry/alert standards and reuse them across pipelines.