CI CD For Data Pipelines

• Why this matters
• Concept explained simply
• Mental model
• Worked examples
• Checklist before you ship
• Exercises
• Common mistakes
• Practical projects
• Who this is for
• Prerequisites
• Learning path
• Next steps
• Mini challenge

This CI runs on every pull request:

name: ci
on: [pull_request]
jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Lint
        run: ruff check .
      - name: Unit tests
        run: pytest -q tests/unit
      - name: Validate dbt project
        run: |
          dbt deps
          dbt compile
      - name: Validate Airflow DAGs
        run: python -m pyflakes dags || true

What this catches: style errors, failing unit tests, SQL compile errors, and obvious DAG import errors before merging.

This CD builds a Docker image, deploys to dev, then stage with smoke tests, then promotes to prod after a canary:

name: cd
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t registry/pipeline:${GITHUB_SHA} .
      - name: Push image
        run: docker push registry/pipeline:${GITHUB_SHA}
  deploy-dev:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to dev
        run: ./infra/deploy.sh dev registry/pipeline:${GITHUB_SHA}
      - name: Dev smoke tests
        run: python checks/smoke.py --env dev
  deploy-stage:
    needs: deploy-dev
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to stage
        run: ./infra/deploy.sh stage registry/pipeline:${GITHUB_SHA}
      - name: Data quality gate
        run: python checks/data_quality.py --env stage --threshold 0.98
  deploy-prod:
    needs: deploy-stage
    runs-on: ubuntu-latest
    steps:
      - name: Canary to prod (10%)
        run: ./infra/deploy.sh prod registry/pipeline:${GITHUB_SHA} --scope canary
      - name: Observe metrics
        run: python checks/observe.py --env prod --minutes 15 --error-rate-threshold 0.01
      - name: Full rollout
        run: ./infra/deploy.sh prod registry/pipeline:${GITHUB_SHA} --scope full

Key idea: each step is conditional on the previous passing, with simple numeric thresholds for quality gates.

Scenario: adding a non-nullable column to a fact table.

• Step 1: Add column as nullable with default; write code to populate it; keep consumers reading old schema.
• Step 2: Backfill in batches; monitor null rate and row counts.
• Step 3: Switch consumers to new column behind a feature flag; keep dual-write temporarily.
• Step 4: After stability, enforce NOT NULL; remove old paths.
• Rollback plan: If metrics dip, revert consumers to old column, stop backfill, and remove feature flag.

name: ci
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - run: ruff check .
      - run: pytest -q tests/unit
      - run: |
          dbt deps
          dbt compile

1. Dev: deploy, run unit tests + dev smoke (sample run), verify logs.
2. Stage: deploy, backfill small subset, validate row counts and null rate < 2%.
3. Prod canary: enable on 10% partitions; monitor 15 minutes for failure rate < 1% and volume within ±5% baseline.
4. Full rollout: expand to 100% after metrics are stable.
5. Rollback: if thresholds breach, revert image/tag, disable feature flag, and restore previous DAG version.

Fix: Add at least row count and null rate checks for critical tables. Treat thresholds as gates, not warnings.

Fix: Use backward-compatible changes first (nullable + default), dual-write, then enforce constraints later.

Fix: Keep previous container tag and DAG version. Test rollback in stage before prod rollout.

Fix: Use environment variables injected by a secret manager. Never commit keys to the repo.