Multi Environment Deployments

Why this matters

Data Platform Engineers ship and operate orchestration (Airflow, Dagster, Prefect, Argo) that moves business-critical data. Multi-environment deployments (dev → staging → prod) let you experiment safely, catch issues early, and meet compliance. In practice, you will:

• Develop and test DAGs/flows without risking production data.
• Promote versions through gates with checks, approvals, and rollbacks.
• Keep secrets and configs isolated per environment.
• Run safe backfills and schema changes.
• Monitor, alert, and debug per environment.

Concept explained simply

Think of each environment as a sandbox with the same shape but different safety levels. You build the same thing in each sandbox, test it, then move it to the next. Promotion is the door between sandboxes that only opens when checks pass.

Mental model

• Parity: Environments feel the same (same code, similar infra), just with tighter controls and scale as you move right.
• Immutability: You promote artifacts (images/packages), not ad-hoc changes.
• Config-as-code: Behavior changes by config/parameters, not code edits per env.
• Isolation: Separate namespaces, credentials, and data paths per env.

Core principles you will use

• Environment isolation: different clusters or namespaces (dev, staging, prod).
• Configuration management: a single codebase; per-env config files/variables.
• Secrets management: store in a vault/secret store; never in code.
• Versioned artifacts: container images or wheels pinned by version/commit.
• Controlled promotion: CI/CD gates, approvals, smoke tests.
• Observability per env: logging, metrics, alert routing per env.
• Data safety rails: read-only in non-prod, sample data, guard flags for destructive steps.

Worked examples

Example 1: Airflow with environment-aware connections

Goal: One DAG codebase; pick the correct database and schedule by environment.

repo/
  dags/
    sales_pipeline.py
  config/
    dev.yaml
    staging.yaml
    prod.yaml
  tests/

# sales_pipeline.py (snippet)
import os
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

ENV = os.getenv("DEPLOY_ENV", "dev")
CONN_ID = f"postgres_{ENV}"  # e.g., postgres_dev, postgres_prod

with DAG(
    dag_id=f"sales_pipeline_{ENV}",
    start_date=datetime(2024, 1, 1),
    schedule_interval=None if ENV == "dev" else "0 * * * *",
    catchup=False,
    tags=[ENV],
):
    def extract():
        # Connect via Airflow connection named by env
        # Defensive: read-only in dev/staging
        pass

    PythonOperator(task_id="extract", python_callable=extract)

Example 2: Argo Workflows with namespaces per environment

Goal: Same workflow; different namespace and schedule per environment.

# dev/cronworkflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: sales-pipeline
  namespace: data-dev
spec:
  schedule: "*/30 * * * *"  # fast feedback in dev
  workflowSpec:
    entrypoint: main
    templates:
      - name: main
        container:
          image: registry.example.com/sales-pipeline:1.2.3
          env:
            - name: DEPLOY_ENV
              value: dev
---
# prod/cronworkflow.yaml
metadata:
  name: sales-pipeline
  namespace: data-prod
spec:
  schedule: "0 * * * *"  # hourly in prod
  workflowSpec:
    templates:
      - name: main
        container:
          image: registry.example.com/sales-pipeline:1.2.3
          env:
            - name: DEPLOY_ENV
              value: prod

Example 3: Prefect with per-environment blocks

Goal: Choose storage/credentials per env using naming conventions.

import os
from prefect import flow, task
from prefect.filesystems import S3

ENV = os.getenv("DEPLOY_ENV", "dev")
BLOCK_NAME = f"s3-{ENV}"  # s3-dev, s3-staging, s3-prod

@task
def extract():
    s3 = S3.load(BLOCK_NAME)
    # Read from environment-specific bucket/prefix

@flow(name=f"sales_pipeline_{ENV}")
def run():
    extract()

if __name__ == "__main__":
    run()

Deployment strategies

Security and compliance essentials

• Secrets in a secret manager; mount by reference, not inline.
• Least privilege per env (separate service accounts, roles, and network rules).
• PII: use masked or synthetic data in non-prod when possible.
• Audit: log promotions, approvers, and artifact versions.

Exercises

Work through these tasks, then compare with the solutions. These mirror the graded exercises below the article.

Exercise 1: Design a 3-environment Airflow deployment plan

Create a plan for dev, staging, and prod with isolated configs and safe promotions.

• Deliverables: repo structure, config matrix (schedule, connections, vars), promotion policy, guardrails, rollback steps.
• Constraints: one codebase; no secrets in code; environment chosen by variable/parameter.

repo/
  dags/
  plugins/
  tests/
  config/
    dev.yaml
    staging.yaml
    prod.yaml
  ci/
    deploy-dev.yml
    deploy-staging.yml
    deploy-prod.yml

Exercise 2: Write environment-aware schedule config

Write a small YAML config that sets schedules and feature flags per env, plus a short code stub that reads the config and prints the chosen schedule.

# config/env.yaml
dev:
  schedule: null
  allow_writes: false
  sample_ratio: 0.1
staging:
  schedule: "*/30 * * * *"
  allow_writes: false
  sample_ratio: 0.5
prod:
  schedule: "0 * * * *"
  allow_writes: true
  sample_ratio: 1.0

# read_config.py
import os, yaml
ENV = os.getenv("DEPLOY_ENV", "dev")
conf = yaml.safe_load(open("config/env.yaml"))
print(conf[ENV]["schedule"])  # None in dev, cron in prod

Checklist before you submit

• One codebase; behavior changes by config/variables.
• No secrets in code; names/paths are environment-specific.
• Clear promotion and rollback steps.
• Safety toggles for destructive operations.

Common mistakes and self-check

• Hard-coding prod resources in code. Self-check: search for prod hostnames/IDs in the repo; move to config/variables.
• Drift between environments. Self-check: diff configs; ensure the same artifact version runs in staging and prod.
• Shared credentials across envs. Self-check: verify separate secrets and service accounts.
• Skipping smoke tests. Self-check: list automated checks gating promotion; add at least one data and one infra check.
• No rollback plan. Self-check: identify previous artifact and the command to redeploy it.

Practical projects

• Build a demo pipeline that runs in dev/staging/prod with different schedules and read/write modes.
• Create a canary rollout for a heavy transform: run on 5% of partitions, compare metrics, then scale to 100%.
• Implement a blue/green Airflow webserver-scheduler switch with a smoke-test DAG.

Mini challenge

Your nightly pipeline must add a new column and backfill. Design a plan to ship it safely across environments and minimize risk in prod. Include: config changes, promotion gates, backfill scope, canary strategy, and rollback.

Who this is for

• Data Platform Engineers responsible for orchestration, CI/CD, and reliability.
• Data Engineers who deploy and operate pipelines end-to-end.

Prerequisites

• Basic orchestration platform knowledge (Airflow, Prefect, Dagster, or Argo).
• Comfort with Git, CI/CD, and containerized deployments.
• Familiarity with environment variables, YAML/JSON configs, and secrets stores.

Learning path

1. Set up three environments with clear isolation.
2. Refactor code to use config-as-code and env-driven connections.
3. Add CI/CD promotions with automated smoke tests.
4. Introduce safety toggles and read-only modes in non-prod.
5. Practice blue/green or canary releases.
6. Document rollback and runbooks.

Next steps

• Complete the exercises above and verify with the checklist.
• When ready, take the quick test below to confirm understanding.
• Note: The quick test is available to everyone; only logged-in users have their progress saved.

Quick Test

Ready? Take the Quick Test below.