Environment Separation Dev Stage Prod

Why this matters

API Engineers ship changes that affect users, partners, and money flows. Clear separation between Development, Staging, and Production protects users while letting teams move fast.

• Dev: rapid iteration, breaking changes are acceptable.
• Stage: realistic pre-production checks with production-like settings.
• Prod: user-facing, strict controls, high reliability.

Your day-to-day tasks that depend on good separation:

• Testing migrations and rollbacks safely.
• Validating feature flags and configs before release.
• Running performance and integration tests without risking users.
• Applying security patches and hotfixes with minimal downtime.

Concept explained simply

Environment separation means you run the same application in isolated spaces with different risk levels and controls. Code and artifacts are identical or near-identical; only configuration and data vary. Changes promote upward: Dev → Stage → Prod.

Mental model

Think of a three-gate runway:

1. Gate 1 (Dev): Anyone on the team can try things. Safety nets are wide; data is fake.
2. Gate 2 (Stage): You must pass checks. Data and configuration mimic Prod. Limited, realistic traffic and integrations.
3. Gate 3 (Prod): Only approved, verified builds enter. Tight access, strict monitoring, immediate rollback plans.

Core principles to follow

• Isolation: Networks, credentials, data stores, and queues are not shared across environments.
• Parity: Keep Stage technically close to Prod (same image, same infra modules, similar sizes when feasible). Differences are intentional and documented.
• Immutability: Promote the same build artifact through Dev → Stage → Prod. Do not rebuild per environment.
• Config as data: Use environment-specific configuration (env vars, config files, secret stores). No hardcoded secrets.
• Data management: Dev uses synthetic data; Stage uses sanitized or synthetically generated realistic data; Prod uses real data only.
• Access control: Least privilege by environment. Manual approvals and protected branches for Prod.
• Change promotion: Changes must pass automated tests and manual reviews before moving up.
• Observability: Separate logs/metrics/traces per environment. Different alert thresholds; same signals.
• Safety nets: Feature flags, canary or blue/green releases, automated smoke tests, and fast rollback.

Worked examples

Example 1: Configuration and secrets by environment

Goal: Same Docker image, different configuration per environment.

# app reads from env vars; no secrets in code
APP_PORT=8080
APP_MODE=production
DB_URL=postgres://user:pass@host:5432/db

# Kubernetes (snippet): different Secret/ConfigMap per env
apiVersion: v1
kind: Secret
metadata:
  name: api-secrets
  namespace: prod
stringData:
  DB_URL: postgres://prod_user:...@prod-db:5432/api
---
apiVersion: v1
kind: Secret
metadata:
  name: api-secrets
  namespace: stage
stringData:
  DB_URL: postgres://stage_user:...@stage-db:5432/api

Why it works: Single image, environment-specific values injected at deploy time.

Example 2: Safe database migrations

Goal: Test schema change safely across environments.

1. Create a forward-compatible migration (e.g., add nullable column).
2. Apply in Dev with synthetic data; run app tests.
3. Apply in Stage with sanitized data; run contract/perf tests.
4. Deploy code that uses new column; verify in Stage.
5. Promote to Prod; apply migration; enable feature gradually.

# Flyway/Liquibase style
-- V20240101__add_index_status.sql
ALTER TABLE orders ADD COLUMN status TEXT;
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_orders_status ON orders(status);

Roll back plan: Application ignores extra column; if needed, revert code first, then clean up later.

Example 3: Promotion gates in CI/CD

Goal: Prevent accidental Prod deployment.

# GitHub Actions-like pseudo YAML
jobs:
  build_and_test:
    runs-on: ubuntu-latest
    steps:
      - run: make test
    outputs:
      artifact: api-image:sha-abc

  deploy_dev:
    needs: build_and_test
    if: github.ref == 'refs/heads/main'
    steps:
      - run: deploy --env dev --image ${{ needs.build_and_test.outputs.artifact }}

  deploy_stage:
    needs: deploy_dev
    environment:
      name: stage
      url: https://stage.example
    steps:
      - run: smoke_tests --env stage
      - run: deploy --env stage --image ${{ needs.build_and_test.outputs.artifact }}

  deploy_prod:
    needs: deploy_stage
    environment:
      name: production
      url: https://api.example
    # Require manual approval via environment protection
    steps:
      - run: deploy --env prod --image ${{ needs.build_and_test.outputs.artifact }}
      - run: post_deploy_checks --env prod

Key: Single artifact, environment protections, smoke tests before and after deploy.

Step-by-step: Set up Dev, Stage, Prod

1. Create separate cloud projects/namespaces and VPCs per environment.
2. Provision separate databases, caches, queues, buckets per environment.
3. Store configuration and secrets per environment in a managed secret store.
4. Build once; tag images with immutable digests.
5. Dev deploys automatically on merge; Stage requires passing integration tests; Prod requires manual approval.
6. Use the same IaC modules (Terraform/CloudFormation), with env-specific variables.
7. Implement feature flags; default off in Prod.
8. Set up separate logging/metrics with clear labels (env=dev|stage|prod).
9. Populate Dev with synthetic data; Stage with sanitized realistic data; never copy Prod raw data.
10. Define promotion checklist (below) and enforce in CI/CD.
11. Add canary/blue-green strategy for Prod.
12. Document rollback per change (schema, app, infra).

ENV   LOG_LEVEL  DB_SIZE   REPLICAS  FEATURE_X
DEV   debug      small     1         on (internal)
STAGE info       medium    2         on (limited)
PROD  warn       prod-tier 4+        off by default, gradual rollout

Exercises

Complete these tasks, then open the solutions to compare. You can take the quick test after the exercises. Note: Test is available to everyone; only logged-in users have progress saved.

Exercise 1: Design a three-environment plan (ID: ex1)

Describe isolation, parity, data strategy, and approvals for Dev, Stage, Prod.

• Isolation: networks, secrets, data stores.
• Parity: image, IaC modules, configs.
• Data: synthetic vs sanitized vs real.
• Approvals: who can deploy where; gates.

Section A: Isolation
Section B: Parity
Section C: Data
Section D: Approvals & Observability

Exercise 2: CI/CD gate rules (ID: ex2)

Write a minimal pipeline snippet that:

• Builds once.
• Auto-deploys to Dev on main merges.
• Deploys to Stage after smoke tests.
• Requires manual approval for Prod.

Pre-promotion checklist

• [ ] Same artifact digest across environments.
• [ ] Config reviewed; no hardcoded secrets.
• [ ] DB migrations tested Dev → Stage; rollback plan documented.
• [ ] Feature flags default safe in Prod.
• [ ] Observability dashboards and alerts updated.
• [ ] Runbook updated with owner and rollback steps.
• [ ] Data privacy validated; no Prod data outside Prod.

Common mistakes and self-check

• Mistake: Rebuilding per environment. Self-check: Does the image digest change across environments? It should not.
• Mistake: Sharing databases. Self-check: Can a Stage incident affect Prod data? If yes, isolation is broken.
• Mistake: Stage not Prod-like. Self-check: Are resource classes, runtime versions, and config keys aligned?
• Mistake: Secrets in code or repo. Self-check: Scan repo and build logs; secrets should only be in secret stores.
• Mistake: Skipping rollback planning. Self-check: Can you revert to previous artifact and schema safely?
• Mistake: Over-logging in Prod. Self-check: Verify PII redaction and log retention per policy.

Practical projects

1. Project: Split a monolithic environment into Dev/Stage/Prod
   • Create three namespaces and separate databases.
   • Move configs to a secret store.
   • Add CI gates and promotion rules.
2. Project: Add feature flags with safe rollout
   • Wrap a new endpoint behind a flag.
   • Enable in Dev and Stage; run smoke and load tests.
   • Enable canary in Prod for 5% of traffic; monitor; widen.
3. Project: Observability by environment
   • Add env labels to logs/metrics/traces.
   • Create dashboards per environment with aligned panels.
   • Set Prod alerts stricter than Stage; none paging in Dev.

Who this is for

• API Engineers building and shipping backend services.
• Platform/DevOps Engineers designing CI/CD and infra.
• Team leads aiming for safer, faster releases.

Prerequisites

• Basic Git and CI/CD knowledge.
• Familiarity with containers or deployment platforms.
• Fundamentals of databases and migrations.

Learning path

1. Understand isolation and parity patterns.
2. Set up secrets and configuration per environment.
3. Implement CI/CD gates and approvals.
4. Add data policies and migration workflows.
5. Introduce feature flags and safe rollout strategies.
6. Strengthen observability and rollback plans.

Mini challenge

Take a recent production incident (real or hypothetical). Write how proper environment separation would have reduced risk or impact. Identify which checklist items would have prevented it.

Next steps

• Finish Exercises 1–2 and compare with solutions.
• Run through the Pre-promotion checklist on your current project.
• Take the Quick Test to check your understanding.

Progress note: The test is available to everyone. If you are logged in, your progress and scores will be saved.