Deployments And Rollbacks

Why this matters

Mental model

Think of deployments like changing the tires on a moving car using lanes:

• Rolling: You change one wheel at a time while the car keeps moving.
• Blue-Green: You prepare a second car in a parallel lane and switch lanes instantly.
• Canary: You test the new lane with a few cars first before sending everyone.
• Feature flags: You deploy the code but keep new features behind switches.

Common deployment strategies

• Recreate: Stop old version, start new. Simple but downtime.
• Rolling: Update a few instances at a time. Minimal disruption.
• Blue-Green: Two identical environments (blue and green). Switch traffic; instant rollback by switching back.
• Canary: Send a small percent of traffic to new version. If healthy, increase; else roll back.
• Feature flags: Deploy dark, toggle on gradually.

Safety checks before and after deploy

• Health probes: readinessProbe prevents traffic to unready pods; livenessProbe restarts stuck pods.
• Automated smoke tests after deploy.
• Metrics gates: error rate, latency, CPU/memory, saturation.
• Database strategy: backward-compatible migrations (expand-then-contract) and ability to roll back or roll forward safely.
• Artifact immutability: each build is versioned and reproducible.

Worked examples

Example 1: Blue-Green with a simple switch

1. Provision two identical stacks: Blue (current) and Green (new).
2. Deploy version 2.0.0 to Green.
3. Run smoke tests against Green (health endpoints, key API calls).
4. Switch traffic from Blue to Green in the load balancer.
5. Monitor error rate and latency for 10–15 minutes.
6. If issues arise, switch back to Blue instantly.

# Pseudocode for a traffic switch step
switch_traffic(target_env: "green")
verify(duration: "15m", metrics: ["5xx_rate", "p95_latency"]) 
if unhealthy: switch_traffic(target_env: "blue")

Example 2: Kubernetes rolling update and rollback

1. Apply Deployment with readiness and liveness probes.
2. Update container image to a new version.
3. Watch rollout and metrics; undo if needed.

kubectl apply -f deployment.yaml
kubectl set image deploy/myapp api=myrepo/myapp:2.0.0 --record
kubectl rollout status deploy/myapp --timeout=5m
# if alerts or errors spike
kubectl rollout undo deploy/myapp
kubectl rollout history deploy/myapp

containers:
- name: api
  image: myrepo/myapp:2.0.0
  readinessProbe:
    httpGet: { path: /healthz, port: 8080 }
    initialDelaySeconds: 5
    periodSeconds: 5
  livenessProbe:
    httpGet: { path: /livez, port: 8080 }
    initialDelaySeconds: 15
    periodSeconds: 10

Example 3: CI pipeline with canary gate

1. Build artifact and push image.
2. Deploy to staging; run tests.
3. Deploy to prod-canary (10% traffic) and run checks.
4. If healthy, ramp to 50% then 100%.
5. If unhealthy at any stage, roll back and alert.

jobs:
  deploy:
    steps:
      - name: Build and push
        run: |
          docker build -t myrepo/myapp:${GIT_SHA} .
          docker push myrepo/myapp:${GIT_SHA}
      - name: Deploy canary (10%)
        run: ./scripts/deploy_canary.sh myrepo/myapp:${GIT_SHA} 10
      - name: Smoke + metrics gate
        run: ./scripts/gate.sh --error-rate-threshold 1 --latency-p95-ms 300
      - name: Ramp to 50% then 100%
        run: ./scripts/ramp.sh 50 && ./scripts/ramp.sh 100
      - name: Rollback if gate fails
        if: failure()
        run: ./scripts/rollback.sh

Rollbacks that actually work

• Keep the previous version warm or quickly deployable.
• Use versioned DB schema with backward-compatible changes. Avoid destructive changes until old version is fully retired.
• Automate the undo path: one command or playbook with steps and owners.
• Verify post-rollback: health endpoints, dashboards, and user flows.

Exercises

Do these hands-on tasks. Then open the solution toggles to compare.

Common mistakes and self-check

• No readiness probe: new pods get traffic before they are ready. Self-check: Does traffic only hit pods after passing readiness?
• Destructive DB migrations with no fallback. Self-check: Can the previous app version run with the new schema?
• Unversioned artifacts: you can’t reproduce the last good build. Self-check: Can you redeploy the exact prior version by ID?
• Skipping canary/metrics: blind deploys. Self-check: What metric would force you to stop? Is it automated?
• Manual, unclear rollback steps. Self-check: Can a new on-call engineer execute the rollback in under 5 minutes?

Practical projects

• Convert a recreate deployment to rolling with probes and verify zero-downtime.
• Implement blue-green for a small service; script the traffic switch and rollback.
• Add feature flags to separate deploy from release for one endpoint.
• Create a one-page rollback playbook and run a tabletop drill.

Who this is for

• Backend Engineers shipping services to staging and production.
• DevOps-minded developers improving reliability and release speed.

Prerequisites

• Basic Git and CI familiarity.
• Containerization basics (Docker images, tags).
• Optional: Kubernetes basics (Deployments, Services, probes).

Learning path

1. Start with rolling deployments and health probes.
2. Add canary or blue-green with a simple traffic switch.
3. Introduce feature flags to decouple deploy and release.
4. Automate gates based on metrics and smoke tests.
5. Practice rollback and document the playbook.

Next steps

• Automate your current deploy with canary and a rollback script.
• Schedule a monthly rollback drill.
• Add DB expand-contract patterns to your contribution guide.

Mini challenge

You deployed 2.1.0 and error rate rose from 0.3% to 3% within 2 minutes. Outline, in 5 short steps, what you do in the next 10 minutes. Include whether you roll back or roll forward and why.

Quick Test

Take the quick test to check your understanding. Everyone can take it; logged-in users get saved progress.