GitOps Concepts

Why this matters

Worked examples

Example 1 — Kustomize overlays per environment

Goal: one base manifest and environment-specific differences.

repo/
  apps/
    payments/
      base/
        deployment.yaml
        service.yaml
        kustomization.yaml
      overlays/
        dev/
          kustomization.yaml
          patch-replicas.yaml
        prod/
          kustomization.yaml
          patch-resources.yaml

# base/kustomization.yaml
resources:
  - deployment.yaml
  - service.yaml

# overlays/dev/kustomization.yaml
resources:
  - ../../base
patches:
  - path: patch-replicas.yaml

# overlays/dev/patch-replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
spec:
  replicas: 1

Prod overlay sets higher replicas and resource limits. The controller syncs the chosen overlay directory for each environment.

Example 2 — Image tag updates via PR

Goal: CI builds an immutable image, then a bot updates the env repo to deploy it.

1. CI builds image registry.example.com/payments:1.7.0 and signs it.
2. CI opens a PR to env repo changing only the image tag in the dev overlay.
3. On merge, the controller pulls the new commit and deploys.

# overlays/dev/kustomization.yaml
images:
  - name: registry.example.com/payments
    newTag: 1.7.0

Example 3 — Fast rollback with Git revert

1. Incident detected after deploy 1.7.0.
2. Create a revert commit restoring tag 1.6.3 in the env repo.
3. Merge the revert; controller reconciles back to 1.6.3.

No cluster access needed for rollback; Git history provides a clear audit trail.

Design a minimal GitOps flow

1. Build: CI builds and scans an image; push with an immutable tag.
2. Propose: open a PR to env repo updating the image tag for a target env.
3. Validate: run lint, policy checks, and deployment dry-runs on PR.
4. Approve: reviewers sign off; required checks pass.
5. Sync: merge PR; controller reconciles cluster to the new desired state.
6. Observe: verify health and metrics; roll forward or revert via Git if needed.

Common mistakes and self-check

• Using mutable tags like latest — makes rollbacks and audits unreliable.
• Bypassing Git with manual kubectl edits — creates drift and surprises.
• Mixing app source and env configs without clear boundaries — noisy history.
• Storing plaintext secrets — always encrypt or externalize.
• Huge PRs that change many services at once — hard to review and rollback.
• No health checks — controller may mark sync as healthy while app is failing.
• Ignoring too many fields in sync — drift hides real issues.

Self-check

• Can you roll back by reverting a single commit?
• Can you answer who approved the last production change?
• Are images and charts pinned to immutable versions?
• Can a new cluster get to the same state by pointing at the same repo and path?

Exercises

Do these hands-on tasks to cement the concepts. Mirror of the exercises list below.

Checklist

• Env repo separated from app source, or clearly partitioned.
• Overlays/values pin immutable versions.
• Promotion uses PRs, not direct pushes to prod.
• Rollbacks are a single revert commit.

Practical projects

• Single service GitOps: Deploy a stateless API to dev and prod using overlays; acceptance: PR to bump image tag auto-deploys and is observable.
• Multi-service rollout: Three services with independent versions; acceptance: promote only one service to prod without touching others.
• Policy gate: Add a check that rejects PRs using latest or missing resource limits; acceptance: unsafe PRs are blocked until fixed.

Who this is for

• Platform Engineers building secure, scalable delivery platforms.
• DevOps/Infra engineers adopting declarative operations.
• Backend engineers who own deployment reliability.

Prerequisites

• Comfort with Git (branches, PRs, tags, revert).
• Basic Kubernetes (deployments, services, namespaces).
• YAML and either Kustomize or Helm fundamentals.

Learning path

1. Understand GitOps principles and pull vs. push deployment models.
2. Create a simple env repo with overlays or values files.
3. Automate image tag bumps via PRs from CI.
4. Add policy checks and required approvals.
5. Practice rollbacks using Git revert.

Next steps

• Introduce commit signing for production env repo.
• Add runtime policy checks and deployment health gates.
• Expand to multi-cluster setups with separate paths per cluster.

Mini challenge

Your CI just built image catalog:2.3.1. Draft the minimal Git diff to promote it to staging only, ensure production remains at 2.3.0, and write one sentence explaining how you would roll back if a regression is found.

Quick Test

Take the quick test to check your understanding. Available to everyone; only logged-in users get saved progress.