Secrets And Config Management

Who this is for

• Data Platform Engineers and DevOps engineers using Terraform, Helm, or Kubernetes.
• Anyone building data pipelines where credentials (DB, cloud, message brokers) must be safe, rotated, and auditable.

Prerequisites

• Basic Terraform or Helm/Kubernetes knowledge.
• Comfortable with YAML and environment variables.
• Familiarity with a cloud provider’s IAM (AWS, Azure, or GCP).

Why this matters

Real tasks you’ll face as a Data Platform Engineer:

• Provision data store credentials (e.g., Postgres, Redshift, BigQuery service accounts) without hardcoding them.
• Inject secrets into jobs (Airflow, Spark, dbt, Flink) safely from CI/CD.
• Rotate keys regularly without breaking pipelines.
• Keep Terraform state and Git repos free of sensitive data.
• Apply different configs across dev/stage/prod cleanly and repeatably.

Concept explained simply

Secrets are sensitive values (passwords, tokens, keys). Config is everything else (URLs, feature flags, timeouts). Store secrets in a secure system and inject them at runtime. Keep configs versioned and environment-specific.

Mental model

• Build-time: code and templates (no secrets).
• Deploy-time: references to secret names/paths and environment configs.
• Run-time: your platform pulls secrets from a vault/manager with least-privileged access.

Key principles

• Separate secrets from config.
• Prefer managed secret stores (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault).
• Encrypt at rest and in transit; enable audit logs.
• Use short-lived credentials where possible (federation/OIDC).
• Apply least privilege IAM to consumers of secrets.
• Never commit secrets to Git; use tooling to prevent this.
• Keep secrets out of Terraform state; read at runtime instead.
• Version and rotate secrets; plan for rollback.
• Layer configs per environment (base -> env overrides).
• Automate CI/CD secret injection; avoid sharing across pipelines.

Worked examples

# terraform.tf (excerpt)
resource "random_password" "db" {
  length  = 32
  special = true
}

resource "aws_secretsmanager_secret" "db_password" {
  name                    = "prod/db/password"
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "db_password_v" {
  secret_id     = aws_secretsmanager_secret.db_password.id
  secret_string = random_password.db.result
}

# IAM policy that allows reading this one secret
data "aws_iam_policy_document" "read_db_secret" {
  statement {
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [aws_secretsmanager_secret.db_password.arn]
  }
}

resource "aws_iam_policy" "read_db_secret" {
  name   = "read-db-secret"
  policy = data.aws_iam_policy_document.read_db_secret.json
}

# Attach to ECS task role (replace with your role)
resource "aws_iam_role_policy_attachment" "ecs_read_secret" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = aws_iam_policy.read_db_secret.arn
}

# config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: platform
data:
  MONGO_HOST: mongo.platform.svc.cluster.local
  MONGO_USER: reports_reader
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: platform
type: Opaque
stringData:
  MONGO_PASS: "CHANGE-ME-LOCALLY-ONLY"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reports-api
  namespace: platform
spec:
  replicas: 1
  selector:
    matchLabels: { app: reports-api }
  template:
    metadata:
      labels: { app: reports-api }
    spec:
      containers:
      - name: api
        image: example/reports:1.0.0
        env:
        - name: MONGO_HOST
          valueFrom: { configMapKeyRef: { name: app-config, key: MONGO_HOST } }
        - name: MONGO_USER
          valueFrom: { configMapKeyRef: { name: app-config, key: MONGO_USER } }
        - name: MONGO_PASS
          valueFrom: { secretKeyRef: { name: app-secrets, key: MONGO_PASS } }

# .github/workflows/deploy.yml (excerpt)
name: Deploy
on: [push]
jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-oidc-deploy
          aws-region: us-east-1
      - name: Read secret
        run: |
          aws secretsmanager get-secret-value \
            --secret-id prod/db/password \
            --query SecretString \
            --output text > db_password.txt
      - name: Deploy
        run: ./deploy.sh

Hands-on exercises

Do these locally or in a sandbox account. Mirror of the graded exercises below.

Exercise 1: Terraform + Secrets Manager

1. Create a Terraform snippet that generates a random password and stores it in a managed secret store. Do not print the password in outputs.
2. Grant read access to only one runtime identity (task role or service account).
3. Verify via plan that no plaintext password appears in state or outputs.

Exercise 2: K8s Config + Secret split

1. Write a ConfigMap with non-sensitive DB settings and a Secret with DB password.
2. Mount them as env vars in a Deployment.
3. Confirm the pod can read both env vars, but only the Secret contains the sensitive value.

Acceptance checklist

• No plaintext secrets in Git files.
• Terraform outputs do not print secret values.
• Least-privileged identity can read the secret; others cannot.
• Configs are per-environment and do not contain sensitive data.

Common mistakes and how to self-check

• Putting secrets in Terraform variables and outputs. Self-check: Search for variable names like password/token in outputs and .tfvars.
• Committing secrets to Git. Self-check: Scan history; rotate any exposed credentials immediately.
• Over-broad IAM permissions (e.g., Read all secrets). Self-check: Policies should reference exact ARNs/paths.
• Storing secrets in ConfigMaps. Self-check: Ensure Secret kind for sensitive keys; ConfigMaps only for non-sensitive.
• Long-lived CI credentials. Self-check: Prefer OIDC/federation and remove static keys.
• Forgetting rotation. Self-check: Add rotation schedules and test roll-forward/roll-back.

Practical projects

• Build a small data ingestion service that reads from a secret-managed API key and writes to a database. Rotate the API key and redeploy without downtime.
• Create a Helm release with separate values files for dev/stage/prod. Secrets are injected at deploy time; configs differ per env.
• Set up CI to assume a cloud role via OIDC and deploy without storing static cloud keys.

Learning path

1. Basics of secrets vs config and environment layering.
2. Managed secret stores and IAM policies.
3. Terraform patterns: avoid secrets in state, sensitive variables.
4. Kubernetes: ConfigMap, Secret, and runtime injection patterns.
5. CI/CD: OIDC, secret injection, and audit.
6. Rotation strategies and incident response for leaked credentials.

Next steps

• Adopt a managed secret store across all services.
• Introduce OIDC in CI to remove static cloud keys.
• Add pre-commit checks to prevent accidental secret commits.
• Document rotation runbooks and practice them.

Mini challenge

Your team needs to rotate the warehouse writer password with zero downtime. Describe, in 5 steps, how you will: create the new secret version, update consumers safely, verify, cut over, and revoke the old version. Include roll-back criteria.

Take the quick test

The quick test is available to everyone; only logged-in users get saved progress.