Secrets And Config Management

Why this matters

As a Platform Engineer, you will ship infrastructure that applications trust. Config mistakes leak credentials; secret mishandling halts deployments or causes breaches. You will:

• Provision and rotate database/API keys safely.
• Inject runtime config to apps across dev/staging/prod without drift.
• Keep secrets out of Git, Terraform plans, logs, and CI artifacts.
• Harden access (RBAC, audit) and automate rotation to reduce human touch.

Who this is for

• Platform/Infra engineers using Terraform, Kubernetes, or CI/CD pipelines.
• Backend engineers maintaining service configs and credentials.
• Team leads setting standards for secret handling and environment management.

Prerequisites

• Basic understanding of Infrastructure as Code concepts.
• Familiarity with at least one cloud (any provider) or Kubernetes.
• Comfort with CI/CD basics (build, deploy, environment variables).

Concept explained simply

Config is information your service needs to run (e.g., feature flags, timeouts, region). Secrets are confidential values (e.g., API keys, database passwords, signing keys) that could cause damage if exposed.

Good practice: keep config and secrets external to code; inject them at runtime from controlled stores; never hardcode; encrypt at rest and in transit; tightly limit who/what can read them.

Core principles (quick checklist)

• Least privilege: grant read-only access to exactly the secrets a workload needs.
• No plaintext in repos: never commit secrets; avoid disguised base64 (it is not encryption).
• Short-lived over long-lived: prefer tokens/credentials that expire automatically.
• Rotation by design: have an automated, tested rotation path for every secret.
• Audit everything: who/what read a secret, when, and from where.
• Keep secrets out of state/logs: protect Terraform state; mask CI logs; mark sensitive values.
• Environment layering: base defaults → environment overrides → runtime injects.
• Separation of duties: IaC provisions stores and policies; pipelines fetch values at runtime.

Worked examples

Example 1: Provision policy-driven access; fetch at runtime

Goal: The app gets a DB password without Terraform ever holding the plaintext.

1. Manually or via a secure pipeline, create/store the DB password in a secret manager (outside Terraform).
2. Use IaC to create: (a) the secret container (if needed), (b) an IAM/RBAC policy allowing the app identity to read the secret, (c) attach that policy to the app's runtime identity (e.g., service account/role).
3. At runtime, the app authenticates with its identity and reads the secret directly from the manager.

# Pseudocode-like Terraform (cloud-agnostic style)
resource "secret_store_secret" "db_password" {
  name = "prod/app/db_password"
  # Value populated outside Terraform by a secure pipeline/process
}

resource "iam_policy" "read_db_password" {
  name   = "app-read-db-secret"
  effect = "allow"
  actions = ["secrets.read"]
  resources = [secret_store_secret.db_password.arn]
}

resource "iam_role" "app_runtime" {
  name = "app-runtime-role"
}

resource "iam_role_policy_attachment" "app_secret_access" {
  role       = iam_role.app_runtime.name
  policy_arn = iam_policy.read_db_password.arn
}

Notes:

• Terraform never sees the plaintext value; it only manages access and identifiers.
• App reads the secret at runtime via its role; audit logs record every read.

Example 2: Encrypted config file pattern (commit encrypted, decrypt in CI)

Goal: Share non-secret config in plain YAML, but protect a few sensitive fields.

1. Keep a config file where secret fields are encrypted client-side with a team key.
2. Commit the encrypted file to Git; never the plaintext.
3. In CI, an identity-based mechanism (e.g., OIDC to KMS) decrypts just-in-time and injects values to the deployment step.

# config.enc.yaml (illustrative)
app:
  name: orders
  timeout_ms: 3000
  db:
    # ciphertext placeholder; decrypted in CI
    password: ENC[AES256_GCM,data:...,iv:...,tag:...]

Benefits: versioned config, code review, no plaintext in repo; access to decrypt is auditable and revocable.

Example 3: CI without stored long-lived secrets (OIDC flow)

Goal: Pipeline needs to read secrets and deploy, but we avoid static CI credentials.

1. Configure CI to request a short-lived token from the cloud using OIDC.
2. The cloud issues a scoped, time-bound credential to CI based on trust policy.
3. CI uses this credential to read needed secrets and perform deployments; nothing long-lived is stored.

# Conceptual trust policy (pseudocode)
trust_policy {
  subject: "repo:org/service:ref:refs/heads/main"
  permissions: ["secrets.read","deploy.write"]
  ttl: "15m"
}

Result: No static keys in CI; blast radius is minimized; access is observable.

Runbooks

Exercises

Do these tasks, then compare with the suggested solutions (hidden below each exercise in the Exercises section at the bottom of this page).

Exercise 1: Design a secrets and config strategy for a 3-environment service

Scenario: You have dev, staging, and prod for a web API using a database and an external payment API. Define how secrets and config will be stored, accessed, rotated, and audited. Include CI/CD handling and Terraform state concerns.

• Deliverable: a one-page plan outlining stores, access policies, rotation, CI integration, and safeguards.

Exercise 2: Fix insecure snippets (Terraform and Kubernetes)

Spot and fix issues that would leak secrets or reduce security.

# Terraform output (snippet)
output "db_password" {
  value = var.db_password
}

# Kubernetes Secret (snippet)
apiVersion: v1
kind: Secret
metadata:
  name: payment-secret
type: Opaque
data:
  apiKey: YXBpX2tleV9wbGFpbnRleHQ=

• Deliverable: list the issues and provide corrected versions.

Self-check checklist

• No plaintext secrets in code, Git, or examples.
• Access is principle-of-least-privilege and auditable.
• Rotation steps are clear and testable.
• Terraform state/logs do not expose sensitive values.
• CI uses short-lived credentials where possible.

Common mistakes and how to self-check

• Committing base64-encoded secrets. Self-check: can you decode with a one-liner? If yes, it's not encrypted.
• Putting secrets in Terraform outputs or locals. Self-check: search for secret names in plan/apply output and state.
• Long-lived CI tokens. Self-check: does the token expire within minutes? If not, replace with short-lived/OIDC.
• Same secret across environments. Self-check: verify unique values and separate access policies per env.
• No rotation plan. Self-check: produce a dated rotation runbook and last rotation timestamp.
• Overbroad permissions. Self-check: enumerate principals with read access; remove unused ones.

Practical projects

• Bootstrap a minimal secret store and rotate a database credential end-to-end in a sandbox environment.
• Convert a service to fetch secrets at runtime using workload identity (no static secrets in CI).
• Add pre-commit and CI scanning for secret leaks; mask sensitive logs; prove a failing case.
• Migrate plaintext Kubernetes Secrets to a controller-based encrypted pattern with audit.

Learning path

• Start: Understand config vs secrets and the least-privilege model.
• Next: Provision secret stores and identity policies with IaC.
• Then: Integrate CI/CD with short-lived credentials and masked logs.
• Finally: Automate rotation and create incident runbooks.

Next steps

• Apply the checklist to one existing service.
• Draft rotation runbooks and schedule test rotations.
• Implement scanning and log masking in your pipelines.

Mini challenge

Your team discovered a secret was printed in CI logs for 30 minutes. Outline the immediate actions (invalidate, purge, scan, harden) and propose two changes that would have prevented it.

Quick Test

The quick test is available to everyone. Only logged-in users will have their progress saved.