Secrets Management

Why this matters

As an API Engineer, you move credentials through build pipelines, container runtimes, and cloud services. Secrets Management keeps tokens, passwords, and keys safe while still making them available to the code that needs them—no leaks, no outages.

• Real tasks you will face:
  • Provision API keys for third‑party services without committing them to Git.
  • Rotate database passwords without downtime.
  • Grant least‑privilege access to CI/CD so pipelines can fetch secrets safely.
  • Audit who accessed which secret, and when.

Concept explained simply

A secret is any value that grants access—think of it like a door key. Secrets Management is the set of practices and tools that store keys in a locked box, control who can take a key, log every access, and let you change the locks without breaking the app.

Mental model

• Safe: Secrets live in a secure store (vault).
• Short‑lived: Access is temporary or rotated often.
• Least privilege: Only the specific service/role gets what it needs.
• Observable: Reads and updates are logged and alertable.

What counts as a secret?

• Database passwords and connection strings
• API tokens and OAuth client secrets
• Private keys, certificates, signing keys
• Encryption keys, JWT signing secrets
• SSH keys, webhook secrets

Core practices (step cards)

1. Identify: Inventory all secrets per service and environment; classify by impact (Low/Medium/High).
2. Store: Put secrets in a central manager; avoid hardcoding and avoid keeping them only in environment files.
3. Access: Use role‑based access control and short‑lived credentials; bind access to service identity (not humans).
4. Rotate: Change secrets regularly and on events (key exposure, staff changes). Automate where possible.
5. Audit: Log reads/writes, set alerts for unusual access, and review periodically.

Worked examples

Example 1 — Local development using environment variables safely

Goal: Keep local .env files out of Git while enabling teammates to run services.

1. Create a template file: .env.example with placeholder values.
2. Add .env to .gitignore.
3. Load secrets at runtime, e.g., in Node.js:

// Node.js
const dbUri = process.env.DB_URI; // do not hardcode
if (!dbUri) throw new Error('Missing DB_URI');

Why this is safe

Secrets do not enter version control. The template documents required keys without exposing values.

Example 2 — Injecting secrets from a secret manager into containers

Goal: A containerized API reads secrets at startup from a central store.

1. Give the service a machine identity (e.g., workload identity) with read access to its secret path only.
2. On startup, fetch secrets via SDK/sidecar and populate environment variables:

# Pseudocode
secrets = secretStore.read("apps/orders-api/prod")
export DB_URI=secrets.DB_URI
exec node server.js

Why this is safe

The container gets only the secrets it needs, for the correct environment, via authenticated identity. No static secrets baked into images.

Example 3 — Zero‑downtime database password rotation

Goal: Rotate DB password without breaking running pods.

1. Create a second DB user or enable dual passwords.
2. Write the new credential to the secret manager under version v2.
3. Roll your deployment so new pods use v2.
4. Confirm traffic is healthy; revoke the old credential (v1).
5. Audit logs for any access using old credentials.

Key idea

Dual credentials allow a rollover period where both old and new work. Once all workloads are updated, remove the old one.

Exercises

Do these hands‑on tasks. You can complete them in any language or stack.

• Exercise 1: Map your service secrets and classify risk. Create a per‑environment inventory with storage and rotation plan.
• Exercise 2: Plan and simulate a safe secret rotation with dual credentials and a rollback strategy.

Checklist — I did it right

• Every secret has an owner, storage location, and rotation frequency.
• No secrets are stored in Git or container images.
• Access is tied to service identity, not personal accounts.
• Rotation includes validation steps and rollback plan.
• Reads and updates are auditable.

Common mistakes and self‑check

• Committing secrets to Git — Self‑check: Search repo history for patterns like AWS_SECRET or password=.
• Long‑lived tokens — Self‑check: List tokens; any older than your policy? Rotate now.
• Too‑broad access — Self‑check: Review policies; can the service read other apps’ secrets?
• Baking secrets into images — Self‑check: Scan Dockerfiles for ENV lines with sensitive values.
• No audit trail — Self‑check: Can you answer who read the production DB password last week?

Practical projects

• Project 1: Secret bootstrap for a microservice. Acceptance: service pulls secrets at startup, no secrets in repo, access is least‑privilege.
• Project 2: Rotation runbook + automation script. Acceptance: script flips credentials, triggers rollout, verifies health, and revokes old secrets.
• Project 3: Secret access audit dashboard. Acceptance: shows who/what accessed each secret and highlights anomalies.

Who this is for

• API Engineers and backend developers deploying services across environments.
• Platform/DevOps engineers maintaining CI/CD and runtime security.

Prerequisites

• Comfort with environment variables and service configuration.
• Basic understanding of CI/CD and containers.
• Familiarity with your programming language’s config loading.

Learning path

1. Inventory and classify secrets per service and environment.
2. Centralize storage and remove secrets from Git and images.
3. Implement role‑based access via service identity.
4. Automate rotation and add audit logging.
5. Test disaster scenarios: leaked secret, invalid rotation, store outage.

Mini challenge

Your API uses a payment gateway key stored as an env var in production. Design a plan to move it to a secret manager and rotate it within 24 hours, ensuring no downtime. Include: identity setup, read path, rollout strategy, verification checks, and rollback.

Next steps

• Complete the exercises and mini challenge.
• Take the Quick Test below. Available to everyone; if you are logged in, your progress will be saved.
• Apply the rotation runbook to a real non‑prod environment.