Who this is for
Backend and platform engineers who build services, deploy to cloud/Kubernetes, or maintain CI/CD and need safe, repeatable ways to store and use passwords, API keys, tokens, and certificates.
Prerequisites
- Comfort with environment variables and configuration files
- Basic understanding of service deployment (containers or VMs)
- Ability to read simple code snippets (any language)
Why this matters
Real tasks you will face:
- Remove hardcoded credentials from a codebase without breaking production
- Rotate an API key with zero downtime
- Inject secrets into CI/CD safely and prevent logging leaks
- Audit who accessed a secret and when
Concept explained simply
A secret is any data that grants access or proves identity: passwords, API keys, OAuth tokens, private keys, signing keys, connection strings, and client certificates.
Good secrets management means: store secrets in a protected place, control who/what can read them, deliver them just-in-time to the process that needs them, avoid accidental exposure (code, logs, crash dumps), and rotate/expire them regularly.
Mental model
Think of a locked vault (storage) with a guard (access control) handing out sealed envelopes (scoped, short-lived credentials). Envelopes auto-expire (rotation/TTL). Cameras record every access (audit). The envelope is opened only inside the service at runtime (just-in-time delivery).
Core principles
- Never commit secrets to version control
- Least privilege: scope each secret to the minimal resource and action
- Prefer short-lived, automatically rotated credentials
- Encrypt at rest and in transit; avoid plaintext copies
- Centralize storage and auditing
- Prevent leakage via logs, error messages, metrics, and screenshots
- Automate: issuance, rotation, revocation, and validation
Where secrets live safely
- Managed secrets stores (central vaults with IAM, audit, rotation)
- Environment variables supplied at deploy/runtime (never committed .env for prod)
- Encrypted files with strict OS permissions when a manager is unavailable
- Orchestrators (e.g., mounting secrets into containers) with encryption-at-rest enabled
- CI/CD secure variables and short-lived tokens rather than long-lived static keys
What NOT to do
- No hardcoded secrets in source code, examples, or tests
- No secrets in plaintext config files or wikis
- No screenshots or logs containing secret values
Worked examples
Example 1 — Remove a hardcoded DB password
- Find the hardcoded secret in code.
- Create a secret in your chosen store and name it clearly (e.g., service/prod/db_password).
- Change application config to read from environment variable or secret provider at startup.
- Inject the secret at deploy time, not in source code.
- Delete the hardcoded value and scan history to ensure it’s gone from recent commits.
Self-check
- The repo contains no secret values, including example configs
- Local dev uses a throwaway dev secret, not production
- Prod pulls the secret at runtime from the manager
Example 2 — Rotate an API key with zero downtime
- Issue a new key but keep the old key valid (dual-secret window).
- Update services to accept/use the new key. Validate traffic with metrics.
- Switch primary usage to new key. Observe for errors.
- Revoke the old key and verify no failures.
Self-check
- There was a defined overlap window
- Dashboards or logs confirmed traffic with the new key
- Old key is revoked and alarms exist for key reuse
Example 3 — Prevent secret leaks in logs
- Identify code paths that log request bodies, env vars, or config.
- Redact common secret patterns (keys, tokens, passwords).
- Mask secrets in CI logs and restrict log access.
- Add tests to ensure sensitive fields are redacted.
Self-check
- No error path prints secrets
- CI shows masked values (e.g., ****)
- Redaction is covered by automated tests
Practical steps you can use today
- Name secrets consistently: app/env/purpose (e.g., billing/prod/stripe_api_key)
- Use per-environment secrets; never share dev/test/prod values
- Give machines identity (workload or OIDC) to fetch secrets without embedding static keys
- Scan repos for secrets before every push
- Set rotation policies and calendar reminders or automation
Exercises
Complete the exercise below. Everyone can access the exercise; if you log in, your progress will be saved.
Exercise ex1 — Remove a hardcoded secret and plan rotation
Goal: Refactor code to load a secret securely and define a safe rotation plan.
- Identify the secret in this snippet:
const cfg = {
dbUser: "appuser",
dbPass: "super-secret-123", // hardcoded
dbHost: "db.internal",
};
module.exports = cfg;
- Refactor so the password comes from an environment variable (e.g., DB_PASSWORD). Add a safe default for local dev that is not production.
- Write a 5-step rotation plan enabling an overlap window and validation metrics.
- Checklist:
- No secret values remain in source
- Production reads from a secure source at runtime
- Rotation has overlap, validation, and revocation steps
Common mistakes and how to self-check
- Committing .env files: Never commit environment files with real secrets. Self-check: your repo ignores .env and contains only .env.example with placeholders.
- Long-lived credentials: Replace with short-lived or rotated secrets. Self-check: list secrets older than 90 days and rotate.
- Logging secrets: Ensure redaction middleware is enabled and tested. Self-check: force an error path and check logs for masking.
- Over-broad access: Secrets readable by many services. Self-check: review IAM policies; each secret should be scoped to one service and environment.
- No audit: Turn on access logs. Self-check: prove who accessed a secret last week.
Practical projects
- Starter: Migrate one service from hardcoded credentials to runtime-injected secrets with a manager
- Intermediate: Implement dual-key rotation for your outbound payment provider without downtime
- Advanced: Add automated secret scanning to pre-commit and CI with redaction tests and on-call alerts
Learning path
- Next: Secret rotation automation and short-lived tokens
- Then: Workload identity (OIDC) for CI/CD and services
- Then: Encryption basics (KMS and envelope encryption)
- Optional: Container orchestrator secret delivery and at-rest encryption
Next steps
- Finish the exercise and compare with the solution
- Take the quick test below (available to everyone; log in to save progress)
- Apply one rotation in your environment this week
Mini challenge
Within 30 minutes, list all secrets your service uses in production, identify owners, last rotation date, and set a rotation reminder for the oldest one.