Secure Secrets Storage

Why this matters

In MLOps, models and pipelines often need access to databases, object stores, feature stores, message brokers, and third-party APIs. Every token, key, or password is a secret that could expose data or infrastructure if mishandled. Secure secrets storage protects training data, prevents supply-chain leaks, and keeps deployments compliant.

• Real tasks you will face: provisioning per-environment API keys; granting training jobs temporary access to object storage; rotating credentials for model registries; preventing secrets from appearing in logs; enforcing audit trails for compliance.

Concept explained simply

A secret is any credential that grants access (passwords, API tokens, certificates, encryption keys). Secure storage means that secrets are encrypted at rest, sent only over secure channels, and made available to workloads just-in-time with least privilege.

Mental model

Think of secrets as radioactive material: handle with tools, minimize exposure, track who touched it, and store it in a shielded container. The lifecycle is: create → store → distribute → use → rotate → revoke → audit.

Core principles

• Least privilege: each job gets only the access it needs, for only as long as needed.
• Short-lived credentials: prefer tokens with automatic expiration.
• Encryption everywhere: at rest with KMS/HSM; in transit with TLS.
• No secrets in code, images, or git history. Never in logs or metrics.
• Automated rotation and revocation.
• Auditability: who accessed what, when, and from where.
• Defense in depth: secret managers, service identity (OIDC), network policies, and runtime isolation.

Worked examples

Example 1: Training job on Kubernetes accessing object storage

1. Authenticate workload: use a service account bound to a workload identity (e.g., via OIDC) so pods get a signed identity instead of static keys.
2. Authorize narrowly: grant read-only to a specific bucket/prefix for the training job namespace.
3. Distribute secret: the pod receives a short-lived token via the cluster identity system; no hardcoded keys in the image.
4. Audit: enable access logs on the bucket and identity provider.

# Key points (pseudo YAML):
# serviceAccountName: trainer-sa
# automountServiceAccountToken: true
# volumes: prefer mounted tokens from the identity provider; avoid embedding static keys
# securityContext: readOnlyRootFilesystem: true

Example 2: CI pipeline retrieves secrets without storing cloud keys

1. Use OIDC federation: the CI runner exchanges its short-lived identity for a temporary cloud token.
2. Fetch needed secret from a secrets manager just-in-time.
3. Pass it to the build step via a file on a tmpfs volume with restrictive permissions (0400), not an environment variable.
4. Ensure logs mask secret values; fail build if secret is printed.

# Pseudo-pipeline
- step: authenticate-with-oidc
- step: fetch-secret name=MODEL_REGISTRY_TOKEN to=/run/secrets/registry token-ttl=15m perms=0400
- step: build-and-push uses /run/secrets/registry
- step: shred-secrets

Example 3: Serving API uses a sidecar to inject secrets

1. Run a secrets sidecar/agent that authenticates using the pod's identity.
2. Template secrets into a file watched by the app (hot-reload if rotated).
3. Mark the secret file as non-world-readable and avoid printing it on startup.
4. Rotate via the manager; agent refreshes the file; app re-reads without restart.

# Application reads from file
DB_PASSWORD_FILE=/run/secrets/db_password
with open(DB_PASSWORD_FILE, 'r') as f:
    db_password = f.read().strip()
# Use password; never log it

Design patterns for ML workloads

• Training: short-lived access to data stores; ephemeral compute identities; scoped buckets.
• Feature engineering: pipeline steps each get their own role; no shared long-lived keys.
• Model registry: write token only for CI publish step; read token for deploy step; both short-lived.
• Serving: sidecar/CSI injection; secret files with strict perms; reload on rotation.
• Notebooks: per-user identity; read-only credentials; session-lifetime tokens.

Hands-on exercises

Note: The quick test is available to everyone. If you are logged in, your progress will be saved.

Exercise 1: Build a secrets inventory and policy

Create a minimal inventory of secrets used by a sample ML service (training, CI, serving), and define handling rules.

• List name, owner, location, rotation interval, and access scope.
• Write rules: storage, distribution method, TTL, and logging guardrails.

Exercise 2: Safer file-based secret for a containerized app

Draft a container runtime plan to load a database password from a file with least exposure.

• Use a read-only mount path like /run/secrets.
• Set permissions to 0400 and ensure the file is not backed by the image layer.
• Show a small code snippet that reads the file and avoids logging it.

Exercise 3: Just-in-time secret in CI/CD

Write a pseudo-pipeline that authenticates using OIDC, fetches a secret at job start, masks it in logs, and deletes it at job end.

• Include explicit TTL and a cleanup step.
• Demonstrate masking patterns for logs.

Common mistakes and self-check

• Putting secrets in environment variables: they can leak via /proc, crash dumps, and diagnostics. Prefer files with restricted perms.
• Committing secrets to git: scan repos and history; rotate immediately if found.
• Long-lived tokens shared across teams: replace with per-service, short TTL credentials.
• Secrets in logs and metrics: use redaction and fail builds if patterns are detected.
• Ignoring rotation: set calendar reminders or automate; test rotation during business hours.
• Single secret for all environments: separate dev/stage/prod with distinct scopes.

Practical projects

• Secure the training pipeline: implement just-in-time access to object storage, with logs proving least privilege.
• Rotate secrets drill: simulate a leak; rotate DB password and update consumers within 30 minutes.
• Secret bootstrapper: a tiny init container that fetches a token, writes to /run/secrets, sets 0400 perms, and exits.

Who this is for, prerequisites, and learning path

Who this is for

• MLOps engineers, platform engineers, and ML engineers deploying models or pipelines.

Prerequisites

• Basic containers and Kubernetes (or another orchestrator)
• Familiarity with CI/CD
• Understanding of IAM roles and service accounts

Learning path

1. Identify secrets and owners; create an inventory.
2. Choose distribution method: identity-based tokens, sidecar/CSI, or manager API.
3. Implement file-based delivery with strict perms and no logging.
4. Add rotation and revocation procedures; test them.
5. Enable auditing and alerting on unusual access patterns.

Next steps

• Automate rotation for at least one critical secret.
• Add a pre-commit and CI step to scan for accidental secrets.
• Pilot OIDC-based federation in one pipeline to remove static cloud keys.

Mini challenge

Pick one running ML service. Replace any long-lived credential with a short-lived, identity-based token. Prove success by showing a log excerpt of the token expiry and an audit record of access.