ConfigMaps And Secrets

Why this matters

As an MLOps Engineer, you ship and operate ML training jobs, batch inference, model servers, and pipelines. Each needs configuration (endpoints, hyperparameters, feature flags) and sensitive data (API tokens, storage credentials). Kubernetes gives you two purpose-built objects to manage these cleanly:

• ConfigMaps: non-sensitive configuration
• Secrets: sensitive configuration (credentials, tokens, keys)

Using them correctly means safer deployments, easier updates, reproducibility, and fewer outages from misconfigured pods.

Concept explained simply

Think of ConfigMaps as labeled notes your app reads (like learning rate or endpoint URLs). Secrets are sealed envelopes with passwords or tokens. You attach these notes/envelopes to Pods as environment variables or as files.

Mental model

• Two drawers: one for plain settings (ConfigMap), one for sensitive settings (Secret).
• You can pass drawer contents into a Pod either as env vars for small values or as mounted files for structured or larger values.
• Updates: env vars are fixed at Pod start; mounted files from ConfigMaps/Secrets can be refreshed on disk, but apps must re-read them.

Core differences and security notes

• ConfigMap is for non-sensitive text data.
• Secret is for sensitive data. Kubernetes stores values base64-encoded; base64 is not encryption. For real protection, enable secret encryption at rest in the cluster and restrict RBAC access.
• Size: Each object should be ≤ ~1 MiB (tied to etcd object size). Store large files (models, datasets) in object storage, not in ConfigMaps/Secrets.

When to use which (ML scenarios)

• ConfigMap examples: training hyperparameters, feature store endpoint URLs, model server flags, path templates.
• Secret examples: S3/GCS credentials, model registry tokens, database passwords, TLS certs, experiment tracking API keys.

Worked examples

apiVersion: v1
kind: ConfigMap
metadata:
  name: hparams
  namespace: default
data:
  hparams.yaml: |
    learning_rate: 0.001
    batch_size: 64
    epochs: 10
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-job
  namespace: default
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: trainer
        image: python:3.10-slim
        command: ["bash", "-lc", "python - <<'PY'\nimport yaml\nprint(open('/config/hparams.yaml').read())\nPY"]
        volumeMounts:
        - name: cfg
          mountPath: /config
      volumes:
      - name: cfg
        configMap:
          name: hparams

apiVersion: v1
kind: Secret
metadata:
  name: s3-cred
  namespace: default
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: <your-id>
  AWS_SECRET_ACCESS_KEY: <your-secret>
  AWS_DEFAULT_REGION: us-east-1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-infer
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels: { app: batch-infer }
  template:
    metadata:
      labels: { app: batch-infer }
    spec:
      containers:
      - name: worker
        image: alpine:3.19
        command: ["sh", "-lc", "env | grep '^AWS_' | sed 's/=.*/=<redacted>/' && sleep 3600"]
        envFrom:
        - secretRef:
            name: s3-cred

apiVersion: v1
kind: Secret
metadata:
  name: api-token
  namespace: default
type: Opaque
stringData:
  token: <very-secret-token>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels: { app: model-server }
  template:
    metadata:
      labels: { app: model-server }
    spec:
      containers:
      - name: server
        image: python:3.10-slim
        command: ["bash", "-lc", "ls -l /etc/creds && head -c 4 /etc/creds/token && echo '...' && sleep 3600"]
        volumeMounts:
        - name: cred
          mountPath: /etc/creds
          readOnly: true
      volumes:
      - name: cred
        secret:
          secretName: api-token
          defaultMode: 0400

How to create and use ConfigMaps & Secrets

1. Create
   • ConfigMap: use data: or a file. For structured config, embed YAML or JSON as a value.
   • Secret: prefer stringData: for authoring; Kubernetes stores it as base64 in data: automatically.
2. Attach to Pod
   • As env vars: env or envFrom.
   • As files: volumes with configMap or secret and volumeMounts.
3. Update safely
   • Change the ConfigMap/Secret, then roll Pods if using env vars.
   • For mounted files, apps must re-read the file to pick up changes.
   • For Deployments, trigger rollout by changing a pod template annotation (e.g., checksum of the resource).

Exercises (practice now)

These mirror the tasks in the Exercises section below. Do them in any namespace you control.

Exercise 1: ConfigMap for ML hyperparameters

• Create a ConfigMap named hparams containing a hparams.yaml with learning_rate: 0.005 and batch_size: 128.
• Run a Job that mounts it at /config and prints file content.
• Update the ConfigMap to learning_rate: 0.01 and roll the Job to read new values.

Exercise 2: Secret for S3 access

• Create a Secret named s3-cred with stringData for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION.
• Run a Deployment that consumes these as env vars and also mounts them at /etc/creds with defaultMode 0400.
• Verify the env vars exist and the files are present; do not print real secrets to logs.

Self-check checklist

• Secrets created using stringData, not pre-base64 data in YAML by hand
• No secrets echoed to logs or stored in git
• ConfigMap used for non-sensitive values only
• Mounted secret files have restrictive permissions
• Rollout performed when env vars must change

Common mistakes and how to self-check

• Putting secrets into ConfigMaps. Fix: Always use Secret for anything sensitive.
• Expecting env vars to update automatically. They do not. Fix: restart Pods or use a rollout trigger.
• Leaking secrets in logs. Fix: Avoid printing env vars; if needed, redact.
• Oversized objects (>1 MiB). Fix: store large files in object storage.
• RBAC too broad. Fix: restrict get/list on Secrets to necessary service accounts and namespaces.
• Relying on base64 for security. Fix: treat it as encoding; enable cluster secret encryption and use strong access controls.

Practical projects

• Build a training Job that reads hparams from a ConfigMap file and writes metrics to stdout.
• Deploy a model server that reads a Secret API token from a file and refreshes it every 5 minutes by re-reading.
• Create a batch pipeline (CronJob) that rotates an access token Secret monthly and performs a zero-downtime rollout.

Learning path

• Start here: ConfigMaps and Secrets (this page)
• Next: Volumes and Persistent Volumes for datasets and models
• Then: Service Accounts and RBAC for least-privilege access
• Finally: Secret rotation strategies and external secret managers

Who this is for

• MLOps Engineers deploying training and inference on Kubernetes
• Data Scientists moving notebooks to scheduled jobs
• Platform Engineers building ML platforms

Prerequisites

• Basic Kubernetes (Pods, Deployments, Jobs)
• kubectl access to a cluster and permissions to create namespaced resources
• YAML familiarity

Next steps

• Automate rollouts when ConfigMaps/Secrets change using annotations.
• Adopt a naming convention (e.g., appname-env-purpose).
• Plan a rotation policy for tokens and keys.

Mini challenge

Create a Deployment that:

• Consumes a ConfigMap of model-server flags (e.g., max_concurrency: 4, log_level: info) mounted at /etc/server
• Consumes a Secret API token as a file at /run/secret/token with permission 0400
• Prints a warning and exits if either file is missing or empty

Ready? Quick Test

The Quick Test is available to everyone. If you are logged in, your score and progress will be saved automatically.