luvv to helpDiscover the Best Free Online Tools
Topic 7 of 7

Secure Deployment Practices

Learn Secure Deployment Practices for free with explanations, exercises, and a quick test (for MLOps Engineer).

Published: January 5, 2026 | Updated: January 5, 2026

Table of Contents

What you will learn

Deploy ML services safely with least privilege, secrets management, signed artifacts, secure containers, network controls, and compliant rollouts.

  • Lock down containers and clusters for inference services
  • Protect data in transit/at rest and handle secrets correctly
  • Use safe rollout strategies (canary, blue/green) with rollback
  • Add CI/CD security gates (scans, SBOM, signing)
  • Monitor, audit, and respond to incidents

Who this is for

  • MLOps engineers responsible for production ML services
  • Data/ML engineers moving models to Kubernetes or cloud
  • Platform engineers adding ML workloads to existing clusters

Prerequisites

  • Basic Linux, containers, and networking
  • Familiarity with CI/CD and environment promotion (dev → staging → prod)
  • Understanding of your organization’s compliance needs (e.g., PII rules)

Why this matters

Real MLOps tasks demand secure-by-default deployments. You will be asked to:

  • Ship a model API to production with zero public data exposure
  • Rotate credentials for a model registry without downtime
  • Prove images are signed and dependencies are tracked
  • Quickly roll back a faulty model version with minimal blast radius
  • Pass audits with clear logs and reproducible builds

Concept explained simply

Secure deployment means every step from build to runtime makes it hard for attackers and easy for you to prove compliance. Think of three layers:

  • Build layer: Dependable code and model artifacts (pin versions, scan, sign, SBOM)
  • Ship layer: Transport safely (private registry, encrypted secrets, controlled access)
  • Run layer: Isolate workloads (least privilege, network policy, TLS, monitoring)

Mental model

Use the Onion of Defense:

  1. Artifact integrity: Reproducible builds, signed images/models
  2. Secrets: Central store, short-lived tokens, rotation
  3. Runtime hardening: Non-root, read-only FS, no extra capabilities
  4. Network: Default deny, allow only what’s needed, mTLS
  5. Observability: Metrics, logs, traces, audits with retention
  6. Release safety: Canary/blue-green + auto rollback

Worked examples

Example 1 — Harden a Kubernetes model API

Goal: Deploy a model API with least privilege and isolation.

  • Image from a private registry, pinned by digest
  • Pod runs as non-root, read-only root FS, no privilege escalation
  • Secrets mounted, not baked into the image
  • NetworkPolicy default deny, only allow from API gateway namespace
  • mTLS for intra-cluster calls where possible

Result: Even if the app is compromised, lateral movement and data exfiltration are constrained.

Example 2 — Safe canary rollout of a new model
  1. Deploy v2 alongside v1
  2. Route 5% traffic to v2; monitor error rate, latency, and key business KPI
  3. If metrics stay healthy for 30 minutes, increase to 25%, then 50%, then 100%
  4. If any SLO breaches, rollback automatically to v1

Result: Controlled exposure limits risk while providing real production feedback.

Example 3 — Secret rotation without downtime
  1. Store credentials in a central secret manager with automatic rotation
  2. Mount secrets into pods via references so updates propagate
  3. Configure app to reload credentials on change (e.g., file watch or short connection TTL)
  4. Rotate keys; verify the app continues serving traffic

Result: Regular rotation reduces risk without service interruption.

Example 4 — Supply chain: SBOM + signed artifacts
  1. Generate SBOM at build time for the inference image
  2. Sign the image and the model artifact
  3. Admission policy only allows verified signatures

Result: Only known, verified artifacts can run in the cluster, and you can trace dependencies quickly.

Deployment patterns that are secure-by-default

  • Least privilege: minimal IAM roles, minimal Kubernetes RBAC per namespace
  • Runtime hardening: non-root, read-only FS, drop capabilities, resource limits
  • Network segmentation: default deny, explicit allow rules, mTLS where applicable
  • Data protection: TLS in transit; KMS-encrypted storage at rest; avoid storing PII unless required
  • Supply chain controls: SBOM, dependency scanning, image signing and verification
  • Safe releases: canary/blue-green, health checks, auto rollback
  • Observability & audit: structured logs, metrics, trace IDs, audit logs with retention

Exercises (hands-on)

Note: Everyone can attempt the quick test; only logged-in users have their progress saved.

Exercise 1 — Write a secure Kubernetes manifest

Draft a Deployment, Service, and NetworkPolicy for a model API that follows these rules:

  • Run as non-root; read-only root filesystem; no privilege escalation; drop all capabilities
  • Use liveness/readiness probes; set CPU/memory requests and limits
  • Env secrets from a secret manager/Kubernetes Secret, not hard-coded
  • Image pinned by digest and pulled from a private registry
  • NetworkPolicy default deny; only allow ingress from a labeled gateway pod
Checklist to self-verify
  • Does securityContext enforce runAsNonRoot and readOnlyRootFilesystem?
  • Are resource requests/limits present for each container?
  • Are probes defined for health checks?
  • Are secrets referenced via secretKeyRef or mounted files?
  • Is there a default-deny NetworkPolicy with a specific allow rule?

Exercise 2 — Add security gates to a CI/CD pipeline

Design a pipeline outline that prevents insecure releases. Include:

  • Dependency and secret scanning
  • Container image build with SBOM generation
  • Image signing
  • Staging deploy + smoke tests
  • Progressive rollout with auto rollback
  • Admission policy that only allows signed images
Checklist to self-verify
  • Are scans blocking on critical issues?
  • Is the image signed before pushing?
  • Is there a separate staging verification step?
  • Is progressive delivery defined with clear rollback triggers?
  • Is signature verification enforced at deploy time?

Common mistakes and self-check

  • Running containers as root — Self-check: ensure runAsNonRoot and user IDs are set.
  • Hard-coding secrets in images or code — Self-check: search repo for secrets; all secrets must come from a secret store.
  • No default-deny network policy — Self-check: confirm an explicit default-deny policy exists in the namespace.
  • Skipping image signing — Self-check: verify an admission rule rejects unsigned images.
  • Single shared service account across namespaces — Self-check: service accounts are per app with minimal permissions.
  • No rollback plan — Self-check: can you revert to the previous version in one command?

Practical projects

  1. Harden a model-serving microservice: Add probes, resource limits, non-root, read-only FS, and network policies; document risk reductions.
  2. Supply chain proof: Build an image, generate SBOM, sign it, and configure your cluster to allow only verified signatures.
  3. Progressive delivery: Implement a canary rollout for a model version with auto rollback on SLO breach; include dashboards and alerts.

Learning path

  1. Foundations: containers, Kubernetes securityContext, RBAC, TLS/mTLS
  2. Secrets: central secret manager, short-lived creds, rotation patterns
  3. Supply chain: dependency scanning, SBOM, artifact signing, private registries
  4. Runtime controls: Pod security, resource policies, network policies
  5. Release safety: canary/blue-green, health checks, rollback
  6. Observability & audit: logs, metrics, traces, audit trails and retention

Next steps

  • Apply the checklists to your staging environment
  • Automate the gates in CI/CD so they cannot be bypassed
  • Schedule a tabletop incident response exercise for a model rollback

Mini challenge

In one page, describe how you would recover from a compromised model image in production within 30 minutes. Include:

  • Detection signal and on-call trigger
  • Containment steps (admission policy, network blocks)
  • Rollback procedure and verification checks
  • Post-incident actions (key rotation, dependency updates, audit review)
See a possible outline
  1. Detect via signature mismatch or anomalous error spikes
  2. Freeze deploys; enforce signed-only policy; isolate namespace
  3. Rollback to last known-good digest; run smoke tests
  4. Rotate registry and DB credentials; re-scan; update SBOM; document findings

Quick test instructions

Answer the questions to check your understanding. Everyone can take the test; only logged-in users have their progress saved.

Practice Exercises

2 exercises to complete

Instructions

Create a Deployment, Service, and NetworkPolicy for a model API named model-api that:

  • Runs as non-root with read-only root filesystem
  • Has liveness/readiness probes and resource requests/limits
  • Uses secrets via secretKeyRef or mounted files
  • Pulls an image by digest from a private registry
  • Denies all ingress by default and only allows traffic from pods labeled role=gateway in the same namespace

Provide YAML.

Expected Output
A valid set of Kubernetes YAML manifests that enforce non-root, read-only FS, no privilege escalation, resource limits, health probes, secrets from Kubernetes Secret, image pinned by digest, and a NetworkPolicy with default deny plus a specific allow rule from role=gateway pods.

Secure Deployment Practices — Quick Test

Test your knowledge with 10 questions. Pass with 70% or higher.

10 questions70% to pass
Model Risk Management BasicsComplete Skill

Have questions about Secure Deployment Practices?

AI Assistant

Ask questions about this tool