Security Scanning For Images

Why this matters

In MLOps, container images move models from notebooks into production. If an image contains vulnerable packages, secrets, or misconfigurations, attackers can pivot into your cluster, exfiltrate data, or tamper with models. Security scanning lets you catch problems early and enforce deployment gates in CI/CD.

• Real tasks you will do: scan images for CVEs, fail builds on high severity issues, generate SBOMs, verify base images, and track fixes across releases.
• Impact: fewer incidents, faster approvals, and compliance-ready pipelines.

Concept explained simply

Security scanning checks what’s inside your container: OS packages, language dependencies, configs, and secrets. It compares them against known vulnerabilities and best practices, then reports risks with severities.

Mental model

Think of your image as a suitcase at an airport. Scanners look for banned items (known CVEs), hidden compartments (secrets), and overweight baggage (unnecessary packages). Lighter, cleaner suitcases pass quickly and safely.

Practical workflow

1. Choose a lean base image. Start with slim/distroless when possible.
2. Install only what you need. Pin versions; avoid compilers and shells in runtime images.
3. Scan locally. Run a scanner before pushing.
4. Scan in CI. Fail on Critical/High; allow Medium/Low with tracking.
5. Generate an SBOM. Store it for traceability.
6. Fix and rescan. Update packages, swap base, or remove unused deps.

Worked examples

Example 1: Scan a Python image

1. Pull or build an image, e.g., python:3.10-slim with a simple app.
2. Run a scan (choose one style):

   # Style A (Trivy-like)
   scanner image my-ml-app:0.1
   
   # Style B (Grype-like)
   scanner -i my-ml-app:0.1
   

3. Read the summary. Expect a count by severity and a list of affected packages.

Severity summary: CRITICAL:1 HIGH:4 MEDIUM:12 LOW:8
Example finding: openssl 1.1.1n - CVE-2022-XXXX - HIGH - fixed in 1.1.1t

Example 2: Reduce vulnerabilities with a better base

1. Original Dockerfile uses python:3.10. Switch to python:3.10-slim and ensure only runtime needs are installed.
2. Rebuild and rescan; compare severity counts.
3. Optionally use a multi-stage build to copy only your app artifacts into a minimal runtime image.

Before: CRITICAL:2 HIGH:8 MEDIUM:20 LOW:15
After (slim + multi-stage): CRITICAL:0 HIGH:2 MEDIUM:6 LOW:5

Example 3: SBOM and policy gate

1. Generate SBOM:

   scanner sbom -o sbom.json my-ml-app:0.2

2. Enforce a gate: fail if any Critical/High found:

   scanner image --exit-on=high my-ml-app:0.2

3. Use the non-zero exit code to stop the pipeline.

Exercises

These match the tasks in the Exercises section below. Do them in order and check off as you go.

• [ ] ex1: Scan an image and summarize results.
• [ ] ex2: Reduce findings by changing the base image and dependencies, then rescan.

Common mistakes

• Using full base images when slim/distroless would work. Fix: start lean.
• Leaving build tools in runtime image. Fix: multi-stage builds.
• Ignoring language-level deps. Fix: scan both OS and app dependencies.
• Not pinning versions. Fix: pin and update intentionally.
• Assuming a pass today means safe forever. Fix: rescan regularly; regenerate SBOMs.
• Silencing findings without time limits. Fix: track exceptions with expiry.

Practical projects

• Harden a model-serving image: convert from full to slim base, add multi-stage build, scan before/after, and record improvement.
• Add a security gate to CI: make the pipeline fail on Critical/High and publish SBOM as an artifact.
• Create a baseline-and-drift report: save an initial scan, then track new vulnerabilities across three releases.

Who this is for

• MLOps Engineers integrating model images into CI/CD.
• Data/ML Engineers packaging models for serving.
• Platform/SRE roles responsible for secure deployments.

Prerequisites

• Basic Docker knowledge: build, tag, push.
• Familiarity with Linux packages and Python or similar runtime.
• Access to run a local container scanner (or a compatible CLI in CI).

Learning path

• Before: Container basics, Dockerfiles, multi-stage builds.
• Now: Image scanning, SBOMs, policy gates.
• Next: Signing and verifying images, runtime security, and admission controls.

Next steps

• Implement scanning in your current project’s CI.
• Set a fail threshold and add SBOM generation.
• Review and reduce vulnerabilities, then keep a weekly rescan routine.

Mini challenge

Take a model-serving image you use today. Replace the base with a slimmer option, convert to a multi-stage build, pin dependencies, and rescan. Can you reduce High/Critical findings by at least 70% while keeping functionality intact?

Quick Test

Ready? Take the quick test below to check your understanding.