Environments And Dependencies

Who this is for

• Aspiring and junior MLOps engineers who need reproducible setups for training and serving models.
• Data scientists moving experiments into production reliably.
• Backend engineers who integrate ML services into broader systems.

Prerequisites

• Comfort with the command line (bash/PowerShell).
• Basic Python knowledge (installing packages, running scripts).
• Optional but helpful: basic Docker familiarity.

Why this matters

In MLOps, you must recreate the same environment on laptops, CI, training clusters, and production. If dependencies drift, models can behave differently or break. Typical tasks you will face:

• Reproduce a training run from 3 months ago to debug a metric change.
• Package an inference service so it runs the same on dev, staging, and prod.
• Pin CUDA/cuDNN versions so GPU training works across machines.
• Minimize image size and build time with smart Docker caching.
• Patch a security vulnerability by bumping a single dependency safely.

Concept explained simply

An environment is the collection of everything your code needs: Python version, packages, OS libraries, and sometimes GPU drivers. Dependencies are the ingredients and their exact versions. Reproducibility means writing down the exact recipe so anyone can bake the same cake later.

Mental model

• Recipe: lock/constraints files (exact versions).
• Pantry: your package indexes and wheels.
• Kitchen: the isolated environment (venv/conda/Docker image).

Key concepts and tools

• Environment isolation: venv/virtualenv, conda, Docker.
• Pinning and locking: requirements.txt (exact pins), constraints files, lock files (e.g., poetry.lock).
• System libraries: apt/yum packages needed by some Python packages (e.g., libgomp).
• CUDA/GPU stacks: match CUDA, cuDNN, and framework wheels.
• Deterministic builds: hash-checking, consistent indexes, and repeatable Dockerfiles.

Worked examples

Example 1: venv + pinned requirements.txt

# 1) Create and activate an isolated environment
python -m venv .venv
# Linux/macOS
source .venv/bin/activate
# Windows
#.venv\Scripts\activate

# 2) Install exact versions
pip install numpy==1.26.4 pandas==2.1.4 scikit-learn==1.3.2

# 3) Freeze to a file for reuse
pip freeze --all > requirements.txt

# 4) Recreate elsewhere
pip install --no-deps --requirement requirements.txt

Tip: Use --no-deps when installing from a fully pinned requirements.txt to avoid resolver surprises.

Example 2: Constraints with pip-tools

# Write top-level requirements (no pins or loose pins)
# file: requirements.in
pandas
scikit-learn>=1.3

# Compile to a fully pinned requirements.txt
pip install pip-tools
pip-compile --generate-hashes requirements.in

# Install with hash checking
pip install --require-hashes -r requirements.txt

pip-compile keeps a clean separation between what you want (requirements.in) and the exact graph you install (requirements.txt).

Example 3: Conda environment for data science

# environment.yml
name: ds-env
channels:
  - conda-forge
dependencies:
  - python=3.10
  - numpy=1.26
  - pandas=2.1
  - scikit-learn=1.3
  - pip
  - pip:
      - mlflow==2.9.0

# Create and use it
conda env create -f environment.yml
conda activate ds-env

Use conda for heavy numeric libs, then pip for pure-Python extras via the pip: section.

Example 4: Docker image for CPU inference (small and cache-friendly)

# .dockerignore
__pycache__
*.pyc
.venv
build/

# Dockerfile
FROM python:3.10-slim AS base
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

# System deps (only what you need)
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
# Separate dependency layer for better caching
COPY requirements.txt .
RUN pip install --no-cache-dir --requirement requirements.txt

# Copy code last to avoid busting the dependency cache
COPY . .

# Non-root user for safety
RUN useradd -m appuser && chown -R appuser /app
USER appuser

CMD ["python", "-c", "import sys,platform,pandas;print('OK', platform.platform(), pandas.__version__)"]

Build once, run anywhere with consistent versions.

# Example idea (do not run without matching wheels)
# FROM nvidia/cuda:12.2.0-cudnn8-runtime-ubuntu22.04
# RUN pip install torch==<version-built-for-12.2>
# Always match framework wheels to CUDA version to avoid runtime errors.

Exercises

Do these hands-on tasks. The quick test is available to everyone; sign in if you want your progress saved automatically.

Exercise 1 (matches ex1 below)

Create a venv and use pip-tools to compile a fully pinned, hash-checked requirements.txt from a simple requirements.in. Reinstall using only the compiled file and confirm versions.

Exercise 2 (matches ex2 below)

Write a minimal Dockerfile that installs from your pinned requirements.txt and prints the package versions at startup. Keep the image small and use caching.

Verification checklist

• Environment can be recreated from files only (no manual steps).
• Exact versions are visible and consistent across runs.
• Docker build is repeatable; dependency layer caches correctly.
• No unnecessary OS packages; image starts as non-root.

Common mistakes (and self-check)

• Not pinning versions: self-check by running pip freeze and comparing to your file.
• Mixing conda and pip haphazardly: prefer conda for heavy libs; add pip-only items under pip: in environment.yml.
• Using latest or floating tags (e.g., python:latest): pin base images to exact versions.
• Missing system libraries: check import errors during build; add minimal apt/yum packages.
• Rebuilding too often: separate dependency and source layers to leverage Docker cache.
• GPU mismatch: ensure framework wheels match CUDA/cuDNN/base image versions.

Practical projects

• Reproducible training run: package a small scikit-learn pipeline with pinned environment; rerun after a week to confirm identical metrics.
• CPU inference image: serve a simple model (e.g., iris classifier) in a slim Docker image with non-root user.
• Dual-env repo: dev environment (loose pins) and release environment (locked pins) using pip-compile or poetry, with a short README on how to switch.

Learning path

1. Local isolation with venv/conda.
2. Pinning and locking (pip-compile or poetry lock files).
3. Deterministic Docker builds for CPU.
4. System libraries and native deps.
5. GPU stacks and CUDA version pinning (when needed).
6. CI reproduction and artifact storage for lock files.

Next steps

• Automate dependency updates and regression tests in CI.
• Introduce security scanning for base images and Python packages.
• Promote locked artifacts (requirements.txt/lock files and images) from dev to prod.

Mini challenge

Timebox 45 minutes: Create a project with a requirements.in, compile to a locked requirements.txt with hashes, and a Dockerfile that prints versions of three key packages on start. Hand it to a teammate to rebuild. If their output matches yours, you nailed reproducibility.