Local Dev With Docker Compose

Why this matters

As an MLOps Engineer, you routinely need to run a small stack locally: model API, training jobs, message broker, vector/feature store, artifact storage, and tracking (e.g., MLflow). Docker Compose lets you boot this up with one command, share it with teammates, and keep dev and CI behavior consistent.

• Spin up a realistic local environment to iterate faster on models and pipelines.
• Reproduce production-like issues on your laptop without touching prod.
• Onboard teammates quickly: one file, one command.
• Run integration tests around your ML services reliably.

Concept explained simply

Think of Compose as a conductor. Each container (service) is a musician. The docker-compose.yml score tells who plays, when, and how they connect (networks and volumes). You press play (up), and the orchestra starts in sync.

Mental model

• Service: a runnable component (API, DB, cache).
• Image/Build: the container blueprint; either pull from a registry or build from a Dockerfile.
• Volumes: persistent data or source code mounts. Named volumes for data; bind mounts for live code.
• Networks: private virtual LANs so services can talk to each other by name.
• Environment: configuration via environment: and env_file:.
• Ports: expose service to your host (e.g., 8000:8000).
• depends_on + healthcheck: control start order and readiness checks.
• Profiles/overrides: toggle optional services or dev-only settings.

# Minimal structure
services:
  api:
    build: .
    ports:
      - '8000:8000'
    environment:
      - 'ENV=dev'
    depends_on:
      db:
        condition: service_healthy
  db:
    image: 'postgres:16'
    volumes:
      - 'pgdata:/var/lib/postgresql/data'
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres']
      interval: 5s
      timeout: 3s
      retries: 10
volumes:
  pgdata:

Setup once: a quick-start checklist

• Install Docker Desktop or Docker Engine + Compose v2.
• Create a project folder with a Dockerfile, .dockerignore, and docker-compose.yml.
• Prefer named volumes for databases (data persistence) and bind mounts for app code (hot reload).
• Keep secrets out of Compose; use .env for local-only variables that are safe to store or use Docker secrets for sensitive data.

1. Create .dockerignore: venvs, __pycache__, large datasets, build artifacts.
2. Write a small Dockerfile (Python base, copy requirements, install, set entrypoint).
3. Add Compose services for your API and dependencies (Redis/Postgres/MinIO).
4. Run docker compose up --build. Iterate.

Worked examples

# docker-compose.yml
name: mlops-local
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
    ports:
      - '8000:8000'
    volumes:
      - './app:/app/app:rw'
    environment:
      - 'REDIS_HOST=redis'
      - 'ENV=dev'
    depends_on:
      redis:
        condition: service_healthy
  redis:
    image: 'redis:7-alpine'
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 5s
      timeout: 3s
      retries: 10

# docker-compose.mlflow.yml
services:
  postgres:
    image: 'postgres:16'
    environment:
      - 'POSTGRES_USER=mlflow'
      - 'POSTGRES_PASSWORD=mlflow'
      - 'POSTGRES_DB=mlflow'
    volumes:
      - 'pgdata:/var/lib/postgresql/data'
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U mlflow']
      interval: 5s
      timeout: 3s
      retries: 10
  minio:
    image: 'minio/minio:latest'
    command: server /data --console-address ':9001'
    environment:
      - 'MINIO_ROOT_USER=minio'
      - 'MINIO_ROOT_PASSWORD=minio123'
    ports:
      - '9000:9000'
      - '9001:9001'
    volumes:
      - 'miniodata:/data'
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:9000/minio/health/live']
      interval: 10s
      timeout: 5s
      retries: 12
  mlflow:
    image: 'ghcr.io/mlflow/mlflow:v2.14.1'
    ports:
      - '5000:5000'
    environment:
      - 'MLFLOW_S3_ENDPOINT_URL=http://minio:9000'
      - 'AWS_ACCESS_KEY_ID=minio'
      - 'AWS_SECRET_ACCESS_KEY=minio123'
    command: mlflow server --backend-store-uri postgresql+psycopg2://mlflow:mlflow@postgres:5432/mlflow --default-artifact-root s3://mlflow-artifacts --host 0.0.0.0 --port 5000
    depends_on:
      postgres:
        condition: service_healthy
      minio:
        condition: service_healthy
volumes:
  pgdata:
  miniodata:

# docker-compose.yml (base)
services:
  api:
    build: .
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000
    ports:
      - '8000:8000'
    environment:
      - 'ENV=ci'
  worker:
    build: .
    command: python worker.py
    profiles: ['worker']

# docker-compose.override.yml (auto-applied locally)
services:
  api:
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
    environment:
      - 'ENV=dev'
    volumes:
      - './app:/app/app:rw'
  worker:
    profiles: ['worker']

# Run with worker profile enabled only when needed
# docker compose --profile worker up --build

Exercises you can do now

These mirror the graded exercises below. Aim to complete them in order.

1. Compose an API + Redis stack with hot reload. Add a health endpoint that returns status ok.
2. Add Postgres with a named volume and a healthcheck. Ensure your API waits for the DB to be healthy.
3. Create a dev profile that adds a worker service and mounts your code for live reload, while CI stays lean.

• Checklist before running:
  • Containers can resolve each other by service name.
  • Persistent data lives in named volumes (not lost on restart).
  • No secrets committed to Git.
  • Healthchecks exist for stateful services.

Common mistakes and self-checks

• Using bind mounts for databases. Fix: use named volumes for DB data.
• Confusing start order with readiness. Fix: add healthchecks and retry logic; do not rely only on depends_on without conditions.
• Hard-coding host IPs. Fix: use service names on the Compose network.
• Forgetting .dockerignore. Fix: exclude venvs, data dumps, build artifacts.
• Ports clash. Fix: change host side of the mapping (e.g., 8001:8000).
• Bloated images. Fix: multi-stage builds, slim bases, no dev deps in prod images.

Practical projects (resume-ready)

• Local ML experimentation stack: API + MLflow + MinIO + Postgres + Redis. Show one-command setup.
• Feature engineering sandbox: Kafka/Redpanda + consumer worker + cache, with profiles to toggle streaming.
• Model retraining lab: Jupyter + worker + scheduler (e.g., Celery/Flower) with artifacts logged to MLflow.

Who this is for

• Early-career MLOps or Data/ML Engineers who need a portable local stack.
• Data Scientists who want fast, reproducible environments for experimentation.

Prerequisites

• Basic Docker knowledge: images, containers, Dockerfile.
• Familiarity with your service (e.g., FastAPI) and at least one datastore (Postgres/Redis).
• CLI comfort: running shell commands.

Learning path

1. Build minimal images for your services.
2. Add Compose for two services (API + cache).
3. Introduce stateful components (DB, MinIO) with healthchecks and volumes.
4. Use profiles and overrides to separate dev from CI.
5. Automate with Makefile or simple scripts for common tasks.

Next steps

• Containerize training jobs that run alongside your API locally.
• Add integration tests that run against your Compose stack in CI.
• Introduce resource limits and logging aggregation.

Mini challenge

Add a new service to your stack: a lightweight dashboard that reads from your API. Use a dev profile so it only runs locally. Include a healthcheck and confirm the API starts first.

Quick Test

The quick test is available to everyone. If you are logged in, your progress will be saved automatically.