Docker Compose For Local Stacks

Who this is for

Machine Learning Engineers and Data Scientists who need reliable, reproducible local environments for APIs, training jobs, experiment tracking, and dependencies like databases, message brokers, and object storage.

Prerequisites

• Basic Docker: images, containers, Dockerfile, volumes, networks
• Command line comfort
• Optional: Python/ML tooling (FastAPI, Jupyter, MLflow)

Why this matters

In real ML work you often run multiple services together: API + Postgres + Redis for feature caching, or Jupyter + MLflow + MinIO for experiments. Docker Compose lets you define and run these as a single, versioned stack so teammates can reproduce your setup with one command.

• Spin up a local inference API with Redis and Postgres for integration tests
• Run Jupyter with MLflow tracking and a database for experiments
• Sandbox data pipelines with a message broker and workers

Concept explained simply

Docker Compose is a YAML file that describes a group of containers (services), how they talk to each other (networks), where they store data (volumes), and how they start up (dependencies). One file, one command: consistent local stacks.

Mental model

• compose.yml = blueprint for your local mini-cloud
• services = apps (api, db, cache)
• networks = private wires between apps
• volumes = hard drives that survive restarts
• environment and env_file = knobs you twist without changing code
• depends_on + healthcheck = boot order and readiness

Set up a reliable local stack (step-by-step)

Worked examples

services:
  api:
    build: ./api
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://mluser:mlpass@db:5432/mldb
      - REDIS_URL=redis://cache:6379/0
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
  db:
    image: postgres:15
    environment:
      - POSTGRES_DB=mldb
      - POSTGRES_USER=mluser
      - POSTGRES_PASSWORD=mlpass
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mluser"]
      interval: 5s
      timeout: 3s
      retries: 10
    volumes:
      - dbdata:/var/lib/postgresql/data
  cache:
    image: redis:7
    command: ["redis-server", "--save", "60", "1"]
volumes:
  dbdata:

services:
  jupyter:
    image: jupyter/scipy-notebook:2023-11-20
    ports: ["8888:8888"]
    volumes:
      - ./notebooks:/home/jovyan/work
    environment:
      - MLFLOW_TRACKING_URI=http://mlflow:5000
    command: ["start-notebook.sh", "--NotebookApp.token="]
  mlflow:
    image: ghcr.io/mlflow/mlflow:v2.8.0
    ports: ["5000:5000"]
    environment:
      - BACKEND_STORE_URI=postgresql://mlflow:mlflow@db:5432/mlflow
      - ARTIFACT_ROOT=/mlruns
    volumes:
      - mlruns:/mlruns
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:15
    environment:
      - POSTGRES_DB=mlflow
      - POSTGRES_USER=mlflow
      - POSTGRES_PASSWORD=mlflow
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mlflow -d mlflow"]
      interval: 5s
      timeout: 3s
      retries: 15
    volumes:
      - trackingdb:/var/lib/postgresql/data
volumes:
  mlruns:
  trackingdb:

services:
  worker:
    build: ./worker
    environment:
      - BROKER_URL=amqp://guest:guest@queue:5672/
      - MINIO_ENDPOINT=http://minio:9000
      - MINIO_ACCESS_KEY=minio
      - MINIO_SECRET_KEY=minio123
    depends_on:
      queue:
        condition: service_healthy
      minio:
        condition: service_started
  queue:
    image: rabbitmq:3-management
    ports:
      - "15672:15672"
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "ping"]
      interval: 10s
      timeout: 5s
      retries: 10
  minio:
    image: minio/minio:RELEASE.2023-12-02T10-51-33Z
    command: ["server", "/data", "--console-address", ":9001"]
    environment:
      - MINIO_ROOT_USER=minio
      - MINIO_ROOT_PASSWORD=minio123
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - minio:/data
volumes:
  minio:

Exercises

Do the following practice to make the concepts stick.

Common mistakes and self-check

• Only using depends_on without healthcheck: The app may start before the DB is ready. Fix: add healthcheck and depends_on with condition: service_healthy.
• Hardcoding hostnames like localhost inside containers: Inside a compose network, use service names (db, cache), not localhost.
• Confusing bind mounts and volumes: Bind mounts map local folders; volumes are managed by Docker. For databases, prefer volumes.
• Exposing too many ports: Only publish what you need (e.g., API). Internal services can stay private on the network.
• Secrets in compose.yml: Prefer env_file: .env for local work; do not commit real credentials.
• Not pinning image tags: Use specific versions (postgres:15) to keep environments reproducible.

Practical projects

• Feature Store Sandbox: API + Redis + Postgres with a nightly batch job container
• Experiment Lab: Jupyter + MLflow + Postgres + MinIO for artifact storage
• Inference Bench: API + Nginx reverse proxy + Redis, with a test client container for load tests

Learning path

• Start: Single-service Dockerfile and docker run
• Then: Two-service Compose (API + DB) with healthchecks
• Add: Caching layer, volumes, and .env
• Advance: Profiles, overrides (compose -f base -f override up), and scaling workers
• Polish: Makefiles or scripts to wrap common compose commands

Next steps

• Convert an existing manual setup into a compose stack
• Add healthchecks and simplify local onboarding with one command
• Introduce a test container that runs integration tests against the stack

Mini challenge

Add a fourth service named tester that waits for api:8000/health to return OK, then runs a simple HTTP check and exits with code 0. Make api depend only on db; tester should depend on api readiness. Hint: use a tiny curl image and a retry loop in the command.

Quick test and progress

You can take the Quick Test below to check your understanding. It’s available to everyone. Only logged-in users will have their progress saved.