Dependency Management

Why this matters

As a Data Engineer, you ship code to many places: Airflow workers, Spark clusters, containers, and servers. Each place needs the exact same dependencies to run reliably. Good dependency management prevents brittle pipelines, avoids “works on my machine,” and makes rollbacks and audits straightforward.

• Keep ETL jobs reproducible across dev, CI, and prod.
• Pin and lock versions to prevent surprise breaks from upstream releases.
• Deploy containers with deterministic builds for Spark/Batch/Streaming tasks.
• Meet security expectations by verifying hashes and using known-good versions.

Concept explained simply

Dependencies are the external pieces your code needs: libraries, system packages, connectors. Managing them means choosing versions, recording them, installing them the same way everywhere, and updating them safely.

Mental model

• Recipe: You write a short list of ingredients you want (top-level dependencies).
• Lock: A precise shopping list the store can fulfill exactly (resolved, pinned, hashed versions).
• Kitchen: An isolated place to cook (virtualenv/conda or a container).
• Meal: The built artifact or image that anyone can run, anywhere.

Core practices (what to actually do)

• Use isolation: virtual environments (Python) or containers for every project.
• Prefer pins and lock files: avoid unbounded ranges like ">=" or "latest" in production builds.
• Separate intent vs resolution: keep a short "requirements.in" (or pyproject dependencies) and a locked "requirements.txt" or lockfile.
• Reproducible installs: use hashes and deterministic flags.
• Handle system deps explicitly: pin OS packages and native libs in Dockerfiles.
• Cache wisely: cache wheels and artifacts; avoid caching the internet (keep lock files authoritative).
• Upgrade on purpose: schedule dependency reviews; update in small chunks; test and rollback.

Worked examples

Example 1 — Python ETL with pip-tools (deterministic installs)

1. Create an isolated env.

   python -m venv .venv
   source .venv/bin/activate  # Windows: .venv\Scripts\activate

2. Write top-level intent in requirements.in.

   # requirements.in
   pandas==2.2.*
   requests>=2.31,<2.33
   pyarrow==14.*

3. Compile a locked file with hashes.

   python -m pip install --upgrade pip pip-tools
   pip-compile requirements.in \
     --generate-hashes \
     -o requirements.txt

4. Install exactly what’s locked.

   pip-sync requirements.txt

5. Sanity check imports.

   python -c "import pandas, requests, pyarrow; print('ok')"

This approach prevents accidental upgrades and ensures every environment matches.

Example 2 — Airflow + provider with a constraints file (no online lookups required)

1. Create a local constraints file for your chosen Airflow version (example versions only).

   # constraints-airflow-2.8.4.txt
   apache-airflow==2.8.4
   apache-airflow-providers-amazon==8.16.0
   click==8.1.7
   pendulum==3.0.0
   # ... include other resolved pins as needed

2. In your Dockerfile, install using the constraints file.

   # Dockerfile (snippet)
   FROM python:3.10-slim
   WORKDIR /app
   COPY constraints-airflow-2.8.4.txt /app/
   RUN python -m pip install --upgrade pip \
       && pip install "apache-airflow==2.8.4" \
              "apache-airflow-providers-amazon==8.16.0" \
              -c constraints-airflow-2.8.4.txt \
       && airflow version

Using a constraints file prevents version drift and keeps provider versions compatible.

Example 3 — Docker with pinned OS packages (Spark job runner)

1. Create a minimal, pinned base.

   FROM python:3.10-slim
   ARG DEBIAN_FRONTEND=noninteractive
   RUN apt-get update \
     && apt-get install -y --no-install-recommends \
        openjdk-17-jre-headless=17.* \
        libkrb5-3=1.20.* \
     && rm -rf /var/lib/apt/lists/*
   ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
   

2. Pin Python deps with a lock file, then install.
3. Copy your Spark job and run with the exact JVM + Python versions you expect.

Pinned OS packages avoid subtle runtime issues on different hosts.

Example 4 — Scala Spark with shaded fat JAR (avoid transitive clashes)

1. In build tool (e.g., sbt/Gradle), fix versions for Spark and connectors.
2. Shade or relocate conflicting transitive deps (e.g., guava) to prevent runtime classpath conflicts.
3. Publish one fat JAR; the cluster sees one deterministic artifact.

Shading isolates your dependency universe from the cluster’s.

Exercises

Complete the tasks below. You can compare with the solutions in each exercise card. Use the checklist to track progress.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip pip-tools
printf "pandas==2.2.*\nrequests>=2.31,<2.33\npyarrow==14.*\n" > requirements.in
pip-compile requirements.in --generate-hashes -o requirements.txt
pip-sync requirements.txt
python -c "import pandas, requests, pyarrow; print('ok')"

# constraints-airflow-2.8.4.txt (example)
apache-airflow==2.8.4
apache-airflow-providers-amazon==8.16.0
click==8.1.7
pendulum==3.0.0

# Dockerfile (example)
FROM python:3.10-slim
WORKDIR /app
COPY constraints-airflow-2.8.4.txt /app/
RUN python -m pip install --upgrade pip \
    && pip install "apache-airflow==2.8.4" \
           "apache-airflow-providers-amazon==8.16.0" \
           -c constraints-airflow-2.8.4.txt \
    && airflow version