Python

Why Python matters for ML Engineers

Python is the backbone of most ML workflows. It lets you explore data quickly, train and evaluate models, write production-grade pipelines, and integrate with services. Mastering Python means you can move from a notebook demo to a reliable, testable system that ships to users.

• Feature engineering and data wrangling
• Training/evaluating models with reproducible code
• Building pipelines and services (batch and real-time)
• Profiling, testing, logging, and packaging for production

Who this is for

Aspiring and junior Machine Learning Engineers who want to build reliable ML pipelines and services, and data scientists transitioning toward production engineering.

Prerequisites

• Comfort with basic programming concepts (variables, loops, functions)
• Familiarity with NumPy/pandas and basic ML concepts (train/test split, metrics)
• Python 3.10+ installed on your machine

Learning path (roadmap)

Worked examples

1) Train a basic classifier with a clean pipeline

Demonstrates: data handling, pipeline, evaluation.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

# Synthetic dataset
data = {
    "x1": np.random.normal(0, 1, 500),
    "x2": np.random.normal(2, 1.5, 500)
}
df = pd.DataFrame(data)
df["y"] = (df["x1"] * 0.8 + df["x2"] * -0.6 + np.random.normal(0, 0.5, 500) < 0).astype(int)

X = df[["x1", "x2"]]
y = df["y"]

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000))
])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
pipe.fit(X_train, y_train)

pred = pipe.predict(X_test)
print(classification_report(y_test, pred))

2) A typed, reusable transformer (no heavy libs)

Demonstrates: type hints, dataclass, API design.

from __future__ import annotations
from dataclasses import dataclass
from typing import Optional
import numpy as np

@dataclass
class Standardize:
    mean_: Optional[np.ndarray] = None
    std_: Optional[np.ndarray] = None

    def fit(self, X: np.ndarray) -> Standardize:
        self.mean_ = X.mean(axis=0)
        self.std_ = X.std(axis=0) + 1e-8
        return self

    def transform(self, X: np.ndarray) -> np.ndarray:
        assert self.mean_ is not None and self.std_ is not None, "Call fit first"
        return (X - self.mean_) / self.std_

    def fit_transform(self, X: np.ndarray) -> np.ndarray:
        return self.fit(X).transform(X)

X = np.random.randn(5, 3)
scaler = Standardize()
Xt = scaler.fit_transform(X)
print(Xt.mean(axis=0).round(5), Xt.std(axis=0).round(5))

3) Async for I/O-bound tasks (simulated)

Demonstrates: asyncio for parallel I/O (e.g., fetching features from services).

import asyncio

async def fetch(name: str, delay: float) -> str:
    await asyncio.sleep(delay)
    return f"{name} ready"

async def main():
    results = await asyncio.gather(
        fetch("embedding", 1.2),
        fetch("classifier", 0.8),
        fetch("vectorizer", 0.5),
    )
    print(results)

asyncio.run(main())

4) Pytest example: parametrized tests

Demonstrates: quick validation of utility functions.

# app/text.py
from typing import List

def tokenize(text: str) -> List[str]:
    return [t for t in text.lower().split() if t]

# tests/test_text.py
import pytest
from app.text import tokenize

@pytest.mark.parametrize("inp, expected", [
    ("Hello  ML", ["hello", "ml"]),
    ("A  B  C", ["a", "b", "c"]),
])
def test_tokenize(inp, expected):
    assert tokenize(inp) == expected

5) Profiling and speeding up code

Demonstrates: cProfile and vectorization.

import cProfile, pstats, io
import numpy as np

N = 2_000_00

def slow_sum_squares(n: int) -> int:
    s = 0
    for i in range(n):
        s += i * i
    return s

pr = cProfile.Profile()
pr.enable()
slow_sum_squares(200_000)
pr.disable()
s = io.StringIO()
pstats.Stats(pr, stream=s).sort_stats("tottime").print_stats(5)
print(s.getvalue())

# Vectorized alternative
arr = np.arange(200_000)
print(int((arr * arr).sum()))

Drills and micro-exercises

• Create a virtual environment, install numpy and scikit-learn, and freeze requirements to a file
• Write a function to compute F1 score from precision and recall with type hints and 3 unit tests
• Add logging to a small script: INFO for progress, DEBUG for shapes/parameters, WARNING for suspicious values
• Use timeit to compare list comprehension vs numpy vectorization for squaring 1e6 integers
• Practice a Strategy pattern: implement Normalizer, Standardizer, and MinMax transformers with a common interface
• Write an async function that concurrently reads three local files (simulate with asyncio.sleep) and aggregates their content

Common mistakes and how to debug

• Data leakage: fitting scalers on full data before split
• Unpinned dependencies: different environments produce different results
• Silent failures: print statements instead of structured logging
• Slow Python loops where vectorization would help
• Unclear project layout: hard to test or package

Mini project: Spam classifier pipeline

Goal: build an end-to-end text classifier with solid engineering practices.

1. Project setup: create a virtual environment; add pyproject.toml or requirements.txt; choose a src/ layout
2. Preprocessing module: tokenize, lowercase, simple n-gram features; add type hints and unit tests
3. Model module: pipeline with a vectorizer and logistic regression; add a CLI entry point to train and evaluate
4. Configuration: support a YAML/JSON config to choose vectorizer hyperparameters
5. Logging: log metrics, class distribution, and key hyperparameters
6. Profiling: profile training on 10k samples; optimize any slow steps
7. Packaging: package as an installable module; run unit tests on install

Subskills

• Production Quality Code Style — Write readable, consistent code (PEP 8) that teammates can maintain.
• OOP And Design Patterns Basics — Structure ML components (transformers, trainers) with clear interfaces.
• Type Hints And Linting — Catch bugs early and enable better tooling and IDE support.
• Virtual Environments And Dependency Management — Reproducible experiments and deployments.
• Data Structures And Algorithms Basics — Choose the right container/algorithm for performance.
• Async And Concurrency Basics — Speed up I/O-bound steps like remote feature fetching.
• Logging And Config Management — Traceability across runs and environments.
• Testing With Pytest — Confidence to refactor and ship.
• Performance Profiling And Optimization — Focus on the bottlenecks that matter.
• Packaging And Project Structure — Ship code as reusable modules.

Next steps

• Pick one subskill below and complete its mini tasks
• Finish the mini project with tests and logging
• When ready, take the skill exam to validate your knowledge

Skill exam

The exam is available to everyone. Only logged-in users will have their progress saved.