Python (pandas and numpy)

Why this skill matters for a Data Scientist

Pandas and NumPy are your daily drivers for turning raw data into reliable, model-ready features. They let you ingest files and databases quickly, clean and reshape data, engineer features, and perform vectorized numerical computation at scale. Mastering these tools makes you faster, more accurate, and more reproducible—key expectations for a Data Scientist.

• Load: Read CSV/Parquet/SQL efficiently with correct dtypes.
• Wrangle: Filter, join, pivot, group, and aggregate without loops.
• Compute: Use NumPy broadcasting and vectorization for speed.
• Time series: Parse dates, resample, and roll windows correctly.
• Quality: Handle missing data, detect outliers, and avoid chained indexing bugs.
• Reproducibility: Write reusable helpers and keep notebooks deterministic.

What you will be able to do

• Load multi-GB datasets with the right dtypes and memory footprint.
• Transform messy raw data into tidy, analyzable tables.
• Engineer features for ML (encodings, aggregations, time-based features).
• Diagnose issues quickly using .info(), .isna(), .value_counts(), and profiling patterns.
• Write small, tested utility functions you can reuse across notebooks.

Practical roadmap

Worked examples

1) Read CSV safely and efficiently

import pandas as pd

# Good: explicit dtypes and date parsing
schema = {
    "user_id": "int64",
    "country": "category",
    "amount": "float64"
}
df = pd.read_csv(
    "transactions.csv",
    usecols=["user_id", "country", "amount", "ts"],
    dtype=schema,
    parse_dates=["ts"]
)

print(df.dtypes)
print("MB:", round(df.memory_usage(deep=True).sum() / 1e6, 2))

2) Vectorize a conditional calculation with NumPy

import numpy as np

x = np.array([10, 20, 30, 40])
mask = x >= 25
# If x>=25, add 5 else subtract 5
y = np.where(mask, x + 5, x - 5)
print(y)  # [ 5 15 35 45]

3) Groupby with multiple aggregations

import pandas as pd

# Suppose df has columns: user_id, amount, ts (datetime)
by_user = (
    df.groupby("user_id")["amount"]
      .agg(txn_count="count", total_spend="sum", avg_spend="mean")
      .reset_index()
)
print(by_user.head())

4) Forward-fill missing values within each user

df = df.sort_values(["user_id", "ts"])  # critical before ffill
df["amount_ffill"] = (
    df.groupby("user_id")["amount"].ffill()
)
# Optionally fill remaining NaN (e.g., first row per group)
df["amount_ffill"] = df["amount_ffill"].fillna(0.0)

5) Daily resample of event counts

# Ensure datetime is the index
s = df.set_index("ts").assign(count=1)["count"]
per_day = s.resample("D").sum().fillna(0)
print(per_day.tail())

Drills and micro-exercises

• Load a CSV with 1M+ rows, set at least two columns to category, and report memory before/after.
• Replace a Python for-loop transformation with a vectorized NumPy or pandas expression.
• Implement a per-group rolling 7-day sum using groupby + rolling.
• Create at least 3 features with groupby-agg (count, mean, unique count) and merge back.
• Prove you avoided chained indexing by making one .loc assignment that updates in place.

Common mistakes and debugging tips

• Chained indexing: df[df.a > 0]["b"] = 1 silently fails. Use df.loc[df.a > 0, "b"] = 1.
• Wrong dtypes: Objects for numbers slow everything. Fix with to_numeric and category for low-cardinality strings.
• Missing data leakage: Using fillna with global means can leak target information. Compute fills within train folds or by group.
• Time chaos: Forgetting timezone conversion or not sorting before resample/ffill produces incorrect sequences.
• Loop over rows: Avoid iterrows/apply row-wise for large data; prefer vectorization or groupby.

Mini project: Customer purchase trends and features

Goal: Build a tidy feature matrix for predicting whether a user will purchase next week.

1. Load transactions (user_id, ts, amount, category) with proper dtypes and parse ts.
2. Create weekly aggregates per user: count, sum, mean, unique categories.
3. Engineer recency (days since last purchase) and rolling 4-week spend.
4. Join user attributes (join on user_id). Ensure left joins preserve all users.
5. Handle missing: ffill per user where sensible; otherwise fill with zeros.
6. Construct X with numeric features and one-hot encodings for key categories.
7. Export to Parquet (features.parquet) for fast reuse.

Who this is for

• Aspiring or practicing Data Scientists who need strong data wrangling speed.
• Analysts and ML Engineers transitioning into model development.
• Anyone working with medium-to-large tabular or time series data.

Prerequisites

• Basic Python (functions, lists, dicts, modules).
• Comfort with Jupyter or a Python IDE.
• Basic statistics (mean, median) helpful but not required.

Learning path

1. Foundation: Data loading, dtypes, memory.
2. Wrangling essentials: filtering, joins, pivots, groupby.
3. NumPy vectorization: broadcasting, masks, reductions.
4. Missing/time series: ffill/bfill, resample, rolling.
5. Feature matrix: encodings, aggregations, leakage-aware splits.
6. Performance and reuse: categories, .loc, functions, reproducible notebooks.

Practical projects to cement skills

• Sales analytics: Build weekly store-level KPIs and a feature matrix for sales forecasting.
• Web logs: Sessionize clickstream data and compute per-user engagement metrics.
• Sensor time series: Resample, interpolate, and compute rolling anomaly scores.

Next steps

• Practice on a real dataset (100k+ rows). Time your operations; optimize.
• Refactor your notebook logic into a small utility module for reuse.
• Take the skill exam below to validate mastery. Exam is free; logged-in users get saved progress.