Experimentation And Evaluation

Why this skill matters for Applied Scientists

Experimentation and evaluation turn models into impact. As an Applied Scientist, you will choose the right metrics, validate improvements with robust offline tests, run trustworthy online experiments, and explain results to product partners. Mastery here prevents costly launches, speeds iteration, and builds confidence in your model decisions.

Who this is for

• Applied Scientists and ML Engineers building user-facing features.
• Data Scientists moving from analysis to model-driven product work.
• Researchers translating prototypes into production improvements.

Prerequisites

• Comfort with Python and common ML libraries (numpy, pandas, scikit-learn).
• Basic statistics: distributions, hypothesis testing, confidence intervals.
• Familiarity with ML tasks (classification, ranking, recommendation, generation).

Learning path (practical roadmap)

1. Align metrics with outcomes
   Map product goals to evaluation metrics (e.g., PR-AUC for rare positives, NDCG for ranking, human rubrics for quality).
2. Establish robust offline evaluation
   Use leakage-safe splits, cross-validation, and ablations to choose models and hyperparameters.
3. Design trustworthy experiments
   Define hypotheses, power and sample size, guardrails, and stopping rules. Plan for multiple comparisons.
4. Deep-dive diagnostics
   Run error analysis by slice, fairness checks, and stress tests across distributions and perturbations.
5. Reproducible tracking
   Version data, code, metrics, and experiment configs. Log seeds and environment. Make results explorable.

Worked examples

1) Choosing the right metric for imbalanced classification

Scenario: Fraud detection with 0.5% positives. Accuracy is misleading.

• Bad: Accuracy (predicting all negatives gives 99.5% accuracy).
• Better: Precision-Recall AUC (PR-AUC); also track recall at fixed precision.

python
# Compute PR-AUC and recall@precision
from sklearn.metrics import average_precision_score, precision_recall_curve

y_true = ...  # 0/1 labels
scores = ...  # predicted probabilities

ap = average_precision_score(y_true, scores)
precision, recall, thresholds = precision_recall_curve(y_true, scores)
# Recall at 95% precision
import numpy as np
idx = np.where(precision >= 0.95)[0]
recall_at_95p = recall[idx].max() if len(idx) else 0.0
print({"ap": ap, "recall@95p": recall_at_95p})

2) Cross-validation without leakage (grouped by user)

Scenario: You predict next purchase for users with many rows each. Splitting randomly leaks user signals across train/test.

python
from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, make_scorer

X = ...  # features
y = ...  # labels
user_id = ...  # group key per row

clf = RandomForestClassifier(n_estimators=200, random_state=0)
cv = GroupKFold(n_splits=5)
auc = cross_val_score(clf, X, y, groups=user_id, cv=cv,
                      scoring=make_scorer(roc_auc_score, needs_proba=True))
print({"mean_auc": auc.mean(), "std_auc": auc.std()})

3) Ablation study to justify complexity

Scenario: Your model stack uses embeddings, features, and a re-ranker. Show which parts matter.

• Train baseline (handcrafted features).
• Add embeddings; measure delta metrics.
• Add re-ranker; measure delta again.

python
configs = [
    {"name": "baseline", "embeddings": False, "reranker": False},
    {"name": "+embeddings", "embeddings": True,  "reranker": False},
    {"name": "+embeddings+reranker", "embeddings": True, "reranker": True},
]

results = []
for c in configs:
    # train_eval(c) returns dict with metrics, e.g., ndcg@10
    res = train_eval(config=c)
    results.append({"config": c["name"], **res})

for r in results:
    print(r)

Conclude with effect sizes and cost/latency trade-offs.

4) Powering an A/B test with guardrails

Scenario: You expect +2% relative lift in conversion, baseline 10%.

python
# Approximate sample size per variant for two-proportion Z-test
# Using normal approximation formula (rough):
import math
alpha = 0.05
power = 0.8
p1 = 0.10
p2 = 0.102  # 2% relative lift

# Z-scores (approx):
Z_alpha = 1.96
Z_beta = 0.84
pbar = (p1 + p2)/2
qbar = 1 - pbar
n = 2 * ((Z_alpha*math.sqrt(2*pbar*qbar) + Z_beta*math.sqrt(p1*(1-p1) + p2*(1-p2)))**2) / ((p2 - p1)**2)
print(int(math.ceil(n)))  # samples per variant (rough)

5) Error slicing and robustness checks

Scenario: A content classifier underperforms on short texts and certain languages.

python
import pandas as pd
from sklearn.metrics import f1_score

# df has: y_true, y_pred, text_len_bin, locale
slices = ["text_len_bin", "locale"]
for col in slices:
    print("--", col)
    for k, mini in df.groupby(col):
        f1 = f1_score(mini.y_true, mini.y_pred)
        print(col, k, "F1=", round(f1, 3), "n=", len(mini))

Drills and exercises

• Rewrite your primary metric so a non-technical PM can explain it in a sentence.
• Create a leakage-safe split for your current dataset (grouped or time-based).
• Run a 3-step ablation and record the metric deltas with confidence intervals.
• Design a minimal A/B test plan: hypothesis, MDE, sample size, guardrails, stopping rule.
• Build a slicing report with at least three user segments and one fairness check.
• Script a stress test that perturbs features or inputs and logs robustness metrics.
• Produce a one-page reproducibility note: data snapshot, commit hash, seed, environment.

Common mistakes and debugging tips

• Metric-product misalignment: Optimize proxy metrics that don’t move the business outcome. Fix: add guardrails and validate correlation to the north-star metric.
• Data leakage in splits: Random splits with user or time leakage. Fix: group or time-based CV; verify zero overlap.
• Peeking in experiments: Repeated looks inflate false positives. Fix: fixed horizon or sequential corrections (alpha spending).
• Underpowered tests: Tiny effects need large samples. Fix: raise sample size, reduce variance (CUPED), or aggregate over longer periods.
• Ignoring slice failures: Overall gains can hide regressions. Fix: report slices with CIs; define fail-fast thresholds.
• Non-reproducible runs: Missing seeds, environment drift, or data changes. Fix: log seeds, environment, data version; store configs.

Mini project: Ship a safe model update end-to-end

1. Define success: Primary metric and 2–3 guardrails; write a one-paragraph hypothesis.
2. Offline plan: Time-based or group CV; select metrics; prepare error slicing and robustness scripts.
3. Ablations: Compare baseline, +feature, +reranker; summarize deltas with CIs.
4. Experiment design: Compute sample size for MDE; set stopping rule; list guardrails and fail criteria.
5. Run & analyze: Launch the test; avoid peeking; report primary metric and guardrails with uncertainty.
6. Decide: Ship, iterate, or roll back; capture learnings in a brief, reproducible report.

Practical project ideas

• Ranking: Build a small search or recommendation demo and compare BM25 vs. neural ranker with NDCG and latency guardrails.
• Classification: Deploy a moderation classifier with recall@precision targets; add language and length slices.
• Generation: Evaluate a summarizer with human rubrics, pairwise preferences, and robustness to noisy input.

Subskills

• Offline Evaluation Metrics Selection — Choose metrics that reflect product goals (e.g., PR-AUC, NDCG, recall@precision, calibration). Avoid accuracy traps.
• Benchmarking And Ablations — Compare baselines to variants; quantify each component’s contribution and cost.
• Cross Validation And Robust Evaluation — Use group/time-aware splits; compute stable estimates and CIs.
• Statistical Significance For Experiments — Hypothesis tests, power, sample size, and stopping rules; control false discoveries.
• Error Analysis And Slicing — Inspect failure modes by segment; track fairness and reliability.
• Robustness And Stress Testing — Perturb inputs, simulate drift, and verify graceful degradation.
• Human Evaluation Design — Build rubrics, pairwise comparisons, and blinded protocols; measure agreement.
• Tracking Experiment Results Reproducibly — Version data/code/config; log seeds; produce auditable reports.

Next steps

• Pick one metric upgrade you can ship this week (e.g., replace accuracy with PR-AUC + recall@precision).
• Create a reusable ablation and slicing template for your team.
• Plan your next experiment with clear power calculations and guardrails.