Applied ML Modeling

What is Applied ML Modeling for Applied Scientists?

Applied ML Modeling is the end-to-end practice of selecting, training, validating, and shipping models that solve real product and business problems. For an Applied Scientist, this means you not only know algorithms—you can choose the right family for the task, make it robust to real data issues, trade off objectives, explain results, and iterate fast.

Why it matters in this role

• Translate ambiguous requirements into measurable modeling goals and metrics.
• Ship models that work in production, not just in notebooks.
• Balance accuracy with latency, fairness, interpretability, and maintainability.

What you’ll be able to do

• Pick appropriate model families for classification, regression, ranking, sequence, and vision tasks.
• Apply classical ML, deep learning, or hybrid approaches based on data scale and constraints.
• Use representation learning, transfer learning, and fine-tuning effectively.
• Handle imbalanced and noisy datasets with practical techniques and metrics.
• Balance multiple objectives (e.g., accuracy, latency, fairness, LTV) in training and evaluation.
• Explain model behavior with interpretable features and model-agnostic tools.

Who this is for

• Early-career Applied Scientists preparing to own modeling projects end-to-end.
• Data Scientists moving from analysis to production ML.
• ML Engineers who want stronger modeling heuristics and evaluation skills.

Prerequisites

• Comfort with Python and NumPy/Pandas.
• Basic ML concepts: train/val/test, overfitting, cross-validation, common metrics.
• Familiarity with scikit-learn; beginner knowledge of PyTorch or TensorFlow helps.

Learning path (roadmap)

1. Frame the problem
   Define target, constraints (latency, memory), and primary/secondary metrics. Establish a simple baseline.
2. Model family selection
   Choose classical vs deep learning vs hybrid based on data size, structure, and constraints.
3. Generalization and regularization
   Use cross-validation, early stopping, weight decay, dropout, and data augmentation where relevant.
4. Data challenges
   Treat imbalance (sampling, class weights, thresholds), label noise (loss tweaks, robust training), leakage checks.
5. Representation and transfer
   Use pretrained embeddings and fine-tune for your domain. Start with head-only, then unfreeze progressively.
6. Multi-task/multi-objective
   Set up shared backbones, task heads, and weighted losses aligned to product goals.
7. Interpretability
   Use permutation importance, partial dependence, SHAP-like reasoning, and honest caveats.
8. Evaluation to decision
   Calibrate probabilities, choose thresholds with cost curves, run A/B guardrails, and plan rollouts.

Mini check: are you ready to move on?

• Baseline and metric defined.
• Rationale for model family documented.
• Cross-validation and leakage checks passing.
• Class imbalance plan chosen.
• Interpretability artifacts drafted.

Worked examples (with code)

1) Tabular binary classification baseline

Goal: Fast, strong baseline with scikit-learn. Start with Logistic Regression and compare to Random Forest.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

X = pd.read_csv('features.csv')
y = pd.read_csv('labels.csv').values.ravel()

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

logit = Pipeline([
    ('scaler', StandardScaler(with_mean=False)),
    ('clf', LogisticRegression(max_iter=200, class_weight='balanced'))
])
rf = RandomForestClassifier(n_estimators=300, random_state=42, n_jobs=-1, class_weight='balanced')

logit.fit(X_train, y_train)
rf.fit(X_train, y_train)

p1 = logit.predict_proba(X_test)[:,1]
p2 = rf.predict_proba(X_test)[:,1]
print('AUC logit:', roc_auc_score(y_test, p1))
print('AUC rf   :', roc_auc_score(y_test, p2))

Why this works

• Logistic is a strong linear baseline; Random Forest adds nonlinearity and interactions.
• class_weight='balanced' deals with class imbalance quickly.

2) Imbalance handling and threshold tuning

Optimize threshold for F1 or precision @ desired recall. Use validation set to select threshold.

import numpy as np
from sklearn.metrics import precision_recall_curve, f1_score

probs = p2  # from previous RF
prec, rec, thr = precision_recall_curve(y_test, probs)
f1s = 2 * (prec * rec) / (prec + rec + 1e-9)
best_idx = np.argmax(f1s)
print('Best F1:', f1s[best_idx], 'Threshold:', thr[best_idx])

# Or fix recall target
target_recall = 0.90
idx = np.where(rec >= target_recall)[0]
if len(idx):
    i = idx[0]
    print('First threshold hitting recall 0.90:', thr[i], 'precision:', prec[i])

Notes

• For rare positives, AUC-PR is often more informative than ROC-AUC.
• Cost-sensitive apps should map threshold to business costs.

3) Regularization and cross-validation

Compare L1 vs L2 and find C via grid search.

from sklearn.model_selection import StratifiedKFold, GridSearchCV

pipe = Pipeline([
    ('scaler', StandardScaler(with_mean=False)),
    ('clf', LogisticRegression(max_iter=500, solver='liblinear'))
])

param_grid = {
    'clf__penalty': ['l1', 'l2'],
    'clf__C': [0.01, 0.1, 1.0, 10.0]
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

gs = GridSearchCV(pipe, param_grid, scoring='roc_auc', cv=cv, n_jobs=-1)
gs.fit(X_train, y_train)
print('Best:', gs.best_params_, gs.best_score_)
print('Test AUC:', roc_auc_score(y_test, gs.best_estimator_.predict_proba(X_test)[:,1]))

Takeaways

• L1 can induce sparsity (feature selection); L2 distributes weight.
• Use CV for fair model comparison; keep a final test set untouched.

4) Transfer learning: text classification (head-then-unfreeze)

Start with a frozen encoder, train the classifier head, then unfreeze top layers for small, controlled fine-tuning.

# Illustrative snippet (requires transformers & torch)
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
import numpy as np

model_name = 'distilbert-base-uncased'
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
for param in model.base_model.parameters():
    param.requires_grad = False  # freeze encoder initially

# ... prepare tokenized datasets: train_ds, val_ds ...
args = TrainingArguments(output_dir='out', per_device_train_batch_size=32, per_device_eval_batch_size=64, num_train_epochs=2, evaluation_strategy='epoch')
trainer = Trainer(model=model, args=args, train_dataset=train_ds, eval_dataset=val_ds)
trainer.train()

# Unfreeze top layers for a short, low LR fine-tune
for param in model.base_model.layer[-2:].parameters():
    param.requires_grad = True
trainer.args.learning_rate = 2e-5
trainer.train()

Heuristics

• Freeze first to stabilize; unfreeze gradually to avoid catastrophic forgetting.
• Use lower LR for the pretrained backbone than the head.

5) Interpretability: permutation importance and partial dependence

Assess global feature influence and marginal effects.

from sklearn.inspection import permutation_importance, PartialDependenceDisplay
import matplotlib.pyplot as plt

rf.fit(X_train, y_train)
r = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
imp = sorted(zip(X.columns, r.importances_mean), key=lambda x: -x[1])[:10]
print('Top features:', imp)

PartialDependenceDisplay.from_estimator(rf, X_test, features=[imp[0][0]])
plt.show()

Use responsibly

• Permutation importance can be biased by correlated features; interpret groups.
• PDP assumes features vary independently; use ICE for heterogeneity.

Drills and exercises

• Reproduce baseline vs tuned model on a public tabular dataset; report AUC-PR and calibration error.
• Implement threshold tuning to meet a recall target; show cost impact with a confusion matrix.
• Run CV comparing L1 vs L2 regularization; plot coefficient paths for top features.
• Take a small text dataset; fine-tune a pretrained encoder with head-only vs partial unfreeze; compare results.
• Compute permutation importance and one PDP; write a 3-sentence caveat on limitations.

Common mistakes and debugging tips

• Leakage from future or target-derived features: Rebuild pipeline so transforms fit only on training folds; verify with CV.
• Overfitting to validation: Too many manual threshold tweaks on the same split. Use nested CV or a fresh holdout.
• Wrong metric for imbalance: Accuracy can mislead; prefer AUC-PR, recall@k, cost-weighted metrics.
• Unstable fine-tuning: Learning rate too high when unfreezing; reduce LR and use layer-wise decay.
• Ignoring latency/size: Measure inference time and memory early; add constraints to model selection.
• Misinterpreting feature importance: Correlations inflate importance. Check redundancy and consider grouped perturbations.

Mini project: Fraud detection pipeline

Build an end-to-end binary classifier for rare-event fraud.

• Data: Transactions with user, merchant, amount, time-based features.
• Goal metric: Maximize AUC-PR; deploy threshold to achieve ≥ 0.9 recall with best precision.
• Constraints: P95 latency < 20 ms per prediction, model size < 50 MB.

Steps

1. Establish a Logistic Regression baseline with class weighting; record metrics.
2. Engineer simple aggregates (user/merchant frequency), then train Gradient Boosting or Random Forest.
3. Tune threshold on validation to meet recall 0.9; report precision and cost impact.
4. Compute permutation importance and one PDP for top feature.
5. Measure inference latency on a sample batch; optimize by pruning features or trees.

Deliverables

• Short report: problem framing, metrics, model choices, and trade-offs.
• Notebook/script with reproducible CV and threshold selection.
• Interpretability artifacts (importance + PDP) with limitations.

Subskills

• Choosing Model Families For Task: Map task type and constraints to linear models, tree ensembles, sequence models, or ranking models.
• Classical ML And Deep Learning Selection: Decide when to use scikit-learn baselines vs neural networks given data scale and structure.
• Representation Learning Concepts: Understand embeddings, encoders, and learned features vs manual features.
• Fine Tuning And Transfer Learning: Freeze, head-train, then unfreeze with careful learning rates and regularization.
• Regularization And Generalization: Apply CV, weight decay, dropout, early stopping, and data augmentation strategies.
• Handling Imbalanced And Noisy Data: Use sampling, class weights, robust losses, and threshold tuning with cost-aware metrics.
• Multi Task And Multi Objective Modeling Basics: Share backbones, tune loss weights, and evaluate across tasks and constraints.
• Interpretability Techniques Basics: Employ permutation importance, PDP/ICE, and transparent reporting.

Practical projects

• Ranking: Build a learning-to-rank model for search results; evaluate NDCG@k vs latency.
• Customer churn: Gradient boosting baseline, calibrate probabilities, and design retention thresholds.
• Image quality triage: Fine-tune a lightweight CNN; trade off accuracy vs on-device constraints.

Next steps

• Take the skill exam below to check mastery. Everyone can take it for free; logged-in users get saved progress.
• Pick one practical project and complete it end-to-end with a written rationale for choices.
• Iterate on interpretability and thresholding to align with realistic constraints.