Why this matters
As an Applied Scientist, you will make decisions that impact product launches and experiments. Cross-validation (CV) and robust evaluation help you avoid overfitting, choose models confidently, and estimate real-world performance before shipping.
- Selecting a ranking model that actually improves conversions, not just offline metrics.
- Forecasting demand without leaking future information.
- Tuning hyperparameters so improvements are real, not lucky noise.
- Reporting uncertainty (e.g., mean and confidence interval) so stakeholders understand risk.
Who this is for
- Applied Scientists and ML Engineers building and evaluating models.
- Data Scientists moving from analysis to production ML.
- Researchers running offline evaluations before live tests.
Prerequisites
- Basic Python/ML workflow knowledge (train/validation/test split, fitting models).
- Understanding of common metrics (accuracy, AUC, RMSE, F1).
- Awareness of data leakage and why it’s harmful.
Concept explained simply
Cross-validation estimates how a model will perform on new data by repeatedly training on one part of the data and validating on another. You rotate which part is held out, then aggregate results to get a stable estimate.
- K-fold CV: split data into K folds; train on K-1, validate on the held-out fold. Repeat for all folds and average.
- Stratified K-fold: preserves class ratios in each fold (important for imbalanced classification).
- Group K-fold: keeps all samples from the same group (e.g., user, session) in the same fold to avoid leakage.
- Time series CV: use past to predict future with ordered, non-overlapping splits; no shuffling.
- Nested CV: inner CV selects hyperparameters; outer CV estimates generalization. Prevents optimistic bias.
Mental model
Imagine your dataset is a limited budget. Every time you validate, you spend some budget to estimate performance. CV stretches that budget by rotating the validation set, giving you multiple, independent looks at performance. The more consistent the results across folds, the more confident you can be.
Robust evaluation
- Report the distribution: mean, standard deviation, and 95% confidence interval across folds.
- Use the right split strategy for your data (i.i.d., grouped, or temporal).
- Pick metrics that reflect business goals (e.g., PR-AUC for rare positives, RMSE for forecasting).
- Prevent leakage: fit preprocessing inside the fold only (e.g., scalers, encoders).
- Stability checks: run repeated CV or different seeds and check variance.
Worked examples
1) Stratified K-fold for imbalanced classification
Task: Predict churn with 8% positives. Metric: PR-AUC or F1 at a chosen threshold.
- Use stratified 5-fold CV to keep the 8% rate per fold.
- Pipeline per fold: fit scaler and model on the training split only.
- Aggregate PR-AUC across folds: report mean and 95% CI.
Why: Random splits could distort class balance; stratification stabilizes estimates.
2) Group K-fold to avoid user leakage
Task: Predict whether a user will click in the next session. Multiple sessions per user.
- Define group by user_id; use GroupKFold with K=5.
- Ensure no user appears in both train and validation split.
- Metric: ROC-AUC or calibrated log loss. Aggregate across folds.
Why: Without grouping, the model may memorize user-specific behavior and overestimate performance.
3) Time series CV with expanding window
Task: Forecast weekly demand.
- Sort by week. Create splits: Train [weeks 1-50] validate [51-55]; Train [1-55] validate [56-60]; Train [1-60] validate [61-65], etc.
- No shuffling, no future leakage in features.
- Metric: MAPE or RMSE. Aggregate across windows.
Why: This mirrors real deployment: learn from the past to predict the future.
4) Nested CV for model selection
Task: Choose between Gradient Boosting and Random Forest with hyperparameter tuning.
- Outer CV (K=5) estimates generalization.
- Inside each outer train split, run inner CV (K=3) to tune hyperparameters.
- Train the best inner config on the outer train split; evaluate on the outer validation split.
- Aggregate outer scores for the final unbiased estimate.
Why: Tuning on the same validation data used for reporting inflates scores. Nested CV fixes this.
Practical setup: choose the right CV
- i.i.d. classification/regression: K-fold (K=5 or 10). For imbalance, use stratified.
- Grouped samples (users, devices, sessions): GroupKFold by group id.
- Time series: Expanding or rolling window CV; never shuffle across time.
- Small data: Prefer higher K (e.g., 10) or repeated CV to reduce variance.
- Large data: 3–5 folds often suffice; consider a single holdout + repeated seeds if compute is tight.
Steps to implement robust CV
- Define target metric(s) that match the business goal.
- Pick a split strategy (K-fold, stratified, group, or time series).
- Build a pipeline so all preprocessing fits only on training folds.
- Run CV, store per-fold scores and, if needed, per-fold predictions.
- Aggregate: report mean, standard deviation, and 95% CI (mean ± 1.96 × SE).
- Sanity checks: compare to simple baselines; check variance across folds.
Common mistakes and self-checks
- Leakage via preprocessing: Did you fit scalers/encoders on the full dataset? Fix: Fit inside each fold only.
- Shuffling time series: Any future data in training? Fix: Use ordered splits.
- Ignoring groups: Do any users appear in train and validation? Fix: GroupKFold.
- Reporting only the best fold: Always report aggregated stats across folds.
- Tuning and evaluating on the same folds: Use nested CV or a separate test set.
- Wrong metric for the problem: For rare positives, prefer PR-AUC or recall at K over accuracy.
Quick self-audit checklist
- [ ] Correct split strategy for data structure (i.i.d., grouped, temporal)
- [ ] Preprocessing inside folds only
- [ ] Aggregated metrics with uncertainty (mean, sd, 95% CI)
- [ ] Sanity baseline compared
- [ ] No leakage (time, target, or identity)
Exercises
Try the tasks below. Open the solutions after attempting.
Exercise 1: Pick the right CV
Three scenarios need the correct CV strategy and metric choice. Write your plan.
- A) Credit default prediction (3% positives) with tabular features.
- B) Recommendation clicks with multiple sessions per user.
- C) Monthly sales forecast for 4 years of data.
Exercise 2: Aggregate CV results
Given 5-fold F1 scores: [0.71, 0.69, 0.74, 0.72, 0.70]. Compute mean, standard deviation, standard error, and 95% CI.
Exercise 3: Spot leakage
Mark which steps create leakage in a classification pipeline:
- 1) StandardScaler fit on full dataset, then split.
- 2) Target encoding computed per fold using only training data for that fold.
- 3) Oversampling positives performed before splitting.
- 4) Time series shuffled before splitting.
Show exercise solutions
Exercise 1 Solution
A) Stratified K-fold (e.g., K=5) with PR-AUC or recall@precision≥X; consider class weighting. B) GroupKFold by user_id; ROC-AUC or calibrated log loss; also CTR@K. C) TimeSeriesSplit with expanding window; RMSE or MAPE; no shuffling.
Exercise 2 Solution
Mean = 0.712. Sample sd ≈ 0.019. SE = sd / sqrt(5) ≈ 0.0086. 95% CI ≈ 0.712 ± 1.96×0.0086 ≈ [0.695, 0.729].
Exercise 3 Solution
Leakage: 1) Yes. 2) No (correct). 3) Yes (must resample inside each training fold). 4) Yes for time series (breaks temporal order).
Practical projects
- Offline ranking evaluation: Use GroupKFold by user to compare two ranking models with NDCG@K and report mean ± 95% CI.
- Nested CV model selection: Compare tuned XGBoost vs. Random Forest with 5×3 nested CV; present unbiased outer-fold AUC and confidence interval.
- Demand forecasting: Implement expanding-window CV for weekly data; compare naive seasonal baseline vs. your model; report RMSE across windows.
Learning path
- Start with K-fold and stratified K-fold on a small classification task.
- Learn to build pipelines so preprocessing happens inside folds.
- Handle grouped data with GroupKFold.
- Handle time-dependent data with TimeSeriesSplit.
- Add uncertainty reporting (sd, SE, CI) and stability checks.
- Adopt nested CV for model selection when you must report unbiased estimates.
Mini challenge
You have 8k labeled rows for churn (5% positives), several categorical features, and some user IDs with multiple rows. Propose an evaluation plan that avoids leakage, selects hyperparameters, and reports uncertainty. Specify split type, metric(s), K, and whether you’ll use nested CV or a final holdout.
Hint
Consider stratification, grouping, and whether your final report should come from nested CV or a held-out test set untouched by tuning.
Next steps
- Use repeated CV to assess stability under different seeds.
- Combine CV with Bayesian hyperparameter optimization.
- Explore cross-fitting and out-of-fold predictions for stacking.
- Add calibration checks (reliability curves, ECE) alongside CV.
- For time series, compare expanding vs. rolling windows and test features that strictly use past data.
Quick Test
You can take the test for free. Log in to save your progress and resume later.