Feature Importance And Explainability Basics

Who this is for

Machine Learning Engineers who need to explain model behavior to teammates, stakeholders, or regulators, and to debug models beyond raw metrics.

Prerequisites

• Basic Python ML workflow (fit, predict, evaluate) in a common framework (scikit-learn, XGBoost, LightGBM).
• Familiarity with classification or regression metrics (accuracy, ROC-AUC, MAE).
• Comfort with tabular datasets and feature preprocessing.

Why this matters

In real work you will:

• Justify why a credit model denied a customer.
• Debug why a churn model relies heavily on a leaked feature.
• Detect bias (e.g., protected attributes influencing predictions).
• Improve performance by identifying useless or unstable features.

Concept explained simply

Feature importance and explainability tools tell you which inputs mattered and how. Global tools summarize the model across the whole dataset. Local tools explain a single prediction.

Mental model

Imagine your model as a committee. Global importance shows who talks most across all meetings. Local explanations show who spoke up for one decision today, and whether they argued for or against it.

Core methods you should know

• Linear coefficients (standardized): global direction and magnitude per feature.
• Impurity-based importance (trees): split gain frequency; fast but biased toward high-cardinality features.
• Permutation importance: drop in metric when shuffling a feature; model-agnostic global measure.
• Partial Dependence (PDP) and ICE: global and per-row view of how changing one feature moves predictions.
• LIME: local surrogate model near one instance.
• SHAP: local additive attributions with consistency; TreeSHAP is efficient for tree models.

Worked examples

Example 1: Linear model (standardized coefficients)

Train a standardized logistic regression for churn. Inspect coefficients: positive means higher churn risk. Because features are standardized, magnitudes are comparable. Age: -0.8 might reduce risk; Tenure: -1.1; Complaints: +1.5 (strong driver).

Example 2: Permutation importance on RandomForest

After fitting a RandomForest, compute permutation importance on a validation set using accuracy or ROC-AUC. You see top drops for Tenure, ContractType, and SupportCalls. When you permute Tenure, ROC-AUC falls from 0.86 to 0.79 (drop 0.07) — strong global impact.

Example 3: Local SHAP for a single prediction

For an XGBoost credit model, compute TreeSHAP for one applicant. Baseline prediction is 0.12 default probability. After attributions: High Utilization (+0.08), Recent Late Payment (+0.06), Long Credit History (-0.03). Final prediction ≈ 0.23. This cleanly explains the decision.

How to choose the right tool

• If you need quick global signals: permutation importance.
• If the model is linear: standardized coefficients + PDP.
• If the model is trees (RF/GBM): TreeSHAP for local and global; sanity-check with permutation importance.
• If stakeholders want a simple storyline for one case: SHAP or LIME (local).
• If you suspect non-linear effects: PDP (global) + ICE (local) to inspect shape and heterogeneity.

Step-by-step workflow

1. Lock a validation set (no peeking).
2. Compute a baseline metric (ROC-AUC/MAE).
3. Get a fast global view: permutation importance (and impurity for trees, but mark bias risk).
4. Drill into shapes: PDP/ICE for top 3 features.
5. Generate local explanations (SHAP/LIME) for correctly and incorrectly predicted cases.
6. Stress-test: correlated features, feature removal, and random label sanity check.
7. Communicate: one global chart + two local case studies + clear caveats.

Hands-on exercises

These mirror the tasks below in the Exercises section (your progress is saved only if logged in).

1. ex1: Compare impurity vs permutation importance on a tree model and explain discrepancies.
2. ex2: Produce a local SHAP explanation for one prediction and summarize it in 3 sentences.

• Checklist: Use a held-out set; report metric drops; check correlated features; include at least one local explanation; document caveats.

Common mistakes and self-check

• Using impurity importance alone. Self-check: do permutation importance disagree? If yes, trust permutation.
• Comparing linear coefficients without standardizing. Self-check: confirm zero mean and unit variance.
• Ignoring correlation. Self-check: compute a correlation matrix; look for importance splitting or instability.
• Reading PDPs literally when features interact heavily. Self-check: plot ICE to see heterogeneity.
• Explaining bad data. Self-check: run a data leakage audit and a random-label sanity test.

Practical projects

• Model report card: one page with metric, top-5 permutation features, two PDPs, two ICE lines, and two SHAP local cases.
• Leakage hunt: remove or mask each top-5 feature and re-evaluate; document drops and conclusions.
• Fairness scan: compare SHAP attributions across groups; if a sensitive proxy appears, mitigate and re-measure.

Learning path

1. Start with permutation importance and standardized coefficients.
2. Add PDP/ICE for top features to learn effect shapes.
3. Adopt SHAP for local explanations, then aggregate globally.
4. Practice communication: concise charts + plain language summaries.

Next steps

• Automate a validation-time explanation report for every model build.
• Establish a checklist for correlation, leakage, and sanity tests.
• Share a short explainer with non-technical stakeholders using local cases.

Mini challenge

Pick one of your models. In 60 minutes, produce: top-5 permutation features, one PDP and one ICE for the most important feature, and a SHAP explanation for a borderline case. Write a 5-line summary a product manager can read.

Quick Test (available to everyone; only logged-in users get saved progress)

Take the quick test below to check your understanding. Aim for 70% or higher.