Model Interpretability

Three distinct reasons, which want different tools: debugging (is the model right for the right reasons?), trust (will a domain expert act on this output?), and compliance (in credit, insurance, hiring and healthcare, an adverse decision often must come with a stated reason).

Conflating the three produces the wrong artifact. A regulator wants a per-decision reason code; a data scientist chasing a bug wants a global ranking to sort features by suspicion; a clinician wants to know whether this particular prediction rests on something clinically plausible.

Global interpretability describes the model's behaviour over the whole input distribution — which features matter, and in what direction, on average. Local interpretability explains one prediction: which feature values pushed *this* row above or below the baseline.

They routinely disagree, and the disagreement is the point. A feature can be unimportant globally and decisive for a small subgroup; a globally monotone effect can reverse locally because of an interaction. Reporting the global story for a local question is the most common interpretability error.

A tree ensemble's feature_importances_ sums the impurity decrease each feature achieved across every split, weighted by the samples reaching that split. It is free — it falls out of the fitted trees with no extra data and no extra predictions.

It is the default that everyone reads first, and it is biased in a specific, predictable way: features with many possible split points — continuous columns, high-cardinality IDs — get more chances to reduce impurity by luck, so they rank high even when they carry no signal.

Shuffle one column of a held-out set, rescore the model, and measure how far the metric drops. Repeat n_repeats times to get a mean and a spread. Because it only needs predict, it works on any model — and because it uses held-out data, it measures reliance on that feature for *generalisation*.

It answers the question people actually mean by 'importance': if this feature became noise, how much worse would the model get? A near-zero drop means the model does not need the column, whatever the impurity ranking says.

For a grid of values of one feature, set *every* row to that value, predict, and average. The resulting curve is the model's average predicted output as a function of that feature, with all other features left at their observed values.

It is the most literal picture of a model's learnt shape. It shows non-linearity, saturation and thresholds that no single importance number can: 'important' says a feature matters, partial dependence says a discount stops helping past 30% and starts hurting past 60%.

An Individual Conditional Expectation plot draws one line per row instead of averaging them: each line is that row's prediction as the feature of interest is swept across its range. The partial dependence curve is exactly the average of all the ICE lines.

If the ICE lines fan out or cross, the feature interacts with something else, and the averaged curve is hiding it. A flat partial dependence curve is perfectly compatible with a feature that raises the prediction sharply for half the rows and lowers it for the other half.

A per-prediction attribution derived from cooperative game theory: each feature's value is assigned the average marginal change it causes to the prediction over all orderings in which features could be added. The attributions are additive — they sum exactly to the gap between this prediction and the baseline (average) prediction.

That additivity is what makes SHAP usable as an explanation rather than a heuristic. A local explanation whose parts do not add up to the decision leaves room for argument; one that does is a complete account of the prediction in terms of its inputs.

Perturb the row of interest to generate a cloud of nearby synthetic points, label them with the black-box model's own predictions, weight them by proximity to the original row, and fit a sparse linear model to that weighted cloud. Its coefficients are the explanation.

It formalises the intuition that a complicated surface is approximately simple if you only look at a small enough patch of it. That gives a model-agnostic local explanation on tabular data, text and images, needing nothing but predict_proba.

Two strategies. Either use a model whose structure is itself the explanation — linear or logistic regression with few features, a shallow decision tree, a scorecard, a generalised additive model — or use the most accurate model available and attach a post-hoc explainer to it.

Post-hoc explanations are approximations of a model, and can be wrong about it; an intrinsically interpretable model cannot misrepresent itself, because there is nothing behind the coefficients to misrepresent. On tabular data with strong features the accuracy gap between the two strategies is often small enough that the choice is genuinely open.

When two features carry nearly the same information, any single-feature importance method divides their shared contribution between them. Shuffle one and the model leans on its twin, so the measured drop is small for each — even though removing both would be ruinous.

This is how a genuinely essential signal gets dropped during 'feature pruning': each of its two representations looks individually negligible. The fix is to group correlated features and permute the group together, or to cluster on the correlation matrix and keep one representative per cluster.