Interpreting a black-box gradient boosting model
model interpretability methods.
global tools like permutation importance or aggregated SHAP rank overall feature influence; local tools like per-instance SHAP or LIME explain one prediction; SHAP unifies both via additive…
What's really being asked
This checks whether you can translate a strong but opaque model into explanations that non-technical stakeholders trust, and whether you know which tools answer global versus local questions.
The full answer
For global interpretation, permutation importance is a solid model-agnostic method: shuffle one feature's values and measure how much performance drops, repeating per feature; bigger drops mean greater importance. It avoids the bias of the built-in gain or split-count importances, which inflate high-cardinality features. Aggregating SHAP values by taking the mean absolute SHAP per feature also gives a reliable global ranking. For explaining a single prediction, SHAP, based on Shapley values from cooperative game theory, fairly distributes the prediction among features as additive contributions that sum to the difference from the baseline. LIME is an alternative that fits a simple interpretable model locally around one instance. SHAP's strength is that the same framework yields both consistent local explanations and a global view.
The mistakes people make
Relying solely on the model's default feature_importances_, which can be misleading. Confusing global importance with local explanation, or claiming one method does both jobs equally. Treating SHAP and LIME as identical. Presenting raw coefficients of an unrelated linear model as the explanation.
What usually comes next
Why can default gain-based importance be biased. How do Shapley values guarantee additivity and consistency. What are the runtime costs of exact SHAP versus TreeSHAP.
A concrete example
A loan-approval model denies an applicant. Globally, permutation importance shows that credit utilization and income are the top drivers across all applicants. For this specific denial, a SHAP force plot reveals that a high debt-to-income ratio pushed the score down by a large amount while a long credit history pushed it up slightly, and the contributions sum to the final decision. The stakeholder sees both the model's overall logic and exactly why this person was declined.
Interview question
Why might you avoid a tree model's built-in gain or split-count feature importance for stakeholder explanations?
- a.It only works for linear models
- b.It cannot be computed without the test set
- c.It is biased toward high-cardinality or frequently split featuresCorrect
- d.It explains individual predictions but not global trends
Why? this is the answer
Impurity- or split-based importance inflates features with many distinct values, giving a distorted ranking. Permutation importance or mean absolute SHAP values provide a less biased global view.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #interpretability
- #shap
- #explainable-ai
- #feature-importance
- #machine-learning
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles