Interpreting a black-box gradient boosting model

WHAT THIS TESTS This checks whether you can translate a strong but opaque model into explanations that non-technical stakeholders trust, and whether you know which tools answer global versus local questions.

A GOOD ANSWER COVERS For global interpretation, permutation importance is a solid model-agnostic method: shuffle one feature's values and measure how much performance drops, repeating per feature; bigger drops mean greater importance. It avoids the bias of the built-in gain or split-count importances, which inflate high-cardinality features. Aggregating SHAP values by taking the mean absolute SHAP per feature also gives a reliable global ranking. For explaining a single prediction, SHAP, based on Shapley values from cooperative game theory, fairly distributes the prediction among features as additive contributions that sum to the difference from the baseline. LIME is an alternative that fits a simple interpretable model locally around one instance. SHAP's strength is that the same framework yields both consistent local explanations and a global view.

COMMON WRONG ANSWERS Relying solely on the model's default feature_importances_, which can be misleading. Confusing global importance with local explanation, or claiming one method does both jobs equally. Treating SHAP and LIME as identical. Presenting raw coefficients of an unrelated linear model as the explanation.

LIKELY FOLLOW-UPS Why can default gain-based importance be biased. How do Shapley values guarantee additivity and consistency. What are the runtime costs of exact SHAP versus TreeSHAP.

ONE CONCRETE EXAMPLE A loan-approval model denies an applicant. Globally, permutation importance shows that credit utilization and income are the top drivers across all applicants. For this specific denial, a SHAP force plot reveals that a high debt-to-income ratio pushed the score down by a large amount while a long credit history pushed it up slightly, and the contributions sum to the final decision. The stakeholder sees both the model's overall logic and exactly why this person was declined.