Explain a loan denial with LIME or SHAP

What's really being asked

Whether you can generate an explanation for one specific prediction from an opaque model and, critically, whether you understand that post-hoc attributions are approximations, not ground-truth reasoning, which matters for high-stakes lending decisions.

The full answer

For a single denied application, both LIME and SHAP produce per-feature contributions. LIME works locally by perturbing the applicant's features, querying the black-box model on those perturbed points, weighting them by closeness to the original, and fitting a simple interpretable model such as sparse linear regression to approximate the decision boundary in that neighborhood. Its coefficients become the explanation. SHAP is grounded in cooperative game theory, treating each feature as a player and assigning Shapley values that fairly distribute the difference between this prediction and a baseline across all features, with desirable properties like local accuracy and consistency. You could tell the stakeholder which factors, such as high debt-to-income ratio and short credit history, pushed the score toward denial and by how much.

The mistakes people make

Presenting the attributions as the model's actual causal reasoning, or as legally sufficient adverse-action reasons without validation. Ignoring that LIME explanations can be unstable across runs and sensitive to the perturbation and neighborhood settings.

What usually comes next

Why can two LIME runs disagree? How does SHAP handle correlated features? Are these explanations sufficient for regulatory adverse-action notices?

A concrete example

For one rejected applicant, SHAP shows debt-to-income contributed most to the denial, followed by recent missed payments, while income pushed slightly toward approval. The analyst presents these contributions but cautions that they approximate the model's behavior locally and can shift if the input changes, so they guide understanding rather than prove causation.