LIME: Explaining Single Predictions from Any ML Model

Why it exists

Complex models like neural networks or gradient boosted trees achieve high accuracy but are 'black boxes,' making it hard to understand why they make a specific prediction. This lack of transparency is a problem for debugging, building trust, and ensuring fairness. LIME was created to provide local, human-understandable explanations for any model's individual predictions without needing to know the model's internal structure.

The mental model

Think of explaining a complex model's decision like asking a foreign language expert for a translation. Instead of asking them to teach you the entire language (a global explanation), you point to one specific phrase and ask, 'What does this mean, and why?' LIME does this for a model prediction. It doesn't explain the whole model, just the behavior around one data point, by creating a simple, local approximation.

How it works

LIME follows a clear recipe. First, select the specific prediction you want to understand. Second, create a new dataset by generating many 'perturbed' variations of your original data point (e.g., slightly changing feature values or removing words from a sentence). Third, get the black box model's predictions for all these new, perturbed points. Fourth, train a simple, interpretable model, like a weighted linear regression, on this new dataset. The key is to give more weight to perturbed samples that are closer to your original data point. Finally, the interpretation of this simple local model (e.g., its feature coefficients) serves as the explanation for the original prediction.

When to use it

Use LIME when you need to explain an individual prediction from any model you can't or don't want to inspect internally. It's ideal for debugging ('Why was this loan application denied?'), providing transparency to users ('Your post was flagged because it contained these words...'), and auditing model behavior on specific, critical cases. It works on tabular data, text, and images.

When not to use it

Do not use LIME if you need a global understanding of the entire model's behavior; for that, use methods like SHAP or Partial Dependence Plots. LIME explanations can also be unstable; slightly different perturbations can sometimes lead to different explanations. It's not suitable for situations requiring a single, mathematically guaranteed 'true' explanation, as the result depends on user-defined settings like the neighborhood size and number of features.

One canonical example

A bank uses an XGBoost model for loan approvals. An applicant is denied. To understand why, they run LIME on the applicant's data. LIME creates thousands of slightly altered profiles (e.g., income $1k higher, credit score 5 points lower) and gets the model's score for each. It then fits a simple linear model to these results, weighted by similarity to the original application. The linear model shows the top factors for the denial were a high debt-to-income ratio and a low number of years at current employment, providing a concrete, local explanation.