Model Interpretability vs. Explainability
Interpretability means a human can grasp a model's logic (e.g., a simple decision tree). Explainability is stronger: it's about why the model made a *specific* choice. This is key for debugging or justifying high-stakes decisions.
Why it exists
Predictive accuracy alone is an incomplete picture of a model's value. For many real-world tasks, especially those with high stakes, simply getting the right answer isn't enough. We need to know why the model made its decision to debug it, trust it, ensure fairness, and comply with regulations. This need for the 'why' drives the demand for interpretability and explainability.
The mental model
Interpretability is understanding a car's engine design; a simple, visible engine is more interpretable than a sealed, complex one. Explainability is the mechanic's report on why your specific car failed its emissions test, pointing to a single faulty part. One is about understanding the system's general logic; the other is about diagnosing a specific outcome.
How it works
Interpretability is the degree to which a human can understand the cause of a model's decisions. A model is 'interpretable' if its internal mechanics are simple enough to be understood directly, like a linear regression model or a small decision tree. Explainability is a stronger requirement that includes interpretability but adds context to justify a single prediction. Complex, 'black box' models like neural networks are not inherently interpretable, so they require post-hoc 'explainability' methods (like SHAP or LIME) to generate explanations for their outputs.
When to use it
The distinction is critical in regulated or high-stakes industries like finance, insurance, and healthcare, where you must be able to justify individual decisions to customers and regulators. It's also vital for model debugging, ensuring fairness by checking for bias, and building user trust. When a model's failure has serious consequences, you need to be able to explain its reasoning.
When not to use it
For low-risk applications, the distinction is less critical. If a movie recommender system suggests a bad film, the consequences are trivial. In these cases, predictive performance (the 'what') might be all that matters, and you might not need to invest in explaining the 'why'. Similarly, for problems that are well-understood and extensively tested, like optical character recognition, the model's performance may be trusted without explicit explanations.
One canonical example
A bank uses a model for loan applications. Interpretability is achieved by using a simple logistic regression model. An auditor can see the model's coefficients and understand that, in general, higher income increases approval odds while higher debt decreases them. Explainability is needed when a specific applicant, Jane, is denied. The system must provide a local explanation: 'Jane's application was denied because her debt-to-income ratio was 55%, which is above the 45% threshold learned by the model.'
Interview question
Which statement most accurately describes the difference between model interpretability and model explainability?
- a.Interpretability refers to understanding a model's general logic, whereas explainability focuses on justifying a specific prediction.Correct
- b.Interpretability is for simple models, while explainability is primarily needed for complex "black box" models.
- c.Interpretability is about model accuracy, while explainability is about ensuring model fairness and compliance.
- d.Interpretability provides reasons for individual model outputs, while explainability describes the overall internal mechanics of the model.
Why? this is the answer
Interpretability allows humans to grasp the model's overall internal workings and general logic. Explainability, a stronger concept, focuses on providing specific reasons for a single, particular prediction or decision made by the model.
Just read this? Test yourself on what you have been reading.
Read the original → christophm.github.io
- #machine learning
- #mlops
- #explainability
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on machine learning — each one lists the topics its interview covers.
See open roles