Practical explainability for an LLM loan summary
Pragmatic LLM explainability under constraints.
Use attribution-by-design with grounded citations, structured rationales, and a deterministic rules layer instead of slow per-token SHAP.
What's really being asked
Whether you know that LIME and SHAP scale poorly on transformers and that in a regulated lending context you need verifiable, faithful explanations, not just plausible narration.
The full answer
Shift from post-hoc attribution to explainability by design. First, make the system retrieval-grounded so every negative claim cites the specific profile span it came from, and validate each citation actually appears in the source. Second, require a structured output that maps each conclusion to named input factors, for example income stability or debt ratio, rather than a free-form paragraph. Third, and most important for legal defensibility, keep the consequential decision in a deterministic, auditable rules or scoring component, and use the LLM only to summarize the already-made decision in plain language. That gives the legal team a stable, reproducible reason code while the LLM handles wording. Where you do want token attribution, attention rollout or cheaper gradient-based saliency can approximate influence faster than perturbation methods, but treat these as supporting evidence, not proof.
The mistakes people make
Claiming chain-of-thought is the explanation; generated reasoning is often a post-hoc rationalization, not the causal path. Just run SHAP anyway, ignoring the stated latency constraint. Asking the model why did you say that, which produces a fresh confabulation. Relying on raw attention weights as faithful importance.
What usually comes next
Why is chain-of-thought not a faithful explanation? How do you verify citations? What reason codes satisfy lending regulations like adverse action notices? How do you test explanation stability across reruns?
A concrete example
The profile shows high debt-to-income. A deterministic scorer flags that field as the rejection driver and emits reason code DTI_HIGH. The LLM then writes the summary, citing the debt-to-income figure from the profile, and the system verifies that exact figure exists in the source. The legal team receives a reproducible reason code plus a citation-checked narrative, instead of a slow, non-deterministic SHAP run over a transformer.
Interview question
Why is it risky to hand the legal team the LLM's own chain-of-thought as the explanation for a loan rejection?
- a.Chain-of-thought is always too short to be useful
- b.Chain-of-thought cannot be logged for audit purposes
- c.Generated reasoning can be a plausible post-hoc rationalization that does not reflect the model's actual decision processCorrect
- d.It is computationally as expensive as running SHAP
Why? this is the answer
An LLM's verbalized reasoning is not guaranteed to be the causal cause of its output; it may rationalize. Routing the decision through a deterministic layer with verified citations gives a faithful, reproducible reason.
Just read this? Test yourself on what you have been reading.
Read the original → aws.amazon.com
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles