Practical explainability for an LLM loan summary

WHAT THIS TESTS Whether you know that LIME and SHAP scale poorly on transformers and that in a regulated lending context you need verifiable, faithful explanations, not just plausible narration.

A GOOD ANSWER COVERS Shift from post-hoc attribution to explainability by design. First, make the system retrieval-grounded so every negative claim cites the specific profile span it came from, and validate each citation actually appears in the source. Second, require a structured output that maps each conclusion to named input factors, for example income stability or debt ratio, rather than a free-form paragraph. Third, and most important for legal defensibility, keep the consequential decision in a deterministic, auditable rules or scoring component, and use the LLM only to summarize the already-made decision in plain language. That gives the legal team a stable, reproducible reason code while the LLM handles wording. Where you do want token attribution, attention rollout or cheaper gradient-based saliency can approximate influence faster than perturbation methods, but treat these as supporting evidence, not proof.

COMMON WRONG ANSWERS Claiming chain-of-thought is the explanation; generated reasoning is often a post-hoc rationalization, not the causal path. Just run SHAP anyway, ignoring the stated latency constraint. Asking the model why did you say that, which produces a fresh confabulation. Relying on raw attention weights as faithful importance.

LIKELY FOLLOW-UPS Why is chain-of-thought not a faithful explanation? How do you verify citations? What reason codes satisfy lending regulations like adverse action notices? How do you test explanation stability across reruns?

ONE CONCRETE EXAMPLE The profile shows high debt-to-income. A deterministic scorer flags that field as the rejection driver and emits reason code DTI_HIGH. The LLM then writes the summary, citing the debt-to-income figure from the profile, and the system verifies that exact figure exists in the source. The legal team receives a reproducible reason code plus a citation-checked narrative, instead of a slow, non-deterministic SHAP run over a transformer.