Logging Model Explanations, Not Just Predictions

Why it exists

Production models fail. Simply logging inputs and outputs (predictions) tells you what went wrong, but not why. To debug subtle issues like data drift or fairness violations, you need to understand the model's reasoning at the point of failure. Prediction explanation logging was created to capture this reasoning.

The mental model

Think of it like a student showing their work on a math test. Just seeing the final answer ("7") isn't enough to know if they understand the concept. Seeing the steps they took ("3 + 4 = 7") lets you debug their logic. Logging explanations (e.g., from SHAP or LIME) is capturing the model's "work" for each prediction.

How it works

When your application requests a prediction, the inference service does two things. First, it runs the model to get the prediction. Second, it uses an explanation technique (like SHAP, LIME, or Integrated Gradients) to generate a set of feature attributions—scores that show how much each input feature contributed to the final prediction. Both the prediction and these attribution scores are then logged to a durable store, often alongside the input features themselves.

When to use it

Use this when the "why" is as important as the "what". This is common in three scenarios. First, for debugging: when model performance degrades, you can analyze historical explanations to see if the model started relying on new or different features (concept drift). Second, for fairness and bias audits: you can check if sensitive attributes are unduly influencing predictions. Third, for regulatory compliance: some laws, like GDPR's "right to explanation," may require you to explain automated decisions to users.

When not to use it

Avoid logging explanations for every single prediction in high-throughput, low-stakes applications. Generating explanations adds computational overhead, increasing latency and cost. Storing them also adds significant storage costs. For a simple product recommendation engine, it's likely overkill. In these cases, consider strategic sampling: log explanations for only a fraction of requests, or for specific cohorts of users or items.

One canonical example

A bank uses a model to approve or deny loan applications. They log not only the "deny" decision but also the SHAP values for that prediction. The log shows that the applicant's debt_to_income_ratio was the primary factor for the denial. When an auditor asks why the loan was denied, the bank can provide a data-driven, feature-based explanation instead of just "the model said so." This provides a clear audit trail and helps ensure the model isn't using prohibited features.