Model Explainability in Production with SHAP

A model that decides about people will be asked why. Customers ask, support teams ask, auditors ask, and engineers debugging a strange decision ask most of all. SHAP is the most common way to answer for a single prediction: it splits the prediction into contributions from each feature.

Getting SHAP values out of a notebook is easy. Running SHAP in production means deciding which explanation is the official one, checking it, storing it and keeping it consistent across releases. This guide works through those decisions on real credit card data.

What SHAP gives you

Lundberg and Lee introduced SHAP, SHapley Additive exPlanations, as a unified framework that assigns each feature an importance value for a particular prediction (Lundberg and Lee: SHAP). The contributions add up: a base value plus every feature's contribution equals the model's output for that case. For tree models, a later paper gave the first polynomial-time algorithm for these explanations, which makes them practical at serving time (Lundberg et al.: explainable AI for trees).

Why services need explanations

Some decisions legally require reasons. In the United States, Regulation B says a statement of reasons for adverse credit action must be specific and indicate the principal reasons, and that saying an applicant failed to reach a qualifying score is insufficient (eCFR: 12 CFR 1002.9). Even where no rule applies, explanations help support staff, speed up debugging and give reviewers something to audit.

The model

The UCI Default of Credit Card Clients data cover 30,000 card holders in Taiwan: credit limit, monthly repayment status, bills and payments from April to September 2005, and whether each defaulted the following month, under CC BY 4.0 (UCI: Default of Credit Card Clients). Repayment status is -1 for paid on time and 1 to 9 for months late.

A gradient-boosted classifier uses only the account history; sex, marital status, education and age are left out. Applicants with a predicted default risk of 50% or more are declined, a simplified policy for illustration.

import time

import numpy as np
import pandas as pd
import shap
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

credit = pd.read_excel("credit.xls", header=1)            # UCI Default of Credit Card Clients, Taiwan, 2005 (CC BY 4.0)
MONTHS = ["Sep", "Aug", "Jul", "Jun", "May", "Apr"]
NAMES = {"LIMIT_BAL": "credit limit",
         **{f"PAY_{n}": f"repayment status, {m}" for n, m in zip((0, 2, 3, 4, 5, 6), MONTHS)},
         **{f"BILL_AMT{n}": f"bill amount, {m}" for n, m in zip(range(1, 7), MONTHS)},
         **{f"PAY_AMT{n}": f"amount paid, {m}" for n, m in zip(range(1, 7), MONTHS)}}
FEATURES = list(NAMES)                                    # sex, marriage, education and age are left out of the model
target = credit["default payment next month"]
train_x, live_x, train_y, live_y = train_test_split(credit[FEATURES], target, test_size=0.3, random_state=0, stratify=target)
model = HistGradientBoostingClassifier(random_state=0).fit(train_x, train_y)
risk = model.predict_proba(live_x)[:, 1]
declined = np.where(risk >= 0.5)[0]
print(f"accuracy {model.score(live_x, live_y):.1%}; {live_y.mean():.1%} of applicants defaulted; "
      f"{len(declined):,} of {len(live_x):,} declined at 50% risk")

start = time.perf_counter()
explanation = shap.TreeExplainer(model)(live_x)           # no background data: the tree_path_dependent algorithm
seconds = time.perf_counter() - start
gap = np.abs(explanation.base_values + explanation.values.sum(axis=1) - model.decision_function(live_x)).max()
print(f"additivity check: base value + contributions = model output, largest gap {gap:.0e}")
print(f"explaining all {len(live_x):,} applicants took {seconds:.1f} s (varies by machine)")


def reasons(values, top=3):
    return list(pd.Series(values, index=FEATURES).nlargest(top).index)


first = declined[0]
row = live_x.iloc[first]
print(f"\nfirst declined applicant: default risk {risk[first]:.0%}; reasons, strongest first:")
for feature in reasons(explanation.values[first]):
    print(f"  {NAMES[feature]:<24} {row[feature]:>8,.0f}   +{explanation.values[first][FEATURES.index(feature)]:.2f} log-odds")

background = train_x.sample(200, random_state=0)
alternative = shap.TreeExplainer(model, data=background, feature_perturbation="interventional")(live_x.iloc[declined])
same_top = np.mean([reasons(explanation.values[i], 1) == reasons(alternative.values[k], 1) for k, i in enumerate(declined)])
same_three = np.mean([reasons(explanation.values[i]) == reasons(alternative.values[k]) for k, i in enumerate(declined)])
print(f"\nsame applicant, interventional algorithm with 200 background rows: {[NAMES[f] for f in reasons(alternative.values[0])]}")
print(f"across all {len(declined):,} declined applicants: same top reason {same_top:.0%}, same three reasons in order {same_three:.0%}")

importance = pd.Series(np.abs(explanation.values).mean(axis=0), index=FEATURES).sort_values(ascending=False)
status = importance[[f for f in FEATURES if f.startswith("PAY_") and "AMT" not in f]].sum() / importance.sum()
print("\nlargest average contributions:", {NAMES[f]: round(v, 2) for f, v in importance.head(4).items()})
print(f"share of all attribution from the six repayment-status fields: {status:.0%}")

accuracy 82.1%; 22.1% of applicants defaulted; 1,050 of 9,000 declined at 50% risk
additivity check: base value + contributions = model output, largest gap 9e-15
explaining all 9,000 applicants took 1.7 s (varies by machine)

first declined applicant: default risk 58%; reasons, strongest first:
  repayment status, Sep           2   +1.85 log-odds
  credit limit               10,000   +0.17 log-odds
  amount paid, Aug            1,600   +0.06 log-odds

same applicant, interventional algorithm with 200 background rows: ['repayment status, Sep', 'credit limit', 'bill amount, Sep']
across all 1,050 declined applicants: same top reason 100%, same three reasons in order 58%

largest average contributions: {'repayment status, Sep': 0.45, 'credit limit': 0.19, 'amount paid, Jul': 0.12, 'bill amount, Sep': 0.11}
share of all attribution from the six repayment-status fields: 47%

What the output shows

• The model is useful but not dramatic. 82.1% accuracy where 22.1% defaulted, with 1,050 of 9,000 applicants declined.
• The explanations add up. Base value plus contributions matched the model's output to within 9e-15. Run this as an assertion on every stored explanation; it catches an explainer paired with the wrong model version.
• The reasons are specific. For the first declined applicant, at 58% risk, September's repayment status of 2, two months late, added 1.85 to the log-odds and dominated the decision. The 10,000 NT dollar credit limit and an August payment of 1,600 added far less. "Your September payment was two months late" is a reason; "your score was too low" is not.
• The explainer setting changed the answer. With the interventional algorithm and 200 background rows, the same applicant's third reason became September's bill. Across all 1,050 declined applicants, the top reason always agreed, but the same three reasons in the same order agreed only 58% of the time. SHAP's documentation describes both options: the interventional approach needs background data, typically 100 to 1,000 samples, while the tree-path-dependent one uses the training examples that reached each leaf (SHAP: TreeExplainer). Neither is wrong, so the choice has to be fixed and versioned.
• Tree explanations are cheap. Explaining 9,000 applicants took about two seconds on this machine.
• The global view shows what the model leans on. September's repayment status had the largest average contribution, followed by the credit limit, and the six repayment-status fields together made up 47% of all attribution.

Putting SHAP into a service

• Explain at decision time for any decision that may need reasons later, and store the reasons with the prediction log; see logging and observability for ML systems.
• Version the explainer with the model: algorithm, background sample and library version, kept together in the model registry.
• Translate features into plain language before anything reaches a customer.
• Watch attributions over time. A feature whose average contribution shifts is a signal worth investigating alongside drift monitoring.
• Assert, do not assume: additivity, no missing values, and the expected number of features in every explanation.

For the wider monitoring plan, see monitoring ML models in production.

The limits

• SHAP explains the model, not the world. A large contribution is not a causal effect.
• Correlated features share credit, and how they share it depends on the algorithm, as the 58% agreement shows.
• Removing sensitive fields does not prove fairness. Other fields can stand in for them; SHAP helps audit that, but it is not a fairness test.

Common mistakes

• Changing explainer settings without a new version.
• Showing raw feature codes to customers.
• Reading SHAP values as causes.
• Explaining decisions only offline, so a past decision cannot be explained later.
• Skipping the additivity check.

Responsible AI practices and compliance are covered in the Security and Governance module of the MLOps course.

Sources

• A Unified Approach to Interpreting Model Predictions (arXiv, NIPS 2017) — Lundberg and Lee
• Explainable AI for Trees: From Local Explanations to Global Understanding (arXiv) — Lundberg et al.
• shap.TreeExplainer (SHAP documentation) — SHAP
• 12 CFR 1002.9 Notifications (Regulation B, eCFR) — US Code of Federal Regulations
• Default of Credit Card Clients dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository