[RFC] Add interpretability API as xgboost.interpret module functions

[RAMitchell]

Summary

We have new work underway on shapley values and other related interpretability concepts. This will add new functionality, however the current feature importance/shapley features are included in the predict API. I propose extending the python api with a module (xgboost.interpret) for interpretability, containing stateless functions exposing upcoming features.

These functions accept either a Booster or an sklearn-style XGB* model, plus DMatrix/array-like inputs, and return well-typed results (arrays and or light-weight result objects).

Motivation

• Minimize disruption to existing Booster / sklearn APIs while adding interpretability features.
• Improve discoverability and documentation (module-level functions are easy to document and test).
• Allow incremental implementation: start as wrappers over existing predict(pred_contribs=..., pred_interactions=...), then evolve internals (esp. top-k) without changing the public API.

Proposed public API

Add a new module:

• xgboost/interpret.py

Functions (accept Booster | XGBModel and DMatrix | array-like | pandas):

• shap_values(model, X,*, X_background=None, output_margin=False, iteration_range=None, approx=False, validate_features=True, feature_names=None, return_bias=False)
• shap_interactions(model, X, *, X_background=None, output_margin=False, iteration_range=None, approx=False, validate_features=True, feature_names=None)
• topk_interactions(model, X, *, X_background=None, k=50, metric="mean_abs", output_margin=False, iteration_range=None, validate_features=True, feature_names=None) - Note: possibly just fold this into shap_interactions
• partial_dependence(model, X, *, features, grid_resolution=50, percentiles=(0.05,0.95), grid=None, sample_weights=None, random_state=0, output="prediction", iteration_range=None)
• Possibly adding shap_values, shap_values methods to booster/sklearn class for convenience

Dispatch/behavior notes

• Internally normalize model to a Booster via model being Booster or having get_booster().
• Normalize X to DMatrix if needed; respect feature names where possible.
• Initial SHAP implementations can wrap existing Booster.predict(..., pred_contribs=True/pred_interactions=True) for compatibility.
• topk_interactions should ideally avoid materializing full (n, p, p) tensors; target a C++ implementation to compute aggregated top-k pairs efficiently.

Return types

Prefer lightweight result objects to keep outputs consistent and extensible:

• ShapValues(values, base_values, feature_names, model_output, ...)
• ShapInteractions(values, feature_names, ...) with helpers for main effects / pair extraction
• TopKInteractions(pairs, scores, pair_names=None, per_row=None, ...)
• PDP(features, grid_values, averages, ...)

Documentation plan (Sphinx)

• Add docs/python/interpretability.rst
  • narrative examples + API reference using .. autofunction:: for each function
  • .. autoclass:: for result types

@ron-wettenstein

[RAMitchell]

The proposal for GroupShap #11493 can be folded into this API - we can simply allow relabeling of the features.

[trivialfis]

interpretability, containing stateless functions exposing upcoming features.

Thank you for the RFC. It would be great if you could provide a scope for the new module. For instance, is visualization included?

ShapValuesResult

Maybe omit the "Result" in its name.

target a streaming / C++ implementation to compute aggregated top-k pairs efficiently.

I'm curious about your plan for this, specifically the return values.

[RAMitchell]

My instinct was to avoid visualisation here, but we do already have graphviz and matplotlib as optional dependencies. Maybe we should just go ahead and make it as easy as possible for the user. In any case that can be for a later time - lets focus on returning the raw data for now.

[RAMitchell]

@trivialfis do you prefer this api as a function that accepts a booster/sklearn model or would you prefer adding methods to booster/sklearn model. I find we have too many methods already and I like the idea of being able to cleanly separate the docs and implementations.

[trivialfis]

I think we should clearly separate computation from visualization (agreeing with your preference), in a similar way that's stated by the mlr3 project. So, I prefer having them to be separated from the booster class. Similar design choice can be found in sklearn visualization module: https://scikit-learn.org/stable/visualizations.html see the RocCurveDisplay.from_estimator.

As for whether these should be functions or classes, I don't have a preference. You have a better understanding of potential states and can make better choices than I.

[ron-wettenstein]

Sorry for joining the discussion late, I was busy with the AAAI conference.

Visualization
I agree that computation and visualization should be separated. Maybe we will start without visualization and add it later. I think it will be useful for the output to be in the same format as the shap package output (a simple np.array) and them researchers can just take the output from xgboost and insert it to shap for visualization.

import shap
from xgboost import interpret

shap_values = interpret.shap_values(model, ...)
shap.summary_plot(shap_values, X_test)

I think the current design already achieve this behaviors.

Adding this as a method
I do want to add the shap_values and shap_interaction_values as methods in the XGBoost model object. I think that will increase the usage and visibility. Also given the fact XGboost model already have feature_importance_ property it makes sense to add also the SHAP approach which considered best practice. This way it will be clear that XGBoost provide a better alternative to the native feature_importance_. Anyway, I agree that there should also be a function under xgboost.interpret and that the logic should be written there. The only question is whether we want also to add a method as a syntactic sugar. The partial_dependence function should stay only in the interpret namespace.

More things (Me and Rory discussed last week):

• We should also support background (interventional) shap. The API should include an optional parameter for the background dataset (say X_background).
• We might not want the topk_interactions to be a method by itself. The shap_interactions parameter can include a topk parameter that when True will return the top k.
• We don't want to support topk per sample. Lets delete the per_row parameter.

Nice to have

• Add functions for Banzhaf values and Banzhaf interaction values.
• Support GroupSHAP using owen values. See the ticket: Feature proposal - groupSHAP #11493. This is not so trivial, I still need to think about it algorithmically. I believe we will add it in a second version.
• The sklearn partial dependence API support also "samples_weights". We can extend our API to support this too (It a trivial algorithmic change). We can also add this parameter to the shap_value and shap_interaction values methods.
  https://scikit-learn.org/stable/modules/generated/sklearn.inspection.partial_dependence.html

Thanks @RAMitchell and @trivialfis. I'm excited to work on this!

[RAMitchell]

@ron-wettenstein I have updated the RFC with your comments. With topk lets just see what the algorithm looks like and make a call on API a bit later.

import shap
from xgboost import interpret

shap_values = interpret.shap_values(model, ...)
shap.summary_plot(shap_values, X_test)

If we only return an array then the bias must be the last column. If so the above fails because shap expects n_features columns only. So I think it still makes sense to return the bias and numpy array separately.

[trivialfis]

Might be relevant, posting for reference:

• Pred_contribs in 2.1.1 takes significantly more gpu memory than in 1.4.2 #10936
• Only analyze features included in model (predcontrib,predinteraction) #4998
• Feature contributions (SHAP) for model with Tweedie objective return values in margin space R package #7448
• SHAP value with refreshed tree. #8199

[RAMitchell]

@trivialfis do we need to make modifications to support vector leaf?

[trivialfis]

At the moment, I store the leaf sum Hessian as a scalar, computed as the sum of all targets. As I understand it, the Hessian is a proxy. If the current sum Hessian is good enough, then I don't expect any significant change. Otherwise, we need to figure out something else.

[RAMitchell]

Thinking more of the tree leaf value. I think it will result in minor changes only to shap code.