Shapley#

class mvpy.model_selection.Shapley(validator: Validator, n_permutations: int = 10, n_jobs: int | None = None, verbose: int | bool = False, on_underspecified: str = 'raise')[source]#

Implements a Shapley value scoring procedure over all feature permutations in \(X\) describing \(y\).

When modeling outcomes \(y\), a common question to ask is to what degree individual predictors in \(X\) contribute to \(y\). To do this, we group predictors according to groups that may, for example, specify:

\[\begin{split}G = \begin{bmatrix} 1 & 1 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 1 \end{bmatrix}\end{split}\]

Or, in other words, here we have three groups of predictors. To compute Shapley values, we always assume that group zero (in this case, including the first three predictors), is some baseline, relative to which we measure the contribution of other predictors. If no such baseline exists, it should simply be an intercept predictor. Next, we perform n_permutations where we fit and score the zero model, then loop over a permutation of our other predictors that we add one by one, measuring how they affect the outcome score relative to our last fitted model.

By repeating this procedure many times, we obtain Shapley values for each predictor that represent the fair contribution of each predictor to outcome scores, invariant of the order in which they may be included. The first score will always correspond to the full baseline performance, whereas the others are relative improvements over baseline.

Warning

This performs \(n k p\) model fits where \(n\) is the number of permutations, \(k\) is the total number of cross-validation steps and \(p\) is the number of unique groups of predictors. For large \(p\) or \(n\), this becomes expensive to solve.

Warning

The default behaviour of this class is to check whether all predictors in \(X\) appear in the group specification groups at least once. If this is not the case, the class will raise an exception. If you would like to mutate this behaviour to either ignore or warn about these cases only, you may want to supply the corresponding on_underspecified value.

Warning

When specifying n_jobs here, be careful not to specify any number of jobs in the model or underlying validator. Otherwise, this will lead to a situation where individual jobs each try to initialise more low-level jobs, severely hurting performance.

Parameters:

validatormvpy.crossvalidation.Validator: The validator object that should be used in this procedure.
n_permutationsint, default=10: How many permutations should we run? A higher number of permutations yields better estimates. Generally, the higher the number of predictor groups, the higher the number of permutations used.
n_jobsOptional[int], default=None: How many jobs should be used to parallelise the hierarchical fitting procedure?
verboseint | bool, default=False: Should progress be reported verbosely?
on_underspecified{‘raise’, ‘warn’, ‘ignore’}, default=’raise’: If we detect an underspecified grouping–i.e., not all available predictors are used–to what level should we escalate things?

Attributes:

validatormvpy.crossvalidation.Validator: The validator object that should be used in this procedure.
n_permutationsint, default=10: How many permutations should we run? A higher number of permutations yields better estimates. Generally, the higher the number of predictor groups, the higher the number of permutations used.
n_jobsOptional[int], default=None: How many jobs should be used to parallelise the hierarchical fitting procedure?
verboseint | bool, default=False: Should progress be reported verbosely?
on_underspecified{‘raise’, ‘warn’, ‘ignore’}, default=’raise’: If we detect an underspecified grouping–i.e., not all available predictors are used–to what level should we escalate things?
validator_List[List[mvpy.crossvalidation.Validator]]: A list containing, per permutation, another list containing Validators for each model group, ordered by group identity.
score_np.ndarray | torch.Tensor | Dict[str, np.ndarray] | Dict[str, torch.Tensor]: The shapley scores of shape (n_permutations, n_sets, n_cv[, ...]) or a dictionary containing each individual Metric.
order_np.ndarray | torch.Tensor: A matrix containing the order in which groups were added to the baseline group of shape (n_permutations, n_groups - 1).

See also

mvpy.model_selection.hierarchical_score, mvpy.model_selection.Hierarchical: An alternative scoring method computing the full permutation over features.
mvpy.model_selection.shapley_score: A shorthand for fitting this class.
mvpy.crossvalidation.Validator: The cross-validation object required by Shapley.

Notes

All entries of scores are relative to the baseline group, except for, of course, the baseline group itself.

Warning

If multiple values are supplied for metric, this class will produce a dictionary of {Metric.name: score, ...} rather than a stacked array. This is to provide consistency across cases where metrics may or may not differ in their output shapes.

Examples

>>> import torch
>>> from mvpy import metrics
>>> from mvpy.dataset import make_meeg_continuous
>>> from mvpy.preprocessing import Scaler
>>> from mvpy.estimators import TimeDelayed
>>> from mvpy.crossvalidation import Validator
>>> from mvpy.model_selection import Shapley
>>> from sklearn.pipeline import make_pipeline
>>> # create dataset
>>> fs = 200
>>> X, y = make_meeg_continuous(fs = fs, n_features = 5)
>>> # setup pipeline for estimation of multivariate temporal response functions
>>> trf = make_pipeline(
>>>     Scaler().to_torch(),
>>>     TimeDelayed(
>>>         -1.0, 0.0, fs, 
>>>         alphas = torch.logspace(-5, 5, 10, device = device)
>>>     )
>>> )
>>> # setup validator
>>> validator = Validator(
>>>     trf,
>>>     metric = (metrics.r2, metrics.pearsonr),
>>> )
>>> # setup groups
>>> groups = torch.tensor(
>>>     [
>>>         [1, 1, 1, 0, 0],
>>>         [0, 0, 0, 1, 0],
>>>         [0, 0, 0, 0, 1]
>>>     ], 
>>>     dtype = torch.long
>>> )
>>> # score individual predictors using Shapley
>>> shapley = Shapley(validator, n_permutations = 3, verbose = True).fit(
>>>     X, y,
>>>     groups = groups
>>> )
>>> shapley.score_['r2'].shape
torch.Size([10, 3, 5, 64, 400])

Fit the models to obtain Shapley values.

Parameters:

Xnp.ndarray | torch.Tensor: The input data of arbitray shape.
ynp.ndarray | torch.Tensor: The output data of arbitrary shape.
groupsOptional[List | np.ndarray | torch.Tensor], default=None: Matrix describing all groups of interest of shape (n_groups, n_predictors). If None, this will default to the identity matrix of (n_predictors, n_predictors).
dimOptional[int], default=None: The dimension in \(X\) that describes the predictors. If None, this will assume -1 for 2D data and -2 otherwise.

Returns:

shapleymvpy.model_selection.Shapley: The fitted shapley model selector.

Shapley#

This Page