Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
42,671 characters · 21 sections · 46 citation commands
Explaining black box decisions by Shapley cohort refinement
Black box prediction models used in statistics, machine learning and artificial intelligence have been able to make increasingly accurate predictions, but it remains hard to understand those predictions. Part of understanding predictions is understanding which variables are important. Variable importance in machine learning and AI has been studied recently in stru:kono:2010,stru:kono:2014, ribe:etal:2016, lund:lee:2017, sund:najm:2019, kuma:2020 as well as the book of moln:2018.
A variable could be important because changing it makes a causal difference, or because changing it makes a large change to our predictions or because leaving it out of a model reduces that model's prediction accuracy jian:owen:2003. Importance by one of these criteria need not imply importance by another.
In order to explain a decision like somebody being admitted to the intensive care unit or turned down for a loan, we are not focused on causality. The variables that are predictive of somebody's outcome ideally should have been the ones that were important to the decision, but they may not have been. Similarly, variables whose inclusion is necessary for a good model fit are not necessarily the ones that best explain an outcome. The explanation should use the model that was deployed, and not any others.
We are left with a problem of explaining the output of one specific fitted function $f(\cdot)$ applied to a vector $\boldsymbol{x}$ of input variables, hereafter predictors. For a target subject $t$ with a $d$-dimensional set of predictors $\boldsymbol{x}_t=(x_{t1},\dots,x_{td})$ we can change some subset of values of $\boldsymbol{x}_t$, perhaps to some baseline levels in $\boldsymbol{x}_b$. Then the variables that best explain why $f(\boldsymbol{x}_t)$ is different from $f(\boldsymbol{x}_b)$ are deemed important for the decision on the target subject. Given $2^d-1$ possible changes, there are strong reasons to favor Shapley value shap:1952 to combine them into an importance measure as sund:najm:2019 and others that they survey do. We note however, that concerns about using Shapley value have been raised by kuma:2020.
Mixing and matching the components of $\boldsymbol{x}_t$ and $\boldsymbol{x}_b$ presents some problems. The variables $x_{ij}$ and $x_{ik}$ may show a strong correlation over subjects $i=1,\dots,n$. Putting $x_{tj}$ and $x_{bk}$ into a single hypothetical point may produce an input combination far from any that has ever been seen. Beyond being unusual, some combinations are physically or even logically impossible. The changes might produce a hybrid data point representing a patient whose systolic blood pressure is lower than their diastolic blood pressure. Somebody's birth date could follow their graduation date. When hospital records show minimum, maximum and average levels of blood oxygen, the hybrid point could have mean $O_2$ below minimum $O_2$, or it could have minimum and maximum that differ along with a variable saying they were only measured once (or never). There could be important reasons to understand effects of longitude and latitude separately, but some combinations will not make sense, perhaps by placing a dwelling within a body of water.
When the function $f(\cdot)$ that made a decision was trained, it would have seen few if any impossible or extremely unlikely inputs. As a result, its predictions there cannot have been regularized suitably. Investigators should be able to choose an importance measure that does not rely on any such values.
In this paper we work under the constraint that only possible values can be used. It is hard to define possible values and so operationally we only use actually observed values. For each predictor, every subject in the data set is either similar to the target subject or not similar. Ways to define similarity are discussed below. Given $d$ predictors, there are $2^d$ different sets of predictors on which subjects can be similar to the target. We form $2^d$ different cohorts of subjects, each consisting of subjects similar to the target on a subset of predictors, without regard to whether they are also similar on any of the other predictors. At one extreme is a set of all predictors, and a cohort that is similar to the target in every way. At the other extreme, the empty predictor set yields the set of all subjects. As usual, for large $d$, Monte Carlo sampling is used to approximate the Shapley value.
We can refine the grand cohort of all subjects towards the target subject by removing subjects that mismatch the target on one or more predictors. The predictors that change the cohort mean the most when we restrict to similar subjects, are the ones that we take to be the most important in explaining why the target subject's prediction is different from that of the other subjects.
We call this approach `cohort refinement Shapley' or cohort Shapley for short. With baseline Shapley, taking account of variable $j$ is done by replacing $x_{bj}$ by $x_{tj}$. Cohort Shapley doesn't change the value. It refines what we know about $\boldsymbol{x}_t$. Variables whose knowledge moves the cohort mean the most are then the important ones.
This knowledge approach has some consequences. If one is studying redlining by an algorithm that simply does not use a protected variable then baseline Shapley can only give that variable zero importance. Cohort Shapley can give that variable a nonzero importance measure even if it was not in the data set at the time the model was trained. It is thus useful for the problem of auditing an algorithm for indirect inferences, a task described by adle:2018. kuma:2020 describe this issue as a catch-22. In their terms, cohort Shapley is a conditional method (changing knowledge) while baseline Shapley is an interventional method (changing values). The conditional approach has the problem of requiring a joint distribution for predictors, while the interventional approach can depend on impossible predictor combinations. The word `importance' does not have a unique interpretation. Multiple ways to quantify it estimate different quantities and involve tradeoffs. We expect that multiple methods must be considered in practice.
sund:najm:2019 and kuma:2020 mention a second problem with baseline Shapley. If two predictors are highly correlated then the algorithm might have used them very unequally, perhaps due to an $L_1$ regularization. It then becomes unpredictable which will be deemed important. The situation is more stable with cohort Shapley. In the extreme case of identical variables and identical similarity notions, cohort Shapley will give them equal importance.
As mentioned above, we define the impact of a variable through the Shapley value. Shapley value has been used in model explanation for machine learning stru:kono:2010,stru:kono:2014,lund:lee:2017,sund:najm:2019 and for computer experiments owen:2014,song:nels:stau:2016,owen:prie:2017. KernelSHAP lund:lee:2017 computes an approximation of baseline Shapley replacing $f$ by a kernel approximation. KernelSHAP provides an option to compute the average of Shapley values for multiple baseline vectors, and one very reasonable choice is to use every training point as a baseline point. We call that `all baseline Shapley' (ABS), below.
Our contribution is to further investigate the differences between baseline and cohort Shapley and to connect them to the global sensitivity analysis literature which emphasizes Sobol' indices sobo:1993. For surveys, see salt:ratt:andr:camp:cari:gate:sais:tara:2008 and gamb:jano:klei:lagn:prie:2016. We show how both baseline Shapley value and cohort Shapley value are derived from functional decompositions. Variable importance for a single target subject $t$ is `local' as distinct from `global' prediction importance, taken over all subjects. We consider aggregation and disaggregation between local and global problems. While Shapley value is additive over games, some unwanted cancellations can arise during aggregation. For instance, the local importances can sum to a global importance of zero. We resolve this by using squared differences for local and global methods. We illustrate these measures on the well known Boston housing and Titanic data sets.
The predictor vector for subject $i$ is $\boldsymbol{x}_i =(x_{i1},\dots,x_{id})$ where $d$ is the number of predictors in the model. Each $x_{ij}$ belongs to a set $X_j$ which may consist of real or binary variables or some other types. There is a black box function $f:\boldsymbol{x}\to\mathbb{R}$ that is used to predict an outcome for a subject with predictor vector $\boldsymbol{x}$. We write $y=f(\boldsymbol{x})$ and $y_i=f(\boldsymbol{x}_i)$. For local explanation we would like to quantify the importance of predictors $x_{tj}$ on $y_t = f(\boldsymbol{x}_t)$. We assume that $t$ is in the set of subjects with available data, although this subject might not have been used in training the model. A baseline point $\boldsymbol{x}_b$ might not be one of the subjects. For instance if $\boldsymbol{x}_b=(1/n)\sum_{i=1}^n\boldsymbol{x}_i$ it will ordinarily not be a data point. For example, $\boldsymbol{x}_{b,j}$ might be in $(0,1)$ for a binary variable $j$.
The set $\{1,2,\dots,n\}$ of subjects is denoted by $1{:}n$ and the set of predictors is $1{:}d$. We will need to manipulate subsets of $1{:}d$. For $u\subseteq 1{:}d$ we let $|u|$ be its cardinality. The complementary set $1{:}d\setminus u$ is denoted by $-u$, especially in subscripts. Sometimes a point must be created by combining parts of two other points. The point $\boldsymbol{y} =\boldsymbol{x}_u{:}\boldsymbol{z}_{-u}$ has $y_j =x_j$ for $j\in u$ and $y_j = z_j$ for $j\not\in u$. Furthermore, we sometimes use $j$ and $-j$ in place of the more cumbersome $\{j\}$ and $-\{j\}$. For instance, $\boldsymbol{x}_{t,-j}{:}\boldsymbol{x}_{b,j}$ is what we get by changing the $j$'th predictor of the target subject to the baseline level. We also use $u + j$ for $u\cup\{j\}$.
Function decompositions, also called high dimensional model representations (HDMR), write a function of $d$ inputs as a sum of functions, each of which depend only on one of the $2^d$ subsets of inputs. Because $f$ and $\boldsymbol{x}$ have other uses in this paper, we present the decomposition for $g(\boldsymbol{z})$. Let $g$ be a function of $\boldsymbol{z}=(z_1,\dots,z_d)$ with $z_j\in Z_j$. In these decompositions we write $$ g(\boldsymbol{z}) = \sum_{u\subseteq1{:}d}g_u(\boldsymbol{z}) $$ where $g_u(\boldsymbol{z})$ depends on $\boldsymbol{z}$ only through $\boldsymbol{z}_u$. Many such decompositions are possible kuo:sloa:wasi:wozn:2010.
The best known decomposition for this problem is the functional analysis of variance (ANOVA) decomposition hoef:1948,sobo:1969. It applies to random $\boldsymbol{z}$ with independent components $z_j\in Z_j$. If $\mathbb{E}( g(\boldsymbol{z})^2)<\infty$, then we write
for non-empty $u\subseteq1{:}d$. The effects $g_u$ are orthogonal in that for subsets $u\ne v$, we have $\mathbb{E}( g_u(\boldsymbol{z})g_v(\boldsymbol{z}))=0.$ Letting $\sigma^2_u = \mathrm{var}(g_u(\boldsymbol{z}))$, it follows from this orthogonality that $$ \sigma^2(g)\equiv \mathrm{var}(g(\boldsymbol{z})) = \sum_{u\subseteq1{:}d}\sigma^2_u(g). $$ We can recover effects from conditional expectations, via inclusion-exclusion,
See owen:2013 for history and derivations of this functional ANOVA.
We will need the anchored decomposition, which goes back at least to sobo:1969. It is also called cut-HDMR alis:rabi:2001 in chemistry, and finite differences-HDMR in global sensitivity analysis sobo:2003. We begin by picking a reference point $\boldsymbol{c}$ called the anchor, with $c_j\in Z_j$ for $j=1,\dots,d$. The anchored decomposition is
We have replaced averaging over $\boldsymbol{z}_{-u}$ by plugging in the anchor value via $\boldsymbol{z}_{-u}=\boldsymbol{c}_{-u}$. If $j\in u$ and $z_j=c_j$, then $g(\boldsymbol{z})_{u,\boldsymbol{c}}=0$. We do not need independence of the $z_j$, or even randomness for them and we do not need mean squares. What we need is that when $g(\boldsymbol{z})$ is defined, so is $g(\boldsymbol{z}_u{:}\boldsymbol{c}_{-u})$ for any $u\subseteq1{:}d$.
The main effect in an anchored decomposition is $g_{j,\boldsymbol{c}}(\boldsymbol{z}) = g(\boldsymbol{z}_j{:}\boldsymbol{c}_{-j})-g(\boldsymbol{c})$ and the two factor term for indices $j\ne k$ is
The version of (ref) for the anchored decomposition is
as shown by kuo:sloa:wasi:wozn:2010.
Shapley value shap:1952 is used in game theory to define a fair allocation of rewards to a team that has cooperated to produce something of value. Suppose that a team of $d$ people produce a value $\mathrm{val}( 1{:}d)$, and that we have at our disposal the value $\mathrm{val}(u)$ that would have been produced by the team $u$, for all $2^d$ teams $u\subseteq1{:}d$. Let $\phi_j$ be the reward for player $j$.
Shapley introduced quite reasonable criteria:
He found that the unique valuation that satisfies all four of these criteria is
Formula (ref) is not very intuitive. Another way to explain Shapley value is as follows. We could build a team from ${\varnothing}$ to $1{:}d$ in $d$ steps, adding one member at a time. There are $d!$ different orders in which to add team members. The Shapley value $\phi_j$ is the increase in value coming from the addition of member $j$, averaged over all $d!$ different orders. For large $d$, random permutations can be sampled to estimate $\phi_j$. From equation (ref) we see that Shapley value does not change if we add or subtract the same quantity from all $\mathrm{val}(u)$. It can be convenient to make $\mathrm{val}({\varnothing})=0$.
When we apply Shapley value to the black box function $f(\boldsymbol{x})$ of the target data subject $\boldsymbol{x}_t$, we define a player as an input predictor $x_{tj}$. The presence of the player $j$ means that the value of $x_{tj}$ is known to the black box function $f(\boldsymbol{x}_t)$. Shapley additive explanation (SHAP) of lund:lee:2017 uses the value function $$ \mathrm{val}(u) = \mathbb{E}(f(\boldsymbol{x})\!\mid\!\boldsymbol{x}_{u}) - \mathbb{E}(f(\boldsymbol{x})). $$ The value of $\mathbb{E}(f(\boldsymbol{x})\!\mid\!\boldsymbol{x}_u)$ depends on the data distribution that is implicitly or explicitly assumed. sund:najm:2019 and kuma:2020 give various definitions.
Interventional Shapley values assume that all combinations of input predictors are possible. That is, the inputs are functionally independent. Baseline Shapley (BS) sund:najm:2019 defines the value function for a predictor set as $$ \mathrm{val}_{\mathrm{BS}}(u) = f(\boldsymbol{x}_{t,u}{:}\boldsymbol{x}_{b,-u}) - f(\boldsymbol{x}_b). $$ It replaces predictors one by one from baseline to subject value. KernelSHAP lund:lee:2017 computes baseline Shapley value for a kernel regression approximation to $f$, for computational efficiency. It also supports ABS, which like the quantitative input influence (QII) measure of datt:2016 assumes statistically independent predictors, using the empirical margins.
Conditional Shapley values such as cohort Shapley allow for predictor variable dependence. When the similarity measures are all identity, then cohort Shapley, defined below, is equivalent to the Conditional Expectation Shapley of sund:najm:2019 on the empirical distribution $\hat D$ of the data, denoted $\mathrm{CES}(\hat D)$. Cohort Shapley extends $\mathrm{CES}(\hat D)$ with similarity measures to better handle continuous variables. CondKernelSHAP aas:2019 implements conditioning using kernel weights derived from the Mahalanobis distance and optionally restricted to $K$-nearest neighbor points only.
The cohort Shapley measure designates subjects as similar or not similar to the target subject $t$ for each of $d$ predictors. Then for each subset of predictors, there is a set of subjects similar to the target for all of those predictors. We call these subsets cohorts. The cohorts are not empty because they all include subject $t$. Cohort Shapley value is Shapley value applied to cohort means. First we describe similarity.
For each predictor $j$, we define a target-specific similarity function $z_{tj}:X_j\to\{0,1\}$. If $z_{tj}(x_{ij})=1$, then subject $i$ is considered to be similar to subject $t$ as measured by predictor $j$. Otherwise $z_{tj}(x_{ij})=0$ means that subject $i$ is dissimilar to subject $t$ for predictor $j$. We always have $z_{tj}(x_{tj})=1$. The simplest similarity is identity: $$ z_{tj}(x_{ij}) =
$$ which is reasonable for binary predictors or those taking a small number of levels. For real-valued predictors, there may be no $i\ne t$ with $x_{ij}=x_{tj}$ and then we might instead take $$ z_{tj}(x_{ij}) =
$$ where subject matter experts have chosen $\delta_{tj}$. Taking $\delta_{tj}=0$ recovers the identity measure of similarity. The two similarity measures above generate an equivalence relation on $X_j$, if $\delta_{tj}$ does not depend on $t$. In general, we do not need $z_{tj}$ to be an equivalence. For instance, we do not need $z_{tj}(x_{ij})=z_{ij}(x_{tj})$ and would not necessarily have that if we used relative distance to define similarity, via $$ z_{tj}(x_{ij}) =
$$ In some settings equivalence is a useful simplification.
We use $1{:}n=\{1,2,\dots,n\}$ to define our set of subjects. Let $$C_{t,u} = \{ i \in 1{:}n \mid z_{tj}(x_{ij})=1,\,\ \text{for all $j\in u$} \},$$ with $C_{t,{\varnothing}}=1{:}n$ by convention. Then $C_{t,u}$ is the cohort of subjects that are similar to the target subject for all predictors $j\in u$ but not necessarily similar for any predictors $j\not\in u$. These cohorts are never empty, because we always have $t\in C_{t,u}$. We write $|C_{t,u}|$ for the cardinality of the cohort. As the cardinality of $u$ increases, the cohort $C_{t,u}$ focuses in on the target subject.
Given a set of cohorts, we define cohort averages $$ \bar y_{t,u} = \frac1{|C_{t,u}|}\sum_{i\in C_{t,u}}y_i. $$ Then the value of set $u$ is $$ \mathrm{val}_{\mathrm{CS}}(u) =\bar y_{t,u}-\bar y_{t,{\varnothing}}=\bar y_{t,u}-\bar y, $$ where $\bar y=(1/n)\sum_{i=1}^ny_i$. The last equality follows because the cohort with $u={\varnothing}$ is the whole data set. The total value to be explained is $$\mathrm{val}_{\mathrm{CS}}(1{:}d) =\bar y_{t,1:d}-\bar y_{t,{\varnothing}} =\bar y_{t,1{:}d}-\bar y. $$ It may happen that $C_{t,1:d}$ is the singleton $\{t\}$. In that case, the total value to be explained is $f(\boldsymbol{x}_t)-\bar y$. In this and other settings, some of the value $\bar y_{t,u}$ may be the average of a very small number of subjects' predictions, and potentially poorly determined but much better determined than points that are extreme extrapolations or are even impossible.
Equation (ref) yields cohort Shapley values $$ \phi_j^{\mathrm{CS}} = \frac1d \sum_{u\subseteq -j} { d-1 \choose |u|}^{-1} ( \bar y_{t,u+j}-\bar y_{t,u}). $$ Here $\bar y_{t,u+j}-\bar y_{t,u}$ is the difference that refining on predictor $j$ makes when we have already refined on the predictor set $u$.
We use Shapley value for local explanation because it uses no impossible data combinations. Here we show that the Shapley effects of song:nels:stau:2016 can be disaggregated into a local squared cohort Shapley effect, when the predictors are given their empirical distribution.
Shapley effects arise from a value function $\mathrm{val}_\mathrm{var}(u)=\mathrm{var}(\mathbb{E}(y\!\mid\!\boldsymbol{x}_u))$. When $\boldsymbol{x}$ has independent components, then owen:2014 shows that the Shapley value for $j$ is
but the dependent data case is much harder.
We may write the value function as
where $P(\boldsymbol{x})$ is the probability density function of $\boldsymbol{x}$ and we retain the same notation for discrete $\boldsymbol{x}$. We take $P(\boldsymbol{x})$ to be the empirical distribution of the data set $\boldsymbol{X}=\{\boldsymbol{x}_1,\dots,\boldsymbol{x}_n\}$ coarsened by a similarity function. Then $P(\boldsymbol{x}\!\mid\!\boldsymbol{x}_{t,u})$ represents the discrete distribution on $\boldsymbol{x}_i$ obtained by choosing subject $i$ uniformly at random from the cohort $C_{t,u}$. Then under $P$, $$\mathbb{E}(f(\boldsymbol{x})\!\mid\!\boldsymbol{x}_{t,u}) = \frac1{|C_{t,u}|} \sum_{i \in C_{t,u}}f(\boldsymbol{x}_i)=\bar y_{t,u}.$$ We also have $\mathbb{E}(f(\boldsymbol{x})) = \bar y$ for this $P$. Finally, $P$ places equal probability $1/n$ on the data and so $$ \mathrm{val}_\mathrm{var}(u) =\frac1n\sum_{t=1}^n(\bar y_{t,u}-\bar y)^2. $$
Next, in addition to ordinary cohort Shapley with value function $\mathrm{val}_{\mathrm{CS}}(t,u)=\bar y_{t,u}-\bar y_{\varnothing}=\bar y_{t,u}-\bar y$ we introduce a squared version with $$ \mathrm{val}_{\mathrm{CS}2}(t,u) = (\bar y_{t,u}-\bar y)^2. $$ Then,
From the additivity axiom of Shapley value, the variance explained Shapley value $\phi^{\mathrm{var}}_j$ is the average of the squared version of cohort Shapley value $\phi^{\mathrm{CS}2}_j$,
The point of equations (ref) and (ref) is that squared values give an aggregation/disaggregation between local and global measures that satisfies the Shapley axioms. Unsquared Shapley values, whether baseline or cohort, give uninteresting aggregate Shapley values of zero.
Here we compare importances of three Shapley values, BS, ABS, and CS. For BS, the baseline is $\boldsymbol{x}_b = (1/n)\sum_{i=1}^n\boldsymbol{x}_i$. For ABS, all subjects are used for the baselines. We also define squared versions. Here, for simplicity, we assume no other subject matches the target on all predictors. Then the total value function of CS and its squared version are
The value functions of BS and its squared difference version are
Now if $f(\boldsymbol{x}_b)=\bar y$, as for instance it would be for linear $f$ and $\boldsymbol{x}_b$ equal to the average of $\boldsymbol{x}_i$, then both $\mathrm{val}_\mathrm{BS}(t,1{:}d)=\mathrm{val}_\mathrm{CS}(t,1{:}d)$ and $\mathrm{val}_{\mathrm{BS}2}(t,1{:}d)=\mathrm{val}_{\mathrm{CS}2}(t,1{:}d)$.
The value function of ABS and its squared difference version are defined as
We see that $\mathrm{val}_{\mathrm{ABS}} = \mathrm{val}_{\mathrm{CS}}$. We can also show that $\mathrm{val}_{\mathrm{ABS}2} \geqslant\mathrm{val}_{\mathrm{CS}2}$ with equality when and only when $f(\boldsymbol{x}_b)=f(\boldsymbol{x}_t)$ for all $t$, a very unlikely possibility. Thus, while ABS decomposes the same value that CS does, their squared counterparts decompose quite different quantities.
stru:kono:2010 derived a Shapley value under independent predictors uniformly distributed over finite discrete sets but they could as well be countable or continuous and non-uniform, so long as they are independent (see the appendix). We give a proof of this different from theirs, and show how both baseline Shapley value and cohort Shapley value are derived from functional decompositions.
For Shapley value, every variable is either `in' or `out', and so binary variables underlie the approach. Here we compute Shapley values based on function decompositions of a function $g$ defined on $\{0,1\}^d$. The $2^d$ values of that function might themselves be expectations, like the cohort mean in cohort Shapley or the quantity stru:kono:2010 use (see the appendix), but for our purposes here they are just $2^d$ numbers.
We use $\boldsymbol{e}_j$ to represent the binary vector of length $d$ with a one in position $j$ and zeroes elsewhere. This is the $j$'th standard basis vector. We then generalize it to $\boldsymbol{e}_u=\boldsymbol{1}_u{:}\boldsymbol{0}_{-u}$ for $u\subseteq1{:}d$. An arbitrary point in $\{0,1\}^d$ is denoted by $\boldsymbol{z}$.
Let $g$ be a function on $\{0,1\}^d$. In our applications, the total value to be explained is $g(\boldsymbol{1})-g(\boldsymbol{0})$, with $\boldsymbol{1}$ corresponding to matching the target in all $d$ ways and $\boldsymbol{0}$ corresponding to no matches at all. The value contributed by $u\subseteq1{:}d$ is $g(\boldsymbol{e}_u)-g(\boldsymbol{0})$.
Because we use the anchored decomposition for functions on $\{0,1\}^d$ instead of the ANOVA, we do not need to define a distribution for $\boldsymbol{z}$. The anchored decomposition on $\{0,1\}^d$ with anchor $\boldsymbol{c}=\boldsymbol{0}$ has a simple structure.
Now we find the Shapley value for a function on $\{0,1\}^d$ in an anchored decomposition. stru:kono:2010 proved this earlier using different methods.
For $u\subseteq1{:}d$ and $\boldsymbol{e}_u=\boldsymbol{1}_u{:}\boldsymbol{0}_{-u}\in\{0,1\}^d$ the $g$ function for baseline Shapley is $$g(\boldsymbol{e}_u)=\mathrm{val}_{\mathrm{BS}}(t,u) = f(\boldsymbol{x}_{t,u}{:}\boldsymbol{x}_{b,-u}) - f(\boldsymbol{x}_b).$$ The corresponding function for cohort Shapley is $$g(\boldsymbol{e}_u) = \mathrm{val}_{\mathrm{CS}}(t,u) = \bar y_{t,u}-\bar y.$$ In variance Shapley $g(\boldsymbol{e}_u)=\mathrm{var}(\mathbb{E}(f(\boldsymbol{x})\!\mid\!\boldsymbol{x}_u))$. It is Shapley value applied to the anchored decomposition of $g$ where the $2^d$ values $g(\boldsymbol{e}_u)$ are in turn defined through an ANOVA decomposition.
In this section we include some numerical examples of cohort Shapley. First we look at variables that predict survival in the Titanic data set. Then we look at the variables that predict value in the Boston housing data set.
Here we consider a subset of the Titanic passenger data set (from \url{http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.csv}) containing 1045 individuals with complete records. This data was collected from Encyclopedia Titanica (\url{https://www.encyclopedia-titanica.org/}) and has been used by Kaggle (see \url{https://www.kaggle.com/c/titanic/data}) to illustrate machine learning. As the function of interest, we construct a logistic regression model which predicts `survival' based on the predictors `pclass', `sex', `age', `sibsp', 'parch', and `fare' (defined at the Vanderbilt web site). Our logistic model outputs an estimated probability of survival, $f(\boldsymbol{x}_t)\in[0,1]$. To calculate the cohort Shapley values, we define similarity as exact for the discrete predictors `pclass', `sex', `sibsp', and `parch' and within a distance of less than 1/10 of the variable range on the continuous predictors `age' and `fare'.
Figure (ref) shows the variance Shapley values, known as `Shapley effects' song:nels:stau:2016. They decompose the total variance into importance of predictor variables. The results indicate that `sex' is the most important followed by `pclass', and `fare'.
Figure (ref) shows the cohort Shapley values for each predictor stacked vertically for every individual. The individuals are ordered by their predicted survival probability. Starting at zero, we plot a blue bar up or down according to the cohort Shapley value for the `sex' variable. Then comes a yellow bar for `pclass' and so on. The squared version of the cohort Shapley is a decomposition of Variance Shapley in Figure (ref). For a comparison of cohort, baseline, all baseline Shapley values, and for their squared versions, see the appendix.
A visual inspection of Figure (ref) reveals clusters of individuals with similar Shapley values for which we could potentially develop a narrative. As just one example, we see passengers with indices between roughly 400 and 600 who have negative Shapley values for `sex' but positive Shapley values for `pclass' while their predicted value is below the mean. Many of the these passengers are men who are not in the lowest class.
The Boston housing dataset has 506 data points with 13 predictors and the median house value as a response harr:rubi:1978. Each data point corresponds to a vicinity in the Boston area. We fit a regression model to predict the house price from the predictors using XGBoost chen:gues:2016.
This dataset is of interest to us because it includes some striking examples of dependence in the predictors. For instance, the variables `CRIM' (a measure of per capita crime) and `ZN' (the proportion of lots zoned over 25{,}000 square feet) can be either near zero or large, but none of the 506 data points have both of them large and similar phenomena can be seen in some other scatterplots.
We will compute baseline Shapley and cohort Shapley for one target point. That one is the 205'th case in the sklearn python library and also in the mlbench R package. This target was chosen to be one for which some synthetic points in baseline Shapley would be far from any real data, but we did not optimize any criterion measuring that distance, and many of the other 506 points share that property. For Shapley values of all subjects, see the appendix. For cohort Shapley, we consider predictor values to be similar if their distance is less than 1/10 of the difference between the 95'th and 5'th percentiles of the predictor distribution.
Figure (ref) shows two scatterplots of the Boston housing data. It marks the target and baseline points, depicts the cohort boundaries and it shows housing value in gray scale. The baseline point is $\boldsymbol{x}_b =(1/n)\sum_{i=1}^n\boldsymbol{x}_i$, the sample average, and it is not any individual subject's point partly because it averages some integer valued predictors. Here, the predicted house prices are 28.38 for the subject and 13.43 for the baseline. The figure also shows some of the synthetic points used by baseline Shapley. Some of those points are far from any real data points even in these two dimensional projections. Model fits at such points are questionable.
Figure (ref) shows cohort Shapley and baseline Shapley values for this target subject. On baseline Shapley, we see that `CRIM', `RM', and `LSTAT' have very large impact and the other variables do not. For cohort Shapley, the most impactful predictors are `RM', `ZN' and `LSTAT' followed by a very gradual decline.
In baseline Shapley `CRIM' was the most important variable, while in cohort Shapley it is one of the least important variables. We think that the explanation is from the way that baseline Shapley uses the data values at the upper orange cross in the left plot of Figure (ref). The predicted price for a house at the synthetic point given by the upper orange cross is 14.17, which is much smaller than that of the subject, and even quite close to the baseline mean. This leads to the impact of `CRIM' being very high. Data like that synthetic point were not present in the training set and so that value represents an extrapolation where we do not expect a good prediction. We believe that an unreliable prediction there gave the extreme baseline Shapley value that we see for `CRIM'.
Related to the prior point, refining the cohort on `RM' reduces its cardinality much more than refining the cohort on `CRIM' does. Because cohort Shapley uses averages of actual subject values, refining the target on `CRIM' removes fewer subjects and in this case makes a lesser change.
The right panel in Figure (ref) serves to illustrate the effect of dependent predictors on cohort Shapley value. The model for price hardly uses `ZN', if at all, and the baseline Shapley value for it is zero. Baseline Shapley atributes a large impact to `LSTAT' and nearly none to `ZN'. For either of those predictors, the cohort mean is higher than the global average, and both `LSTAT' and `ZN' have high impact in cohort Shapley.
We can explain the difference as follows. As `ZN' increases, the range of `LSTAT' values narrows, primarily by the largest `LSTAT' values decreasing as `ZN' increases. Refining on `ZN' has the side effect of lowering `LSTAT'. Even if `ZN' itself is not in the model, the cohort Shapley value captures this effect. Baseline Shapley cannot impute a nonzero impact for a variable that the model does not use.
Cohort Shapley resolves two conceptual problems in baseline Shapley and other interventional methods. First, it does not use any impossible or even unseen predictor combinations. Second, if two predictors are identical then it is perfectly predictable that their importances will be equal rather than subject to details of which black box model was chosen or what random seed if any was used in fitting that model. Cohort Shapley also allows the user to study indirect influence which is impossible via baseline Shapley.
It does not escape the catch-22 that kuma:2020 mention. Cohort Shapley requires the user to specify a distribution on predictors. That distribution is the empirical distribution coarsened by a similarity definition. The importance measure will depend on the chosen similarity measure which can be an advantage but may also be a burden. Baseline Shapley and cohort Shapley have different counterfactuals and they address different problems.
There is a related literature on how finely a continuous variable should be broken into categories. gelm:park:2009 suggest as few as three levels for the related problem of choosing a discretization prior to fitting a model. We have used more levels but this remains an area for future research.
Cohort Shapley supports another choice not possible with baseline Shapley. We can replace the function values $f(\boldsymbol{x}_i)$ by the original response values in the cohort averages. The resulting sensitivity measures are for a similarity-based nearest neighbor rule.
In our appendix we investigate numerically the proportion of unrealistic data points created when sampling independently from marginal distributions. Using a working definition of `realistic' we find that marginal sampling produces many more unrealistic values than one gets in hold out data sets.
Our cohort Shapley method can be used to detect unethical practices such as redlining, even when the protected variable or variables were not used in the prediction model. We take care to point out that a large cohort Shapley value for a protected variable is not in itself proof of redlining. Followup investigations would be required to reach that conclusion.
We thank Masashi Egi of Hitachi for valuable comments. This work was supported by the U.S. National Science Foundation under grant IIS-1837931.
\eject