Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
35,789 characters · 16 sections · 46 citation commands
Cohort Shapley value for algorithmic fairness
Machine learning is now commonly used to make consequential decisions about people, affecting hiring decisions, loan applications, medical treatments, criminal sentencing and more. It is important to understand and explain these decisions. A critical part of understanding a decision is quantifying the importance of the variables used to make it. When the decisions are about people and some of the variables describe protected attributes of those people, such as race and gender, then variable importance has a direct bearing on algorithmic fairness.
As we describe below, most variable importance measures work by changing a subset of the input variables to a black box. We survey such methods below and argue against that approach. The first problem is that the resulting analysis can depend on some very unreasonable variable combinations. A second problem with changing inputs to a black box is that it necessarily attributes zero importance to a variable that the algorithm did not use. If a protected variable is not actually used in the black box then the algorithm would automatically be considered fair. However this `fairness through unawareness' approach is not reliable, as adle:2018 and many others have noted. Information about the protected variables can leak in through others with which they are associated. The practice known as `redlining' involves deliberate exploitation of such associations. When studying fairness we must have the possibility of studying a variable not included in the black box. In this paper we develop and illustrate an approach to algorithmic fairness that does not use impossible values and can detect redlining.
Variable importance measures have a long history and there has been a recent surge in interest motivated by problems of explainable AI. The global sensitivity analysis literature studies black box functions used in engineering and climate models among others. Much of that work is based on a functional ANOVA model of the function relating the output to inputs. Variance explained is partitioned using Sobol' indices sobo:1993. salt:etal:2008 is an introductory textbook and raza:etal:2021 is a current survey of the field. wei:lu:song:2015 provide a comprehensive survey of variable importance measures in statistics. They include 197 references of which 24 are themselves surveys. moln:2018 surveys variable importance measures used in explainable AI. Prominent among these are SHAP lund:lee:2017 and LIME ribe:etal:2016. LIME makes a local linear approximation to a black box function $f(\cdot)$ and one can then take advantage of the easier interpretability of linear models. SHAP is based on Shapley value from cooperative game theory shap:1953, that we describe in more detail below.
Ordinarily we can compare two prediction methods by waiting to see how accurate they are on unknown future data or, if necessary, use holdout data. This does not carry over to deciding whether LIME or SHAP or some other method has made a better explanation of a past decision. Suppose that we want to understand something like why a given applicant was turned down for a loan. When $f(\cdot)$ is available to us in closed form then there is no doubt at all about whether the loan would have been offered under any assortment of hypothetical feature combinations that we choose. We can compare methods by how they define importance. Choosing among definitions is a different activity, essentially a philosophical one, and we face tradeoffs. Our problem is one of identifying the causes of given effects when we have perfect knowledge of $f(\cdot)$ for all input variable combinations. This is different from the more common learning task of quantifying the effects of given causes. That distinction was made by holl:1988 in studying causal inference. It goes back at least to mill:1843.
A problem with many, but not all, measures of variable importance is that they are based on changing some feature values from one level to another, while holding other features constant and then looking at the changes to $f(\cdot)$. When the underlying features are highly correlated, then some of these combinations can be quite implausible, casting doubt on any variable importance methods that use them. In extreme cases, the combinations can be physically impossible (e.g., systolic blood pressure below diastolic) or logically impossible (e.g., birth date after graduation date). The use of these combinations has also been criticized by hook:ment:2019:tr.
To avoid using impossible values, Cohort Shapley method\footnote{https://github.com/cohortshapley/cohortshapley} from mase:owen:seil:2019 uses only observed data values. For any target subject $t\in\{1,\dots,n\}$ and any feature variable $j=1,\dots,d$ we obtain a Shapley value $\phi_j=\phi_j(t)$ that measures the impact of the value of feature $j$ on $f(\cdot)$ for subject $t$. As we describe below, those impacts can be positive or negative. Using one of the Shapley axioms we will be able to aggregate from individual subjects to an impact measure for the entire data set. We can also disaggregate from the entire data set to a subset, such as all subjects with the protected level of a protected variable.
In this paper we use the cohort Shapley method from mase:owen:seil:2019 to measure variable importance. One of the features of cohort Shapley is that it can attribute importance to a variable that is not actually used in $f(\cdot)$. This is controversial and not all authors approve but it is essential in the present context of algorithmic fairness.
Our contribution is to provide a method of quantifying bias at both individual and group levels with axiomatic consistency between the individual and group measures and without requiring any impossible or unobserved combinations of input variables. The approach can even be used when the prediction algorithm is not available as for instance when it is proprietary.
We investigate the COMPAS data angw:2016 which include predictions of who is likely to commit a crime. There is great interest in seeing whether the algorithm is unfair to Black subjects. In the given context, the prediction code $f(\cdot)$ is a proprietary algorithm, unavailable to us and so we cannot change any of the values in it. What we have instead are the predictions on a set of subjects. Cohort Shapley can work with the predictions because it does not require any hypothetical feature combinations. Figure (ref) shows cohort Shapley impacts for the race of the subjects in the COMPAS data, computed in a way that we describe below. As it turns out the impact of race on whether a subject was predicted to reoffend was always positive for Black subjects and it was always negative for White subjects. For some other measures that we show, the histograms overlap. The average impact for Black subjects was $0.067$ and the average for White suspects was $-0.101$. For context, the response values were $1$ for those predicted to reoffend and $0$ for those predicted not to reoffend.
An outline of this paper is as follows. Section (ref) introduces our notation, defines Shapley value, and presents a few of the variable importance measures and fairness definitions from the literature. Section (ref) describes the COMPAS recidivism data set. Section (ref) has our analysis of that data including a Bayesian bootstrap for uncertainty quantification. Section (ref) has our conclusions. An appendix presents some additional COMPAS results like the ones we selected for discussion.
For subjects $i=1,\dots,n$, the value of feature $j=1,\dots,d$ is $x_{ij}\in\mathcal{X}_j$. We consider categorical variables $x_{ij}$. Continuously distributed values can be handled as discussed in Section (ref). The features for subject $i$ are encoded in $\boldsymbol{x}_i\in\mathcal{X} =\prod_{j=1}^d\mathcal{X}_j$. For each subject there is a response value $y_i\in \mathbb{R}$. The feature indices belong to the set $1{:}d\equiv\{1,\dots,d\}$ and similarly the subject indices are in $1{:}n\equiv\{1,\dots,n\}$. For $u\subseteq1{:}d$ the tuple $\boldsymbol{x}_u$ is $(x_j)_{j\in u}\in\mathcal{X}_u=\prod_{j\in u}\mathcal{X}_j$, and $\boldsymbol{x}_{iu}=(x_{ij})_{j\in u}$.
There is also an algorithmic prediction $\hat y_i$. It is usual in variable importance problems to have $\hat y_i =f(\boldsymbol{x}_i)$ for a function $f(\cdot)$ that may be difficult to interpret (e.g., a black box). This $f(\cdot)$ is often an approximation of $\mathbb{E}(y\!\mid\! \boldsymbol{x})$ though it need not be of that form. Cohort Shapley uses only the values $\hat y_i$ so it does not require access to $f(\cdot)$. Since it only uses a vector of values, one per subject, it can be used to find the important variables in $y_i$, or $\hat y_i$ or combinations such as the residual $y_i-\hat y_i$.
In many settings $y_i$ and $\hat y_i$ are both binary. The value $y_i=1$ may mean that subject $i$ is worthy of a loan, or is predicted to commit a crime, or should be sent to intensive care and $\hat y_i\in\{0,1\}$ is an estimate of $y_i$. In this case, measures such as $\mathrm{FP}_i = 1\{\hat y_i=1\ \&\ y_i=0\}$ describing a false positive are of interest.
Here are some typographic conveniences that we use. When $j\not\in u$, then $u+j$ is $u\cup\{j\}$. For $u\subseteq1{:}d$ we use $-u$ for $1{:}d\setminus u$. The subscript to $\boldsymbol{x}$ may contain a comma or not depending on what is clearer. For instance, we use $\boldsymbol{x}_{iu}$ but also $\boldsymbol{x}_{i,-u}$.
Shapley value shap:1953 is used in game theory to define a fair allocation of rewards to a team that has cooperated to produce something of value. Many variable importance problems can be formulated as a team of input variables generating an output value or an output variance explained. We then want to apportion importance or impact to the individual variables.
Suppose that a team of $d$ members produce a value $\mathrm{val}( 1{:}d)$, and that we have at our disposal the value $\mathrm{val}(u)$ that would have been produced by the team $u$, for all $2^d$ teams $u\subseteq1{:}d$. Let $\phi_j$ be the reward for player $j$. It is convenient to work with incremental values $\mathrm{val}(j\!\mid\! u)=\mathrm{val}(u+j)-\mathrm{val}(u)$ for sets $u$ with $j\not\in u$.
Shapley introduced quite reasonable criteria:
He found that the unique valuation that satisfies all four of these criteria is
Formula (ref) is not very intuitive. Another way to explain Shapley value is as follows. We could build a team from ${\varnothing}$ to $1{:}d$ in $d$ steps, adding one member at a time. There are $d!$ different orders in which to add team members. The Shapley value $\phi_j$ is the increase in value coming from the addition of member $j$, averaged over all $d!$ different orders. From equation (ref) we see that Shapley value does not change if we add or subtract the same quantity from all $\mathrm{val}(u)$. It can be convenient to make $\mathrm{val}({\varnothing})=0$.
We discuss some variable importance measures based on Shapley value below. We use the framework from sund:najm:2019. They survey many uses of Shapley value in explainable AI.
Most mechanisms for investigating variable importance proceed by changing some, but not all, of the input variables to $f(\cdot)$. For $u\subseteq1{:}d$ the hybrid point $\boldsymbol{w}=\boldsymbol{x}_{u}{:}\boldsymbol{z}_{-u}$ has $$ w_j =
$$ We can investigate changes to $\boldsymbol{x}_{u}$ by examining $f(\boldsymbol{x}_{u}{:}\boldsymbol{z}_{-u})-f(\boldsymbol{x})$ for various points $\boldsymbol{x},\boldsymbol{z}\in\mathcal{X}$.
Global sensitivity analysis raza:etal:2021 works with an analysis of variance (ANOVA) defined in terms of $\boldsymbol{x}\in\mathcal{X}$ with independent components for which $\sigma^2 = \mathrm{var}( f(\boldsymbol{x}))<\infty$. The sets $\mathcal{X}_j$ can be discrete or continuous. It is common there to measure the importance of a subset of variables by variance explained. For instance, the lower Sobol' index for variables $\boldsymbol{x}_u$ is $\underline\tau^2_u=\mathrm{var}( \mathbb{E}(f(\boldsymbol{x})\!\mid\!\boldsymbol{x}_u))$ which is usually normalized to $\underline\tau^2_u/\sigma^2$. The upper Sobol' index is $\overline\tau^2_u=\sigma^2-\underline\tau^2_{-u}$, that is, everything not explained by $\boldsymbol{x}_{-u}$ is attributed to $\boldsymbol{x}_u$. These measures are very natural in settings where the components of $\boldsymbol{x}$ can all vary freely and independently, but such is usually not the case for inputs to a black box machine learning model.
The quantitative influence function (QII) method of datt:2016 uses a model in which the features are statistically independent from their marginal distributions.
In baseline Shapley (see sund:najm:2019), there is a baseline tuple $\boldsymbol{x}_b\in\mathcal{X}$ of $\boldsymbol{x}$, such as the sample average $\bar\boldsymbol{x}$. Then the value for variable subset $u$ and subject $t$ is $f(\boldsymbol{x}_{t,u}{:}\boldsymbol{x}_{b,-u})$. We get the same Shapley values using $\mathrm{val}(u)=f(\boldsymbol{x}_{t,u}{:}\boldsymbol{x}_{b,-u})-f(\boldsymbol{x}_b)$. Then the total value to explain for subject $t$ is $f(\boldsymbol{x}_t)-f(\boldsymbol{x}_b)$. The counterfactuals are changes in some subset of the values in $\boldsymbol{x}_t$. Important variables are those that move the value the most from the baseline $f(\boldsymbol{x}_b)$ to subject $t$'s value. Shapley's combination provides a principled weighting of the effect of changing $x_{b,j}$ to $x_{t,j}$ given some subset $u$ of variables that have also been changed.
Conditional expectation Shapley sund:najm:2019 uses a joint distribution $D$ on $\mathcal{X}$ and then for subject $t$ $\mathrm{val}(u)=\mathbb{E}_D(f(\boldsymbol{x})\mid\boldsymbol{x}_u=\boldsymbol{x}_{t,u})$ under $D$. SHAP lund:lee:2017 is a conditional expectation Shapley using a distribution $D$ in which the features are independent. Cohort Shapley mase:owen:seil:2019 that we describe next is very nearly conditional expectation Shapley with $D$ equal to the empirical distribution.
In cohort Shapley the counterfactual does not involve replacing $\boldsymbol{x}_{tu}$ by $\boldsymbol{x}_{bu}$. It is instead about concealing the values of $\boldsymbol{x}_{tu}$. It is what kuma:2020 call a conditional method because it requires specification of a conditional distribution on the features. The version we present here is a form of conditional expectation Shapley using the empirical distribution of the data.
For every pair of values $x_{tj},x_{ij}\in\mathcal{X}_j$ we can declare them to be similar or not. In the present context we take similarity to just be $x_{tj}=x_{ij}$. For a target subject $t\in1{:}n$ and set $u\subseteq1{:}d$ we define the cohort $C_u = C_{tu} =\{i\in1{:}n\mid \boldsymbol{x}_{iu}=\boldsymbol{x}_{tu}\}$. These are the subjects who match subject $t$ on all of the variables $j\in u$. They may or may not match the target on $j\not\in u$. None of the cohorts is empty because they all contain $t$. The value function in cohort Shapley is the cohort mean $$ \mathrm{val}(u) = \frac1{|C_{tu}|} \sum_{i\in C_{tu}}\hat y_i. $$ Here $\hat y_i=f(\boldsymbol{x}_i)$ but we write it this way because in practice we might not have $f(\cdot)$ at our disposal, just the predictions $\hat y_i$.
The incremental value $\mathrm{val}(j\!\mid\! u)$ is then how much the cohort mean moves when we reveal $x_{tj}$ in addition to previously revealed variables $\boldsymbol{x}_{tu}$. Having specified the values, the Shapley formula provides attributions. Note that $\mathrm{val}({\varnothing})$ is the plain average of all $\hat y_i$ and Shapley value is unchanged by subtracting $\mathrm{val}({\varnothing})$ from all of the $\mathrm{val}(u)$. Furthermore, when there are many variables, then very commonly $C_{t,1{:}d}=\{t\}$ and then the total value to be explained, $\mathrm{val}(1{:}d)-\mathrm{val}({\varnothing})$ is simply $\hat y_t-(1/n)\sum_{i=1}^n\hat y_i$.
Just as there are many ways to define variable importance, there are multiple ways to define what fairness means. Some of those definitions are mutually incompatible and some of them differ from legal definitions. Here we present a few of the issues. See corb:goel:2018, chou:roth:2018, berk:2018, and frie:etal:2019 for surveys. We do not make assertions about which definitions are preferable.
For $y,\hat y\in\{0,1\}$, let $n_{y\hat y}$ be the number of subjects with $y_i=y$ and $\hat y_i=\hat y$. These four counts and their derived properties can be computed for any subset of the subjects. The false positive rate (FPR) is $n_{01}/n_{0\text{\tiny$\bullet$}}$, where a bullet indicates that we are summing over the levels of that index. We ignore uninteresting corner cases such as $n_{0\text{\tiny$\bullet$}}=0$; when there are no subjects with $y=0$ then we have no interest in the proportion of them with $\hat y=1$. The false negative rate (FNR) is $n_{10}/n_{1\text{\tiny$\bullet$}}$. The prevalence of the trait under study is $p=n_{1\text{\tiny$\bullet$}}/n_{\text{\tiny$\bullet$}\text{\tiny$\bullet$}}$. The positive predictive value (PPV) is $n_{11}/n_{\text{\tiny$\bullet$}1}$. As chou:2017 notes, these values satisfy
See also klei:mull:ragh:2016.
Equation (ref) shows how some natural definitions of fairness conflict. FPR and FNR describe $\hat y\!\mid\! y$, while PPV describes $y\!\mid\!\hat y$. If two subsets of subjects have the same PPV but different prevalences $p$, then they cannot also match up on FPR and FNR. Fairness in $y\!\mid\! \hat y$ terms and fairness in $\hat y\!\mid\! y$ terms can only coincide in trivial settings such as when $\hat y=y$ always or empirically unusual settings with equal prevalence between subjects having different values of a protected attribute.
hard:2016 writes about measures of demographic parity, equalized odds and equal opportunity. Demographic parity, requires $\Pr( \hat y=1\!\mid\! \boldsymbol{x}_p)=\Pr(\hat y=1)$ where $\boldsymbol{x}_p$ specifies the levels of one (or more) protected variables. Equalized odds requires $\Pr(\hat y=1\!\mid\! y=y', \boldsymbol{x}_p) = \Pr(\hat y=1\!\mid\! y=y'), y' \in {0,1}$, so that false positive rates and true positive rates are both equal across groups. Because $\mathrm{FNR} = 1 - \mathrm{TPR}$ the false negative rates are also equal. Equal opportunity is defined in terms of a preferred outcome. When $y=0$ is preferred, it requires that $\mathrm{Pr}(\hat y=1\!\mid\! y=0,\boldsymbol{x}_p)=\mathrm{Pr}(\hat y=1\!\mid\! y=0)$ That is, the false positive rate is unaffected by the protected variables.
chou:2017 considers a score $s$ (such as $\hat y$) to be well calibrated if $\mathrm{Pr}(y=1\!\mid\! s ,\boldsymbol{x}_p)=\mathrm{Pr}(y=1\!\mid\! s)$ for all levels of $s$. A score attains predictive parity if $\mathrm{Pr}(y=1\!\mid\! s>s_*,\boldsymbol{x}_p)=\mathrm{Pr}(y=1\!\mid\! s>s_*)$ for all thresholds $s_*$. The distinction between well calibratedness and predictive parity is relevant when there are more than two levels for the score.
There is some debate about when or whether using protected variables can lead to improved fairness. See xian:2020 who gives a summary of legal issues surrounding fairness and corb:etal:2017 who study whether imposing calibration or other criteria might adversely affect the groups they are meant to help.
The additivity axiom of Shapley value is convenient for us. It means that importances for the residual $y_i-\hat y_i$ is simply the difference of importances for $y_i$ and $\hat y_i$. We can aggregate importances from subjects to the whole data set by summing, or more interpretably averaging over $t=1,\dots,n$. We can also disaggregate from the whole data set to subsets of special interest by averaging cohort Shapley values over target subjects $t\in v\subset 1{:}n$.
COMPAS is a tool from Northpointe Inc. for judging the risk that a criminal defendant will commit another crime (re-offend) within two years. Each subject is rated into one of ten deciles with higher deciles considered higher risk of reoffending. angw:2016 investigated whether that algorithm was biased against Black people. They obtained data for subjects in Broward County Florida, including the COMPAS decile, the subjects' race, age, gender, number of prior crimes and whether the crime for which they were charged is a felony or not. angw:2016 describe how they processed their data including how they found followup data on offences committed and how they matched those to subjects for whom they had prior COMPAS scores. They also note that race was not one of the variables used in the COMPAS predictions.
The example is controversial. angw:2016 find that COMPAS is biased because it gave a higher rate of false positives for Black subjects and a higher rate of false negatives for White subjects. flores2016false and dieterich2016compas disagree raising the issue of $\hat y\!\mid\! y$ fairness versus $y\!\mid\! \hat y$ fairness. The prevelance of reoffences differed between Black and White subjects forcing $y\!\mid\!\hat y$ and $\hat y \!\mid\! y$ notions to be incompatible. flores2016false also questioned whether the subjects in the data set were at comparable stages in the legal process to those for which COMPAS was designed.
Following chou:2017 we focus on just Black and White subjects. That provides a sample of 5278 subjects from among the original 6172 subjects. As in that paper we record the number of prior convictions as a categorical variable with five levels: 0, 1--3, 4--6, 7--10 and $>$10. Following angw:2016, we record the subjects' ages as a categorical variable with three levels: $<$25, 25--45, $>$45. Also, following chou:2017, we consider the prediction $\hat y_i$ to be $1$ if subject $i$ is in deciles 5--10 and $\hat y_i=0$ for subject $i$ in deciles 1--4.
In this section we use cohort Shapley to study some fairness issues in the COMPAS data, especially individual level metrics. We have selected what we found to be the strongest and most interesting findings. The larger set of figures and tables from which these are drawn are in the appendix. The appendix also compares some conventional group fairness metrics for Black and White subjects.
We have computed Shapley impacts for these responses: $y_i$, $\hat y_i$, $y_i-\hat y_i$, $\mathrm{FP}_i = 1\{ y_i=0\,\&\, \hat y_i=1\}$ and $\mathrm{FN}_i = 1\{ y_1=1\,\&\, \hat y_i=1\}$. If $\mathrm{FP}_i=1$ then subject $i$ received a false positive prediction. Note that the sample average value of $\mathrm{FP}_i$ $$\hat\mathbb{E}(\mathrm{FP}_i)=\frac{n_{01}}{n_{\text{\tiny$\bullet$}\text{\tiny$\bullet$}}} =\mathrm{FPR}\times \frac{n_{0\text{\tiny$\bullet$}}}{n_{\text{\tiny$\bullet$}\text{\tiny$\bullet$}}} =\mathrm{FPR}\times (1-p).$$ Being wrongly predicted to reoffend is an adverse outcome that we study. Its expected value is the FPR times the fraction of non-reoffending subjects. Similarly, $\hat\mathbb{E}(\mathrm{FN}_i)=\mathrm{FNR}\times p$.
Figure (ref) shows histograms of Shapley impacts of race for the subjects in the COMPAS data. The first panel there reproduces the data from Figure (ref) showing a positive impact for every Black subject and a negative one for every White subject for the prediction $\hat y$. For the actual response $y$, the histograms overlap slightly. By additivity of Shapley value the impacts for $y-\hat y$ can be found by subtracting the impact for $\hat y$ from that for $y$ for each subject $i=1,\dots,n$. The histograms of $y_i-\hat y_i$ show that the impact of race on the residual is typically positive for White subjects and negative for Black subjects.
Overlap of distributions is seen in equal opportunity (false positive rate: FPR) and false negative rate (FNR). In there, small number of adversely affected White and beneficially affected Black are observed. We can inspect further on these overlapped subjects to understand which conditions are exceptional cases. Also we can inspect on tail of the distribution to understand the extreme cases where bias is very likely observed.
Figure (ref) shows histograms of Shapley impacts for race but now they are color coded by the subjects' gender. We see that the impact on $\hat y$ is bimodal by race for both male and female subjects, but the effect is larger in absolute value for male subjects. The impacts for the response, the residual and false negative and positive values do not appear to be bimodal by race for female subjects, but they do for male subjects. The case of FN is perhaps different. The modes in Figure (ref) are close together and not so apparent in Figure (ref) which has slightly different bin boundaries. It is clear from these figures that the race differences we see are much stronger among male subjects.
The appendix includes some figures that each include 25 histograms. They are Shapley impact histograms for all five features color coded by each of the five features. There is one such figure for each of $y$, $\hat y$, $y-\hat y$, $\mathrm{FP}$ and $\mathrm{FN}$.
Tables (ref) through (ref) in the appendix record mean Shapley impacts for the five predictors and our responses measures, disaggregated by race and gender.
We take a particular interest in the residual $y_t-\hat y_t$. It equals $1$ for false negatives and $-1$ for false positives and $0$ when the prediction was correct. Table (ref) shows a subset of the largest values for the residual $y_i-\hat y_i$.
What we see there is that revealing that a subject is Black tells us that, on average, that subject's residual $y_t-\hat y_t$ is decreased by $3.6$%. Revealing that the subject is White increases the residual by $5.4$%. Revealing race makes very little difference to the residual averaged over male or over female subjects. Revealing gender makes quite a large difference of $-8.3$% for female subjects and $+2.0\%$ for male subjects.
To judge the uncertainty in the values in Table (ref) we applied the Bayesian bootstrap of rubi:1981. That algorithm randomly reweights each data point by a unit mean exponential random variable. By never fully deleting any observation it allows one to also bootstrap individual subjects' Shapley values though there is not space to do so in this article. Figure (ref) shows violin plots of 1000 bootstrapped cohort Shapley values.
The risk of being falsely predicted to reoffend involves two factors: having $y_i=0$ and having $\hat y_i=1$ given that $y_i=0$. FPR is commonly computed only over subjects with $y_i=0$. Accordingly, in this section we study it by subsetting the subjects to $\{ i\mid y_i=0\}$ and finding cohort Shapley value of $\hat y_i=1$ for the features.
Figure (ref) shows cohort Shapley impact of race factor conditioned on race factor for FPR and FNR. The distribution of impacts conditioned on race are separated in the figure. This conditional analysis shows a stronger disadvantage for Black subjects.
Variable importance in statistics and machine learning ordinarily considers three issues: whether changing $x_j$ has a causal future impact on $y$, whether omitting $x_j$ from a data set degrades prediction accuracy and whether changing $x_j$ has an important mechanical effect on $f(\boldsymbol{x})$ jian:owen:2003. These are all different from each other and the third choice is the most useful one for explaining the decisions of a prediction method.
Cohort Shapley mase:owen:seil:2019 looks at a fourth issue: whether learning the value of $x_{tj}$ in a sample is informative about the prediction $\hat y_t$. It is well suited to settings such as the COMPAS data from Broward County, where: the original algorithm is not available to researchers, the protected variable of greatest interest was not included in the model, and the set of subjects for which fairness is of interest is different from the set on which the algorithm was trained.
Cohort Shapley does not address counterfactuals, such as whether subject $t$ would have had $\hat y_t=0$, but for the fact that $x_{tj}=B$ instead of $W$. At face value, such counterfactuals directly address the issue of interest. Unfortunately there can be many other variables and combinations of variables that would also have changed the outcome, providing an equally plausible explanation. Taking account of them all combinations can bring in implausible or even impossible variable combinations.
Cohort Shapley avoids using unobserved combinations of variables. In a fairness setting we might find that every combination of variables is logically and physically possible, though some combinations may still be quite unlikely, such as the largest number of prior offences at the youngest of ages.
We do not offer any conclusions on the fairness of the COMPAS algorithm. There is a problem of missing variables that requires input from domain experts.
If some such variable is not in our data set then accounting for it could change the magnitude or even the sign of some of the effects we see. Some other issues that we believe require input from domain experts are: choosing the appropriate set of subjects to study, determining which responses are relevant to fairness and which combination of protected and unprotected variables are most suitable to include. We do however believe that given a set of subjects and a set of variables, that cohort Shapley supports many different graphical and numerical investigations. It can do so for variables not used in a black box at hand, and it does not even require access to the black box function.
Cohort Shapley value can be used to detect algorithmic bias in an algorithm circumventing some limitations described above. As with any testing method, there is the possibility of false positives and false negatives. Any bias that is detected must be followed up by further examination. The positive social outcomes of our method arise from providing a tool that can detect and illustrate biases that might have otherwise been missed. Negative social outcomes can also arise from false positives and false negatives due to missing variables.
This work was supported by the US National Science Foundation under grant IIS-1837931 and by a grant from Hitachi, Ltd.