The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
35,789 characters
Cohort Shapley value for algorithmic fairness
\maketitle
\begin{abstract}
Cohort Shapley value is a model-free method of
variable importance grounded in game theory that
does not use any unobserved and potentially
impossible feature combinations.
We use it to evaluate algorithmic fairness, using
the well known COMPAS recidivism
data as our example.
This approach allows one to identify
for each individual in a data set the extent to
which they were adversely or beneficially
affected by their value of a
protected attribute such as their race.
The method can do this even if race was not
one of the original predictors and even if
it does not have access to a proprietary
algorithm that has made the predictions.
The grounding in game theory lets us define aggregate
variable importance for a data set consistently
with its per subject definitions.
We can investigate variable importance for multiple
quantities of interest in the fairness literature
including false positive predictions.
\end{abstract}
\section{Introduction}
Machine learning is now commonly used to make consequential decisions about people, affecting
hiring decisions, loan applications,
medical treatments,
criminal sentencing and more.
It is important to understand and explain these decisions.
A critical part of understanding a decision is
quantifying the importance of the variables
used to make it. When the decisions
are about people and some of the variables describe
protected attributes of those people, such as race and
gender, then variable importance has a direct bearing
on algorithmic fairness.
As we describe below, most variable importance
measures work by changing a subset of
the input variables to a black box. We survey
such methods below and argue against that approach.
The first problem is that the resulting analysis
can depend on some very unreasonable variable
combinations. A second problem with changing inputs
to a black box is that it necessarily attributes zero
importance to a variable that the algorithm did not
use. If a protected variable is not actually used in
the black box then the algorithm would automatically
be considered fair. However this `fairness through
unawareness' approach is not reliable,
as \cite{adle:2018} and many others have noted.
Information about the protected variables can leak
in through others with which they are associated.
The practice known as `redlining' involves
deliberate exploitation of such associations.
When studying fairness we must have the possibility
of studying a variable not included in the black
box. In this paper we develop and illustrate
an approach to algorithmic fairness that does not
use impossible values and can detect redlining.
\subsection{Variable Importance}
Variable importance measures have a long
history and there has been a recent surge in
interest motivated by problems of
explainable AI.
The global sensitivity analysis literature studies black box functions
used in engineering and climate models among others.
Much of that work is based
on a functional ANOVA model of the function relating
the output to inputs. Variance explained
is partitioned using Sobol' indices
\citep{sobo:1993}.
\cite{salt:etal:2008} is an introductory
textbook and \cite{raza:etal:2021} is a current survey
of the field. \cite{wei:lu:song:2015} provide
a comprehensive survey of variable importance measures in statistics. They
include 197 references of which 24 are
themselves surveys.
\cite{moln:2018} surveys variable importance
measures used in explainable AI.
Prominent among these are SHAP \citep{lund:lee:2017}
and LIME \citep{ribe:etal:2016}. LIME makes a local linear
approximation to a black box function $f(\cdot)$
and one can then take advantage of the
easier interpretability of linear models.
SHAP is based on Shapley value from
cooperative game theory \citep{shap:1953}, that
we describe in more detail below.
Ordinarily we can compare two prediction methods
by waiting to see how accurate they are on unknown
future data or, if necessary, use
holdout data.
This does not carry over to deciding
whether LIME
or SHAP or some other method has made a
better explanation of a past decision.
Suppose that we want to understand something like why a
given applicant was turned down for a loan.
When $f(\cdot)$ is available to us in closed form
then there is no doubt at all about whether the
loan would have been offered under any assortment
of hypothetical feature combinations that we choose.
We can compare methods by
how they define importance. Choosing among
definitions is a different activity, essentially
a philosophical one, and we face tradeoffs.
Our problem is one of identifying the causes
of given effects when we have perfect knowledge
of $f(\cdot)$ for all input variable
combinations. This is different from the more
common learning task of quantifying the effects
of given causes. That distinction was made by
\cite{holl:1988} in studying causal inference. It
goes back at least to \cite{mill:1843}.
A problem with many, but not all, measures of
variable importance is that they are based
on changing some feature values from one level
to another, while holding other features constant
and then looking at the changes to $f(\cdot)$.
When the underlying features are
highly correlated, then some of these combinations
can be quite implausible, casting doubt on any
variable importance methods that use them.
In extreme cases, the combinations can be physically
impossible (e.g., systolic blood pressure below
diastolic) or logically impossible (e.g., birth date
after graduation date). The use of these combinations
has also been criticized by \cite{hook:ment:2019:tr}.
To avoid using impossible values, Cohort Shapley method\footnote{https://github.com/cohortshapley/cohortshapley}
from \cite{mase:owen:seil:2019} uses only observed data values.
For any target subject $t\in\{1,\dots,n\}$ and any
feature variable $j=1,\dots,d$ we obtain
a Shapley value $\phi_j=\phi_j(t)$ that measures
the impact of the value of feature $j$ on $f(\cdot)$
for subject $t$. As we describe below, those
impacts can be positive or negative. Using one of
the Shapley axioms we will be able to aggregate from
individual subjects to an impact measure for the
entire data set. We can also
disaggregate from the entire data set to a subset,
such as all subjects with the protected level of a
protected variable.
\subsection{Contributions}
\begin{figure}
\centering
\includegraphics{fig/compas_dp_cs_r_hist.png}
\caption{
Histograms of the cohort Shapley
impact of race on whether a person in the COMPAS
data is predicted to reoffend. Orange bars represent
Black subjects; blue represent White subjects.
}
\label{fig:compas_dp_cs_r_hist}
\end{figure}
In this paper we use the cohort Shapley method
from \cite{mase:owen:seil:2019} to measure
variable importance.
One of the features of cohort Shapley is
that it can attribute importance to a variable
that is not actually used in $f(\cdot)$.
This is controversial and not all authors
approve but it is essential in the present
context of algorithmic fairness.
Our contribution is to provide a method of quantifying bias at
both individual and group levels with axiomatic consistency between
the individual and group measures and without requiring any impossible
or unobserved combinations of input variables.
The approach can even
be used when the prediction algorithm
is not available as for instance when it is proprietary.
We investigate the COMPAS data \citep{angw:2016}
which include predictions of who is likely
to commit a crime. There is great interest
in seeing whether the algorithm is unfair
to Black subjects. In the given context,
the prediction code $f(\cdot)$ is a proprietary
algorithm, unavailable to us and so we cannot
change any of the values in it. What we have
instead are the predictions on a set of subjects.
Cohort Shapley can work with the predictions
because it does not require any
hypothetical feature combinations.
Figure~\ref{fig:compas_dp_cs_r_hist} shows cohort
Shapley impacts for the race of the subjects
in the COMPAS data, computed in a way that we
describe below. As it turns out the impact
of race on whether a subject was predicted to
reoffend was always positive for Black subjects
and it was always
negative for White subjects. For some other
measures that we show, the histograms
overlap. The average impact for
Black subjects was $0.067$ and the average
for White suspects was $-0.101$.
For context, the response values were $1$ for
those predicted to reoffend and $0$ for those
predicted not to reoffend.
An outline of this paper is as follows.
Section~\ref{sec:basics} introduces
our notation, defines Shapley value,
and presents a few of the variable
importance measures and fairness definitions from the literature.
Section~\ref{sec:compas} describes
the COMPAS recidivism data set.
Section~\ref{sec:dataanalysis} has
our analysis of that data including
a Bayesian bootstrap for uncertainty quantification.
Section~\ref{sec:conclusions} has
our conclusions.
An appendix presents some additional COMPAS
results like the ones we selected for
discussion.
\section{Notation and definitions}\label{sec:basics}
For subjects $i=1,\dots,n$, the value of feature
$j=1,\dots,d$ is $x_{ij}\in\mathcal{X}_j$.
We consider categorical variables $x_{ij}$.
Continuously distributed values can
be handled as discussed in
Section~\ref{sec:conclusions}.
The features for subject $i$ are
encoded in $\boldsymbol{x}_i\in\mathcal{X} =\prod_{j=1}^d\mathcal{X}_j$.
For each subject there is a response value
$y_i\in \mathbb{R}$.
The feature indices belong to the set $1{:}d\equiv\{1,\dots,d\}$ and similarly
the subject indices are in $1{:}n\equiv\{1,\dots,n\}$.
For $u\subseteq1{:}d$ the tuple $\boldsymbol{x}_u$
is $(x_j)_{j\in u}\in\mathcal{X}_u=\prod_{j\in u}\mathcal{X}_j$,
and $\boldsymbol{x}_{iu}=(x_{ij})_{j\in u}$.
There is also an algorithmic
prediction $\hat y_i$.
It is usual in variable importance problems
to have $\hat y_i =f(\boldsymbol{x}_i)$ for a function $f(\cdot)$
that may be difficult to interpret (e.g., a black box).
This $f(\cdot)$ is often an approximation of
$\mathbb{E}(y\!\mid\! \boldsymbol{x})$ though it need not be of that form.
Cohort Shapley uses only the values $\hat y_i$ so it
does not require access to $f(\cdot)$. Since it
only uses a vector of values, one per subject, it
can be used to find the important variables in $y_i$,
or $\hat y_i$ or combinations such as
the residual $y_i-\hat y_i$.
In many settings $y_i$ and $\hat y_i$ are both
binary. The value $y_i=1$ may mean that subject $i$
is worthy of a loan, or is predicted to commit a crime,
or should be sent to intensive care and $\hat y_i\in\{0,1\}$
is an estimate of $y_i$. In this case,
measures such as $\mathrm{FP}_i = 1\{\hat y_i=1\ \&\ y_i=0\}$
describing a false positive are of interest.
Here are some typographic conveniences
that we use.
When $j\not\in u$, then $u+j$
is $u\cup\{j\}$. For $u\subseteq1{:}d$
we use $-u$ for $1{:}d\setminus u$.
The subscript to $\boldsymbol{x}$ may contain
a comma or not depending on what is
clearer. For instance, we use $\boldsymbol{x}_{iu}$
but also $\boldsymbol{x}_{i,-u}$.
\subsection{Shapley values}
Shapley value \citep{shap:1953} is used in game theory
to define a fair allocation of rewards to a team that
has cooperated to produce something of value.
Many variable importance problems can be formulated
as a team of input variables generating an output
value or an output variance explained. We then
want to apportion importance or impact to the
individual variables.
Suppose that a team of $d$ members produce
a value $\mathrm{val}( 1{:}d)$, and that we have at our disposal
the value $\mathrm{val}(u)$ that would have been produced
by the team $u$,
for all $2^d$ teams $u\subseteq1{:}d$.
Let $\phi_j$ be the reward for player $j$.
It is convenient to work with incremental
values $\mathrm{val}(j\!\mid\! u)=\mathrm{val}(u+j)-\mathrm{val}(u)$
for sets $u$ with $j\not\in u$.
Shapley introduced quite reasonable criteria:
\begin{compactenum}[\quad 1)]
\item Efficiency: $\sum_{j=1}^d\phi_j = \mathrm{val}(1{:}d)$.
\item Symmetry: If $\mathrm{val}(i\!\mid\! u)=\mathrm{val}(j\!\mid\! u)$ for all $u\subseteq1{:}d\setminus\{i,j\}$,
then $\phi_i=\phi_j$.
\item Dummy: if $\mathrm{val}(j\!\mid\! u) = 0$ for all $u\subseteq1{:}d\setminus\{j\}$,
then $\phi_j=0$.
\item Additivity:
if $\mathrm{val}(u)$ and $\mathrm{val}'(u)$ lead to values $\phi_j$ and $\phi_j'$
then the game producing $(\mathrm{val}+\mathrm{val}')(u)$ has values $\phi_j+\phi'_j$.
\end{compactenum}
He found that the unique valuation that satisfies all
four of these criteria is
\begin{align}\label{eq:shapj}
\phi_j = \frac1d\sum_{u\subseteq -j}
{d-1\choose |u|}^{-1}\mathrm{val}(j\!\mid\! u)
\end{align}
Formula~\eqref{eq:shapj} is not very intuitive.
Another way to explain Shapley value is as follows.
We could build a team from ${\varnothing}$
to $1{:}d$ in $d$ steps, adding one member at a time.
There are $d!$ different orders in which
to add team members. The Shapley value $\phi_j$
is the increase in value coming from the addition of
member $j$, averaged over all $d!$ different orders.
From equation~\eqref{eq:shapj} we see that Shapley value does not change if we add or subtract the same quantity from all $\mathrm{val}(u)$. It can be convenient to make $\mathrm{val}({\varnothing})=0$.
We discuss some variable importance measures
based on Shapley value below.
We use the framework from \cite{sund:najm:2019}.
They survey many
uses of Shapley value in explainable AI.
\subsection{Changing variables}
Most mechanisms for investigating variable
importance proceed by changing some, but not all,
of the input variables to $f(\cdot)$.
For $u\subseteq1{:}d$ the hybrid
point $\boldsymbol{w}=\boldsymbol{x}_{u}{:}\boldsymbol{z}_{-u}$ has
$$
w_j = \begin{cases}
x_{j}, & j\in u\\
z_{j}, & j\not\in u.
\end{cases}
$$
We can investigate changes to $\boldsymbol{x}_{u}$
by examining $f(\boldsymbol{x}_{u}{:}\boldsymbol{z}_{-u})-f(\boldsymbol{x})$
for various points $\boldsymbol{x},\boldsymbol{z}\in\mathcal{X}$.
Global sensitivity analysis \citep{raza:etal:2021} works with
an analysis of variance (ANOVA) defined
in terms of $\boldsymbol{x}\in\mathcal{X}$ with independent
components for which
$\sigma^2 = \mathrm{var}( f(\boldsymbol{x}))<\infty$.
The sets $\mathcal{X}_j$ can be discrete or
continuous. It is common there to measure
the importance of a subset of variables by
variance explained.
For instance, the lower Sobol' index
for variables $\boldsymbol{x}_u$ is
$\underline\tau^2_u=\mathrm{var}( \mathbb{E}(f(\boldsymbol{x})\!\mid\!\boldsymbol{x}_u))$
which is usually normalized to $\underline\tau^2_u/\sigma^2$.
The upper Sobol' index is $\overline\tau^2_u=\sigma^2-\underline\tau^2_{-u}$,
that is, everything not explained by $\boldsymbol{x}_{-u}$
is attributed to $\boldsymbol{x}_u$.
These measures are very natural in settings
where the components of $\boldsymbol{x}$ can all
vary freely and independently,
but such is usually not the
case for inputs to a black box machine
learning model.
The quantitative influence function (QII)
method of \cite{datt:2016}
uses a model in which the features
are statistically independent from
their marginal distributions.
In baseline Shapley (see \cite{sund:najm:2019}),
there is a baseline tuple $\boldsymbol{x}_b\in\mathcal{X}$
of $\boldsymbol{x}$, such as the sample average
$\bar\boldsymbol{x}$. Then the value for
variable subset $u$ and subject $t$
is $f(\boldsymbol{x}_{t,u}{:}\boldsymbol{x}_{b,-u})$.
We get the same Shapley values using
$\mathrm{val}(u)=f(\boldsymbol{x}_{t,u}{:}\boldsymbol{x}_{b,-u})-f(\boldsymbol{x}_b)$.
Then the total value to explain for
subject $t$ is $f(\boldsymbol{x}_t)-f(\boldsymbol{x}_b)$.
The counterfactuals are changes
in some subset of the values in $\boldsymbol{x}_t$.
Important variables are those that move
the value the most from the baseline
$f(\boldsymbol{x}_b)$ to subject $t$'s value.
Shapley's combination provides a principled
weighting of the effect of changing
$x_{b,j}$ to $x_{t,j}$ given some subset $u$
of variables that have also been changed.
Conditional expectation Shapley
\citep{sund:najm:2019} uses a joint
distribution $D$ on $\mathcal{X}$ and then
for subject $t$
$\mathrm{val}(u)=\mathbb{E}_D(f(\boldsymbol{x})\mid\boldsymbol{x}_u=\boldsymbol{x}_{t,u})$
under $D$.
SHAP \citep{lund:lee:2017} is a conditional
expectation Shapley using
a distribution $D$ in which the
features are independent.
Cohort Shapley \citep{mase:owen:seil:2019} that we
describe next is very nearly
conditional expectation Shapley
with $D$ equal to the empirical
distribution.
\subsection{Changing knowledge}
In cohort Shapley the counterfactual
does not involve replacing
$\boldsymbol{x}_{tu}$ by $\boldsymbol{x}_{bu}$. It is instead
about concealing the values of $\boldsymbol{x}_{tu}$.
It is what \cite{kuma:2020} call a
conditional
method because it requires specification
of a conditional distribution on
the features.
The version we present here is a form
of conditional expectation Shapley
using the empirical distribution of
the data.
For every pair of values $x_{tj},x_{ij}\in\mathcal{X}_j$
we can declare them to be similar or not.
In the present context we take
similarity to just be $x_{tj}=x_{ij}$.
For a target subject $t\in1{:}n$ and
set $u\subseteq1{:}d$ we define
the cohort
$C_u = C_{tu} =\{i\in1{:}n\mid \boldsymbol{x}_{iu}=\boldsymbol{x}_{tu}\}$.
These are the subjects who match subject
$t$ on all of the variables $j\in u$.
They may or may not match the target
on $j\not\in u$.
None of the cohorts is empty because
they all contain $t$.
The value function in cohort Shapley is
the cohort mean
$$
\mathrm{val}(u) = \frac1{|C_{tu}|}
\sum_{i\in C_{tu}}\hat y_i.
$$
Here $\hat y_i=f(\boldsymbol{x}_i)$ but we write
it this way because in practice we might
not have $f(\cdot)$ at our disposal,
just the predictions $\hat y_i$.
The incremental value
$\mathrm{val}(j\!\mid\! u)$ is then how much
the cohort mean moves when we reveal
$x_{tj}$ in addition to previously
revealed variables $\boldsymbol{x}_{tu}$.
Having specified the values, the Shapley
formula provides attributions.
Note that $\mathrm{val}({\varnothing})$
is the plain average of all $\hat y_i$
and Shapley value is unchanged by
subtracting $\mathrm{val}({\varnothing})$
from all of the $\mathrm{val}(u)$.
Furthermore, when there are many variables,
then very commonly $C_{t,1{:}d}=\{t\}$
and then the total value to be
explained, $\mathrm{val}(1{:}d)-\mathrm{val}({\varnothing})$
is simply $\hat y_t-(1/n)\sum_{i=1}^n\hat y_i$.
\subsection{Definitions of fairness}
Just as there are many ways to define variable
importance, there are multiple ways to define
what fairness means. Some of those definitions
are mutually incompatible and some of them differ
from legal definitions. Here we present a few of
the issues.
See \cite{corb:goel:2018}, \cite{chou:roth:2018}, \cite{berk:2018}, and \citet{frie:etal:2019} for surveys.
We do not make assertions about which
definitions are preferable.
For $y,\hat y\in\{0,1\}$, let $n_{y\hat y}$ be
the number of subjects with
$y_i=y$ and $\hat y_i=\hat y$.
These four counts and their derived properties can
be computed for any subset of the subjects.
The false positive rate (FPR) is
$n_{01}/n_{0\text{\tiny$\bullet$}}$, where a bullet indicates that we are summing over the levels of that index.
We ignore uninteresting corner cases such
as $n_{0\text{\tiny$\bullet$}}=0$; when there are no
subjects with $y=0$ then we have no interest
in the proportion of them with $\hat y=1$.
The false negative rate (FNR) is
$n_{10}/n_{1\text{\tiny$\bullet$}}$.
The prevalence of the trait under study
is $p=n_{1\text{\tiny$\bullet$}}/n_{\text{\tiny$\bullet$}\text{\tiny$\bullet$}}$.
The positive predictive value (PPV) is
$n_{11}/n_{\text{\tiny$\bullet$}1}$.
As \citet[equation (2.6)]{chou:2017} notes,
these values satisfy
\begin{align}\label{eq:fprandp}
\mathrm{FPR} = \frac{p}{1-p}\frac{1-\mathrm{PPV}}{\mathrm{PPV}}(1-\mathrm{FNR}).
\end{align}
See also \cite{klei:mull:ragh:2016}.
Equation~\eqref{eq:fprandp} shows how some natural
definitions of fairness conflict.
FPR and FNR describe $\hat y\!\mid\! y$,
while PPV describes $y\!\mid\!\hat y$. If two
subsets of subjects have the same PPV but
different prevalences $p$, then they cannot
also match up on FPR and FNR. Fairness in
$y\!\mid\! \hat y$ terms and fairness in $\hat y\!\mid\! y$
terms can only coincide in trivial settings
such as when $\hat y=y$ always or empirically unusual
settings with equal prevalence between
subjects having different values of a protected
attribute.
\cite{hard:2016} writes about measures
of demographic parity, equalized odds and equal opportunity.
Demographic parity, requires
$\Pr( \hat y=1\!\mid\! \boldsymbol{x}_p)=\Pr(\hat y=1)$
where $\boldsymbol{x}_p$ specifies the levels of
one (or more) protected
variables.
Equalized odds requires $\Pr(\hat y=1\!\mid\! y=y', \boldsymbol{x}_p)
= \Pr(\hat y=1\!\mid\! y=y'), y' \in {0,1}$, so that false
positive rates and true positive rates are
both equal across groups. Because $\mathrm{FNR} = 1 - \mathrm{TPR}$ the
false negative rates are also equal.
Equal opportunity is defined in terms
of a preferred outcome.
When $y=0$ is preferred, it requires that
$\mathrm{Pr}(\hat y=1\!\mid\! y=0,\boldsymbol{x}_p)=\mathrm{Pr}(\hat y=1\!\mid\! y=0)$
That is, the false positive rate
is unaffected by the protected variables.
\cite{chou:2017} considers a score $s$
(such as $\hat y$) to be well calibrated if
$\mathrm{Pr}(y=1\!\mid\! s ,\boldsymbol{x}_p)=\mathrm{Pr}(y=1\!\mid\! s)$ for all levels of $s$.
A score attains predictive parity if
$\mathrm{Pr}(y=1\!\mid\! s>s_*,\boldsymbol{x}_p)=\mathrm{Pr}(y=1\!\mid\! s>s_*)$ for all thresholds $s_*$.
The distinction between well calibratedness
and predictive parity is relevant when
there are more than two levels for
the score.
There is some debate about when or whether
using protected variables can lead to
improved fairness.
See \cite{xian:2020} who gives a summary of
legal issues surrounding fairness
and \cite{corb:etal:2017}
who study whether
imposing calibration or other criteria
might adversely affect the groups they
are meant to help.
\subsection{Aggregation and disaggregation}
The additivity axiom of Shapley value
is convenient for us. It means that
importances for the residual $y_i-\hat y_i$
is simply the difference of importances
for $y_i$ and $\hat y_i$.
We can aggregate importances from subjects
to the whole data set by summing, or more
interpretably averaging
over $t=1,\dots,n$.
We can also disaggregate from
the whole data set to subsets of
special interest by averaging cohort
Shapley values over target subjects
$t\in v\subset 1{:}n$.
\section{COMPAS data}\label{sec:compas}
COMPAS is a tool from Northpointe Inc.
for judging the risk that a criminal defendant will
commit another crime (re-offend)
within two years. Each subject is rated into one of
ten deciles with higher deciles considered higher
risk of reoffending.
\cite{angw:2016} investigated whether that algorithm
was biased against Black people. They obtained data
for subjects in Broward County Florida,
including the COMPAS decile, the subjects' race,
age, gender, number of prior crimes and whether the
crime for which they were charged is a felony
or not. \cite{angw:2016} describe how they
processed their data including how they found
followup data on offences committed and how
they matched those to subjects
for whom they had prior COMPAS scores.
They also note that race was not one of the variables
used in the COMPAS predictions.
The example is controversial.
\cite{angw:2016} find that COMPAS
is biased because it gave a
higher rate of false positives for Black
subjects and a higher rate of false negatives
for White subjects.
\cite{flores2016false} and \cite{dieterich2016compas} disagree raising the issue of $\hat y\!\mid\! y$
fairness versus $y\!\mid\! \hat y$ fairness.
The prevelance of reoffences differed
between Black and White subjects
forcing $y\!\mid\!\hat y$ and $\hat y \!\mid\! y$
notions to be incompatible.
\cite{flores2016false} also questioned whether the
subjects in the data set were at comparable
stages in the legal process to those for
which COMPAS was designed.
Following \cite{chou:2017} we focus on just Black
and White subjects. That provides a sample of
5278 subjects from among the original 6172 subjects.
As in that paper we record the number of prior
convictions as a categorical variable with
five levels: 0, 1--3, 4--6, 7--10 and $>$10.
Following \cite{angw:2016}, we record the
subjects' ages as a categorical variable
with three levels:
$<$25, 25--45, $>$45.
Also, following \cite{chou:2017}, we consider the
prediction $\hat y_i$ to be $1$ if subject
$i$ is in deciles 5--10 and $\hat y_i=0$
for subject $i$ in deciles 1--4.
\section{Exploration of the COMPAS data}\label{sec:dataanalysis}
In this section we use cohort Shapley
to study some fairness issues in the COMPAS
data, especially individual level metrics.
We have selected what we found to be
the strongest and most interesting findings.
The larger set of figures and tables from which
these are drawn are in the appendix.
The appendix also compares some conventional
group fairness metrics for Black and White
subjects.
We have computed Shapley impacts for these
responses: $y_i$, $\hat y_i$, $y_i-\hat y_i$,
$\mathrm{FP}_i = 1\{ y_i=0\,\&\, \hat y_i=1\}$ and
$\mathrm{FN}_i = 1\{ y_1=1\,\&\, \hat y_i=1\}$.
If $\mathrm{FP}_i=1$ then subject $i$ received a
false positive prediction. Note that the
sample average value of $\mathrm{FP}_i$
$$\hat\mathbb{E}(\mathrm{FP}_i)=\frac{n_{01}}{n_{\text{\tiny$\bullet$}\text{\tiny$\bullet$}}}
=\mathrm{FPR}\times \frac{n_{0\text{\tiny$\bullet$}}}{n_{\text{\tiny$\bullet$}\text{\tiny$\bullet$}}}
=\mathrm{FPR}\times (1-p).$$
Being wrongly predicted to reoffend is an
adverse outcome that we study. Its expected
value is the FPR times the fraction of
non-reoffending subjects.
Similarly, $\hat\mathbb{E}(\mathrm{FN}_i)=\mathrm{FNR}\times p$.
\subsection{Graphical analysis}
Figure~\ref{fig:compas_cs_race} shows histograms
of Shapley impacts of race for the subjects
in the COMPAS data. The first panel there reproduces
the data from Figure~\ref{fig:compas_dp_cs_r_hist}
showing a positive impact for every Black subject
and a negative one for every White subject
for the prediction $\hat y$.
For the actual response $y$, the histograms
overlap slightly. By additivity of Shapley
value the impacts for $y-\hat y$ can be
found by subtracting the impact for $\hat y$
from that for $y$ for each subject $i=1,\dots,n$.
The histograms of $y_i-\hat y_i$ show that
the impact of race on the residual is typically
positive for White subjects and negative
for Black subjects.
Overlap of distributions is seen in equal opportunity (false positive rate: FPR) and false negative rate (FNR). In there, small number of adversely affected White and beneficially
affected Black are observed.
We can inspect further on these overlapped subjects to understand which conditions are exceptional cases.
Also we can inspect on tail of the distribution to understand the extreme cases where bias is very likely observed.
Figure~\ref{fig:compas_cs_race_gender} shows
histograms of Shapley impacts for race
but now they are color coded by
the subjects' gender.
We see that the impact on $\hat y$ is bimodal
by race for both male and female subjects,
but the effect
is larger in absolute value for male subjects.
The impacts for the response, the residual
and false negative and positive values do not
appear to be bimodal by race for female subjects,
but they do for male subjects. The case of FN is
perhaps different. The modes in Figure~\ref{fig:compas_cs_race}
are close together and not so apparent in
Figure~\ref{fig:compas_cs_race_gender} which
has slightly different bin boundaries.
It is clear from these figures that the
race differences we see are much stronger
among male subjects.
\begin{figure}[t]
\centering
\includegraphics[width=1.0 \textwidth]{fig/compas_cs_race5.png}
\caption{Histogram of cohort Shapley value of race factor on COMPAS recidivism data.}
\label{fig:compas_cs_race}
\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=1.0 \textwidth]{fig/compas_cs_race_gender5.png}
\caption{Histogram of cohort Shapley value of race factor for each gender on COMPAS recidivism data.}
\label{fig:compas_cs_race_gender}
\end{figure}
The appendix includes some figures that each
include 25 histograms.
They are Shapley impact histograms for all five
features color coded by each of the five
features. There is one such figure for each
of $y$, $\hat y$, $y-\hat y$, $\mathrm{FP}$ and $\mathrm{FN}$.
\subsection{Tabular summary}
Tables~\ref{tab:csi_prediction} through~\ref{tab:csi_fn}
in the appendix record mean Shapley
impacts for the five predictors
and our responses measures, disaggregated
by race and gender.
\begin{table}[t]
\centering
\begin{tabular}{lcccc}
\toprule
Variable & White & Black & Male & Female \\
\midrule
race\_factor &$\phantom{-}$0.054 & $-$0.035 & $-$0.001 &$\phantom{-}$0.003 \\
gender\_factor & $-$0.001 &$\phantom{-}$0.001 &$\phantom{-}$0.020 & $-$0.083 \\
\bottomrule\\
\end{tabular}
\smallskip
\begin{tabular}{lcccc}
\toprule
Variable & White-Male & White-Female & Black-Male & Black-Female \\
\midrule
race\_factor &$\phantom{-}$0.057 &$\phantom{-}$0.042 & $-$0.036 & $-$0.030 \\
gender\_factor &$\phantom{-}$0.023 & $-$0.083 &$\phantom{-}$0.018 & $-$0.083 \\
\bottomrule\\
\end{tabular}
\caption{\label{tab:csi_residual_subset}
Mean cohort Shapley impact of groups on residual $y - \hat y$.
}
\end{table}
We take a particular interest in the
residual $y_t-\hat y_t$.
It equals $1$ for false negatives and $-1$
for false positives and $0$ when the prediction
was correct.
Table~\ref{tab:csi_residual_subset} shows
a subset of the largest values for
the residual $y_i-\hat y_i$.
What we see there is that revealing that
a subject is Black tells us that, on average,
that subject's residual $y_t-\hat y_t$
is decreased by $3.6$\%. Revealing that
the subject is White increases the residual
by $5.4$\%. Revealing race makes very little
difference to the residual averaged over
male or over female subjects.
Revealing gender makes quite a large difference
of $-8.3$\% for female subjects and $+2.0\%$
for male subjects.
To judge the uncertainty in the values in
Table~\ref{tab:csi_residual_subset}
we applied the Bayesian bootstrap of \cite{rubi:1981}.
That algorithm randomly reweights each data point
by a unit mean exponential random variable. By
never fully deleting any observation it allows
one to also bootstrap individual subjects' Shapley
values though there is not space to do so in this
article. Figure~\ref{fig:agg_race_impact_violin} shows
violin plots of 1000 bootstrapped cohort Shapley values.
\begin{figure}[t]
\centering
\includegraphics[width=1.0 \textwidth]{fig/agg_race_impact_violin2.png}
\caption{Bayesian bootstrap violin plot of aggregated cohort Shapley race factor impact of groups on residual $y - \hat y$.
Red crosses represent aggregated cohort Shapley values of groups without bootstrap.}
\label{fig:agg_race_impact_violin}
\end{figure}
\subsection{FPR and FNR revisited}
The risk of being falsely predicted
to reoffend involves two factors:
having $y_i=0$ and having $\hat y_i=1$
given that $y_i=0$. FPR is commonly
computed only over subjects with $y_i=0$.
Accordingly, in this section we study
it by subsetting the subjects to
$\{ i\mid y_i=0\}$ and finding cohort Shapley
value of $\hat y_i=1$ for the features.
Figure~\ref{fig:compas_cs_fprfnr} shows cohort Shapley impact of race factor conditioned on race factor for FPR and FNR.
The distribution of impacts conditioned on race are separated in the figure.
This conditional analysis shows a stronger disadvantage for Black subjects.
\begin{figure}[t]
\centering
\includegraphics[width=1.0 \textwidth]{fig/compas_cs_impact_fprfnr_race.png}
\caption{Histogram of cohort Shapley value of race factor on COMPAS recidivism data for false positive subsetting to $y_i = 0$ and false negative subsetting to $y_i = 1$.}
\label{fig:compas_cs_fprfnr}
\end{figure}
\section{Conclusions}\label{sec:conclusions}
Variable importance in statistics and machine
learning ordinarily considers three issues:
whether changing $x_j$ has a causal future
impact on $y$, whether omitting $x_j$ from
a data set degrades prediction accuracy
and whether changing $x_j$ has an
important mechanical effect on $f(\boldsymbol{x})$
\citep{jian:owen:2003}. These are all
different from each other and the third
choice is the most useful one for
explaining the decisions of a
prediction method.
Cohort Shapley \citep{mase:owen:seil:2019}
looks at a fourth issue: whether
learning the value
of $x_{tj}$ in a sample is
informative about the prediction
$\hat y_t$. It is well suited to
settings such
as the COMPAS data from Broward County,
where: the original algorithm is
not available to researchers, the
protected
variable of greatest interest was not
included in the model,
and the set of subjects for which fairness
is of interest is different from the set on
which the algorithm was trained.
Cohort Shapley does not address counterfactuals,
such as whether subject $t$ would
have had $\hat y_t=0$, but for the fact
that $x_{tj}=B$ instead of $W$.
At face value, such counterfactuals directly
address the issue of interest. Unfortunately
there can be many other variables and combinations
of variables that would also have changed
the outcome, providing an equally plausible
explanation. Taking account of them
all combinations can bring in
implausible or even
impossible variable combinations.
Cohort Shapley avoids using
unobserved combinations of variables.
In a fairness setting we might find
that every combination of variables is
logically and physically possible,
though some combinations
may still be quite unlikely, such as the
largest number of prior offences at
the youngest of ages.
We do not offer any conclusions on the
fairness of the COMPAS algorithm.
There is a problem of missing variables
that requires input from domain
experts.
If some
such variable is not in our data set
then accounting for it could change
the magnitude or even the sign of
some of the effects we see.
Some other issues that we believe
require input from domain experts
are: choosing the appropriate set of
subjects to study, determining which
responses are relevant to fairness
and which combination of protected
and unprotected variables are most
suitable to include.
We do however
believe that given a set of subjects
and a set of variables, that cohort
Shapley supports many different
graphical and numerical investigations.
It can do so for variables not used
in a black box at hand, and it does
not even require access to the black
box function.
Cohort Shapley value can be used to detect algorithmic
bias in an algorithm circumventing some
limitations described above.
As with any testing method, there is the possibility
of false positives and false negatives.
Any bias that is detected must be followed up by
further examination.
The positive social outcomes of our method
arise from providing a tool that can detect
and illustrate biases that might have otherwise been missed.
Negative social outcomes can also arise from false
positives and false negatives due to missing variables.
\section*{Acknowledgments}
This work was supported by
the US National Science Foundation under grant IIS-1837931
and by a grant from Hitachi, Ltd.
\bibliographystyle{apalike}
\bibliography{paper}
\newpage