Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
47,266 characters · 13 sections · 61 citation commands
Debiasing classifiers: is reality at variance with expectation?
Artificial intelligence and machine learning (AI/ML) are now used for many high-stakes decision-making processes at scale Mehrabi2019,Rudin2019, such as credit decisions Chen2018,Turner2019, medical diagnoses Topol2019, and criminal sentencing compas,corbettcompas,richardcompas. In these use cases, unfairness is not just an ethical concern, but has legal and regulatory dimensions as well Barocas2016,Chen2018,Xiang2019,Kurshan2020,ico. As such, regulators have signalled their interests in detecting and remediating bias in these real-world applications ico,usrfi.
Bias can originate from any part of the machine learning modeling process, ranging from exclusionary biases Barocas2016,Buolamwini2018,Richardson2019 in training data, to problem definitions or feedback cycles that reinforce historical and systemic discrimination Freeman2017,liu19delayed,barabas20,decolonialai. To address bias in a model, one must first identify the relevant fairness metrics narayanan2018translation,Verma2018 and then select a method to debias the model with respect to those metrics. However, both aspects are challenging in practice. It is not always obvious which fairness definitions are relevant for a particular application compas,richardcompas, and remediating bias usually comes at a cost. For example, a credit decisioning model has to be accurate in order to be profitable, which motivates fairness notions like equality of opportunity. At the same time, there are reputational and regulatory risks associated with bias in incorrect decisions, leading to considerations of equalized false negative rate and equalized false positive rate Hardt2016-zi. These different definitions of fairness cannot be satisfied simultaneously due to well-known impossibility theorems kleinberg2016inherent,chouldechova2017fair. Furthermore, a debiased model will not be used in practice if its performance degrades too much. Therefore, in practice, we have to consider not only fairness--fairness trade-offs, but also the fairness--performance trade-offs to determine the best debiased model Pleiss2017,fact.
\paragraph{Assumptions.} The setting for our paper assumes that
Despite this restricted setting, we find that the practicalities of debiasing are already sufficiently rich for in-depth study. Other work identified technical challenges resulting from the lack of native support for fairness or debiasing concerns in major machine learning software libraries, but do not consider variance or sensitivity issues Biswas2020-jk. More recently, rodolfa2020machine show that trade-offs between recall parity and precision$@k$ are often small in real-world projects.
Given this body of work, we were therefore surprised to it is surprising to see results like those in (ref), which suggest that classifiers can exhibit any combination of improved or worsened fairness, and also improved or worsened performance, after using standard debiasing algorithms. In this paper, we show that out-of-sample generalization error is responsible for the fluctuations observed in the aforementioned figure, and that careful estimation of such error is essential for proper evaluation of debiasing methods. While previous works have studied distributionally robust optimization for fairness 2020LinBeyondTraining and data-dependent constraint generalization Cotter2019WellGeneralizing, we focus on the generalization of fairness algorithms.
\paragraph{Our Contributions.} In (ref), we show how to generalize existing debiasers to apply to fairness metrics other than what they were originally defined for. We introduce generalized reweighing which can apply to other fairness definitions beyond demographic parity and identify fairness definitions for which reweighing cannot generalize to. We also introduce a new NLinProg debiaser which generalizes the equalized odds debiaser, and is capable of handling multiple fairness and performance metrics simultaneously. In (ref), we present a detailed empirical study across nine different models, showing that debiasing methods generally fail to achieve perfect fairness in out-of-sample measurements, and produce large variance in the actual metrics, and tend to overfit on training data. In (ref), we present our main theoretical result, (ref), showing that the fluctuations we observed empirically can be attributed to bias-variance trade-off. In (ref), we verify a prediction from this analysis, that the ability to debias varies with the base rate of the protected class. In (ref), we show how an explicit consideration of the fairness--performance trade-offs motivates the notion of partial debiasing. We also show experimentally the somewhat counter-intuitive result that a partial debiasing treatment can actually yield classifiers with better out-of-sample fairness. We introduce other related work throughout the exposition of this paper, in lieu of a dedicated section.
\paragraph{Notation.} Sets. In general, calligraphic letters like $\mathcal A$ denote a set, capital letters $A$ denote a variable that is an element of a set $\mathcal A$, and small letters $a$ denote a value that the variable $A$ can take. Let $S\in\mathcal S = \{0,1\}$ be a binary protected class, $X\in\mathcal X$ be some set of features that explicitly excludes $\mathcal S$, $Y\in\mathcal Y = \{0,1\}$ be a binary outcome variable, and $\hat Y\in\mathcal Y$ be an estimator for $Y$. While we specialize to the cases of binary $\mathcal Y$ and $\mathcal S$ for the ease of presentation, our results generalize to larger finite classes. Furthermore, let $Z = (X, Y) \in \mathcal Z = \mathcal X\ensuremath{\!\times\!}{}\mathcal Y$ and $W = (X, Y, S) \in \mathcal W = \mathcal X\ensuremath{\!\times\!}{}\mathcal Y\ensuremath{\!\times\!}{}\mathcal S$, Additionally, define $\mathcal D \in \mathcal W^n$ to be in-sample (training) data with $n$ points, $\mathcal D^\star \in \mathcal W^{n^\star}$ to be out-of-sample \textit{(testing) data} with $n^\star$ points, and $\Delta^k = \{z\in\mathbb R^{k+1}: z\ge 0, \Vert z \Vert_1 = 1 \}$ be the standard non-negative simplex of dimensionality $k$. \textbf{Classification functions.} Let $\mathcal F : \mathcal X\ensuremath{\!\times\!}{}\mathcal S \rightarrow \mathcal Y$ be the function space of $\mathcal S$-\textit{aware classifiers}, where each element $f \in \mathcal F$ is a classification function, and $\mathcal F_0 : \mathcal X \rightarrow \mathcal Y$ be the function space of $\mathcal S$-\textit{oblivious classifiers}. Each $\mathcal S$-oblivious classifier $f_0 \in F_0$ has a 1:1 relation to a trivial $\mathcal S$-aware classifier $f\in F: f(x, s) = f_0(s)$ which simply ignores the $s$ argument. We differentiate between aware and oblivious classifiers only where necessary. Also, let $\mathcal H \subseteq \mathcal F$ be some family of classifiers, and $\textrm{id}_\mathcal{A}: \mathcal A \rightarrow \mathcal A$ be the identity function over the set $\mathcal A$. \textbf{Debiasing functions.} Let $g_\textrm{pre}: \mathcal X\ensuremath{\!\times\!}{}\mathcal S \rightarrow \mathcal X$ be a pre-processing debiasing function, $g_\textrm{post}: \mathcal Y\ensuremath{\!\times\!}{}\mathcal S \rightarrow \mathcal Y$ be a post-processing debiasing function, and $G: \mathcal F \rightarrow \mathcal F$ be an in-processing debiaser, which is a higher-order function that is an endomorphism over $\mathcal F$. \textbf{Loss functions.} Let $\ell : \mathcal F \ensuremath{\!\times\!}{} \mathcal W \rightarrow \mathbb R^+_0$ be a performance loss such as the hinge or binomial deviance, and $\phi_h : \mathbb R^2 \rightarrow = R^+_0$ be a loss function associated with the fairness definition $h$. \textbf{Metrics.} Let $\gamma : \mathcal F \ensuremath{\!\times\!}{} \mathcal W^{\mathbb N} \rightarrow [0, 1]$, $\gamma(f, \mathcal D)$ be the accuracy of the classifier $f$ on the data set $\mathcal D$, and $\tau_{h} : \mathcal F \ensuremath{\!\times\!}{} \mathcal W^{\mathbb N} \rightarrow [0, 1]$ , $\tau_h(f, \mathcal D)$ the fairness metric as defined in (ref) corresponding to the fairness definition $h$. When clear from context, the arguments $f$ and $\mathcal D$ will be dropped for brevity.
Many technical definitions of fairness exist and they have been reviewed elsewhere narayanan2018translation,Verma2018,richardcompas,fact. We present only the definitions of fairness that we will study in this paper in (ref). In addition to choosing a suitable fairness definition, we also have to choose some loss function, $\phi$, to quantify the discrepancy from perfect fairness. One such function is the Calders-Verwer gap Calders2010 $\Delta_\textsc{DP} = \Pr(\hat Y\ensuremath{\!=\!}1 | S\ensuremath{\!=\!}1) - \Pr(\hat Y\ensuremath{\!=\!}1 | S\ensuremath{\!=\!}0)$, which is simply the difference of the two sides of the equation that define demographic parity, and vanishes when perfect fairness exists. In addition to absolute differences, other metrics based on ratios, relative differences, or other more complicated losses have have been proposed. In this paper, we focus on symmetrized ratio metrics as defined in (ref).
Toolkits such as Aequitas aequitas, IBM AI Fairness 360 aif360, Microsoft Fairlearn fairlearn, and Amazon SageMaker Clarify clarify provide many debiasing algorithms. These algorithms are traditionally classified as pre-processing, in-processing and post-processing methods, which are depicted at the functional level in (ref). In this section, all primed quantities have been debiased. A pre-processing debiaser first transforms the input features $\mathcal X$ using some function $g_\textrm{pre} : \mathcal X \ensuremath{\!\times\!}{} \mathcal S \rightarrow \mathcal X$, then feeds the transformed features as input to an oblivious classifier $f: \mathcal X \rightarrow \mathcal Y$. The debiased classifier is then the composition $f^\prime : \mathcal X \ensuremath{\!\times\!}{} \mathcal S \rightarrow \mathcal Y$, $f^\prime = f \circ g_\textrm{pre}$. A post-processing debiaser takes the output of some oblivious classifier, $\hat Y \in \mathcal Y$, then transforms this output using some function $g_\textrm{post} : \mathcal Y \ensuremath{\!\times\!}{} \mathcal S \rightarrow \mathcal Y$, The debiased classifier is then the composition $f^\prime : \mathcal X \ensuremath{\!\times\!}{} \mathcal S \rightarrow \mathcal Y$, $f^\prime = g_\textrm{post} \circ (f \ensuremath{\!\times\!}{} \textrm{id}_\mathcal{S})$, where $\textrm{id}_\mathcal{S}$ is the identity function over protected class. Finally, an in-processing debiaser transforms some oblivious classifier $f$ into an $\mathcal S$-aware but debiased classifier $f^\prime = G(f)$, using the function-to-function mapping $G:(\mathcal X \rightarrow \mathcal Y) \ensuremath{\!\times\!}{} \mathcal W \rightarrow (\mathcal X \ensuremath{\!\times\!}{} \mathcal S \rightarrow \mathcal Y)$. In general, the resulting classifier cannot be written as a function composition involving the original oblivious classifier $f$. Some in-processing debiasers like prejudice removal prejudiceremover further require that the debiased classifier $f^\prime$ be $\mathcal S$-oblivious, which is equivalent to the defining the debiased classifier $f^\prime(X, S) = f(X)$ to be independent of $S$ always. In other words, pre-processing debiasers transform the features $\mathcal X$, post-processing debiasers transform the predictions $\hat {\mathcal Y}$, and in-processing debiasers transform the classifiers $f$.
We conclude the introduction with two debiasing algorithms as illustrations of the general principle. Reweighing (RW) is a pre-processing debiaser introduced to enforce demographic parity (DP) kamiran2012data. Since DP is satisfied when $Y$ and $S$ are independent, reweighing assigns each data point $i$ a weight $ w_{\textrm{DP},i} = {\Pr(\hat Y\!=\!y_i) \Pr(S\!=\!s_i)} / {\Pr(\hat Y\!=\!y_i, S\!=\!s_i)} $, altering the measure associated with the sampled distribution of $(Y, S)$ to match what would be expected from statistical independence. Equalized odds (EOd) is a post-processing debiaser Hardt2016-zi,Pleiss2017 that calculates probabilities $\Pr(\hat Y^\prime | \hat Y, S)$ that the predictions $\hat Y$ should be flipped to yield the debiased predictions $\hat Y^\prime$ that satisfy equalized odds fairness, while having $\hat Y'$ as close as possible to $\hat Y$.
The reweighing pre-processor and equalized odds post-processsor are specialized to specific fairness definitions, demographic parity and equalized odds respectively. In this section, we show how these debiasers can be generalized to other fairness definitions.
The reweighting pre-processor of (ref) can be easily extended to some, but not all, other definitions of group fairness. For example, considering $\hat Y \perp \!\!\! \perp S | Y\ensuremath{\!=\!}1$ instead of $\hat Y \perp \!\!\! \perp S$ gives an immediate generalization of reweighing for equality of opportunity (EOp) instead of DP. As EOp fairness requires $ \Pr(\hat Y\ensuremath{\!=\!}1 | S = 0, Y = 1) = \Pr(\hat Y\ensuremath{\!=\!}1 | S = 0, Y = 1), $ the corresponding reweighing scheme is simply $w_{\textrm{EOp}, i} = {\Pr(\hat Y = y_i)} / {\Pr(\hat Y = y_i | S = s_i, Y\ensuremath{\!=\!}1)}. $ However, there is no such reweighing scheme for equalized odds (EOd), which requires that both EOp and PE hold. Each equation demands its own reweighing scheme, with the first as before and the second as in $ w_{\textrm{PE}, i} = {\Pr(\hat Y = y_i)} / {\Pr(\hat Y = y_i | S = s_i, Y\ensuremath{\!=\!}0)}, $ which will in general differ from the weights $w_{\textrm{EOp}, i}$. Thus, reweighing as a method for exact debiasing works for neither composite fairness definitions that require multiple equality constraints, nor situations requiring multiple fairnesses to be satisfied simultaneously. It is therefore natural to consider the possibility of some interpolation scheme between different weighting schemes. We will revisit this idea later in (ref).
We now introduce NLinProg (NLP), a generalization of the equalized odds post-processor to allow for arbitrary combinations of group fairnesses to be debiased simultaneously.
We implement (ref) in the JuMP jump framework for the Julia programming language julia, which uses Ipopt ipopt for interior point optimization. The MIT-licensed open source implementation is available on GitHub. \footnote{URL redacted for double-blind peer review.}
Unless otherwise specified, our subsequent experiments specialize to one accuracy loss $\ell^{(1)}(z) = 1 - \gamma(z)$, where $\gamma(z) = \sum_i (TP_i + TN_i) / N$ is the usual definition of accuracy, and fairness loss $\phi^{(1)}(z) = 1 - \tau_{h}(z)$, where $\tau_h$ is a quantity we will now define.
It is easy to show that $\tau_{h} \in [0, 1]$; we omit the proof of this simple fact. Furthermore, $\tau_{h}$ is symmetric in its arguments, which removes the need to assume that either class is generally privileged. Where clear from context, we will (with abuse of notation) also write the above as $\tau_{h}(z)$.
We do not recommend NLinProg for general use---as we will see in (ref), its performance is generally Pareto suboptimal, in that it yields neither the most fair classifiers nor the most accurate classifiers. However, for our experiments, NLinProg serves as a useful construct for investigating the general behavior of post-processing methods.
\paragraph{Methodology.} We now evaluate the performance of three representative debiasers, RW, EOd (as described in (ref)), and NLP ((ref)) on nine different debiasing experiments as stated in (ref), representing different fairness criteria, data sets and debiasing strategies. We observe the phenomena in this section when running similar experiments using the Python toolkits Aequitas aequitas and Fairness 360 aif360, and have carefully reimplemented the algorithms in our own Julia implementation (provided in the Supplement) to verify that these effects are not the results of undiagnosed implementation bugs. We present results from our own implementations, which corroborate similar findings from the Python codes.
The classifier trained for each experiment is a random forest classifier estimated using the MIT-licensed DecisionTree.jl DecisionTree.jl Julia package, which implements the standard classification and regression trees (CART) CART and random forest algorithms Breiman2001. While hyperparameter tuning is an important part of developing fair real-world models Schelter2019,Perrone2020, we keep all hyperparameters at the same default values to facilitate comparison across these varied experiments, eliminating variation due to hyperparameter choice. Our evaluation criteria are the ratio of out-of-sample fairnesses $\tau/\tau_0$ for the debiased and original classifiers, with $\tau$ as defined in (ref), and the ratio of out-of-sample accuracies $\gamma/\gamma_0$ respectively. Unlike many previous studies, we focus on the out-of-sample behavior of the original and debiased classifiers, and estimate the generalization error by computing metrics across 100 different train--test splits computed from ten times ten-fold cross-validation (10 CV 10). Such extensive evaluation is necessary to reduce the error bars on the fairness metrics $\tau$ to determine if a debiaser had a statistically meaningful treatment effect; our experiments demonstrating such necessity are detailed in the Supplement.
\paragraph{Results.} (ref) summarizes the results of our experiments. Within each subplot and point type, each point corresponds to the exact same classifier type, debiased the exact same way, but repeated over 100 different train--test splits arising from ten times ten-fold cross-validation (10 CV 10). The only variation thus comes from the specific subset of data used for model training, and the test data for evaluation. The thick cross-hairs represent the ideal perfect fairness and accuracy, with the grey regions representing a one standard deviation spread across the folds. The narrow cross-hairs pinpoint the point where $\gamma = \gamma_0$ and $\tau = \tau_0$, i.e., where the debiaser had no treatment effect whatsoever.
Na\"ively, we would expect that $\tau > \tau_0$ and $\gamma \approx \gamma_0$, i.e., that the fairness should improve while the accuracy stays roughly constant or perhaps decreases due to an implicit fairness--accuracy trade-off. Instead, we see that for Experiments A, E, H, and I, no debiaser was able to attain the target of maximal fairness. In fact, some experiments (like NLP in A or EOd in D) show essentially no change in the fairness and accuracy metrics at all. More worryingly, nearly all the experiments show large scatter in the out-of-sample fairness, with many points below the $\tau = \tau_0$ line. Our results therefore show that not only are debiasers unable to guarantee fairness out-of-sample, but even when it can do so for a particular train--test split, the effect can disappear entirely for different test data.
Experiments A, C, G, H and I also show evidence of an fairness--accuracy trade-off: as the fairness improves, the accuracy worsens, and the graphs generally trace out a negative slope. The satisfiability analysis of fact shows that perfect accuracy and fairness can be attained in theory for all the experiments; however, we can understand this effect as arising from change in Bayes rate due to the additional fairness constraint imposed fact. Nevertheless, we also see evidence of overfitting, not just in the variance of $\gamma/\gamma_0$, but also in many points with $\gamma > \gamma_0$, where debiasing increased the accuracy of the classifier, but not in a robust way. Our results agree with Friedler2019, who showed that debiasing methods are prone to overfit on the training set, in that debiasing outcomes vary depending on the details of the train/test split, albeit without an explanation for this phenomenon. Our results are also consistent with Schelter2019, who showed that retuning hyperparameters is necessary to improve generalizability, also we do not investigate the effect of hyperparameter tuning in our work.
In summary,
Below in (ref), we provide a theoretical analysis of these phenomena in (ref) in terms of bias-variance trade-off. In additional experiments in (ref) and the Supplement, we also demonstrate the somewhat counter-intuitive result that a partial debiasing treatment can actually yield more fair classifiers in practice, which is the case for 12 out of the 27 combinations of experiment and debiaser.
We now present a theoretical analysis of the phenomena we have observed above. To simplify our approach, we consider the penalized (or dual) version of machine learning problems involving fairness constraints. Our starting point is the scalarized optimization program $\lambda \ell + (1-\lambda) \phi$, with fairness loss $\phi: \mathbb R^2 \rightarrow \mathbb R^+_0$, for example, $\phi(x, y) = |x - y|$. The trade-off is parameterized by $\lambda$, interpolating linearly between considering only fairness ($\lambda=0$) and only performance ($\lambda=1$).
We want to know how the empirical trade-off, as measured on some test set $\mathcal{D}^\star$, converges to the true trade-off, as measured on the true underlying distribution $(Z,S) \sim \mathcal{P}$.
An example of $\ell$ would be misclassification error $\ell = \mathbf{1}_{\lbrace \hat Y \neq Y \rbrace}$ (the complement of accuracy, $\mathbf{1}_{\lbrace \hat Y \ensuremath{\!=\!}{} Y \rbrace})$, while an example of $\mu$ would be predictive parity, $\mu = \mathbf{1}_{\lbrace \hat Y \ensuremath{\!=\!}1 \rbrace}$, corresponding to the fairness constraint $\Pr(\hat Y\ensuremath{\!=\!}1 | S\ensuremath{\!=\!}1) = \Pr(\hat Y\ensuremath{\!=\!}1 | S\ensuremath{\!=\!}0)$, i.e., demographic parity. The fairness loss $\phi$ is related to the symmetrized fairness gap $\Delta_h$ defined in (ref), since we can take $\phi(z_{|S\ensuremath{\!=\!}0}, z_{|S\ensuremath{\!=\!}1}) = |h(z_{|S\ensuremath{\!=\!}0}) - h(z_{|S\ensuremath{\!=\!}1})| = \Delta_h(z)$.
We now derive the limiting distribution of $L_{\mathcal{D}^\star}(f)$ and show that it exhibits some form of bias--variance decomposition.
This result can be proved with repeated use of the central limit theorem, the delta method and Slutsky's lemma. The full proof is included in the Supplement.
The first three terms in the limiting variance $\mathbb{V}_{\lim}(f)$ can be interpreted as
The last terms grow with $(1-\lambda)^2 k_i^2$, which intuitively captures how sensitivity to fairness constraints leads to increased variance. Interestingly, these terms are also inversely proportional to the base rates $\pi_s = \Pr(S=s)$, meaning that imbalance in the \smash{protected class} increases the variance.
(ref) predicts that the estimation variance depends on protected class imbalance, specifically, that the standard deviation of the estimated classifier $\sigma(\hat f) \sim 1/\Pr(S=1)$ as $\Pr(S=1)$ goes to zero. We should therefore expect a similar behavior for the fairness metric $\sigma(\tau) \sim 1/\Pr(S=1)$ computed for the estimated classifer. We now confirm this dependency on a simple synthetic data generating process that allows us to vary the base rates in both outcome class $\Pr(Y)$ and protected class $\Pr(S)$. The details of the synthetic data and experimental setup are given in the Supplement. We use 10 times repeated 10-fold cross-validation on 20,000 data points and report the standard deviation of $\tau_\textsc{EFPR}$ across the replications.
(ref) shows how the standard deviation of the fairness metric changes with the base rate $\Pr(S=1)$ for three different values of $\Pr(Y=1)$. Each curve has the same qualitative shape consistent with an inverse dependence on $\Pr(S=1)$. Our results support the theoretical analysis above that the variance in debiasing is strongly affected by class imbalances, both with respect to the class imbalance and the fraction of data points in the two protected attribute groups.
(ref) implies that when considering the trade-off between performance and fairness, it is possible to construct a variance-minimizing debiaser that does not perfectly debias a model, but has better generalization properties. Minimizing the limiting variance (ref) with respect to $\lambda$ will in general not yield a full debiaser $\lambda=1$, but rather some intermediate debiasing strength. This observation motivates our introduction of the notion of partial debiasing in this section. We will now describe two specific examples of partial debiasing.
\paragraph{Partial reweighing.} The reweighing pre-processor of (ref) can be easily generalized to yield a partial debiaser, simply by interpolating between the weight 1 (for $\lambda = 0$) and the weight $w_{h, i}$ for the fairness constraint $h$ in (ref) (for $\lambda=1$). The simplest such partial reweighing scheme is to simply perform linear interpolation, $ w_i = (1-\lambda) 1 + \lambda w_{h, i} $, although more exotic interpolation method could also be used.
\paragraph{Partial post-processing.} Similarly, for post-processing methods like equalized odds ((ref)) and NLinProg ((ref)), we can define a partial debiasing scheme simply by interpolating the flipping probabilities $\Pr(\hat Y^\prime | \hat Y\ensuremath{\!=\!}{}y, S\ensuremath{\!=\!}{}s)$ between 0 and their original values defined previously. Again for linear interpolation, this corresponds to replacing the flipping probabilities by $\lambda \Pr(\hat Y^\prime | \hat Y, S)$.
As we show below, we find some surprising and nontrivial behaviors of this simple partial reweighing scheme, including the result that partial debiasing is in general preferable to full debiasing ($\lambda = 1$) to produce a low-variance debiased classifier.
We finish with another follow-up experiment, reporting out-of-bag metrics after 10-fold cross-validation and train an initial logistic regression model. We then debias this same model multiple times with respect to predictive parity (PP) fairness using three different partial debiasers ((ref)): partial equalized odds post-processing (EOd), partial reweighing (RW), and partial NLinProg (NLP) for accuracy and PP fairness. (ref) shows three different trajectories from parametrically increasing the debiasing strength from $\lambda = 0$ (no debiasing) to $\lambda = 1$ (full debiasing), starting from the initial model ($\lambda = 0$) at coordinates $(\gamma/\gamma_0, \tau/\tau_0)= (1, 1)$, again aggregated across 10CV10 folds.
As in our earlier experiments in (ref), we see that the behavior of the different debiasers are markedly different. As before, it should be possible in theory to improve a classifier's fairness $\tau$ without compromising accuracy $\gamma$ in this experiment. On the contrary, we observe that reweighing barely changes the metrics of the model, whereas EOd steadily leads to worsened PP unfairness and worse accuracy. The worsened fairness is to be expected, however, since we are debiasing with respect to a different metric that we are measuring. In contrast, debiasing and measuring the same metric of fairness in NLinProg leads to improved fairness $\tau$, but at the expense of worsened accuracy $\gamma$. In this example, none of the debiased models come close to perfect fairness with metric $1/\tau_0 = 1.842$, implying that the training data and model family simply do not admit a perfectly fair classifier.
We have presented detailed empirical studies throughout the paper (especially (ref)) showing that classifiers treated with debiasing methods generally suffer from worse out-of-sample generalization behavior, so much so that the out-of-sample fairness can worsen relative to the original classifier. We need many test--train--validate splits to make a statistically significant determination of the treatment effect. As shown in the Supplement, the uncertainty in the fairness metric appears to be usually an order of magnitude larger than that for accuracy, which could reflect rare protected classes in (ref). We showed in (ref) that this increased variance can be explained by bias--variance trade-off. To remove the statistical bias in the classifier that corresponds to discriminatory bias, we have to impose a fairness constraint, but satisfying that constraint increases the uncertainty of where the best decision boundary can be drawn, especially when the baseline model is already carefully estimated with attention paid to out-of-sample generalization error. Furthermore, we confirmed empirically in (ref) that the estimation variance (ref) is particularly severe when any of the protected classes is rare, i.e., when the base rate $\Pr(S)$ approaches zero. In practice, full debiasing is also not desirable if the performance of the debiased classifier degrades too much. We showed in (ref) that the fine-grained control afforded by partial debiasing allows us to learn new classifiers that have desirable out-of-sample fairness properties.
The empirical results, while mostly negative, have motivated the theoretical analysis of (ref), which gives us detailed insight into the origins of the large variance in classifiers. In particular, (ref) states that the variance varies dramatically with protected class imbalance, which to our knowledge is a new result. Furthermore, (ref) suggests that that partial debiasing can let us find a variance minimizing estimator that, while not applying the full debiasing treatment, can yield better fairness properties that generalize in practice. Since it is in general difficult to vary this trade-off parameter $\lambda$ explicitly to find this estimator, finding practical ways to compute this minimal variance estimator seems like a promising research direction that could improve the practical utility of debiasing methods. Conversely, our results also show that fundamental limits exist to the ability to debias arbitrary models in a purely black box manner. Identifying underlying causal connections linking protected classes to features Zhang2018 may therefore be a more promising direction for successful mitigation of bias.
Our results signal caution to avoid the risk of fairwashing Aivodji2019,Anders2020, in the sense of believing that one is using a fair model resulting from some debiasing treatment, when in fact the model is overfit and does not generalize well out-of-sample Friedler2019. Rather than blindly trusting that a debiased classifier is now fair, our results demonstrate that debiasing treatments need to be carefully tested in order to verify that the desired fairness properties hold in practice.
This work has been funded by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. The authors of this work take full responsibilities for its content. We also thank our generous funding agencies IQVIA, UNIVERSITY of Auckland, Turing and tools practices, and Microsoft.
\paragraph{Disclaimer} This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates (“JP Morgan”), and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.