Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
134,579 characters · 15 sections · 101 citation commands
Binary Choice under Asymmetric Loss in a Data-Rich Environment: Theory and an Application to Algorithmic Fairness
\affil[1]{UNC Chapel Hill} \affil[2]{IIT Delhi}
\thispagestyle{empty} \setcounter{page}{0}
Many economic decisions are inherently binary. This includes life-changing decisions such as pretrial release, college admission, job hiring, medical testing and treatment---areas increasingly influenced by automated algorithmic processes based on vast data inputs. It also extends to routine tasks such as loan or credit card approvals, fraud detection, or spam filters. The topic has gained interest from a diverse set of fields, ranging from economics and computer science, to machine learning (ML), among others, and depending on the discipline is also known as classification or screening problems.
In this paper, we focus on data-driven binary decision problems characterized as a solution to the following risk-minimization problem:
where $Y\in\{-1,1\}$ is a binary outcome, $X\in\mathcal{X}\subset\ensuremath{\mathbf{R}}^d$ is a vector of covariates, $f:\mathcal{X}\to\{-1,1\}$ is a binary decision (or prediction) function, $\ell:\{-1,1\}^2\times\mathcal{X}\to\ensuremath{\mathbf{R}}_+$ is a loss function, and the expected value $\ensuremath{\mathbb{E}}_{X,Y}$ is taken with respect to the distribution of $(X,Y)$. The problem in ((ref)) is minimized over all measurable functions of covariates and the loss function $\ell$ describes the asymmetric consequences in different states of the world. For example, pretrial detention decisions may reflect the asymmetric cost of (a) keeping someone wrongly in jail before his/her trial, (b) releasing a recidivist, versus the social benefits of (a) keeping potential recidivists in jail, and (b) releasing non-recidivists who could be productive members of society. In the loan approval application, the losses may depend on loan size and other considerations such as equal credit opportunities.
The importance of asymmetries in economic decision problems has been recognized for a long time; see granger1969prediction, manski1989estimation, granger2000economic, diebold1996optimal,diebold1997optimal, elliott2016economic, among others. The combination of high-dimensional data and general loss functions in binary decision problems is challenging and not well understood.\footnote{In contrast, the asymmetric decisions in the regression setting can be achieved via quantile regression, see koenker1978regression, the asymmetric least-squares, see newey1987asymmetric, or more generally with M-estimators based on a suitable asymmetric loss function; see elliott2016economic.} Econometricians have studied nonparametric binary choice problems for a long time, but the literature does not offer computationally attractive solutions for high-dimensional datasets. For example, elliott2013predicting consider the binary utility maximization problem similar to ((ref)) and propose to solve its empirical counterpart by extending the maximum score estimator of manski1975maximum. These examples typically involve non-smooth and non-convex combinatorial optimization which is known to be NP-hard and, hence, challenging to apply in modern data-rich environments.\footnote{Several efforts have been made to make these problems computationally feasible. For example, horowitz1992smoothed proposes to use kernel smoothing which addresses the lack of smoothness, but not the lack of convexity. florios2008exact considers the mixed-integer optimization that requires special solvers without guarantees of finding the global optimum.}
In this paper, we formally show that the valid binary decisions corresponding to the problem ((ref)) can be obtained by solving simple weighted logistic regression or more generally weighted machine learning problems including deep learning, boosting, and LASSO, where the weights are computed directly from the loss function. Our approach can be understood as solving a convexified counterpart to the empirical utility maximization problem of elliott2013predicting similarly to modern classification methods solving the convexified counterparts to the maximum score estimator of manski1975maximum.
The practical implementation of our method is remarkably simple, which is why it can be discussed here. The first ingredient comes from a policy/decision-maker who has to provide a quartet of loss functions pertaining to (a) true positives, denoted by $\ell_{1,1}(x)$, (b) true negatives, $\ell_{-1,-1}(x)$, (c) false positives, $\ell_{-1,1}(x)$, and (d) false negatives, $\ell_{1,-1}(x)$, where we define $\ell_{f,y}(x):=\ell(f,y,x)$. This requires a careful selection of relevant covariates, $x\in\mathcal{X}$, and functional forms describing the economic decision problem. These are usually determined by either a cost/utility function of the social planner (policy/decision-maker), or are obtained through some type of cost-benefit analysis. Given these economic inputs, the task of the econometrician is to compute weights from $\omega(y,x)$ with $y\in\{-1,1\}$ and $x\in\mathcal{X}$ that will be applied to the appropriate classification procedures. We show that this very simple loss-based reweighting leads to valid binary decisions without strong distributional assumptions. In some cases, e.g., for carefully crafted deep learning architectures, these decisions are optimal from the minimax point of view.
Our methodology can incorporate various preferences through the loss function including the treatment of protected groups which is related to the ongoing debate about algorithmic biases. As more and more decisions affecting our daily lives have become digitized and automated by data-driven algorithms, concerns have been raised regarding such biases. The potential for ML algorithmic outcomes to reproduce and reinforce existing discrimination against legally protected groups has been of great concern. Gender and race are two leading examples. cowgill2019economics discuss the case of computer scientists at Amazon who developed powerful new technology to screen resumes and discovered the algorithm placed a negative coefficient on terms associated with women and as a result appeared to be amplifying and entrenching male dominance in the technology industry. Another example of gender discrimination is discussed by datta2015automated who studied AdFisher, an automated tool that explores how user behaviors, Google's ads, and Ad Settings interact. They found that setting the gender to female resulted in fewer instances of an ad related to high-paying jobs than setting it to male.
Related to our empirical illustration, journalists at the news website ProPublica reported on a commercial software used by judges in Broward County, Florida, that helps to decide whether a person charged with a crime should be released from jail prior to the trial. They have found that the software tool called Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) resulted in a disproportionate number of false positives for African American defendants who were classified as high risk but subsequently not charged with another crime. We apply our covariate-driven classification methods to pretrial decisions with an emphasis on algorithmic fairness and show how these biases can be eliminated naturally when the decision-maker incorporates preferences for false positive mistakes for protected groups.\footnote{It is also worth mentioning that human decision-makers can also be biased. For example, posner2025judge reports that large language models tend to rely more on legal arguments than human judges who may be influenced by sympathy and emotions.}
Therefore, we make contributions to different strands of the literature. First, our paper is related to the general social planner setting studied by rambachan2020economic as well as the special cases studied by corbett2017algorithmic and kleinberg2018algorithmic, among others. rambachan2020economic argue that the algorithmic fairness constraints can be addressed using the tools of welfare economics, where a social planner has group-specific welfare weights. Our paper adopts the same point of view, but we take a step further showing how to map these welfare functions to the specific ML methods. Note that corbett2017algorithmic and menon2018cost suggest incorporating fairness constraints in ML algorithms which is equivalent to using the plug-in classifiers, where the fitted probability is compared to race-specific thresholds. agarwal2018reductions and cotter2019two consider the symmetric classifiers with non-convex fairness constraints which are challenging to use in practice. In contrast, our work focuses on convexified asymmetric classifiers that do not require fitting the conditional probabilities. kitagawa2023constrained use convexification techniques similar to ours and show that the standard symmetric binary classification can incorporate various constraints with a Hinge convexifying function. In contrast, our work focuses on incorporating fairness through the loss function similarly to rambachan2020economic which can be also done with the logistic and exponential convexifying functions.
Second, our paper is related to the literature on empirical utility and welfare maximization; see manski2004statistical, hirano2009asymptotics, bhattacharya2012inferring, zhao2012estimating, kitagawa2018should, athey2021policy, mbakop2021model, kallus2021more, viviano2023fair, adjaho2022externally among others. This literature usually relies on the potential outcomes framework and deals with continuous outcomes. In contrast to this work, we consider the unconstrained asymmetric binary classification. Our paper is also related to su2021model,su2021utility who highlights connections between elliott2013predicting and the cost-sensitive classification literature. su2021model studies the model selection problem for the non-smooth non-convex empirical utility maximization of elliott2013predicting while su2021utility considers the utility-maximizing nearest neighborhood method. In contrast, our work proposes to solve a convexified problem and establishes the performance guarantee for asymmetric Logit, deep learning, LASSO, and boosting which has not been done in the aforementioned papers.
Regarding deep learning for economic applications, our paper is related to the recent work of farrell2018deep who show that under weak assumptions the deep learning estimators of the regression function can achieve sufficiently fast convergence rates for semiparametric inference and establish supporting non-asymptotic results; see also chen2007sieves for an earlier review of shallow neural networks and dell2024deep for economic applications of deep learning.
Lastly, we also contribute to classification literature in statistics and computer science; see zhang2004statistical, bartlett2006convexity, boucheron2005theory, koltchinskii2011oracle, for the theory of symmetric binary classification. The classification with asymmetric losses is also known as cost-sensitive or example-dependent classification and has been developing largely independently from the related econometric literature, e.g.\ see elkan2001foundations. While this literature is quite large, few formal results exist justifying proposed methods. For example, bahnsen2014example introduces weights in logistic regression heuristically without providing formal justifications that the weights actually correspond to the asymmetric decision problem. Several papers were written subsequently, see bahnsen2015example, Mehta2020deepcatch and xia2017cost among others, mostly with practical computer science applications such as fraud detection, e-commerce, or marketing. There is also a body of literature on classification with imbalanced or corrupted classes, where weighting is also done heuristically to improve the accuracy of a classifier, which can be highly sensitive to over-represented and/or noisy classes; see ting1998inducing and sun2007cost among others. This literature considers weights that are different from ours, e.g., based on the empirical number of instances in a given class, or proportional to the inverse of expected costs that are simulated with the importance or rejection sampling. scott2011calibrated,scott2012calibrated, and bao2020calibrated are examples that attempt to deal rigorously with several instances of asymmetric classification problems. Neither of these papers address the problem ((ref)) in full generality studied in our paper. For example, scott2012calibrated covers a special case of our problem with covariate-independent false positive/negative losses while scott2011calibrated focuses on predicting a sign of a real-valued random variable. Moreover, these papers do not provide specific estimators and do not establish the supporting theoretical results. In contrast, our paper shows how the weights coming from the asymmetric loss function can be incorporated in the cost-sensitive asymmetric Logit, deep learning, boosting or LASSO.
The paper is organized as follows. Section (ref) introduces notation, describes the binary decision problem in terms of risk/payoff, and illustrates a convenient characterization of the optimal binary decision. Examples of several asymmetric binary decision problems are also provided. Section (ref) covers the main convexification theorem and provides examples of convexifying functions. In Section (ref), we provide the excess risk bounds for the binary decision rules constructed from the data and discuss several cases, including logistic regression, LASSO, and deep learning. Section (ref) presents Monte Carlo simulation results. Finally, we provide an empirical application to pretrial detention in Section (ref) followed by conclusions. The proofs of key results appear in the Appendix, while the Online Appendix provides additional results for boosting, LASSO, and shallow learning as well as the detailed data description for the empirical application.
Let $Y\in\{-1,1\}$ be the target variable and let $X\in\mathcal{X}\subset\ensuremath{\mathbf{R}}^d$ be covariates.\footnote{We use capitals for random variables and lowercase letters for realizations.} A measurable function $f:\mathcal{X}\to\{-1,1\}$ is called the binary decision/choice/prediction. The decision-making process amounts to minimizing a risk function that describes its consequences in different states of the world
where $\ell:\{-1,1\}^2\times\mathcal{X}\to\ensuremath{\mathbf{R}}$ is a loss function specified by the decision-maker. Note that the decision $f$ can be random and the expectation is taken with respect to the distribution of $(Y,X)$ only. The loss function can be asymmetric and may also depend on the covariates $X$, which is economically a more realistic scenario faced by the decision-maker than the one provided by the standard classification setting.
In the Introduction, we alluded to many examples in economic decision-making. To motivate the first example, we can think of a credit risk application, where with the false negative mistakes ($f(X)=-1$ and $Y=1$) the bank suffers a loss from the borrower's default, while with the false positive mistakes ($f(X)=1$ and $Y=-1$), the bank simply foregoes its potential earnings. Moreover, the size of the loan and other economic factors may determine the loss, see also lieli2010construction:
As noted in the Introduction, an important policy debate pertains to the fairness and the discrimination bias of machine learning algorithms towards, e.g., low income groups, gender, or race. The following example suggests that the unfair treatment of individuals can be eliminated if the social planner assigns group-specific weights in the loss function, see also rambachan2020economic:
Computer scientists have focused on defining fairness-aware algorithms by imposing restrictions on $f$.\footnote{Alternatively, one could fit two different classifiers for two groups and use the plug-in approach with group-specific cut-off. This is not the route taken in our paper. We provide some comparisons to plug-in classifiers in simulations.} Often a formal criterion of fairness is defined, and a decision rule is developed to satisfy the criterion, e.g., the statistical parity across groups. We follow the economist's arguments that fairness is naturally defined through welfare/losses.
In the remainder of the paper, we will continue to work with settings involving a generic vector of covariates $X\in\ensuremath{\mathbf{R}}^d$ which may contain the binary group membership variable $G\in\{0,1\}$ and some other covariates $Z\in\ensuremath{\mathbf{R}}^{d-1}$. The point worth keeping in mind, however, is that our framework covers much discussed preference-based notions of fairness characterized by general covariate-driven loss functions.
Following the decision-theoretic perspective, we define the optimal binary decision $f^*:\mathcal{X}\to\{-1,1\}$ as a solution to
where the minimization is done over all measurable functions $f:\mathcal{X}\to\{-1,1\}$. We also use $\mathcal{R}^*=\mathcal{R}(f^*)$ to denote the corresponding smallest achievable risk. Note that this framework also covers the utility/welfare maximization problems, in which case, we can define $\ell(f(X), Y,X)=-U(f(X),Y,X)$, where $U(f(x),y,x)$ is a utility/welfare function of a binary decision $f(x)$ when the outcome is $Y=y$ and covariates are $X=x$.
The following proposition provides an alternative convenient for us characterization of the optimal binary decision, also known as the Bayes' decision; see Appendix for a proof.
Proposition (ref) is related to elliott2013predicting, but in contrast to their work, it recasts the asymmetric classification problem in a way that is amenable to convexifiction. Note that $a$ can be interpreted as a difference of net losses $\ell_{-1,1}-\ell_{1,1}$ and $\ell_{1,-1}-\ell_{-1,-1}$ from wrong decisions when $Y=1$ and $Y=-1$ respectively. Similarly, $b$ can be interpreted as a sum of two net losses. Proposition (ref) shows that the optimal binary decision minimizes the objective function involving the indicator function $z\mapsto \ensuremath{\mathds{1}}_{z\geq 0}$, which is discontinuous and not convex. This leads to a difficult NP-hard empirical risk minimization problem
The purpose of this section is to convexify the binary decision problem with a generic loss function making it amenable to modern ML algorithms. It is easy to see that the risk function of a binary decision rule $f:\mathcal{X}\to\{-1,1\}$ can be written as
with $d(y,x) = 0.25(\ell_{1,1}(x) + \ell_{-1,1}(x))(1+y) + 0.25(\ell_{1,-1}(x) + \ell_{-1,-1}(x))(1-y) - 0.25\omega(y,x)$; see the proof of Proposition 2.1 in the Appendix. Replacing the indicator function with a convex function $\phi:\ensuremath{\mathbf{R}}\to\ensuremath{\mathbf{R}}$, we obtain the convexified risk
Therefore, minimizing the convexified risk amounts to solving
Let $f^*_\phi$ be a solution to the convexified problem and let $\mathcal{R}^*_\phi = \inf_{f:\mathcal{X}\to\ensuremath{\mathbf{R}}}\mathcal{R}_\phi(f)$ be the optimal convexified risk. Next, we can define the excess convexified risk of a decision $f:\mathcal{X}\to\{-1,1\}$ as
The excess convexified risk measures the deviation of the convexified risk of a given decision rule $f:\mathcal{X}\to\{-1,1\}$ from the optimal convexified risk and can be controlled. Unfortunately, the convexified excess risk tells us little about the performance of the binary decision in terms of the actual excess risk that the decision-maker cares about, namely:
In the following subsection, we show that the excess risk is bounded from above by the convexified excess risk, a result that we refer to as the main convexification theorem.
Let $\eta(x)=\Pr(Y=1|X=x)$ be the conditional probability. We impose the following assumption on the conditional probability $\eta$ and the loss function:
Assumption (ref) (i) requires that the losses from wrong decisions exceed those of correct decisions. It ensures that weights are $\omega(Y,X)\geq 0$ a.s., so that the convexification is possible (ii) requires that $\eta(x)$ is non-degenerate for almost all states of the world $x\in\mathcal{X}$ and is often imposed in econometrics literature. (iii) requires that the loss function is bounded and is satisfied in our empirical application.
Next, we restrict the class of convexifying function $\phi$.
Assumption (ref) (i) and (ii) are relatively mild restrictions imposed on the convexifying function. (iii) is a technical condition which we verify for the logistic, exponential, and hinge convexifications; see Lemmas (ref), (ref), and (ref) in the Online Appendix. We focus on these three convexifications since they cover the majority of ML algorithms used in practice, including the (penalized) logistic regression, boosting, deep learning, and support vector machines. Note also that Assumption (ref) does not impose any restrictions on the loss function $\ell$.
We will also use the following covariate-driven threshold rule defined by the asymmetry of the loss function $\ell:$
where $\ell_{f,y}(x)=\ell(f,y,x)$.
For $z\in\ensuremath{\mathbf{R}}$, put $\ensuremath{\mathrm{sign}}(z)=\ensuremath{\mathds{1}}_{z\geq 0} - \ensuremath{\mathds{1}}_{z<0}$. Our first result establishes the link between the optimal decision $f^*$ and the solution to the convexified risk minimization problem $f^*_\phi.$
The optimal decision rule $f^*(x)=\ensuremath{\mathrm{sign}}(\eta(x)-c(x))$ is well known, see elliott2013predicting and references therein.\footnote{For completeness of presentation, we provide a concise proof in the Online Appendix.} More importantly, Proposition (ref) shows that the optimal decision rule for the convexified problem that can be easily solved in practice coincides with $f^*$. The threshold $c(x)$ corresponds to the fraction of net losses from false positives in the total net losses, hence, the decision $f(x)=1$ is made whenever the choice probability $\Pr(Y=1|X=x)$ exceeds the fraction of false positive losses. Note that in the symmetric binary classification case $\ell_{1,-1}(x)=\ell_{-1,1}(x)=1$ and $\ell_{1,1}(x)=\ell_{-1,-1}(x)=0$, so that $c(x)=1/2$ and the optimal decision is $f^*(x)=1$ if $\Pr(Y=1|X=x)$ exceeds $\Pr(Y=0|X=x)$ and $f^*(x)=-1$ otherwise. Interestingly, the optimal decision rules depend on $\eta(x)=\Pr(Y=1|X=x)$, which is related to the heteroskedasticity $\ensuremath{\mathrm{Var}}(Y|X=x)=4\eta(x)(1-\eta(x))$ and all other moments; cf. diebold1996optimal,diebold1997optimal.
It is worth mentioning that our starting point is a fixed decision problem described by the loss function $\ell$, which in conjunction with $\eta$ determines the boundary separating the two decisions $\{x\in\mathcal{X}:\; \eta(x)-c(x) = 0 \}$. The following condition generalizes the so-called Tsybakov noise or margin condition, see boucheron2005theory, and quantifies the complexity of the decision problem.
Assumption (ref) describes the complexity of the classification problem around the decision boundary $\{x:\eta(x)=c(x) \}$. If $\alpha=0$, then it does not impose any restrictions as we can always take $C_m=1$, while larger values of $\alpha$ are more advantageous for classification. In the special case of the symmetric binary classification, $c(x)=1/2$, and the extreme case of $\alpha=\infty$ corresponds to $\eta(x)=\Pr(Y=1|X=x)$ being bounded away from $1/2$ which roughly means that there are no observations close to the decision boundary. In this case, we can interpret this as a perfect separation between recidivists and non-recidivists in the space of covariates. We refer to boucheron2005theory, Section 5.2, for additional discussion of the margin condition as well as for equivalent formulations in the special case of symmetric binary classification; see also ponomarev2024lower.
Our next result relates the convexified excess risk to the excess risk of the binary decision problem a generic loss function under the margin condition.
The formal proof of this result with the explicit expression for the constant $C_\phi$ appears in the Appendix. It is worth noting that our result nests the symmetric binary classification as a special case; see e.g.\ zhang2004statistical, Theorem 2.1. Theorem (ref) relates the excess risk of the sign of $f$ to the convexified risk of $f$ (note that the convexified risk is well-defined for arbitrary $f:\mathcal{X}\to\ensuremath{\mathbf{R}}$). Therefore, instead of solving the NP-hard empirical risk minimization problem, we can solve the following weighted classification problem
where $\mathscr{F}$ is a class of measurable functions $f:\mathcal{X}\to[-1,1]$, called a soft decision. Note that we restrict the range of the soft decision to $[-1,1]$ since the risk is determined by the sign of $f$ only.
We consider three examples of convexifying functions widely used in empirical applications. They are the logistic, exponential, and hinge functions. These convexifying functions appear in various ML algorithms including boosting, support vector machines, neural networks, and simple logistic regression; see Figure (ref). Applying Theorem (ref) allows us to implement suitably reweighted versions.
For the symmetric loss function and linear function class, $f(X_i) = X_i^\top\theta$, the problem in equation ((ref)) becomes the standard logistic regression. The deep learning consists of minimizing the problem in equation ((ref)) over the class of deep neural networks. Lastly, the state-of-the-art Extreme Gradient Boosting (XGBoost) algorithm of chen2016xgboost can be understood as a functional gradient descent algorithm minimizing the problem in equation ((ref)) with some additional regularization; see also friedman2001elements, Ch. 10.
For the symmetric loss function, this objective function is usually minimized iteratively in the adaptive boosting (AdaBoost); see also friedman2001elements, Ch. 10.
For the symmetric loss function, this objective function is usually minimized subject to a reproducing kernel Hilbert space constraint within support vector machines; see friedman2001elements, Ch. 12.
The convexified empirical risk minimization problem consists of minimizing the empirical risk
over some class of functions $f:\mathcal{X}\to[-1,1]$, denoted $\mathscr{F}_n$. Let $\hat f_n$ be a solution to $\inf_{f\in\mathscr{F}_n}\widehat{\mathcal{R}}_{\phi}(f)$, and let $f_n^*$ be a solution to $\inf_{f\in\mathscr{F}_n}{\mathcal{R}}_{\phi}(f)$. Put also $\|f\|_q = (\ensuremath{\mathbb{E}}|f(X)|^q)^{1/q}$ with $q\geq 1$ and $\|.\|_2=\|.\|$. The following assumption requires that the risk function is sufficiently curved around the minimizer.
It is worth mentioning that in light of Assumption (ref) imposed on the loss function $\ell$, Assumption (ref) is mainly about the curvature of the convexifying function $\phi$. In particular, it is typically satisfied with $\kappa=1$ when $\phi''$ is strictly positive, which is the case for the logistic and exponential convexifications; see Lemma (ref) in the Online Appendix. On the other hand, it holds with $\kappa=1 + 1/\alpha$ in the hinge case, where $\alpha$ is the margin parameter from the Assumption (ref); see Lemma (ref). It is worth mentioning that the curvature condition in Assumption (ref) is typically used to establish bounds on $\|f-f_\phi^*\|$ and may not be needed to establish the slow convergence rates of the excess risk. However, this condition is typically used to obtain the fast convergence rates for the excess risk in the symmetric classification and regression cases; see koltchinskii2011oracle.
We first state the oracle inequality for the excess risk in terms of a fixed point of the local Rademacher complexity of the class $\mathscr{F}_n$ defined as
where $R_n(f-f_n^*) = \frac{1}{n}\sum_{i=1}^n\varepsilon_i(f(X_i)-f_n^*(X_i))$ is a Rademacher process, i.e., $(\varepsilon_i)_{i=1}^n$ are i.i.d. in $\{-1,1\}$ with probabilities $1/2$. An attractive feature of this complexity measure is that it only depends on the local complexity of the parameter space in the neighborhood of the minimizer and provides a sharp description of the learning problem; see koltchinskii2011oracle. At the same time, the local Rademacher complexities are general enough to provide a unified theoretical treatment for different methods and can be used to deduce oracle inequalities in many interesting examples. For a function $\psi:\ensuremath{\mathbf{R}}_+\to\ensuremath{\mathbf{R}}_+$, put $\psi^\flat(\sigma) = \sup_{\delta\geq\sigma} [\psi(\delta)/\delta]$ and for a constant $\kappa\geq 1$, put $\psi_\kappa^{\sharp}(\epsilon) = \inf\left\{\sigma>0:\;\sigma^{1/\kappa-1}\psi^\flat(\sigma^{1/\kappa})\leq \epsilon \right\}$. The transform $\psi^\sharp_{\kappa}$ is a generalization of the $\sharp$-transform considered in koltchinskii2011oracle and describes the fixed point of the local Rademacher complexity in our setting. The following result holds:\footnote{Proofs for all results in the Appendix.}
The proof of this result appears in the Appendix and provides the explicit expression for all constants. Theorem (ref) tells us that the accuracy of the binary decision $\ensuremath{\mathrm{sign}}(\hat f_n)$ depends on the fixed point of the local Rademacher complexity of the class, $\psi^\sharp_{n,\kappa}$, and the approximation error to the convexified risk of the optimal decision. In the remaining part of this section, we will specialize this result to parametric decisions and deep neural networks. In each case it can be seen that for a fixed $\epsilon$, we have $\psi_n^\sharp(\varepsilon)\to 0$ as long as $n\to\infty$ and the complexity of the class $\mathcal{F}_n$ does not increase “too fast". Therefore, Theorem (ref) illustrates the variance/bias trade-off, i.e. more complex classes of decision rules reduce the approximation error (bias) at costs of increasing the Rademacher complexity (variance) and vice versa. Note that in the special case when $\kappa=1$, Theorem (ref) recovers koltchinskii2011oracle, Proposition 4.1. In the following subsections, we illustrate this result for parametric and nonparametric binary decision rules. Additional results on boosting and LASSO are available in the Online Appendix.
We start with illustrating our risk bounds for parametric binary decision rules. The decision rule is defined $\ensuremath{\mathrm{sign}}(f_{\hat\theta})$ with $\hat\theta$ solving
where $f_\theta(x)=\sum_{j=1}^p\theta_j\varphi_j(x)$, $\theta\in\Theta\subset\ensuremath{\mathbf{R}}^p$, and $(\varphi_j)_{j\geq 1}$ is a collection of functions in $L_2(P_X)$, called the dictionary. This covers the linear functions $f_\theta(x)=x^\top\theta$ as well as nonlinear functions of covariates provided that the dictionary $(\varphi_j)_{j\geq 1}$ contains nonlinear transformations. The most popular choice of convexifying function is the logistic function $\phi(z)=\log_2(1+e^z)$. More generally, we have the following result for any convexifying function satisfying Assumption (ref).
It follows from Lemmas (ref), (ref), (ref), (ref), and (ref) that for the logistic and the exponential functions $\gamma=1/2$ and $\kappa=1$ while for the hinge function $\gamma=1$ and $\kappa=1+1/\alpha$. Therefore, in all three cases, for parametric decisions we obtain
uniformly over a set of distributions restricted in Theorem (ref). For a fixed $p$ the convergence rate of the first term can be anywhere between $O(n^{-1/2})$ and $O(n^{-1})$ depending on margin parameter $\alpha$. Since this rate is independent of the convexification, one can use logistic regression reweighted for the asymmetries of the loss function when the signal-to-noise ratio is low and the approximation error is dominated by the first term. Besides simplicity, this choice is also attractive because: 1) the objective function is differentiable; 2) it recovers logistic MLE in the symmetric case; 3) it has a slightly better constant in Theorem (ref) than the exponential function. However, from the nonparametric point of view, the choice of the convexifying function is more subtle and the hinge convexification can lead to a better rate of the approximation error as we shall see in the following section.
In the high-dimensional case, when $p$ can be large relative to $n$, we can consider the weighted empirical risk minimization problem with the LASSO penalty
where $|.|_1$ is the $\ell_1$ norm and $\lambda_n\downarrow 0$ is a tuning parameter. We can deduce from the Online Appendix, Theorem (ref) that for the hinge, logistic, and exponential convexifications with probability at least $1-\delta$
where $s$ is the number of non-zero coefficients in a certain oracle vector $\theta^*\in\ensuremath{\mathbf{R}}^p$. According to this bound: 1) the dimension $p$ may increase exponentially with the sample size if $s$ is small; 2) we can use the logistic convexification in the low signal-to-noise settings, where the approximation error is relatively small.
In this final subsection, we discuss how to construct the asymmetric deep learning architecture and discuss the corresponding risk bounds; see farrell2018deep and references therein for more details. The deep learning amounts to fitting a neural network with several hidden layers, also known as a deep neural network. Fitting a deep neural network requires choosing an activation function $\sigma:\ensuremath{\mathbf{R}}\to\ensuremath{\mathbf{R}}$ and a network architecture. We focus on the ReLU activation function, $\sigma(z) = \max\{z,0\}$, which is the most popular choice for deep networks.\footnote{Other activation functions used in the deep learning include: leaky ReLU, $\sigma(z)=\max\{\alpha z,0\},\alpha>0$; exponential linear unit (ELU), $\sigma(z) = \alpha(e^z - 1)\ensuremath{\mathds{1}}_{z<0} + z\ensuremath{\mathds{1}}_{z\geq 0}$; and scaled ELU.}
The architecture consists of $d$ neurons corresponding to covariates $X=(X_1,\dots,X_d)\in\ensuremath{\mathbf{R}}^d$, one output neuron corresponding to the soft prediction $f\in[-1,1]$, and a number of hidden neurons. The final decision is obtained with $\ensuremath{\mathrm{sign}}(f)\in\{-1,1\}$. Hidden neurons are grouped in $L$ layers, known as the depth of the network. A hidden neuron $j\geq 1$ in a layer $l\geq 1$ operates as $z\mapsto \sigma(z^\top a_j^{(l)} + b_j^{(l)})$, where $z$ is the output of neurons from the layer $l-1$ and $a_j^{(l)},b_j^{(l)}$ are free parameters. The last layer and the output neuron produce together $z\mapsto \sigma(z + b^{(L)}c(x) + 1) - \sigma(z + b^{(L)}c(x) - 1) - 1\in\ensuremath{\mathbf{R}}$, where $b^{(L)}$ is the parameter to be estimated and $c(x)$ is a known decision cut-off function. The network architecture $(L,\mathbf{w})$ is described by the number of hidden layers $L$ and a width vector $\mathbf{w}=(w_1,\dots,w_{L})$, where $w_{l}$ denotes the number of hidden neurons at a layer $l=1,2,\dots,L$. For completeness, put also $w_0=d$ and $w_{L+1}=2$. Our final deep learning architecture $(L,\mathbf{w})$ is
where $\Theta_n^{\rm DNN} = \left\{\theta(x) = A_{L-1}\sigma_{\mathbf{b}_{L-1}}\circ \dots \circ A_1\sigma_{\mathbf{b}_1}\circ A_0x:\; \|\theta\|_\infty\leq F \right\}$, each $A_l$ is $w_{l+1}\times w_{l}$ matrix of network weights and for two vectors $y=(y_1,\dots,y_r)$ and $\mathbf{b}=(b_1,\dots,b_r)$ (a bias vector), we put $\sigma_{\mathbf{b}}\circ y = (\sigma(y_1 + b_1), \dots,\sigma(y_r + b_r))^\top$. Our deep learning architecture can be arranged on a graph presented in Figure (ref). Note that the asymmetries of the loss function are incorporated in the neural network architecture via the orange neuron which is fed directly to the last layer consisting of 2 ReLU neurons. Interestingly, this additional connection can be obtained using the so-called “residual connection" which is also often used in Transformer models, such as ChatGPT and AlphaFold, cf. targ2016resnet.
The following assumption restricts the smoothness of the conditional probability and imposes some assumptions on how the network architecture should scale with the sample size. For simplicity, we define the width of the network as the maximum width across all layers and denote it as $W_n = \max_{0\leq l\leq L}w_l$.
It is worth mentioning that we allow for neural networks, where the product of the width and the depth increase at the rate specified in Assumption (ref) (ii). Let $\mathscr{F}_n^{\rm DNN}$ be a set of neural networks with the architecture satisfying Assumption (ref), where weights and biases $\{A_0,A_l,b_l,l=1,\dots,L \}$ are allowed to take arbitrary real values.
The soft deep learning decision $\hat f_n$ is a solution to the empirical risk minimization problem with the hinge convexification
The following result holds for the binary decision estimated with the deep learning.
Note that in the special case of symmetric binary classification this result recovers the convergence rate recently obtained in kim2021fast, Theorem 3.3, who in turn assume that the weights are bounded and allow for very slowly diverging depth of order $L_n = O(\log n)$. In particular, this rate matches the minimax lower bound up to the $\log n$ factor; see audibert2007fast.
Anticipating the empirical application to the economic prediction of recidivism, we report on a simulation study pertaining to pretrial detention decisions. As explained in the next section, we present a design pertaining to judges who need to decide whether to release an offender, facing the possibility that the defendant might commit other crimes versus keeping in jail a defendant who would obey the law and the terms of the release. A first subsection describes the design and the second the simulation results.
There are two groups, $G=0$ and 1, with one group assumed to be a protected segment of the population. We set $Y=-1$ if the person does not commit a crime upon pretrial release, and $Y=1$ otherwise. This is a much-studied topic and we approach it from a social planner point of view using a simplified stylized example with a loss function taken from Example (ref):
where $z$ is a vector of observable characteristics and $\psi_g,\varphi_g>0$ for $g\in\{0,1\}$. From the above, we do not suffer any losses (or gains) when a defendant being released becomes a productive member of society or when a defendant who would commit another crime is kept in jail.
Keeping someone in jail who does not intend to commit another crime comes with costs $\varphi_g$ depending on the group membership $g\in \{0,1\}$. If $\varphi_1>\varphi_0$, then the cost of keeping an individual in jail who does not intend to commit another crime is higher if that individual is in group $g=1$. Similarly, releasing a recidivist comes with costs $\psi_g,g\in\{0,1\}$, so that if $\psi_1\ne \psi_0$, the costs of releasing a recidivist are different for the protected group and everyone else.\footnote{It is worth mentioning that in credit risk applications, the bank might care more about the false negative mistakes (failing to predict defaults), while the social planner might be more concerned with the false positive mistakes (failing to predict that the loan will be repaid) and equal credit opportunities regardless of the group membership. Since our general framework can be applied to different binary prediction problems in economics, in this simulation study, we will look at how both mistakes change with $\varphi_g$ and $\psi_g$ for $g\in\{0,1\}$.}
According to equation ((ref)), the threshold between the two binary decisions is:
and according to Proposition (ref) the optimal decision rule is $f^*(g,z)$ = $\ensuremath{\mathrm{sign}}(\eta(g,z) - c(g,z))$ with $\eta(g,z)=\Pr(Y=1|G=g,Z=z)$. Note also that $a(g,z)=\psi_g - \varphi_g$ and $b(g,z)=-(\psi_g + \varphi_g)$. The design of the data generating process is
where $\varepsilon\sim N(0,1)$, $G\sim \mathrm{Bernoulli}(\rho)$, and $Z_1,\dots,Z_d\sim_{i.i.d.} N(0,1)$. Therefore, the protected segment is a fraction $\rho$ of the population (determined by the Bernoulli distribution parameter) and the conditional probability is
where $\Phi_\sigma$ is the CDF of $N(0,\sigma^2)$. Note that the DGP may feature a very simple example of nonlinearities with quadratic terms and interactions involving $Z_1.$ In addition, the DGP covers the linear model with $\tau = 0.$ Lastly, we also set $\gamma = (1,0.9,0.8,0,0,\dots,0)^\top$ and the dimension $d$ of covariates $Z$ is set to 15.
Let $(Y_i,G_i,Z_i)_{i=1}^n$ be i.i.d.\ draws of $(Y,G,Z)$. To evaluate the performance of our approach, we split the sample into the training or estimation sample $(Y_i,G_i,Z_i)_{i=1}^{n_e}$ and the test sample $(Y_i,G_i,Z_i)_{i=n_e+1}^n$. For parametric predictions, we estimate the decision rule solving the weighted logistic regression over the class of linear functions $\{(g,z)\mapsto \theta_0 + \theta_1g + z^\top\gamma:\;\theta_0,\theta_1\in\ensuremath{\mathbf{R}},\;\gamma\in\ensuremath{\mathbf{R}}^{d} \}$. Note that according to our theory if $\tau=0$, then the weighted linear logistic regression provides valid binary predictions even if the choice probabilities are not logistic, cf.\ equation ((ref)). However, since in practice we typically do not know the parametric class that can capture all the relevant nonlinearities (i.e., that $\tau\ne 0$), we would often estimate the linear prediction rule
Then the estimated prediction rule is $(g,z)\mapsto \ensuremath{\mathrm{sign}}(\hat \theta_0 + \hat\theta_1g + z^\top\hat\gamma)$, where $(\hat \theta_0,\hat\theta_1,\hat\gamma)$ are estimated above, and the binary predictions evaluated on the test sample are
To obtain binary predictions with neural networks, we solve
where $\mathscr{F}_n^{\rm NN}$ is a relevant neural network class, see Figure (ref) and targ2016resnet. Then the estimated prediction rule is $z\mapsto \ensuremath{\mathrm{sign}}(\hat f(z))$, and the binary predictions evaluated on the test data are
We set the sample sizes $n$ = 1,000 and 10,000 with 30% set aside as a test sample in the simulations. To benchmark our asymmetric binary choice approach, we focus first on the unweighted approach with logistic regression, gradient-boosted trees, shallow and deep learning, and support vector machines. For each method, we compute the group-specific false positive (FP) and false negative (FN) by mistakes estimating:
for $g\in\{0,1\}$ on the test sample. We also compute the total misclassification error by estimating
on the test sample. We use the TensorFlow, scikit-learn, and XGBoost packages in Python to implement the machine learning methods and consider simple linear Logit, Logit with quadratic polynomial features and LASSO, XGBoost, support vector machines, as well as shallow and deep learning.
The tuning parameters are computed using $5$-fold cross-validation, including the LASSO penalty, the SVM penalty, and the number of trees. All other parameters are kept at their default values in the XGBoost package. We also use the radial basis function and the default value of the scaling parameter in the scikit-learn package. The neural networks have a width of 15 neurons and all other parameters are set at their default values (batch size, number of epochs, etc.); i.e. we do not use additional regularization ($\ell_1$/$\ell_2$ or dropout). For shallow learning, we use the sigmoid activation function. The deep ReLU neural network has 5 hidden layers.
The simulation results appear in Table (ref). We find that in terms of the total misclassification error, the ML methods outperform logistic regression in the nonlinear DGP case (except for shallow learning). The best performance is achieved by LASSO with nonlinearities with boosting and support vector machines coming in second. Importantly, we observe a disproportionate number of false positive and false negative mistakes across the two groups in many cases.
Next, we investigate whether group-specific misclassification rates can be equalized across the two groups with weighted Logit and Boosting. For simplicity, we focus on the setting with $\tau=0$, $\rho=0.2$, and $n=1,000$, as in this case, we observe a disproportionate number of FP and FN across the two groups. Note that in this case Logit has larger FP rates for group $G=1$ while Boosting for group $G=0$. Figure (ref) shows that the asymmetric weighted logistic regression can equalize the FP probabilities across the two groups at $\varphi_1\approx 1.6$ and FN probabilities across the two groups for $\psi_0\approx 1.6$. Note that equalizing the FP probabilities comes with the increase in FN probabilities in the group $G=1$ and equalizing the FN probabilities comes with the increase in the FP probabilities in the group $G=0$. Therefore, the decision-maker or the social planner has to think carefully about all these trade-offs when calibrating the asymmetric loss function. The results for the gradient boosting are similar, except for the fact that larger weight factors are required to equalize FP/FN probabilities across groups.
Next, we compare the performance of the standard logistic regression approach, which ignores the asymmetric loss function, with the asymmetric logistic regression in terms of the average loss of a social planner. The social planner loss for the former will be denoted by $\ell_{logit}$ while our new estimator yields $\ell_{w-logit}.$
The simulation results are reported in Table (ref). We report several measures to appraise the findings. First, we report P($\ell_{logit}$ $>$ $\ell_{w-logit}$) which is the percentage that the standard logistic regressions generate larger social planner costs compared to our weighted regression. Hence, this measure reports how often our estimator outperforms the standard procedure, where the probabilities are computed from the Monte Carlo simulated samples. Next, we report summary statistics for the ratio $\ell_{logit}/\ell_{w-logit}.$ When the ratio is above 1.00 then the weighted logistic approach is better. We report the minimum, maximum, mean, and three quartiles of the simulation distribution. All simulations involve 5000 replications.
We start from a baseline case for the parameter setting, namely: $\psi_0$ = 3, $\psi_1$ = 1, $\varphi_0$ = 1.7, and finally $\varphi_1$ = 1. For the baseline case and $n$ = 1,000, P($\ell_{logit}$ $>$ $\ell_{w-logit}$) is 0.52, meaning that in more than 50% of cases, our procedure is superior (lower social costs) to the standard logistic regression. A more detailed summary statistics is reported in the Online Appendix, Table (ref). We also examine various deviations from the baseline case. For the larger sample size $n$ = 10,000 the gains are larger, which is now 0.62, meaning that the probability that the symmetric logistic regression leads to larger losses increases as we get more data.
Next, we report two columns in Table (ref) where we change the fraction of the protected population $\rho$ from 0.2 to respectively 0.5, introducing nonlinearities with $\tau=1$, and changing the cost structure. We can see that in most of the cases, our approach is superior with a larger margin compared to the baseline case. Introducing more asymmetries in the loss function implies better performance of asymmetric logistic regression, as expected.
Lastly, we compare the performance of the weighted ERM decisions to simple plug-in rules motivated by Proposition (ref). The plug-in rules predict $\hat Y=1$ when $\hat\eta(x)\geq c(x)$, where $\hat\eta(x)$ is an estimator of $\eta(x)=\Pr(Y=1|X=x)$. Note that in the special case of the symmetric binary classification with the logistic convexification, the plug-in decisions are actually equivalent to the ERM decisions in population since
However, this equivalence does not hold in our asymmetric setting.\footnote{It is known that the plug-in rules can be inferior or superior relative to ERM in the symmetric case depending on the distribution of the data; see audibert2007fast for more detailed comparison.} In addition, some of the popular and successful ML techniques, e.g., support vector machines and neural networks with hinge convexification, are ERM-based and do not estimate $\eta$. For simplicity, we focus on the parametric case and compare the asymmetric Logit to the asymmetric plug-in rules based on the symmetric Logit. These results are presented in Table (ref). In most of the cases, the plug-in rules are inferior to our approach, and the difference is more pronounced in small samples. Lastly, we provide more detailed summary statistics for the ratio $\ell_{plugin}/\ell_{w-logit}$ in the Online Appendix, Table (ref).
The purpose of this section is to illustrate our novel econometric methods with an application to the problem of algorithmic fairness in pretrial detention. We will show how ML taking into account social planner preferences can be incorporated directly into the digital decision process.\footnote{Among economists, the idea to apply ML to pretrial detention decisions has recently been explored by kleinberg2018human. } In particular, we will illustrate that valuing outcomes of protected groups allows us to reduce various forms of algorithmic discrimination.
Across the nation, judges are increasingly using algorithms to assess the risk of whether a defendant, if released, would fail to appear in court or be rearrested for a new crime, a term called recidivism. Journalists at ProPublica have investigated a commercial algorithm, called Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), developed by Northpointe, Inc. The algorithm assigns defendants recidivism risk scores based on more than 100 factors, including age, sex, and criminal history.
COMPAS does not explicitly use race as an input, hence, there are no disparate treatment issues.\footnote{The legal scholars distinguish between the disparate treatment and the disparate impact and there is often a tension between these two. The disparate treatment pertains to using race explicitly in various decisions and is often illegal, including bail, hiring, admission, loans, etc. However, even when the decision is carefully made to avoid the disparate treatment, it may still have unintended disparate impact consequences with substantial legal implications. } Nevertheless, the aforementioned ProPublica article revealed that African American defendants are substantially more likely to be classified as high risk. Therefore, the ProPublica article pertains to the disparate impact issues, and remedying it would require using race explicitly. This bring us to the legal issues related to algorithmic affirmative action; see bent2019algorithmic for further discussion. As a simple illustration of our methodology, we will focus on equalizing the false positive rates, computed as
or positive predictive values
for African American versus other individuals, i.e. $g\in\{\mathrm{AF,O}\}$. Equalizing the false positive rates in equation (ref) achieving the error rate balance fairness while equalizing the positive predictive values in equation (ref) achieves the predictive parity fairness; see mitchell2021algorithmic for further discussion. Therefore, we will show that our methodology can mitigate the disparate impact problem. Note that it is not the purpose of our study to compare ML outcomes with human decisions (as in kleinberg2018human) or to compare ML outcomes with COMPAS.
We use data from Broward County, Florida, originally collected by ProPublica. Following their analysis, we only consider defendants who were assigned COMPAS risk scores within 30 days of their arrest and define the recidivism as a new felony or misdemeanor charge within two years after the initial screening. In addition, we restrict our analysis to only those defendants who spent at least two years (after their COMPAS evaluation) outside a correctional facility without being arrested for a violent crime or were arrested for a violent crime within these two years. Following standard practice, we use this two-year violent recidivism metric and set $Y_i=1$ for those who re-offended within this window, and $Y_i = -1$ for those who did not. In the Online Appendix Section (ref), we provide a detailed description of data cleaning and report the summary statistics for the cleaned dataset.\footnote{As we explain in Online Section (ref), while working with the COMPAS data, we discovered several issues regarding how the dataset was constructed by ProPublica journalists. Ideally, for each defendant we should only have one entry included in the data set with that record containing all the information including current charges and past crimes, to predict recidivism. However, we found multiple records for the same arrest in the COMPAS data set, with only differences in charges. As a result we undertook a thorough cleaning process described in the Online Appendix.} The total number of individuals is 8,227 with 495 variables making the problem high-dimensional, hence, calling for ML techniques as it may not be clear what types of crimes are predictive of recidivism. A large number of these variables are dummies for different kinds of crimes from the criminal code. The rest include variables like age, sex, race, crime history, marriage status, etc. We have a demographic mix dominated by African Americans and Caucasians, respectively 4,109 and 2,797 in numbers. We stratify race along two groups, African American versus all other, which given our sample is roughly African American versus Caucasian. The binary outcome $is\_recid$ indicates that 31.3% of cases resulted in a recidivism. The minimum age is 16 with a median of 29.
To understand whether the high-dimensional machine learning methods are needed for this dataset, we fit a LASSO-Logit regression and plot the area under the curve (AUC) for the 20% of the data left out as the test sample. The results appear in Figure (ref). We find that the highest test AUC, namely $0.74$, is achieved for the model with 152 covariates.\footnote{It is worth mentioning that simple linear Logit using all 495 covariates is not feasible because the matrix of covariates is close to singular. See also Online Appendix Table (ref) for the post-LASSO Logit estimates.}
Next, we fit the standard symmetric LASSO-Logit and Boosting classifiers using 494 covariates (excluding the race dummy). We tune the penalty of LASSO-Logit and the learning rate of Boosting using 5-fold cross-validation on the training sample. The test errors of both classifiers are reported in Table (ref). The standard symmetric classifiers have disproportionate biases even though the race variable has not been used to fit these models. African American defendants have almost 3 times higher false positive rates compared to the rest of the population. We also note that false negative rates are enormously large compared to the false positive rates, and calling for additional balancing. The performance of boosting is slightly inferior to LASSO-Logit, suggesting the limited role of additional nonlinearities. This is probably not suprising given that our covariates already include interactions between various crime types and degrees as well as a quadratic age term.
To balance the false positive and false negative rates rates, we fit the asymmetric classifiers with a loss function
where $f\in\{-1,1\}$ is recidivism prediction and $y\in\{-1,1\}$ recidivism outcome. The cost of false positive mistakes is normalized to $1$ and the costs of false negative mistakes denoted by $\psi$.
The performance of asymmetric classifiers for various values of $\psi\in[0,10]$ appears in Figure (ref). We find that changing the cost of false negative mistakes allows us to achieve balanced false positive and false negative rates both for LASSO-Logit and Boosting. The false positive and false negative rates are equalized approximately at $\psi=2.25$ for both classifiers.
However, changing $\psi$ is not enough to balance the false positive rates across the two groups; see Online Appendix Figure (ref) and Table (ref) for more details.
To address the algorithmic fairness issue, we rely on a two-group setup also used in the previous section where the protected group ($G$ = 1) are African American offenders. We focus on the loss function from Example (ref):
where $f\in\{-1,1\}$ is a recidivism prediction, $y\in\{-1,1\}$ is a recidivism outcome, and $g\in\{0,1\}$ is the group membership. Therefore, $\varphi_1$ is a false positive cost for African Americans which we will vary while $\psi=2.25$ is a false negative cost calibrated to balance the false positive and false negative rates.
First, we look at balancing the false positive rates across the two groups by varying $\varphi_1$. The results appear in Figure (ref). We find that the false positive rates for African Americans and Others can be equalized if the decision-maker values the outcomes of African American defendants more highly, i.e., the costs of false positive mistakes for African Americans, $\varphi_1$, are sufficiently high. The two rates are equalized when $\varphi_1\approx 1.9$ (LASSO-Logit) and $\varphi_1\approx2.25$ (Boosting). Therefore, varying $\varphi_1$ allows us to mitigate the disparate impact problem highlighted by ProPublica, achieving balanced false positive rates for the two groups. We also note that this type of fairness comes with a cost of increased false negative rates for African Americans. In other words, to reduce the number of wrongly detained African American defendants, we have to accept a larger number of wrongly released African American defendants.
Next, we look at balancing the positive predictive values (PPV) across the two groups by varying $\varphi_1$. The results appear in Figure (ref). We find that the PPV for African Americans and others can be equalized if the decision-maker imposes higher costs for false positive mistakes for African American defendants. The PPVs for two groups are equalized when $\varphi_1\approx 1.25$ for both methods.\footnote{This fairness measure roughly corresponds to what the COMPAS algorithm was originally designed to achieve; see dieterich2016compas for Northpointe's reply to the ProPublica article.} Therefore, varying $\varphi_1$ allows us to mitigate the disparate impact problem, achieving balanced PPV for the two groups. We also note that this type of fairness comes with a cost of increased negative predictive values (NPV) for Others. In other words, to balance the PPVs, we need to increase the likelihood that an African American defendant predicted to reoffend actually does so, which leads to a larger likelihood that Other defendants predicted not to reoffend actually do so.\footnote{See liang2021algorithm and liu2024inference for recent work on quantifying the fairness-accuracy frontier.}
In conclusion, algorithmic fairness can be achieved by placing a higher value on the outcomes of protected groups within our loss-based framework. This approach is transparent and can be directly incorporated into the digital decision process. However, the exact value of the loss function parameters that achieves balancing depends on the classifier and the fairness notion used; hence, it should be carefully calibrated in practice. It is impossible to achieve all fairness notions simultaneously, cf. chouldechova2017fair.
This paper provides a new perspective on data-driven binary decisions with losses/utilities/welfare driven by economic factors, and contributes more broadly to the growing literature at the intersection of econometrics and machine learning. We show that the economic costs and benefits lead to a very simple loss-based reweighting of logistic regression or state-of-the art ML techniques. Our approach constitutes a significant advantage relative to others previously considered in the literature and leads to theoretically justified binary decisions for high-dimensional datasets frequently encountered in practice. We adopt a distribution-free approach and show that the loss-based reweighted logistic regression may lead to valid binary decisions even when the choice probabilities are not logistic. We also show that the carefully crafted asymmetric deep learning architectures are optimal from the minimax point of view.
Our work opens several directions for future research. First, it calls for a more disciplined application of ML classification methods to economic decisions based on the costs/benefit considerations. For example, the false positive and false negative mistakes in pretrial detention decisions can be associated with different monetary costs for the society while the correct decisions may also have additional social benefits. In our empirical application we opted for a stylized cost differentiation scheme to make the empirical exercise transparent, but in practice one can use costs that are covariate-driven and based on a comprehensive cost-benefit analysis for the U.S.\ pretrial detention decision, such as in baughman2017costs, to build a preference-based approach for this particular application. Second, various economic applications often involve multi-class decisions and to that end some of the results of farrell2018deep could potentially be extended to the asymmetric case provided that a suitable convexification result is established for generic covariate-driven losses.
\setcounter{page}{1} \setcounter{section}{1} \setcounter{equation}{0} \setcounter{table}{0} \setcounter{figure}{0}
In this Appendix we provide the proofs of all the main theorems, lemmas, and propositions. We will use $a\lesssim b$ if $a\leq Cb$ for a constant $C<\infty$ that does not depend on a particular distribution in the class of distributions restricted by our assumptions.