Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
74,636 characters · 17 sections · 44 citation commands
Fair Prediction with Endogenous Behavior
\affil[1]{Department of Computer and Information Sciences, University of Pennsylvania} \affil[2]{Department of Economics and Department of Electrical & Systems Engineering, University of Pennsylvania} \affil[3]{Department of Economics, Rice University}
Algorithms to automate consequential decisions such as hiring hiring, lending lending, policing policing, and criminal sentencing sentencing are frequently suspected of being unfair or discriminatory. The suspicions are not hypothetical. The 2016 ProPublica study propublica of the COMPAS Recidivism Algorithm (used to inform criminal sentencing decisions by attempting to predict recidivism) found that the algorithm was significantly more likely to incorrectly label black defendants as recidivism risks compared to white defendants, despite similar overall rates of prediction accuracy between populations. Since then, discoveries of “algorithmic bias” have proliferated, including a recent study of racial bias by algorithms that prioritize patients for healthcare sendhilScience. Thus spurred, policymakers, regulators, and computer scientists have proposed that algorithms be designed to satisfy notions of fairness (see for instance mathbabe16,fairmlbook,ethicalalgorithm for overviews).
This raises a question: what measure(s) of fairness should designers be held to, and how do these constraints interact with the original objectives the algorithm was designed to target? The COMPAS case illustrates that the answer is not clear. ProPublica and Northpointe (the company that designed COMPAS) advocated for different measures of fairness. ProPublica argued that the algorithm's predictions did not maintain parity in false positive and false negative rates between white and black defendants, \footnote{Northpointe's algorithm had differing Type-1 and Type-2 error rates across the two groups.} while Northpointe countered that their algorithm satisfied predictive parity. \footnote{Roughly, the accuracy of COMPAS scores was the same for both groups at all risk levels.} Subsequent research identified hard trade-offs in the choice of fairness metrics: under some mild conditions, the two requirements above cannot simultaneously be satisfied (KMR16, Chou17). This inspired a literature proposing (or criticizing) notions of fairness based on ethical/ normative grounds. The literature evaluates algorithms on the basis of these measures, and/or proposes novel algorithms that better trade-off the goals of the original designer (decision accuracy, algorithmic efficiency) with these fairness desiderata. \footnote{See e.g. DworkFairness,HPS16,sharad1,sharad2,sharad3,impossibility,gerrymander,multicalibration,implicit for a small sample of an enormous literature.} In general, the different proposed fairness measures are fundamentally at odds with one another. For example, in addition to the impossibility results due to KMR16,Chou17, enforcing parity of false positive or false negative rates for e.g. parole decisions typically requires making parole decisions using different thresholds on the posterior probability that an individual will commit a crime for different groups. This has itself been identified by sharad1 as a potential source of “unfairness”.
This line of research is subject to two criticisms. First raised by, for example sharad1: these notions of fairness are disconnected from and lead to unpalatable tradeoffs with other economic and social quantities and consequences one might care about. Second, the literature almost exclusively assumes that the agent types, which are relevant to the decision at hand, are exogenously determined, i.e. unaffected by the decision rule that is selected. For instance, in the criminal justice application described, individual choices of whether to commit a crime or not, and therefore the overall crime rates, are fixed and not affected by policy decisions made at a societal level (e.g. what legal standards are used to convict, policing decisions etc). In settings like this where agent decisions are exogenously fixed, sharad1 and implicit observe that optimizing natural notions of welfare and accuracy (incarcerating the guilty, acquitting the innocent) are achieved by decision rules that select a uniform threshold on “risk scores” that are well calibrated --- for example, the posterior probability of criminal activity --- which tend not to satisfy statistical notions of fairness that have been proposed in the literature. Does this mean that setting uniform thresholds on equally calibrated risk scores is better aligned with natural societal objectives than is asking for parity in terms of false positive and negative rates across populations?
In this paper, we consider a setting in which agent decisions are endogenously determined and show that in this model, the answer is no: in fact, parity of false positive and negative rates (sometimes known in this literature as equalized odds HPS16) is aligned with the natural objective of minimizing crime rates. Parity of positive predictive value and posterior threshold uniformity are {\em not}. Although the model need not be tied to any particular application, we develop it using the language of criminal justice. We treat agents as rational actors whose decisions about whether or not to commit crimes are {\em endogenously} determined as a function of the incentives given by the decision procedure society uses to punish crime. The possibility for unfairness arises because agents are ex-ante heterogeneous: their demographic group is correlated with their underlying incentives--- for example each individual has a private outside option value for not committing a crime, and the distribution of outside options differs across groups. Our key result is that policies that are optimized to minimize crime rates are compatible with a popular measure of demographic fairness --- equalizing false positive and negative rates across demographics --- and are generally incompatible with equalizing positive predictive value and uniform posterior thresholds. Thus, which of these notions of fairness is compatible with natural objectives hinges crucially on whether one believes that criminal behavior is responsive to policy decisions or not.
Our results have direct implications for regulatory testing for unfairness. Often, in settings of interest, a regulator does not directly observe the decision rule used by an adjudicator. However, the regulator may wish to test whether the adjudicator is using a “fair” rule, i.e. whether the adjudicators choices are biased towards or against some demographic group. Following a tradition starting with becker, one standard used is called an outcome test, i.e. comparing, ex-post, the classification assigned by the adjudicator to observed outcomes. For instance, in a criminal justice setting, one may compare the judge's decision to the (somehow obtained) actual innocence or guilt of the defendants, or in a lending setting, compare the lender's decision on whom to extend loans to with the actual repayment outcomes of loan applicants etc.
In this context, a given prescription on what constitutes a “fair” or non-discriminatory rule maps into a corresponding outcome test. In particular, a test that is popularly used by researchers and regulators corresponds to the common-posterior-threshold rule described above. As already mentioned, this is not the best test in our model. When used, this test attempts to evaluate whether the adjudicator is using a common posterior threshold across groups by evaluating whether the marginal agents across groups have similar probabilities of different outcomes. \footnote{For a discussion of this in the context of evaluating the fairness of lending standards, see ferguson1995constitutes.} However, implementing this test is difficult: identifying (and being sure that one has correctly identified) the marginal agent in each group is hard (this is roughly the infra-marginality problem, see e.g. simoiu2017problem). For instance, there may be information observed by the decision maker but not by the regulator/ econometrician (an oft cited example is that police observe a suspect's demeanor, and use this as a factor, but this cannot be quantified). By contrast, if our maintained assumptions are valid, then an adjudicator wishing to minimize crime should use a rule that equalizes false positive and false negative rates across demographic groups. This is easy to estimate and test: there is no need to identify a marginal agent.
We first derive our results in an extremely simple baseline model to highlight the underlying intuition. We then show that our conclusions are robust to a number of elaborations and generalizations of the model.
Our baseline model (in Section (ref)) has a mass of agents who each belong to one of two demographic groups. Each agent has a single choice on the extensive margin, for instance a binary choice of whether or not to commit a crime; or whether or not to acquire human capital, etc. To fix ideas, in this paper we frame the matter as a decision about whether to commit a crime.
An adjudicator has to classify each agent as guilty or innocent. This classification is based on a noisy signal that the adjudicator receives of each agent's choice; the distribution of this signal depends only on the agent's choice, and not on her group. Further, the adjudicator observes the group membership of each agent. The adjudicator commits ex-ante to a classification rule, i.e. how it will classify agents as a function of the signal received, and potentially the agent's group membership.
Agents are expected payoff maximizers who enjoy a monetary benefit from crime but also a cost of being declared guilty of the crime. In choosing whether to commit a crime, they compare their expected net benefit from the crime to an outside option. The costs and benefits are privately known to the agent, but not to the adjudicator (who only sees group membership). The only distinction between groups is that the distributions of costs and benefits may be different for different groups. For example, individuals from different groups might have different legal employment opportunities, different costs of incarceration (e.g. differences in stigma), etc. The model is flexible enough to allow (potentially different) fractions of the population in each group who are rigidly law-abiding (i.e. do not commit a crime regardless of circumstance) or hardened criminals (i.e. will commit a crime regardless of circumstance), and a variety of responses to incentives in between these two extremes. We don't model the source of this heterogeneity: it is exogenous, and the distribution is known to the adjudicator. Given these preferences, the adjudicator's decision rule determines their choices, which in turn determines the overall crime rate in each population.
The adjudicator's objective is to minimize the overall crime rate, i.e. the total mass of agents that choose to commit a crime. While we model the adjudicator as knowing that the underlying groups are heterogeneous (i.e. knowing the above distributions that describe each group), the adjudicator is not biased for or against any group, nor is there any underlying preference for fairness. Our main result (see Section (ref)) is that the classifier that minimizes the crime rate is fair according to a metric that has attracted attention in the literature: setting different thresholds on the posterior probability of crime for each group so as to guarantee equality of false positive and negative rates. This corresponds to setting the same threshold on signals across groups.
To dig a little deeper into this result, the equilibrium crime rate in each population can be viewed as the adjudicator's prior belief in equilibrium that an agent has committed a crime, given knowledge only of her group membership. Given the noisy signal, the adjudicator has a posterior belief that the agent has committed a crime. In a static environment, the optimal classification rule for an adjudicator who wishes to optimize classification accuracy will be a group-independent threshold on her posterior belief that an agent has committed a crime. Note that priors will generally differ between populations (because outside option distributions differ, crime rates in the two populations differ). Therefore policies corresponding to group-independent thresholds on posterior beliefs will typically correspond to applying group-dependent thresholds on signals and vice versa.
Equalizing false-positive and false-negative rates across groups then corresponds to setting identical thresholds on the raw signal for each group. It can be viewed as a commitment to avoid conditioning on group membership, even when group membership information is ex-post informative for classification. The intuition is that if the adjudicator uses the same threshold on the posterior belief that an agent has committed a crime for each group, the decision rule is making use of information contained within each group's prior. Although this information is statistically informative of the decision made by the agent, it is not within the agent's control. Using this information therefore only distorts the (dis-)incentives to commit crime. On the other hand, if the adjudicator uses the same threshold on agents' signals for each group (and hence different thresholds on posterior beliefs), decisions are made only as a function of information under the control of the agents, and hence are more effective at discouraging crime. The equalizing of false-positive and false-negative rates across the groups follows from this.
The main insights of our baseline model continue to hold even when many of its core assumptions are relaxed. We summarize them here.
First, the baseline model assumes that a signal is observed by the adjudicator for every agent (or equivalently, at equal rates across populations). There is significant empirical evidence, however, that this is often not the case: for example, arrest rates (and hence prosecution rates) are substantially higher in minority populations for certain drug offenses, despite evidence that the underlying prevalence is more uniform across groups drugdisparities. Section (ref) introduces an elaboration of the baseline model based on Persico02.
In this variant, the adjudicator must rely on an intermediary (police) to inspect agents and generate a signal. The adjudicator observes a signal from an individual only if the police inspect them, and individuals are punished only if they are inspected and their signal crosses the adjudicator's threshold for establishing guilt. The police have their own objectives: to maximize the number of successful inspections (e.g. inspections that result in an arrest). Formally, we study the following game: The adjudicator commits to a decision rule on guilt/ innocence based on signals. Then police and agents play a simultaneous move game: The police choose an inspection intensity for each group to maximize their objectives, given the adjudicator's rule and crime rates in each group, subject to an overall capacity constraint. Agents of different groups commit crime based on both the adjudicator's rule, and the police's inspection rate for their group. We show that similar results continue to hold in this model, i.e. the optimal rule for the adjudicator will continue to equalize the disincentive to commit crime across the groups. This will result in equalizing conditional false-positive and false-negative rates across the groups, i.e. the rate conditional on being inspected.
In Section (ref), we consider a setting in which the signal distribution depends not just on the action chosen by the agent (crime or no crime) but may also depend on their group. For example, the underlying signal generating process may be less noisy for agents from certain groups, and noisier for others. Of course, in general, the structure of the optimal solution is closely tied to the relationship between the signal distributions, and our results cannot carry over without further assumptions.\footnote{Consider, for example, the case in which there is a perfectly informative signal for one group but not for another. Then, the optimal solution will have zero error for the former group, whereas error will be inevitable for the latter, for whom we only have a noisy signal.} Nevertheless, the insights provided by our previous analysis allow us to study the tradeoff between the adjudicator's objective (minimizing overall crime) and various fairness notions. In particular, Theorem (ref) gives conditions under which our baseline insights continue to hold, i.e. conditions under which the adjudicator's optimal rule will continue to equalize the disincentive to commit crime across groups. Conversely, Theorem (ref) shows conditions under which rules that equalize false-positive or false-negative rates across groups outperform rules that equalize incentives.
Finally, note that crime rate in our model (or what is referred to in the classification literature as “base rate”) is endogenous, where it is normally modeled as exogenously fixed. This allows us to consider an additional notion of fairness that is ill-defined in the existing literature; namely classification rules that equalize base rate across groups. In Section (ref), we study when this may be better or worse than the other notions we considered above in terms of the adjudicator's objective (overall crime rate).
The economic literature starting from arrow1972some has considered models of discrimination where agent decisions (e.g. to gain education) are endogenous and their incentives determined by a principal's choice (e.g. employer's hiring rule). CL93,FV92 study models in which individuals have a choice about how much effort to exert, and identical populations can have different outcomes in (e.g. hiring markets) because of asymmetric self-confirming equilibria. There has also been extensive interest in the design of affirmative action policies for example in higher education. loury makes the case that affirmative action may be necessary to correct historical inequity by constructing a dynamic model in which heterogeneity between two groups may persist if the principal uses a non-discriminatory rule going forward. The subsequent literature is too large to comprehensively cite, see fang2011theories for a survey.
There has also been substantial interest in evaluating outcome data for evidence of discrimination, using (or developing) an underlying theoretical prediction of how such discrimination would manifest: see e.g. knowles2001racial, Persico02 or anwar2006alternative in the context of policing/ traffic stops. A large literature has studied lending data for evidence of discrimination against women and minorities: see e.g. ferguson1995constitutes or ladd1998evidence for overviews of both the debate on what measures of (un-)fairness to use, and an overview of the existing research.
More recently, in the computer science literature, several papers consider effort-based models that are similar in spirit to CL93,FV92. HC18 propose a two-stage model of a labor market with a “temporary” (i.e. internship) and “permanent” stage, and study the equilibrium effects of imposing a fairness constraint (“statistical parity”, which corresponds to hiring from two populations at equal rates) on the temporary stage. fat20 consider a model of the labor market with higher dimensional signals, and study equilibrium effects of “subsidy” interventions which can lessen the cost of exerting effort. downstream study the effects of admissions policies on a two-stage model of education and employment, in which a downstream employer makes rational decisions, but student types are exogenously determined. Two recent papers delayed,delayed2 study non game-theoretic models by which classification interventions in an earlier stage can have effects on individual type distributions at later stages, and show that for many commonly studied fairness constraints (including several that we consider in this paper), their effects can either be positive or negative in the long term, depending on the functional form of the relationship between classification decisions and changes in the agent type distribution.
Each agent belongs to a group $g \in \mathcal{G}$. A group corresponds to some observable characteristic of the agent, for instance race or gender. There are $N_g$ agents in group $g$. For simplicity assume just two groups $\{1,2\}$ though the results extend straightforwardly to any finite number. Each agent makes a single binary decision to either commit a crime ($c$) or remain innocent ($i$).
Then, for each agent, the adjudicator observes a random signal $s \in \mathbb{R}$ which is informative of the agent's guilt/innocence. The distribution of the signal depends only on whether the agent has committed a crime (and is therefore conditionally independent of their group). Criminals' signals are drawn according to the distribution $F^{\text{c}}$ (with pdf $f^{\text{c}}$) and innocents' signals drawn from $F^{\text{i}}$ (with pdf $f^{\text{i} }$). It is without loss of generality (reordering signals if necessary) to assume that the signal distributions satisfy the Monotone Likelihood Ratio Property (MLRP), i.e. higher signals imply a higher likelihood of guilt.
The adjudicator commits to a decision rule $\beta$, which labels an agent in group $g$ with signal $s$ as guilty with probability $ \beta_g(s)\in [0,1] $. Note that implicitly this means the adjudicator perfectly observes an agent's group membership, along with the signal, at the time of adjudication. We write $q=1$ to indicate that the agent is labeled guilty and $q=0$ otherwise.
Now we describe agents' incentives to commit a crime in the first place. The agent receives a reward $ \rho $ when he commits a crime, but pays a penalty of $ \kappa $ if he is labeled as guilty. An agent who does not commit a crime receives his outside option value $ \omega $. All three quantities are privately known only to the agent and are drawn independently from a distribution that may potentially differ across the groups. An agent in group $ g $ commits a crime if his net utility from committing a crime is higher than not:
as the disincentive for committing a crime: it is the group specific additional probability of being found guilty having committed a crime relative to not. Then, the crime rate of group $g$ can be expressed in terms of $H_g$, the survivor function (i.e. 1-CDF) associated with the relevant quantity on the right hand side of (ref) given the joint distribution of $\rho, \kappa$ and $\omega$:
The adjudicator's objective is to minimize the overall crime rate, i.e. to solve
where of course, $\beta$ determines $\Delta_g$ by (ref) and therefore $CR_g$ as per (ref). Here, $B$ is the set of all feasible policies for the adjudicator, i.e. $B=\{\beta_g, g\in \mathcal{G}: \beta_g: \mathbb{R} \rightarrow [0,1]\}$.
We are interested in how decision rules $\beta$ respecting various notions of fairness perform relative to the optimal policy, and to each other. There are five main notions of fairness that we discuss throughout this paper, each of which corresponds to equalizing some statistical quantity across groups. Three of them have been considered both in the literature and in the popular press: equalizing false positive rates, false negative rates, and positive predictive value. These three notions of fairness are of particular interest to us because it has been shown that attaining all three measures simultaneously is impossible (KMR16,Chou17).
Given a policy $ \beta_g $ for group $g$, true positive rate (TPR), false positive rate (FPR), false negative rate (FNR), and positive predictive value (PPV) are defined as
Additionally, we propose two {\em new} notions of fairness: equalizing disincentives (denoted $\Delta$) and equalizing crime rates (denoted $\text{CR}$). We say the policy $\beta$ achieves fairness notion $\xi \in \{\mathrm{FPR}, \mathrm{FNR}, \mathrm{PPV}, \Delta, \text{CR}\}$ if the resulting respective quantity is the same across the groups when the adjudicator chooses policy $\beta$. We write $ B_{\xi} $ to be the set of all policies that achieve fairness notion $\xi$.
Given this framework we are interested in two questions. First, which of these fairness notions is compatible with the adjudicator's problem (ref)? Second, under what conditions is a particular fairness notion better than another in terms of the objective of minimizing overall crime rate, i.e. when do we have that for fairness notions $\xi, \xi'$: \[ \min_{\beta \in B_{\xi}} \sum_{g \in \mathcal{G}} N_g \text{CR}_g \le \min_{\beta \in B_{\xi'}} \sum_{g \in \mathcal{G}} N_g \text{CR}_g. \]
One final piece of notation will be useful. We denote by $\beta^\star$ the solution to the adjudicator's problem (ref), i.e. the policy that minimizes crime overall. We sometimes refer to this as the optimal policy. Further, for fairness notion $\xi$, we denote as $\beta^\star_\xi$ the solution to the adjudicator's problem among all rules that satisfy fairness notion $\xi$,i.e. the rule that solves
The main result we build around is that the solution to the adjudicator's problem (ref) is naturally “fair” in terms of three of the five measures above. It provides an interesting counterpoint to the impossibility results of KMR16 and Chou17. Those results state that it is impossible to simultaneously equalize false negative rates, false positive rates, and positive predictive value across groups. This raises a question of which of the fairness measures should be preferred over the others. By endogenizing the base rate of criminal activity, we find that equalizing false positive rates and equalizing false negative rates are preferred to equalizing positive predictive value in the sense that the former two are compatible with the optimal policy while the latter is not. Formally,
Rather than thinking of an arbitrary function $\beta_g(s)$, it is more natural to think of the adjudicator as selecting a threshold $T_g$ for each group $g$ so that any member of group $g$ whose signal $s$ exceeds $T_g$ is labeled guilty. That is \[ \beta_g(s)=
\]
Under a threshold policy, group $g$'s true positive rate reduces to $\mathrm{TPR}_g =F^{\text{c}}(T_g)$ and the false positive rate simplifies to $\mathrm{FPR}_g =1-F^{\text{i}}(T_g)$.
It is interesting to contrast the policy of setting equal thresholds on the signal, which we show to be the optimal policy here, as opposed to equal thresholds on the `posterior' or another calibrated risk score. The latter is advocated in sharad2, for example.
In that paper, the authors consider a setting where crime choices are exogenous/ fixed, and study the choice of policy that minimizes weighted misclassification rates (i.e. acquittal of the guilty and incarceration of the innocent). They show that an optimal policy involves a common `threshold' on the posterior across groups, i.e., first, the adjudicator estimates the prior probability that an individual has committed a crime by considering the base rate of crime for the individual's group and then uses the observed signal to update her prior probability to her posterior belief that the individual in question has committed a crime. Second, the individual is deemed guilty if the posterior probability of guilt exceeds some threshold.
Of course, in our setting, choice of crime is endogenous, and the planner's objective function is minimizing crime rate rather than minimizing mislabeling costs. Nevertheless, it is interesting to inquire into the implications of equalizing the thresholds on the posterior in our setting.
In our setting, the posterior after observing the signal $ s $ and the group $ g $ is \[ \Pr(c\mid s,g) = \frac{f^{\text{c}}(s)\text{CR}_g}{f^{\text{c}}(s) \text{CR}_g + f^{\text{i} }(s)(1-\text{CR}_g)} \] which increases in $ s $ when the signal structure satisfies monotone likelihood ratio property. Thresholding the posterior corresponds to choosing a value $ \pi_g\in [0,1] $ and classifying as guilty whenever $ \Pr(c\mid s,g)\geq \pi_g $.
Let $ T^* $ be the threshold on the signal under the planner's optimal policy. The optimal policy classifies the defendant as guilty if the signal $ s $ exceeds the threshold $ T^* $. With the monotone likelihood ratio property, this implies a threshold rule on the posterior which is to classify the defendant as guilty whenever the posterior exceeds \[ \pi_g = \Pr(c \mid T^*,g). \] By observation, the posterior thresholds are equalized across groups if and only if $\text{CR}_g = \text{CR}_{g'}$, i.e. if and only if crime rates are equalized under the optimal policy $T^*$. This in turn will only occur if $H_g( \Delta) = H_{g'} (\Delta)$ which of course will not obtain in general because $H_g$ need not be the same as $ H_{g'}$.
The baseline model assumes that when an agent commits a crime, the adjudicator observes the signal generated and adjudicates. What if the signals generated by members of each group are observed at different rates? This can happen if the adjudicator relies on an intermediary to record the signals. For example, in the crime application, the groups may be policed at different rates. Their (dis)incentives to commit crime will then differ as a result. Critically, we suppose that the police's incentives differ from the adjudicator's. The adjudicator's choice of rule therefore influences the police's choice on how to divide their manpower across different groups. Both the adjudicator's rule and the police's choice influence the incentives of agents to commit crime, and ultimately determine the overall crime levels in society.
To model this, we build upon upon the framework of Persico02. There are a continuum of police officers who choose inspection intensities for each group $\{\theta_g\}_{g \in \mathcal{G}}$ given a search capacity $S$. The choice of inspection intensity determines the rate at which signals are observed from the two groups: upon inspection, a police officer observes a signal about whether a crime was committed or not. In Persico02, the signal is assumed to be perfect. We depart from this assumption in that in our setting, the observed signal is {\em noisy}, as in our baseline model. The adjudicator, as before, wishes to minimize the overall crime rate. As in Persico02, the police have different incentives. Specifically, each police officer tries to maximize the number of `successful' inspections, i.e. where the signal recorded exceeds the threshold set by the adjudicator. As in Persico02, we motivate this incentive as driven by the career concerns of individual police officers (who are e.g. promoted if they have many successful arrests etc.).
The timing of the game is therefore: (1) The adjudicator chooses the function $\beta$, (2) the police takes this as fixed and chooses inspection intensities $ \theta_g\in [0,1] $ for each group subject to the constraint $ N_1 \theta_1 + N_2 \theta_2 \leq S $ (recall that $N_g$ is the number of agents in group $g$),\footnote{To make this model non trivial, we assume that search capacity is limited, i.e., $S < N_1 + N_2 $.} and then (3) given the adjudicator's choice $\beta$, and inspection probability $ \theta $, an agent of group $g$ with crime reward $\rho$, cost of being found guilty $\kappa$ and outside option commits a crime if (analogous to ((ref)), but now taking into account also the probability of inspection):
where $\Delta_g$ is as defined in (ref). By analogy with (ref) an agent of group $g$ commits crime with probability $H_g (\theta_g \Delta_g)$.
As a benchmark, consider a setting where the adjudicator can choose both $\beta$ and $\theta$. Here the objective function is to minimize the overall crime rate just as in (ref), and the additional constraint simply reflects that the choice of inspection rule must be feasible, i.e. the total level of inspection cannot exceed total search capacity $S.$ That is to say the adjudicator's problem in this benchmark can be written as:
We refer to the solution of ((ref)) as the {\bf first-best} solution.
Now return to our setting above where the adjudicator chooses $\beta$ but not $\theta$. The police take $\beta$ as given and choose $\theta$ to maximize the number of successful inspections. Note that an inspection in group $g$ successful with probability $H_g(\theta_g \Delta_g).$ As in Persico02, we assume an interior equilibrium solution, i.e., $ \theta_g>0\ \forall g$. \footnote{A corner solution entails the police completely ignoring a group. In this case, the setting is trivial because conditional false positive rates are undefined for the ignored group.} Intuitively, in this case, the optimal strategy for the police will equalize the crime rate between the two groups. Otherwise, police that are trying to maximize their successful inspection probability will exclusively search the group with the highest crime rate. Recognizing this, the adjudicator will solve the following problem:
The solution to problem (ref) is the {\bf second-best} solution.
We note that because groups will be inspected at different rates, the TPR and FPR as we have defined them should correctly be called the conditional true and false positive rates respectively, i.e., the rates conditional on being inspected.\footnote{We observe that these conditional rates are implicitly what has been studied in the fairness in machine learning literature, because these are the rates that can be computed from the data.} Theorem (ref) now carries over mutatis mutandis, i.e. in both the first and second best solution, the thresholds will be set so as to equalize the conditional false and true positive rates $CFPR$ and $CTPR$. Proofs for this and subsequent theorems are in the appendix.
While the optimal $\beta$ under the first and second best outcomes coincide, the optimal inspection intensities need not. In particular, the first- and second-best solutions coincide when $ H $ is convex. However, when $H$ is concave, then the inspection intensities under the second-best outcome maximize the average number of crimes out of all possible search intensities given the optimal signal thresholds.
We now examine the extent to which the conclusion of Theorem (ref) holds if we allow the signal structure $F_g = (F^{\text{c}}_g,F^{\text{i}}_g)$ to be different across groups $g \in \mathcal{G}$. The signal structure $ F_g $ and the strategy $ \beta_g(s) $ matter to the extent they discourage crime. Recall, that the relevant sufficient statistic of a strategy $\beta_g(s)$ is what we called the disincentive to commit crime: \[\Delta_g = \mathrm{TPR}_g - \mathrm{FPR}_g = \int_{\mathbb{R}} (f^{\text{c}}(s)-f^{\text{i} }(s))\beta_g(s)ds.\]
The set of achievable disincentives given a signal structure is $ [\underline \Delta_g, \overline \Delta_g] $ where
The relevant sufficient statistics for the signal structure $ (F^{\text{c}}_g,F^{\text{i}}_g) $ for group $g$ are its minimal and maximal disincentive $ \underline \Delta_g $ and $ \overline \Delta_g $ which determines the range of disincentives a classification rule is able to provide.
In this section, we give some insight into what happens when the signal structure varies across groups without making further assumptions on how the signal structure varies. First, as should be clear from the intuition previously, what really matters for our results in the baseline model is not that the signal distributions are identical across populations, but rather that the maximal disincentive $\overline \Delta_g$ is the same across groups: if we have this, then the basic insight of Theorem (ref) holds as before (maximizing disincentives across groups). This is summarized in Theorem (ref). Note that since the signal structures are different, the implication of Theorem (ref), i.e. that FPR/ FNR will also be equalized across groups, will not hold in general. Further, if the maximal disincentive differs, then in general the result does not hold, as we show in Example (ref). Theorem (ref) then provides conditions under which various “natural” fair policies are ranked under the adjudicator's objective to minimize overall crime.
Let us start with analyzing the optimal policy that minimizes average crime. As in Section (ref), average crime is minimized by maximizing the disincentive for crime in each group, which is attained by setting $ \beta_g(s) $ such that $ \Delta_g = \overline \Delta_g $ for every $g$. Unlike in Section (ref), the optimal policy does not guarantee any of the fairness notions described in section (ref) --- equalizing disincentives, equalizing false positive rates, equalizing false negative rates or equalizing positive predictive value --- when the signal structures differ across the groups.
An immediate observation is that when the signal structures have the same maximal disincentives $ \overline \Delta_g $, then the optimal effective policy equalizes disincentives.
Theorem (ref) follows from the core insight of Theorem (ref) that the optimal rule maximizes the disincentive to commit crime for each group. When the signal structures across the groups are identical as in Theorem (ref), the signal structures have the same maximal disincentives, and therefore, the optimal policy equalizes disincentives. Equalizing disincentives coincides with equalizing false positive rates and equalizing false negative rates in this case. However when the distributions of signals are different, equalizing disincentives need not be be the same as equalizing false positive/ false negative rates. Indeed, equalizing disincentives may yield a strictly lower crime rate than equalizing false positive rates and equalizing false negative rates even when $ \overline \Delta_1=\overline \Delta_2 $. To see this, consider the following example.
When the signal structures across the groups have different maximal disincentives, we identify conditions under which both equalizing false positive rates and equalizing false negative rates yields a strictly lower crime rate than equalizing disincentives. Without loss of generality, let us assume that $ \overline\Delta_2 > \overline \Delta_1 $.
In general, the optimal policy does not guarantee any of the fairness notions. Theorem (ref) provides a sufficient and necessary condition under which equalizing false positive rates and equalizing false negative rates attains lower crime rates than equalizing disincentives. However, condition 3 is hard to interpret. Further, it is unclear which of equalizing false positive rate and false negative rate would be better overall for the adjudicator. Without additional structure on the signal structures, it is hard to proceed further. To explore these issues, we restrict attention to signal distributions that are members of location-scale families of distributions.
Note that the underlying distribution $ f $ is identical across the groups as in contrast to Example (ref) where the underlying distribution $ f_g $ differed across the groups. Combined with the functional form (ref), log-concavity of $ f $ is equivalent to the signal structure satisfying the monotone likelihood ratio property for each group $g$ which implies that the optimal $ \beta $ is a threshold strategy. The log-concavity of $ f $ also implies that $ f $ is unimodal, which guarantees the uniqueness of the threshold that attains the optimal policy. There are many natural location-scale families of distributions satisfying log-concavity including normal distributions, logistic distributions, and extreme value distributions.
A property that makes location-scale family particularly tractable is that the disincentives engendered by a threshold depend only on $ \frac{m_g}{\sigma_g} $, i.e. is the ratio between the scale shift $\sigma_g$ and the marginal effect of crime $ m_g $. For this class of distributions, we can say that it is always preferable to equalize either false positive rates or false negative rates compared to equalizing disincentives.
The formal proof is in the appendix. For some intuition, note that the maximum disincentive $ \overline \Delta_g $ is determined by and increasing in $ \frac{m_g}{\sigma_g} $. Intuitively, this is because the larger the normalized marginal effect of crime on the signal is, the better the adjudicator is able to distinguish between the criminal and the non-criminal based on the signal, and therefore the adjudicator can increase the disincentive to commit crime. If this term is equal across groups, then Theorem (ref) applies and part (1) follows as a corollary. Now, without loss of generality, suppose instead that $ \frac {m_1} {\sigma_1} < \frac {m_2} {\sigma_2} $. Then $ \overline \Delta_2 > \overline \Delta_1 $. It can also be verified that the condition $ (3) $ in Theorem (ref) holds, so that equalizing false positive rates and equalizing false negative rates always yield a lower crime rate than equalizing disincentives. Furthermore, we can also verify that equalizing false positive rates and equalizing false negative rates never attains the crime rate under the optimal policy.
A natural question to ask is whether one of equalizing false positive rates or equalizing false negative rates will have a lower crime rate than the other. It is a hard question to answer in general. When the underlying distribution $ f $ is symmetric around $0$ ($0$ is without loss of generality since the $\mu_g$'s can always be shifted if $f$ is symmetric around some other number), however, the set of feasible disincentives $ (\Delta_1, \Delta_2) $ are identical under equalizing false positive rates and false negative rates, and therefore, the two notions of fairness yield the same crime rate.
In our model, the crime rates are endogenously determined by agents' decisions as a response to the policy implemented. This is unlike most of the fairness literature that assumes that the underlying rates are fixed. It motivates us to study another fairness measure that previous papers could not have asked for: equalizing crime rates.
To understand the implications of equalizing crime rates, let us assume that group $ 2 $ is `riskier' than group $ 1 $ without loss of generality. Specifically, assume that $H_2$ stochastically dominates $H_1$ -- that is to assume $H_2(\Delta) \ge H_1(\Delta)\ \forall \Delta$.
In this section, we focus on the comparison between equalizing crime rates and equalizing disincentives, while allowing any arbitrary signals structures $F_g = (F^{\text{c}}_g, F^{\text{i}}_g)$. Equalizing disincentives is an appropriate fairness measure to compare because it is attained by the optimal policy when $ \overline \Delta_1 = \overline \Delta_2 $.
The first question to ask is whether equalizing crime rates can ever attain a lower crime rate than equalizing disincentives. We find that equalizing crime rates attains a lower crime rate than equalizing disincentives if and only if $ \overline \Delta_2 $ is sufficiently larger than $ \overline \Delta_1 $.
The theorem is best demonstrated using the diagrams, although the formal proof is provided in the appendix. For theorem (ref), we consider 4 cases: (i) $\overline{\Delta}_1 + \epsilon \le \overline{\Delta}_2$, (ii) $\overline{\Delta}_1 < \overline{\Delta}_2 < \overline{\Delta}_1 + \epsilon$, (iii) $\overline{\Delta}_1 =\overline{\Delta}_2$, and (iv) $\overline{\Delta}_1 > \overline{\Delta}_2$. For initial illustration purposes, we will focus on figure (ref) which corresponds to the first case, $\overline{\Delta}_1 + \epsilon < \overline{\Delta}_2$. Each red and blue curve represents $ H_1(\cdot) $ and $ H_2(\cdot) $, respectively. Note the `thicker' segments of each curve are the set of crime rates $H_g(\Delta_g)$ that can be achieved by varying the disincentive, $\Delta_g \in [\underline \Delta_g, \overline \Delta_g] $. For each group, we denote the optimal policy by a triangle. The optimal policy while equalizing disincentives, which is denoted by `X', is obtained as intersections of the black line and the outside option distribution functions. The optimal policy while equalizing crime rates, which is denoted by `o', is obtained as intersections of the orange line and the outside option distribution function. In figure (ref), note that for group $1$, the crime rate stay the same under equalizing disincentives and crime rates, but the crime rate increases once one changes from the optimal policy that equalizes disincentives to the one that equalizes crime rates. Therefore, the optimal policy subject to equalizing crime rates is more preferred than under equalizing disincentives.
Figure (ref) shows the other cases. Note that as we decrease $\overline{\Delta}_2$ (or equivalently increase the crime rate of group $2$), there comes a point determined by $N_1$ and $N_2$ at which equalizing crime rates is more preferred than equalizing disincentives. More specifically, in figure (ref), imagine moving the right most blue triangle to the left and hence raising the orange line; as this happens, both `X' marks, which denote the optimal policy that equalizes crime rate, need to go up, while the optimal policy that equalizes incentive stays the same. Therefore, depending on the ratio of the number of people ($N_1$ and $N_2$), there exists some $\epsilon$ such that $\overline{\Delta}_2 \le \overline{\Delta}_1 + \epsilon$ if and only if equalizing crime rate attains lower crime rate than equalizing crime rates. And it's easy to see from figure (ref) and (ref) that equalizing disincentives achieves lower crime rate than equalizing crime rates in the corresponding cases. Therefore, these arguments together imply that equalizing crime rates attains a lower crime rate than equalizing disincentives if and only if $ \overline \Delta_2 $ is sufficiently higher than $ \overline \Delta_1 $.
Having verified that equalizing crime rates can attain lower crime rates than equalizing disincentives, the next question is whether equalizing crime rates can ever attain a lower crime rate than any of other fairness notions. The answer to this question is positive which we establish by finding a condition under which the optimal policy attains equalizing crime rates.
Figure (ref) depicts the case when $ H_1(\overline \Delta_1) = H_2(\overline \Delta_2) $. By construction, the optimal policy equalizes crime rates. As it can be seen from Figure (ref), equalizing disincentives attains a strictly higher crime rates than equalizing crime rates. Furthermore, it can be shown that other notions of fairness --- equalizing false positive rates, false negative rates and positive predictive value - are not satisfied in general.
This paper gives a general model in which classification rules which equalize false positive and false negative rates can be compatible with natural objectives, in spite of failing to capitalize on statistically relevant information. We derived the model using the language of criminal justice, but one could just as easily apply the base model to settings in which the principle was making some other binary decision based on partial information, such as a lending or employment decision. The underlying reason is that conditioning on demographic information, while statistically useful, leads to decision rules that incentivize different groups differently --- because demographic information is not under individual control. Hence, in settings in which the underlying objective depends on the decisions of rational agents, the decision rule should explicitly commit not to condition on information that relates to an individual's demographic group, and instead use only information that is affected by the choices of the individual. Abstracting away, the necessary conditions under which our conclusions hold are that:
Here, conditions (2) and (3) are unlikely to hold precisely in most situations, but we give settings under which they can be relaxed.
More generally, if we are in a setting in which we believe that individual decisions are rationally made in response to the deployed classifier, and yet the deployed classifier does not equalize false positive and negative rates, then this is an indication that either the deployed classifier is sub-optimal (for the purpose of minimizing base rates), or that one of conditions (2) and (3) fails to hold. Since in fairness relevant settings, the failure of conditions (2) and (3) is itself undesirable, this can be a diagnostic to highlight discriminatory conditions earlier in the pipeline than the adjudicator's decision rule. In particular, if conditions (2) or (3) fail to hold, then imposing technical fairness constraints on a deployed classifier may be premature, and instead attention should be focused on structural differences in the observations that are being fed into the deployed classifier.