Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
121,043 characters · 26 sections · 70 citation commands
A Discrimination Report Card
\epigraph{Sunlight is said to be the best of disinfectants; electric light the most efficient policeman.} {Louis Brandeis }
Though half a century has passed since the enactment of the Civil Right Act, there is ample evidence that American employers still discriminate based upon legally protected characteristics including sex and race charles_guryan_jel,bertrand_duflo_review, quillian2017meta, quillian2020evidence. One reason for the stubborn persistence of employer discrimination is the paucity of reliable information regarding the tendencies of specific organizations to engage in such illicit behavior. Job seekers cannot direct their search effort away from biased firms if the identities of these firms are unknown. Corporate executives may have little incentive to search for more equitable recruiting practices if they cannot benchmark their organizations' biases to those of their peers.
This paper constructs a discrimination report card that summarizes objective information regarding the relative biases of a broad collection of Fortune 500 companies. Our analysis leverages a massive correspondence experiment sending up to 1,000 job applications to each of 108 firms. In a previous analysis of this experiment kline2021systemic, we established that applications listing distinctively Black names received seven percent fewer employer contacts than those listing distinctively white names. This contact penalty displays striking heterogeneity across companies: applying Empirical Bayes (EB) deconvolution methods, we found that the top 20% of discriminating firms were responsible for nearly half of the employer contacts lost to racial discrimination.
While this earlier work established that discrimination is highly concentrated among a small subset of employers, the experimental estimate for any particular firm is subject to significant sampling uncertainty. This statistical noise presents an obstacle to reliable estimation of discrimination by specific firms. Though scientific communication is generally aided by transparency andrews2021model, lay audiences can find it challenging to translate point estimates and standard errors into conclusions of interest. Experimental evidence from psychology suggests audiences frequently depart from Bayesian reasoning, which may generate over- or under-reactions to statistical evidence mullainathan2002thinking,mullainathan2008coarse,bordalo2016stereotypes. These considerations suggest that a low-dimensional, easily-digestible summary of the experimental findings is important for effectively communicating the results to a broad audience. Likely for similar reasons, scholars, policymakers, and private businesses increasingly report simple “report cards" summarizing estimates of quality for various types of institutions, including colleges chetty2017college, K-12 schools bergman2020housing,angrist2021gs, teachers bergmanhill2018teachers,pope2019teachers, healthcare providers randhealthcarereportcards2002,pope2009hospitals,kolstad2013, and neighborhoods chettyhendren2018part2,chetty2018atlas.
This paper develops new methods for grading firms (or other units) based upon noisy measures of their conduct while maintaining statistical guarantees on the reliability of the resulting grades. The information content of our classification scheme is quantified by Kendall's $\tau$ measure of correlation kendall1938new between our proposed ranking and the true ranking of firms' latent discriminatory behavior. The reliability of the report card grades is quantified by an analogue of the False Discovery Rate benjamini1995controlling,storey2002direct that we term the Discordance Rate (DR). The DR between two grades gives the expected probability that a firm assigned the worse grade discriminates less than a firm assigned the better grade. We also develop an extension of DR that weights mistakes by their cardinal magnitudes, which captures the idea that large mistakes are more costly than small ones.
We show that the tradeoff between these notions of information and reliability arises naturally from a series of lotteries over firm pairs where an analyst guesses which member of each pair exhibits worse conduct. When facing multiple gambles of this form, the analyst faces an optimization problem subject to logical transitivity constraints requiring all pairwise comparisons to be consistent with a coherent underlying ranking. A parameter $\lambda$ trades off the gains of correctly ranking pairs of firms against the costs of misclassifying them. When $\lambda=1$, it is optimal to assign every firm a unique grade that maximizes the expected rank correlation with the true latent discrimination levels. These maximally-informative grades turn out to be closely connected to classic proposals for preference aggregation via pairwise elections found in the social choice literature borda1784memoire,condorcet1785essay,young1978consistent,young1986optimal, with the posterior probability that one firm is more biased than another serving as a “vote share.” When $\lambda<1$ it is only optimal to strictly rank firm pairs that can be distinguished with sufficiently high posterior probability, potentially yielding ties and therefore a low number of distinct “grades." These coarse grades protect against misinterpretation at the cost of losing information, thereby reducing correlation with the true ranks.
The grades generated by our procedure yield a simple classification of firms into groups that facilitates pairwise comparisons of firm conduct. The grades allow an assessment not only of which firms are discriminating most, but also which firms are discriminating least. The latter sort of information may prove especially useful in the hunt for best practices that lead to inclusive hiring. For example, our past work documented smaller contact gaps at firms that contacted applicants from fewer phone numbers, a proxy for centralized human resources (HR) practices. berson2020outsourcing provide corroborating evidence that contact gaps are lower among French firms with centralized recruiting practices, while challeeffect report encouraging experimental evidence on the potential for HR reforms to foster inclusive hiring at large firms. Our report card results suggest directions for work comparing these and other policies across firms graded as high and low performers.
To supplement the ordinal information conveyed by our grades, we report Empirical Bayes posterior means and credible intervals summarizing the level of discriminatory conduct at each employer. This ordinal and cardinal information is visualized in a single report card that provides a powerful and easily-accessible rubric for assessing absolute and relative discriminatory conduct. Although this rubric was designed to communicate information about discrimination, we expect the methods proposed here to prove useful in other settings where noisy measures of institutional performance and conduct have been studied, including assessments of school value added angristqje2017, hospital quality chandra2016health, health insurance plans abaluckhealthinsurance2021, and regional intergenerational mobility chetty2018impacts2. Routines for implementing our ranking procedure are available online at \url{https://github.com/ekrose/drrank}.
As an introductory illustration of the method, we rank the contact rates of the first names used in our correspondence experiment. A non-parametric deconvolution suggests that name-specific contact rates cluster around two distinct values capturing mean contact rates for distinctively white and Black names. Weighing the loss from incorrectly ordering a pair of names four times as heavily as the gain from correctly ordering them, our ranking procedure stratifies the names into two groups with distinct grades. These grades are shown to be strongly predictive of a name's nominal race but not its sex. Allowing additional grades has little impact on these correlations, suggesting that our ranking procedure is suitable for recovering “missing labels” with a low-dimensional structure.
Proceeding to our primary application of ranking firm biases against Black applicants, we compute optimal grades subject to the same preferences over correct and incorrect rankings used for first name pairs. In a single pairwise gamble, these preferences (which correspond to a particular choice of $\lambda$) require at least 80% posterior confidence to justify a strict ordering of firms. In our baseline specification, applying this choice of $\lambda$ to generate a transitive ordering over all firms yields three unique grade levels. These grades capture roughly 19% of the between-firm variation in proportional contact penalties and yield an expected rank correlation with the true penalties of 0.21. Although our grading system reflects only ordinal considerations, we estimate that the average white/Black gap in contact rates among firms awarded the worst grade is 24%, while the gap among firms awarded the best grade is only 3%.
Our earlier work found that industry explains roughly half of the variation in racial discrimination levels across firms kline2021systemic. Motivated by this finding, we extend our procedure to build industry information into the report card grades. This extension is achieved by augmenting efron2016empirical's log-spline deconvolution approach to flexibly estimate separate distributions of discrimination within and between industries. Consistent with our past work, we find that industry affiliation accounts for roughly half of the cross-firm variation in proportional contact penalties. Incorporating industry affiliation into the ranking procedure with the same choice of $\lambda$ yields four grades. These improved grades explain 47% of the variation in contact penalties across firms and yield a correlation with the latent ranks of 0.32, while limiting the expected share of firm pairs that are mis-ranked to 5.2%.
These more informative industry-dependent grades constitute our preferred ranking of the companies in our experiment. Firms assigned the worst grade in this ranking contact white applicants 22% more often than Black applicants, similar to the lowest category in the ranking without industry effects. However, 5 firms receive this label in the model with industry effects compared to only 2 in the baseline model, an indication of the extra information conveyed by industry. Similar to the specification without industry, firms receiving the best grade in the industry effects model (11 firms) exhibit very small racial biases. To the extent that these differences are driven by HR practices or other firm policies, there may be opportunities for the substantial set of firms that scored poorly to improve their behavior by imitating the practices of those who scored more highly.
Our work extends a burgeoning literature on EB ranking methods. A large empirical literature ranks teachers, schools, hospitals, and neighborhoods using James-Stein style shrinkage rules chetty2014measuring,chetty2018opportunity. portnoy1982maximizing established conditions under which ranking based on such rules maximizes the probability of a correct ordering, while laird1989empirical proposed directly computing posterior mean ranks under a normality assumption on the latent heterogeneity. Both sorts of ranks may be noisy, however, leading to a proliferation of ranking mistakes when the number of units grows large. A recent econometrics literature confirms that this problem can become severe in practice and proposes approaches to testing hypotheses regarding either ranks themselves or the levels of highly-ranked units andrews2019winners,mogstad2020rankings.
Building on the analogy with multiple testing, gu2020invidious consider the use of non-parametric EB methods to select tail performers subject to constraints on the False Discovery Rate, which limits the number of ordering mistakes expected when selecting top performers. Our proposal generalizes the approach in gu2020invidious by accommodating more than two grades and avoids the requirement to treat one of the grades as a null hypothesis. More recent work by gu2022ranking considers a ranking of journals based on pairwise citation counts using a penalized Bradley-Terry model bradley1952rank. While our proposed approach shares gu2022ranking's focus on pairwise differences, the method does not require pairwise data on tournaments and allows users to trade off transparent notions of the information content and reliability of the resulting grades.
After undergoing peer review, we plan to release the names of the firms in this study to the public. Regulatory agencies such as the Equal Employment Opportunity Commission (EEOC) and the Office of Federal Contract Compliance (OFCCP) have broad discretion to launch investigations into possible violations of equal employment opportunity laws, especially violations by federal contractors. Many of the firms receiving poor grades turn out to be federal contractors, suggesting this information may be of help in targeting future compliance efforts. However, compliance efforts are inevitably long and costly and many firms remain out of compliance even after having been fined maxwell2013using.
As the introductory quote by Brandeis suggests, shining some statistical light on the problem of discrimination may have a more immediately salutary effect than regulatory enforcement efforts. Little scientific information about the discriminatory conduct of particular firms is available to the public. The most powerful “disinfectant” may well be the decentralized reactions of employees, customers, and leaders of these organizations to the provision of such information.
We construct discrimination report cards based on the resume correspondence experiment analyzed in kline2021systemic. The experiment's sampling frame began with the 2018 list of companies in the Fortune 500. Attention was then restricted to firms with sufficient geographic variation and entry-level job posting for our experiment to be feasible. Over the course of the study, 125 entry-level job vacancies were sampled from each of these employers, with each vacancy corresponding to an establishment in a different U.S. county. This restriction was intended to ensure nation-wide coverage of each firm's recruitment conduct and to minimize the chances that multiple sampled job vacancies were being managed by the same individual.
Sampling was organized in a series of 5 waves, with a target of 25 jobs sampled for each firm in each wave. The majority of firms (72) were sampled in all waves; the rest were excluded in some waves due to COVID-19 and technological interruptions. We attempted to send each sampled job four pairs of applications, with each pair consisting of one Black applicant and one white applicant. Some vacancies received fewer than 8 total applications because the job opening closed while applications were still in progress. Our final sample included roughly 84,000 applications.
To signal race and gender, we followed previous correspondence experiments and used distinctive names. Our set of names started with that of bertrand_mullainathan_2004, who used 9 unique names for each race and gender group. This list was supplemented with 10 additional names per group from a database of speeding tickets issued in North Carolina between 2006 and 2018. We classified a name as racially distinctive if more than 90% of individuals with that name are of a particular race, and selected the most common distinctive Black and white names for those born between 1974 and 1979. Distinctive last names were taken from the 2010 U.S. Census. We selected names with high race-specific shares among those that occur at least 10,000 times nationally.
One application within each pair was randomly assigned a distinctively white name while the other was randomly assigned a distinctively Black name. Fifty-percent of names were distinctively female and the rest distinctively male, but assignment of sex was not stratified. Each fictitious applicant was independently randomly assigned a large set of additional characteristics, including educational and previous employment histories.
Our primary outcome is whether an employer attempted to contact the fictitious applicant. Phone numbers and e-mail addresses assigned to the fictitious applicants were monitored to determine when employers reached out for an interview. Contact information was assigned to ensure that no two applicants to the same firm shared an e-mail address or phone number. Our analysis focuses on whether the employer attempted to contact an applicant by any method within 30 days of applying. Further details on the experimental design are available in kline2021systemic.
Consider the problem of ranking a collection of $n$ firms, indexed by $i\in\{1,\dots,n\}\equiv[n]$, according to their values of a scalar measure of discrimination $\theta_{i}\in \mathbb{R}$. The decision variable $d_i\in[n]$ gives the grade assigned to firm $i$. Larger values of $d_i$ indicate a firm is more biased. Hence, when $d_i>d_j$ for two firms $i$ and $j$, we say that firm $i$ received a “worse” grade than firm $j$.
For each firm $i\in[n]$, we have the measurements $Y_{i}=(\hat \theta_i, s_i)$, where $\hat \theta_i$ is a consistent estimate of $\theta_i$ and $s_i$ is that estimate's asymptotic standard error. We assume the $\{\hat \theta_i\}_{i=1}^n$ are mutually independent and that $\hat \theta_i \sim \mathcal{N}(\theta_i, s_i^2)$. The normality assumption can be justified by an asymptotic approximation as each $\hat{\theta}_{i}$ is based on the contact rate of a large number of underlying applications.
It is convenient to recast the problem of ranking $n$ firms as that of ranking all $\binom{n}{2}$ pairs of firms subject to a set of transitivity constraints. Correctly ranking the bias of a pair of firms yields a concordance while ranking the pair incorrectly yields a discordance. A pair can also be deemed a tie, which yields neither a discordance nor a concordance.
To build intuition it is helpful to first consider the problem of deciding on the rank of a single pair of firms $i$ and $j$ based upon realizations $(y_i,y_j)$ of independent signals $(Y_i,Y_j)$. Suppose that correctly ranking the pair yields payoff $\lambda\in[0,1]$ while reversing their true rank yields payoff -1. We can also declare the comparison a draw by assigning the firms equal ranks, which amounts to abstaining from the gamble and yields certain payoff 0.
In considering this lottery, we view the pair $(\theta_i,\theta_j)$ of firm types as $i.i.d.$ draws from a continuous prior distribution $G:\mathbb{R}\rightarrow[0,1]$. The posterior probability that firm $i$ is more biased than firm $j$ will be denoted $\pi_{ij}=\Pr\left(\theta_{i}>\theta_{j}|Y_i=y_{i},Y_j=y_{j}\right)$. Because $G$ is continuous, ties are measure zero and $\pi_{ij}=1-\pi_{ji}$.
The expected utility of assigning grades $d=(d_1,d_2)\in\{1,2\}^2$ to these firms can therefore be written
The optimal policy is a simple posterior threshold rule:
When $\lambda=1$, it is optimal to follow a maximum a posteriori (MAP) rule, assigning the higher rank to whichever firm has a greater probability of having the largest value of $\theta$. But when $\lambda<1$, it is better to assign pairs of firms with $\pi_{ij}$ near 1/2 equal grades rather than risk ranking them incorrectly. The quantity $(1-\lambda)$ can therefore be thought of as measuring discordance aversion.
Now consider the case where we can gamble on the relative rank of all $\binom{n}{2}$ pairs of firms. kendall1938new's classic $\tau$ measure of rank correlation can be defined as the share of pairs yielding a concordance minus the share yielding a discordance. The loss function we propose is a generalization of $\tau$ indexed by a scalar $\lambda\in[0,1]$ that controls the benefit of a concordance relative to the cost of a discordance.
Letting $\theta=(\theta_1,\dots,\theta_n)'$ denote the vector of latent biases and $d=(d_1,\dots,d_n)'$ a vector of assigned grades, our loss function can be written:
While every discordant pair yields a loss of 1, every concordant pair reduces loss by $\lambda$. When $\lambda=1$ the loss function equals minus one times Kendall's $\tau$ measure of rank correlation between $d$ and $\theta$, which we will denote $\tau(d,\theta)$. When $\lambda<1$, ranking mistakes are more costly than forgone concordances, which creates an incentive to deem the pair a tie.
Building on the insight that $\tau(d,\theta)=-L\left(d,\theta;1\right)$, we can also write the loss function:
where the quantity $DP(d,\theta)=\binom{n}{2}^{-1}\sum_{i=2}^{n}\sum_{j=1}^{i}[1\left\{ \theta_{i}>\theta_{j},d_{i}<d_{j}\right\} +1\left\{ \theta_{i}<\theta_{j},d_{i}>d_{j}\right\}]$ is the Discordance Proportion. The Discordance Proportion gives the share of firm pairs that are misranked according to their grades. Interpreting our decision problem as a series of tests of the null hypothesis that each member of a pair discriminates equally, the Discordance Proportion may be seen as a directional (sometimes called type III) error rate. That is, we reject equality in favor of an erroneous alternative, yielding a discordance. This representation clarifies that the parameter $\lambda$ trades off the desire to accurately communicate information to the audience by maximizing $\tau(d,\theta)$ against concern about misclassifying the ranks of firms, as reflected by $DP(d,\theta)$.
While one would ideally like to choose grades $d$ that balance the rank correlation $\tau(d,\theta)$ against the Discordance Proportion $DP(d,\theta)$, these quantities are not directly observed. Suppose, however, that we have a continuous $i.i.d.$ prior $G:\mathbb{R}\rightarrow[0,1]$ over the elements of $\theta$ and that we observe a realization $y$ of the random vector $Y=(Y_1,\dots,Y_n)'$. The posterior probability that firm $i\in[n]$ is more biased than firm $j \neq i$ will again be denoted by $\pi_{ij}=\Pr\left(\theta_{i}>\theta_{j}|Y=y\right)$.
The posterior expectation under $G$ of both the unknown $\tau(d,\theta)$ and $DP(d,\theta)$ can be expressed in terms of the pairwise probabilities $\pi_{ij}$. The posterior expected rank correlation $\bar \tau(d)=\mathbb{E}[\tau(d,\theta)|Y=y]$ is given by \[ \bar \tau(d) = \binom{n}{2}^{-1}\sum_{i=2}^{n}\sum_{j=1}^{i}1\left\{ d_{i}<d_{j}\right\}\cdot \left(\pi_{ij}-\pi_{ji}\right) +1\left\{ d_{i}>d_{j}\right\} \cdot \left(\pi_{ji}-\pi_{ij}\right). \] Likewise, the posterior expected value of $DP(d,\theta)$, a quantity we term the Discordance Rate (DR), is
Consequently, the posterior expected loss (i.e., the Bayes risk) of assigning grades $d\in[n]^n$ can be written:
The optimal grades $d^*(\lambda)$ minimize $\mathcal{R}(d;\lambda)$. To simplify this minimization problem, it is convenient to recast the relevant decision variables as pairwise indicators $d_{ij}=1\left\{ d_{i}>d_{j}\right\} $ and $e_{ij}=1\left\{ d_{i}=d_{j}\right\}$. Transitivity requires that for any triple $\left(i,j,k\right)\in[n]^{3}$ the following constraints hold:
\[ d_{ik} +\left(1-d_{jk}\right)\leq1+ d_{ij}, \] \[ e_{ij} + e_{jk} \leq 1 + e_{ik}. \]
Hence, we can rewrite the problem of choosing $d\in[n]^n$ to minimize (ref) as that of choosing the binary indicators $\{d_{ij},e_{ij}\}_{i=2,j=1}^{i=n,j=i}$ to minimize
subject to the transitivity constraints in (ref) and the logical constraint that $e_{ij}+d_{ij}+d_{ji} = 1$ for all $(i,j)\in[n]^2$. Note that both the objective (ref) and the constraints are linear in the control variables. This reformulation therefore yields an integer linear programming problem, the solution to which can be computed with standard optimization packages.
To develop intuition for the role that $\lambda$ plays in the nature of the solution to our linear programming problem, it is useful to consider the task of ranking a single pair in (ref) while ignoring cross-pair constraints. Setting $d_{ij}=1$ minimizes risk whenever \[ \underbrace{\pi_{ji}-\lambda\pi_{ij}}_{d_{ij}=1 \text{ } (i \succ j)}<\min\left\{ \underbrace{0}_{e_{ij}=1 \text{ (tie)}},\underbrace{\pi_{ij}-\lambda\pi_{ji}}_{d_{ij}=e_{ij}=0 \text{ } (i \prec j)}\right\}. \]
For $\lambda\in\left[0,1\right]$, we can rewrite this condition as
which is equivalent to the rule derived in section (ref). Hence, with $\lambda=1$, it is optimal to choose $d_{ij}=1\{\pi_{ij}>1/2\},$ which can be thought of as a MAP estimate of the pairwise rank. When $\lambda\in\left(0,1\right)$, greater posterior certainty is required to conclude that $d_{ij}=1$ and values of $\pi_{ij}$ near 1/2 will yield ties even though $G$ is continuous. As $\lambda$ approaches zero, fewer distinct grades will be assigned. When $\lambda=0$, all $n$ firms are assigned the same grade because $\pi_{ij}\leq1$.
The coarse grades that result from applying the pairwise thresholding rule in (ref) when $\lambda<1$ can generate a form of Condorcet cycle in indifferences that violates the transitivity constraints in (ref) even if they would be satisfied under $\lambda=1$. The following three firm example illustrates the problem.
Note that if we had set $\lambda=1$ in the above example transitivity would have been satisfied because the posteriors themselves are transitive in the sense that for any triple $(i,j,k)$ of firms, $\pi_{ij}>\pi_{ji}$ and $\pi_{jk}>\pi_{kj}$ imply $\pi_{ik}>\pi_{ki}$. This transitivity derives from the scalar index structure of the posteriors, revealed by the fact that \[ \pi_{ij}>\pi_{ji} \Longleftrightarrow \omega_i>\omega_j. \] sobel1993bayes establishes the transitivity of posteriors in a broader exponential family subject to a corresponding index restriction. In general, however, when the observations are heteroscedastic, such index representations are not available and transitivity is not assured. When transitivity fails, the constraints in (ref) will bind and multiple units may receive the same grade even when $\lambda=1$.
Finally, note that coarse grades need not be a consequence of transitivity violations. If $\omega_2 \in (-1.19,0.81)$ in the preceding example, it is optimal to rank $\theta_1 > \theta_3$, $\theta_1 > \theta_2$, and $\theta_2 = \theta_3$. Thus pairwise thresholding yields two grades and no transitivity violations. Whether the transitivity constraints bind therefore depends on the structure of the pairwise posterior contrasts.
The literature on ranking methods bears a close connection to problems of social choice. If we re-interpret $\pi_{ij}$ as the share of votes for firm $i$ over firm $j$ in a pairwise election then a number of standard preference aggregation schemes suggest themselves.\footnote{In developing this analogy, we temporarily depart from the convention that $d_i>d_j$ implies firm $i$ has been assigned a “worse” grade than firm $j$, referring instead to firms with high $d_i$ as highly ranked.} For example, borda1784memoire's voting method simply ranks each firm $i$ based on the number of elections it has won; i.e., based upon $\sum_{j\neq i} 1\{\pi_{ij}>1/2\}$. If (as we have assumed) $G$ is continuous, then the Borda measure is equivalent to the posterior mean rank, a quantity studied by laird1989empirical.
The ranking procedure devised in section (ref) turns out to be closely tied to condorcet1785essay's voting scheme. To develop this connection, it is useful to define the kemeny1959mathematics distance between the vectors $\theta$ and $d$, which can be written
Integrating out $\theta$ yields
When ties are not possible (i.e., when $\pi_{ij}=1-\pi_{ji}$ for all $i\neq j$) we obtain the simplification
young1978consistent show that condorcet1785essay's voting scheme is equivalent to choosing a ranking $d$ that minimizes (ref). young1986optimal establishes that this vote aggregation scheme is the unique rule that is unanimous, neutral, and satisfies reinforcement and independence of remote alternatives. It can also be viewed as a type of maximum likelihood estimator giving “the ranking of all candidates that is most likely to be correct” young1986optimal.
Note that $\left(2\pi_{ij}-1\right)\left(d_{ji}-d_{ij}\right)$ is minimized by the MAP thresholding rule $d_{ij}^{MAP}=1\{\pi_{ij}>1/2\}$. When $\lambda=1$, our objective in (ref) reduces to (ref). Consequently, the most granular version of our grading scheme minimizes the expected Kemeny distance between the assigned grades and the true rankings. When $\lambda<1$ we depart from the Kemeny criterion by calling elections a draw when they are close. Here, a close election is one where $\pi_{ij}<(1+\lambda)^{-1}.$
Condorcet rankings obey the famous Condorcet winner criterion: a unit that wins all pairwise elections between candidates (that is, satisfies $\pi_{ij}>1/2$ $\forall j \neq i$) will be ranked first. The following proposition reveals that when $\lambda<1$ our grades fulfill a modified version of the Condorcet winner criterion.
We leave the short proof for (ref). By symmetry of the objective in (ref), the firm assigned the lowest grade by our method must achieve the highest grade when the sign of the estimand being ranked is reversed. Hence, Proposition (ref) also implies that any Condorcet loser -- i.e., any candidate firm $i$ with $\pi_{ji}>(1+\lambda)^{-1}$ for all $j\neq i$ -- must be assigned the lowest grade.
Another well-known property of Condorcet rankings is that when no Condorcet winner exists, the top ranked candidate must be a member of the smith1973aggregation set: the smallest non-empty subset of candidates such that every candidate in the subset is majority-preferred over every candidate not in the subset. The following proposition establishes a corresponding property of our grades in the case where $\lambda<1$.
The proof is again left for the appendix. Symmetrically, Proposition (ref) implies the firm assigned the lowest grade must be a member of the Smith loser set of candidates that are majority non-preferred to all others. Finally, we note that when $\lambda<1$ and no ordering is possible within the Smith set, all firms in the set will receive equal grades.
As with the preceding propositions, the proof appears in (ref).
Ranking mistakes are likely to be more costly when the magnitude of the mistake is larger. The following family of loss functions weight pairwise concordances and discordances by the $p$'th power of the difference between the cardinal biases of the two firms:
A loss function corresponding to the $(p=2,\lambda=1)$ case was previously considered by sobel1990complete. The corresponding family of risk functions take the form
where $\mu^p_{ij}=\mathbb{E}\left[\max\{(\theta_i-\theta_j),0\}^p\mid Y=y\right]$. Note that $\lim_{p\rightarrow0} \mu_{ij}^p = \pi_{ij}$. Hence, one can think of our baseline risk function in (ref) as a limiting case of $\mathcal{R}^p$ as $p$ approaches zero. In what follows, we focus on this limiting case where $p\rightarrow0$ (“binary loss”) but also report results analyzing the case where $p=2$ (“square-weighted loss”) in (ref).
To summarize the reliability of the grades it is useful to report the Discordance Rate $DR(d^*)$, which gives the posterior expected frequency of discordances between pairs of grades. From (ref), this quantity is trivial to compute, as it depends only on the optimized decisions $\{d_i^*\}_{i=1}^n$ and the posterior probabilities $\{\pi_{ij}\}_{i\neq j}$.
We can also define the conditional Discordance Rate between a specific pair of grades $g$ and $g'<g$ as
The denominator of each pairwise rate can be thought of as giving the number of rejections of the null hypothesis that a pair of firms discriminate equally in favor of the alternative that the firm assigned to group $g'$ is more biased than the firm assigned to group $g$. Hence, $DR_{g,g'}$ is an analogue of the directional false discovery rate benjamini1995controlling,benjamini2005false, giving the expected share of pairs with differing grades that are misranked.
Because the conditional DRs are symmetric ($DR_{g,g'}=DR_{g',g}$), we report them as a lower triangular matrix. Note that the overall $DR$ is a weighted average of the conditional DRs with positive weight put on the “on-diagonal” terms $DR_{g,g}$, which are necessarily zero. When working with $p$-weighted loss a corresponding weighted version of the conditional discordance rate can be employed:
where $m^p_{ij}=\mathbb{E}_{G}\left[(\theta_{i}-\theta_{j})^{p}\mid Y=y\right].$ The $p$-weighted discordance rate nests the corresponding unweighted rate as $DR^0_{g,g'}=DR_{g,g'}$. For any $p>0$, $DR^0_{g,g'}$ is guaranteed to lie in the unit interval.
If we view the grades $\{d_i^{*}\}_{i=1}^n$ as exchangeable random variables and take $DR_{g,g'}$ as an assessment of the conditional misranking probability $\Pr(\theta_i>\theta_j \mid d_{i}^{*}=g,d_{j}^{*}=g')$ we can convert the conditional DR's into Bayes Factors expressing the informativeness of assigned grade pairs about the relative ranks of firms. By Bayes' rule,
where we have used the fact that under a smooth $i.i.d.$ prior $\Pr\left(\theta_{i}>\theta_{j}\right)=1-\Pr\left(\theta_{i}\leq\theta_{j}\right).$ The Bayes Factor $BF_{g,g'}$ conveys the odds that a randomly selected firm assigned grade $g$ is more biased than a randomly selected firm assigned grade $g'<g$. Corresponding $p$-weighted Bayes Factors can be formed in the same manner using $DR^p_{g,g'}$.
As an introductory illustration of the methods developed in the previous section, we now rank the employer contact propensities of the names used in our correspondence experiment. The experiment utilized 76 first names, which were split equally between the nominal categories of: Black male, Black female, white male, and white female.
Table (ref) lists the mean contact rates of names in each of these categories, along with the number of applications. Distinctively white and female names were called back most often in the experiment, followed by white male names, then Black male names, with Black female names being called back least often. Though the same names were intended to be sent to each firm, the COVID-19 epidemic and other disruptions led to minor imbalances reflected in the sample counts. In our past work kline2021systemic we were unable to reject the null hypothesis that the first names randomly assigned to our applications exhibited equal employer contact probabilities conditional on their nominal race and sex, which suggests these imbalances had little effect on mean contact rates by race or sex. For completeness, Table (ref) reports for each demographic group the $p$-value of a Wald test of the null hypothesis that the name specific contact rates are equal. The smallest such $p$-value is 0.24.
To get a sense of what share of the variation in name contact probabilities is likely explained by the names' putative race and sex labels, we compute a bias-corrected estimate of the between demographic group variance using the formula $\frac{G-1}{G}\left(S^2-\overline{s^{2}}\right)$ where $G=4$ is the number of demographic groups, $S^2$ is the sample variance across demographic groups of the point estimates reported in Table (ref), and $\overline{s^{2}}$ is the average squared standard error across those groups. This calculation yields an unbiased estimate of the between demographic group variance of $(0.011)^2$. Applying a corresponding calculation to the name specific means and standard errors yields a bias-corrected estimate of the variability in contact probabilities across all 76 first names of $(0.010)^2$. The finding that our between demographic group variance estimate exceeds our between name variance estimate suggests that the nominal race and sex of a first name explains nearly all of the variation across first names in contact probabilities; that is, we expect that a regression of the latent name specific contact probabilities on race and sex interactions would yield an $R^2$ very close to one.
In principle, even if race and sex perfectly predict employer treatment of names, the causal factors generating this association could be other features of names that correlate strongly with race and sex. A candidate factor that has attracted substantial attention from social scientists is employer stereotypes about the likely productivity of individuals with different names fryer2004causes,gaddis2017names. This hypothesis was evaluated by bertrand_mullainathan_2004, who found that the average socioeconomic status of the first names considered in their experiment (as proxied by average maternal education) varied widely within race but were insignificantly related to contact rates.\footnote{Recent work by crabtree2022racially directly elicits perceptions of educational attainment and income by first name on a variety of online platforms. Remarkably, they find that perceptions of social class exhibit variability across racially distinctive first names in the same race category comparable in magnitude to the variability found between race categories (see their Figure 4).} Because roughly half (40 out of 76) of our first names coincide with those studied by bertrand_mullainathan_2004, our finding of insignificant contact probability differences within race and gender casts further doubt on the view that employer responses are driven primarily by features of names other than their likely race or sex.
The finding that race and sex provide an accurate low dimensional summary of the 76 name specific contact probabilities suggests it is possible to build a highly informative ranking of the names involving just a few grades. Below, we investigate this conjecture in two ways. First, we examine how the expected Kendall's $\tau$ produced by our procedure scales with the number of grades assigned. Second, we treat each name's nominal race and sex as “missing labels” and study the extent to which the coarse grades assigned to first names by our ranking algorithm can recover these labels from data on firms' sample contact rates.
Abusing notation somewhat, let $i$ in this section refer to a first name and denote the number of applications with name $i$ sent in the experiment by $N_i$. The number of employer contacts received within 30 days by those applications is denoted by $C_i$. If the contacts are viewed as independent Bernoulli trials with name specific contact probabilities $p_i$ then the contact rate $C_i/N_i$ of name $i$ has mean $p_i$ and variance $p_i(1-p_i)/N_i$. This dependence of the variance on the contact probability complicates ranking exercises, as contact rates for names that deserve the best and worst grades -- that is, those with $p_{i}$ near one or zero -- will also be estimated with the least noise.
To stabilize the variance, we rank names according to a bartlett1936square transformation of their contact rates \[ \hat \theta_i = \sin^{-1} \sqrt{C_i/N_i}. \] The logic of this transform follows from the observation that $\frac{d}{dx} \sin^{-1} \sqrt{x}=\left[2\sqrt{x(1-x)}\right]^{-1}$. Consequently, the Delta method implies $\hat \theta_i$ has asymptotic distribution $\mathcal{N}(\theta_i, (4N_i)^{-1}),$ where $\theta_i=\sin^{-1} \sqrt{p_i}$.
To estimate the distribution $G$ from which $\theta_i$ was drawn we first apply the non-parametric maximum likelihood (NPMLE) estimator developed by koenker2014convex as implemented by the `GLmix' command in the R package REBayes koenker2017rebayes. The NPMLE estimates a discrete approximation to the distribution of $\theta_i$ assuming that $\hat \theta_i\mid \theta_i,N_i \sim \mathcal{N}(\theta_i,(4N_i)^{-1})$. Supporting the maintained independence of $\theta_i$ from $N_i$, a regression of $\hat \theta_i$ on $\ln N_i$ yields a statistically insignificant relationship ($p$=0.17).\footnote{The variation in $N_i$ is primarily attributable to the fact that a subset of our first and last name pairs were taken from the study of bertrand_mullainathan_2004, while the remaining name pairs were drawn from North Carolina data on speeding tickets and Census data. The number of last names considered differed across the two data sources, leading to imbalances in the average number of last names (and hence applications) per first name.}
A plot of the estimated marginal distribution $\hat G$ of $\theta_i$ produced by the NPMLE is provided in Figure (ref). The bars correspond to histograms of $\hat \theta_i$ while the yellow spikes represent the estimated probability mass function $d \hat G$ of the $\theta_i$. This discrete distribution does an excellent job matching the mean value of the $\hat \theta_i$ and its bias corrected variance, which we compute as its sample variance minus its average squared standard error $n^{-1}\sum_{i=1}^ns_i^2=n^{-1}\sum_{i=1}^n(4N_i)^{-1}$.
Figure (ref) also plots the estimated density of $\theta_i$ produced by efron2016empirical's log-spline estimator, which models $G$ as a 5th order spline in the exponential family. Estimation of the spline parameters is conducted via penalized maximum likelihood, where $N_i$ is treated as independent of $\theta_i$. The penalization parameter has been chosen by GMM to match unbiased estimates of $G$'s first two moments as closely as possible. Despite being continuous, the bimodal shape of the log-spline estimate is remarkably consistent with that of the NPMLE. For reference, the sample mean values of $\hat \theta_i$ for each nominal race and sex category are portrayed on the Figure as vertical lines. Despite not using race in the estimation procedure, the two modes of the log-spline estimate fall near the race specific means as do the modes of the NPMLE estimate.
The lower panel of Figure (ref) converts these estimates back into probability points via the inverse transform $\theta \mapsto \sin(\theta)^2$. The NPMLE finds two large mass points: one at $p=0.226$ and another at $p=0.244$. The 1.8 percentage point gap between these mass points is very near the Black-white contact gap in the experiment of 2.1 percentage points. Likewise, the distance between the modes of the log-spline estimate is roughly 2.1 percentage points. The NPMLE also finds a third mass point at $p=0.260$, which lies just below the estimated average contact rate for distinctively white female names.
As noted earlier, the discrete $\hat G$ produced by the NPMLE is a data dependent approximation to the true $G$. Even if differences in the treatment of names are driven primarily by employer perceptions of their race and sex, it seems unlikely that the true $G$ is literally characterized by a few mass points, as small differences across names in their perceived race should generate corresponding contact rate differences. In fact, although three discrete modes are visible from the Figure, the NPMLE solution involves dozens of atoms with positive mass, most of which are imperceptible. In what follows, we rely on the log-spline estimate $\hat G$ of $G$ which, as in the theoretical analysis of Section (ref), implies that ties are measure zero.
The top left panel of Figure (ref) depicts the posterior contrast probabilities $\pi_{ij}=\Pr(\theta_i>\theta_j \mid \hat \theta_i,\hat \theta_j,N_i,N_j)$; see the Appendix for details on how these posteriors were computed. Names have been ordered according to their Condorcet rank (i.e., their grade when $\lambda=1$). To ease interpretation, we have labeled the name with the highest ranked contact probability 1 and that with the lowest ranked contact probability 76. Name pairs with adjacent ranks exhibit $\pi_{ij}$'s near 1/2, indicating that we have little confidence in their relative order. Reassuringly, names with vastly different ranks exhibit $\pi_{ij}$'s near 0 or 1, indicating that the experimental data are informative about the relative ordering of these pairs.
The top right panel of Figure (ref) depicts the Discordance Rate that arises from minimizing $\mathcal{R}(d;\lambda)$ -- that is, from solving (ref) subject to (ref) -- for different choices of $\lambda$. The number used to demarcate each solution corresponds to the number of distinct grades the integer linear programming procedure produces for that choice of $\lambda$. A sharp “elbow” emerges around $\lambda=0.18$, above which the $DR$ grows rapidly.
The bottom panel depicts the trade off between grade reliability $1-DR$ and informativeness $\bar \tau$ associated with our choice of $\lambda$. The data are potentially quite informative about name rankings: as $\lambda$ approaches 1, the expected rank correlation $\bar \tau$ approaches 0.44. However, the reliability of such a report would be fairly low, yielding a Discordance Rate of 0.28. For comparison, we also show the results of naively ranking based on $\hat \theta$ or the EB posterior mean $\bar \theta_i = \mathbb{E}[\theta_i|Y=y]$. Remarkably, both naive approaches yield ranks with $\bar \tau$ and $DR$ very similar to those produced by our procedure when $\lambda=1$.
To improve the reliability of the grades, we set $\lambda=.25$, implying via equation (ref) that, in the absence of transitivity considerations, we would require 80% posterior certainty of each pairwise ranking decision that is not an abstention. This choice yields two grades that exhibit a remarkably high level of informativeness ($\bar \tau=0.29$) and an acceptable level of reliability ($DR=.07$). For comparison, lowering the implicit posterior threshold to 70% by setting $\lambda=0.41$ would yield three grades and increase their informativeness by 11% (to $\bar \tau=0.32$) at the expense of a 21% increase in $DR$. Conversely, requiring $\lambda < 0.18$ would generate only one grade, yielding both $\bar \tau$ and $DR$ of zero by construction.
Figure (ref) lists the first names according to their Condorcet ranking, along with the posterior mean of each name's contact probability $p_i=\sin(\theta_i)^2$. In addition to the posterior means, which are depicted as dots, we report posterior credible intervals connecting the 2.5th percentile of each name's posterior distribution of contact probabilities to the 97.5th percentile of its posterior distribution. One should expect that approximately 72 (i.e., 95%) of these 76 intervals contain their name's true latent contact rate.\footnote{The asymmetry of the credible intervals reflects both that the estimated mixing distribution $\hat G$ of $\theta_i$ is asymmetric and that we have fed the interval limits through the nonlinear transformation $\theta \mapsto \sin(\theta)^2$.} While the credible intervals tend to be fairly short---spanning between two and three percentage points in most cases---there is clearly enough uncertainty about each name's contact probability to significantly complicate the task of ranking them.
The Condorcet ranks can be thought of as a one dimensional encoding of the $\binom{76}{2}=2,850$ pairwise contrasts depicted in the upper left panel of Figure (ref). When the posteriors are transitive, as they are in this case, this reduction amounts to the most likely ordering of the name contact probabilities. Variation in $N_i$ across names, and hence the precision with which contact rates are measured, could in principle generate substantial non-monotonicity of the posterior mean in the Condorcet rank. In practice, however, names' Condorcet rankings are very nearly monotone in their posterior means.
The Condorcet ranks are extremely correlated with race. Of the top 38 ranked first names, only 8 are distinctively Black. Though the three top ranked names “Misty,” “Heather,” and “Laurie” are all distinctively female, the presumptive sex of a name turns out to be only weakly related to its Condorcet rank: 19 of the top 38 names are distinctively male. Hence, the Condorcet ranks manage to recover the race labels from contact rates with very little error but serve as unreliable proxies of a name's sex.
By construction, the Condorcet ranks exhibit the strongest expected rank correlation with the latent $\theta$ ranks. The coarse ranks that emerge when $\lambda<1$ sacrifice rank correlation in exchange for the prospects of incurring fewer mistakes. Each name's color reflects its assigned grade (i.e., its coarse rank). A depiction of how the grades vary with name-specific contact rates and their standard errors is provided in Appendix Figure (ref). As expected, names with higher contact rates tend to earn the better grade $\star\star$. However, heteroscedasticity in the estimates prevents the grades from being characterized by a single cutoff contact rate.
Though we saw earlier that the expected rank correlation of our grades with the true latent ranks is 0.29, it is also of interest to know how much $p_i$ varies across grades. As described in (ref), we can use our EB posteriors to compute an estimate of the variance of $p_i$ across grades. Though our procedure assigns only two grades to the names, we estimate that the (name-weighted) between grade standard deviation in contact probabilities is 0.006. Since the marginal standard deviation of $p_i$ is roughly 0.010, a regression of the latent $p_i$ on our grades should yield an $R^2$ of 35%.
The coarse grades that emerge from our procedure continue to align closely with our race labels: 35 of the 53 names (66%) in the top grade are distinctively white, while just 3 of the 23 names (13%) in the second grade are white. Notably, the top two names are also female; however, they do not appear in their own grade. Hence, a two-group ranking recovers the missing race label with limited error and, consistent with our findings in Table (ref), suggests that white female names are particularly favored.
It is natural to wonder if a solution with more grades would be more predictive of sex. Appendix Figure (ref) reports the pseudo-$R^2$ and Area under the Curve (AUC) from a series of logistic regressions of the name's sex on grade indicators for different choices of $\lambda$. Note that if we were to set $\lambda=1$, this regression would necessarily predict sex perfectly, as every name would receive its own dummy indicator. However, the four-grade solution with the smallest value of $\lambda$ yields a pseudo-$R^2$ for sex of 0.012. With five grades we find a pseudo-$R^2$ for sex of 0.034. By contrast, a corresponding logistic regression of race on assigned grades yields pseudo-$R^2$s for four- and five-grade solutions of 0.28 and 0.23, respectively.
Appendix Figure (ref) shows the grades that result from minimizing expected square-weighted loss $\mathcal{R}^2(d,\lambda)$, which weights large mistakes in pairwise rankings more heavily than small mistakes. As above, we set $\lambda=0.25$, which now yields five distinct grades. Once again, the grade categories are better predictors of race than sex. As shown in Appendix Figure (ref), a logistic regression of sex on these five grade categories yields a pseudo-$R^2$ of roughly 0.05, while a corresponding race regression yields a pseudo-$R^2$ of 0.24. (ref) provides further details on the square-weighted rankings.
We conclude that our grades are fairly strong predictors of a name's race but not its sex. Given that the overall gender gap in contact rates is indistinguishable from statistical noise in our experiment, the failure to predict gender is not particularly surprising. The ability to predict race for a wide range of choices of $\lambda$, however, suggests that our grading scheme can be effective at detecting latent group structure even when the number of units being ranked is relatively modest.
We turn now to ranking firms in their relative treatment of Black versus white names. The conduct of each firm $i$ in our experiment is characterized by the race-specific contact probabilities $(p_{iw},p_{ib})$. These probabilities represent the hypothetical 30 day contact rates that would arise for applications with distinctively white and Black names, respectively, if we were to sample an infinite number of job vacancies from firm $i$ and send each job four pairs of applications. The sample contact rates $(\hat p_{iw},\hat p_{ib})$ provide unbiased estimates of these contact probabilities.
To mitigate the potential influence of firm heterogeneity in baseline contact rates on our measure of discrimination, we focus on the following proportional measure of bias against Black names at firm $i$:
Our estimator of $\theta_i$ will be the plug-in analog $\hat \theta_i = \ln(\hat p_{iw})-\ln(\hat p_{ib})$. Because the number of applications sent to each firm is large, we employ the Delta method to construct a standard error $s_i$ for each $\hat \theta_i$ based on the job-clustered sampling covariance matrix of the sample contact rates. Although $\hat \theta_i$ is not fully variance-stabilized, the log transform removes any direct dependence of the variance on $\theta_i$ itself.\footnote{Specifically, a second-order Taylor expansion of $\hat{p}_{iw}/\hat{p}_{ib}=\exp\left(\hat{\theta}_{i}\right)$ around the point $(p_{iw},p_{ib})$ yields the approximation $\mathbb{V}\left[\hat{p}_{iw}/\hat{p}_{ib}\right]\approx\theta_{i}^{2}\left\{ \frac{\mathbb{V}\left[\hat{p}_{iw}\right]}{p_{iw}^{2}}+\frac{\mathbb{V}\left[\hat{p}_{ib}\right]}{p_{ib}^{2}}-2\frac{\mathbb{C}\left[\hat{p}_{iw},\hat{p}_{ib}\right]}{p_{iw}p_{ib}}\right\}.$ Consequently, the Delta method implies that $\mathbb{V}\left[\hat{\theta}_{i}\right]\approx\frac{\mathbb{V}\left[\hat{p}_{iw}\right]}{p_{iw}^{2}}+\frac{\mathbb{V}\left[\hat{p}_{ib}\right]}{p_{ib}^{2}}-2\frac{\mathbb{C}\left[\hat{p}_{iw},\hat{p}_{ib}\right]}{p_{iw}p_{ib}}.$}
In what follows, we exclude the eleven firms in the original experiment with callback rates below 3% or fewer than 40 total sampled jobs, since the estimated contact ratios for these firms may be unreliable. Summary statistics for the remaining estimation sample of 97 firms are provided in Table (ref). The unweighted average value of $\hat \theta_i$ across these 97 firms is 0.095, implying the typical firm in our sample exhibits a bias against Black names of roughly 10%. Detailed point estimates and uncertainty measures for all 97 firms used in our analysis are provided in Appendix (ref).
Twenty-one of the 97 estimated contact gaps are negative, indicating a preference for distinctively Black names. The firm-specific estimates are noisy, however, possessing an average standard error of 0.104. To test whether all firms in fact weakly prefer white to Black names (i.e., the joint null that $\theta_i\geq0$ $\forall i\in[n]$) we apply the high dimensional inequality testing procedure of bai2021two. This procedure yields a $p$-value of 0.94, suggesting the observed negative point estimates are likely attributable to chance.
Although the asymptotic variance of $\hat \theta_i$ does not mechanically depend on $\theta_i$, it is possible for $\theta_i$ and $s_i$ to be correlated. The top panel of Appendix Figure (ref) plots $\hat \theta_i$ against $s_i$, revealing that firms with more precise estimates tend to show less bias against Black names. The Spearman correlation between between $\hat \theta_i$ and $s_i$ is 0.36 ($p<0.001$).
In light of the above findings, we assume that each $\theta_i$ is non-negative and may depend (statistically) on its standard error $s_i$. A simple model satisfying these criteria is:
The parameter $\beta$ governs how the conditional distribution of bias varies with the standard error $s_i$. The latent variable $v_i$ captures heterogeneity in discriminatory conduct among firms with similar standard errors, and follows a distribution $G_v$ with strictly positive support. When $\beta$ is positive, both the mean and variance of $\theta_i$ increase monotonically with $s_i$. To complete the model, we link our estimates $\hat \theta_i$ to $\theta_i$ as follows: \[ \hat \theta_i = \theta_i + s_ie_i, \quad e_i \mid s_i,v_i \sim \mathcal{N}(0,1) \text{ for all } i\in [n], \] where $s_ie_i$ is the noise in $\hat{\theta}_{i}$ attributable to the fact that a finite number of jobs were sampled from firm $i$.
To evaluate the plausibility of this model, we scrutinize some of the moment conditions it implies. Letting $\mathbb{E}[v_{i}|s_{i}]=\mu_{v}>0$ and $\mathbb{V}(v_{i}|s_{i})=\sigma_{v}^{2}>0$, consider the following “studentized” version of $\hat \theta_i$: \[ T_{i}=\dfrac{\hat{\theta}_{i}-s_{i}^{\beta}\mu_{v}}{\sqrt{s_{i}^{2\beta}\sigma_{v}^{2}+s_{i}^{2}}}. \] Our model implies that $T_i$ has mean zero and marginal variance one. Moreover, $T_i$ should be independent of $s_i$. These restrictions imply the following four moment conditions:
Imposing these moment conditions via two-step efficient GMM yields the parameter estimates reported in Table (ref). The minimized value of the GMM criterion function suggests the model's over-identifying restrictions -- which test the joint requirement that the $T_i$ have mean zero and constant variance across all values of $s_i$ -- are satisfied ($p=0.97$). The GMM estimate of $\beta$ is $\hat \beta \approx 1/2$, indicating that $\theta_i$ is roughly proportional to $\sqrt{s_i}$. The large estimated value of $\sigma_v$ reveals that discriminatory conduct varies substantially among firms with similar standard errors.
The top panel of Appendix Figure (ref) superimposes the estimated conditional expectation function $\mathbb{\hat E} [\theta_i|s_i]=s_i^{\hat \beta} \hat \mu_v$ on the scatterplot of $\hat \theta_i$ against $s_i$. Consistent with the $J$-test from GMM estimation, the estimated conditional mean fits the cloud of points closely. The bottom panel of Appendix Figure (ref) plots values of the estimated residual $\hat T_i=\dfrac{\hat{\theta}_{i}-s_{i}^{\hat \beta}\hat \mu_{v}}{\sqrt{s_{i}^{2\hat\beta}\hat \sigma_{v}^{2}+s_{i}^{2}}}$ against $s_i$. Consistent with our model, $\hat T_i$ exhibits roughly constant variance and a mean near zero throughout the observed range of $s_i$.
To estimate the population distribution of contact penalties, we deconvolve the residual $\hat v_i= \hat \theta_i / s_i^{\hat \beta}$ which, by the Delta method, obeys
Relying again on a variant of efron2016empirical's log-spline estimator, we parametrize $G_v$ as a fifth order natural spline with strictly positive support. The spline parameters are estimated by penalized maximum likelihood with the penalty term chosen to minimize the distance to our earlier GMM estimates $(\hat \mu_v, \hat \sigma_v^2)$ of the first two moments of $v_i$. We then integrate over the empirical distribution of $s_{i}$ to convert the estimated $\hat{G}_{v}$ into an estimate $\hat{G}_{\theta}$ of the distribution of contact gaps $\theta_{i}$ .
Figure (ref) plots the log-spline estimate $\hat{G}_\theta$ overlaid against the histogram of contact gap estimates. $\hat{G}_\theta$ is less dispersed than the histogram, reflecting the noise in the estimates $\{\hat{\theta}_i\}_{i=1}^n$. Unlike with our earlier analysis of names, the density $\hat{G}_\theta$ is unimodal but highly skewed. While most firms exhibit little bias against Black names, some exhibit large biases of 20-40%. By construction, no firms are estimated to discriminate against white names.
As a robustness check, we also compute NPMLE estimates using the GLVmix procedure developed by koenker2017rebayes, which estimates a bivariate discrete distribution for $(\theta_i,N_is_i^2)$ under the assumption that $\theta_i$ is independent of $N_i$. The resulting marginal distribution of $\theta_i$ exhibits many mass points and is also unimodal. The NPMLE estimate of the variance of the $\theta_i$'s departs somewhat from both the log-spline estimate and a simple bias-corrected variance estimator $n^{-1}\sum_i [(\hat \theta_i - \bar \theta)^2 -s_i^2]$. However, the NPMLE and log-spline estimates appear comparable in their overall shape. Since a discrete distribution with exact ties seems implausible, we rely again on the log-spline estimates in what follows. The EB posterior distribution of $\theta_{i}|Y_{i}$ inherits the continuity of the log-spline deconvolution estimate of the prior distribution, which has the added benefit of simplifying computation of posterior credible intervals for each $\theta_i$.
In kline2021systemic we found large differences in the magnitude of contact gaps across 2-digit NAICS industries. Many of these industries have only 2 or 3 firms, precluding a fixed effects approach to incorporating industry affiliation into the model. We therefore employ a hierarchical random effects specification of $v_i$ taking the form: \[ v_{i} = \eta_{k(i)}\xi_{i}, \label{eq:doubleG} \] \[ \xi_{i} \ | \ s_{i},\eta_{k(i)} \sim G_{\xi}, \ \ i\in\{1,...,n\}, \] \[ \eta_k \ | \ \mathbf{s}_{k} \sim G_{\eta}, \ \ k\in\{1,....,K\}, \] where the function $k:\{1,\dots,n\}\rightarrow\{1,\dots,K\}$ returns a firm's industry and $\mathbf{s}_{k}$ is the vector of standard errors for all firms with $k(i)=k$. The industry effect $\eta_{k(i)}$ captures correlation in discriminatory conduct among firms in the same industry, while the firm effect $\xi_i$ captures departures from the industry average. As a normalization we assume $\mathbb{E}[\eta_k]=1$, which implies $\mathbb{E}[\xi_i]=\mu_v$.
The marginal variance of $v_{i}$ in this model is $\sigma_{v}^{2}=\sigma_{\eta}^{2}\sigma_{\xi}^{2}+\sigma_{\eta}^{2}\mu_{v}^{2}+\sigma_{\xi}^{2}$, where $\sigma_{\xi}^{2}$ gives the variance of $\xi_i$ and $\sigma_{\eta}^{2}$ the variance of $\eta_k$. To separately identify the between and within industry variance components, we add two new moment conditions to the set listed in (ref). Denote the average value of $\hat v_i$ in industry $k$ by \[ \ensuremath{\bar{v}_{k}= n_{k}^{-1}\sum_{i:k(i)=k}\hat{\theta}_{i}/s_{i}^{\beta}}=n_{k}^{-1}\sum_{i:k(i)=k}v_{i}+n_{k}^{-1}\sum_{i:k(i)=k}e_{i}/s_{i}^{\beta}, \] where $n_k$ gives the number of firms in industry $k$. The variance of $\bar{v}_k$ in this model can be shown to be $V_k \equiv \left(\sigma_{\eta}^{2}\sigma_{\xi}^{2}/n_{k}+\sigma_{\eta}^{2}\mu_{v}^{2}+\sigma_{\xi}^{2}/n_{k}\right)+n_{k}^{-1}\sum_{i:k(i)=k}s_{i}^{2\left(1-\beta\right)}$. Letting $\ensuremath{\bar{s}_{k}=n_{k}^{-1}\sum_{i:k(i)=k}s_{i}}$ denote the average standard error in industry $k$, our two new moment conditions can be written \[ \mathbb{E}\left[\left(\bar{v}_{k}-\mu_{v}\right)^{2}-V_{k}\right]=0,\quad\mathbb{E}\left[\left\{ \left(\bar{v}_{k}-\mu_{v}\right)^{2}-V_{k}\right\} \bar{s}_{k}\right]=0. \] The first condition simply equates the empirical squared deviations of the $\bar{v}_{k}$ around the model implied mean to the model implied variance. The second condition prohibits heteroscedasticity with respect to $\bar{s}_k$.
GMM estimates of the parameters of this hierarchical model are reported in the second column of Table (ref). The model's over-identifying restrictions again appear to be satisfied ($p=0.95$). While the variance $\sigma^2_{\eta}$ of the industry component is estimated to be nearly 10 times as large as the variance $\sigma^2_{\xi}$ of the firm specific component, the multiplicative influence of these components on $v_i$ implies that roughly half of the marginal variance in $v_{i}$ stems from within industry variation.\footnote{By the law of total variance $\mathbb{V}\left[v_{i}\right]=\mathbb{V}[\mathbb{E}[v_{i}|k(i)]]+\mathbb{E}[\mathbb{V}[v_{i}|k(i)]].$ Plugging in $v_i=\eta_{k(i)}\xi_i$ reveals that the within share $\mathbb{V}[\mathbb{E}[v_{i}|k(i)]]/\mathbb{V}\left[v_{i}\right]$ evaluates to $(\sigma_{\eta}^2+1)\sigma_{\xi}^2/\sigma_v^2.$}
To identify the marginal distribution of $\theta_i$, we assume that both $G_{\eta}$ and $G_{\xi}$ lie in the exponential family parameterized by a 5th order spline. Generalizing efron2016empirical's log-spline estimator to the hierarchical case, these distributions are estimated by penalized maximum likelihood (see (ref) for details). The two penalty parameters in this likelihood function are chosen so that the resulting distributions match GMM estimates of the between-industry and total variances of $\theta_{i}$.
Estimates of $G_{\xi}$ and $G_{\eta}$ are displayed in the top panel of Figure (ref). Table (ref) reports moments of the within- and between-industry distributions implied by the log-spline estimates as well as moments of the overall contact ratio $\theta_{i}=s_i^{\beta}\eta_{k(i)}\xi_i$. Standard errors for the moments are computed via the Delta method, treating the fifth-order splines as correctly specified models for the log density functions. The mean contact gap, between-industry standard deviation, and total standard deviation reported in Table (ref) closely match the corresponding GMM estimates of these parameters in Table (ref).
As can be seen in Figure (ref), the industry component $\eta_{k}$ is more variable than the firm component $\xi_i$ and exhibits positive skew and excess kurtosis, reflecting that some industries feature particularly heavy discrimination against Black names. Recall however that the location of the industry effect distribution is not informative as we have normalized $\mathbb{E}[\eta_k]=1$. The bottom panel of Figure (ref) shows that the implied distribution of $\theta_i$ is similar to the estimate from the model without industry effects in Figure (ref), with a peak at small contact penalties and a long right tail. As expected, the deconvolved distribution is more compressed than the empirical distribution of estimated contact gaps.
Figure (ref) plots the pairwise posterior ranking probabilities $\pi_{ij}$ with firms ordered by their rank under $\lambda=1$. Following our earlier convention with the names, these ranks range from 1 (the largest contact penalty) to 97 (the smallest contact penalty). Panel (a) shows results from our baseline specification with the log-spline estimate of the marginal mixing distribution as prior, while panel (b) reports results based on the hierarchical log-spline model with industry effects. Because the firm assigned rank 1 is deemed most discriminatory, many other firms are more likely than not to have lower values of $\theta$. Firms of middling rank, on the other hand, are more difficult to distinguish from others. Including industry effects tightens the posteriors, which leads the $\pi_{ij}$'s to become more dispersed around 1/2. This phenomenon is apparent in the more distinct ridge along the diagonal of the matrix in panel (b) compared to the baseline specification in panel (a).
Figure (ref) displays only the pairwise probabilities that satisfy the naive thresholding rule $\pi_{ij}>(1+\lambda)^{-1}$ when $\lambda$ has been set to 0.25. The resulting frontier implies numerous transitivity violations. For example, in panel (a), firm \#9 cannot be distinguished from firm \#4 or firm \#49, suggesting each of these pairs in isolation would be labeled a tie. However, firm \#49 is clearly distinguishable from firm \#4, yielding a contradiction. Super-imposed on the figure we show a frontier corresponding to the three grades that solve (ref) subject to (ref) when $\lambda=0.25$. These frontiers can be viewed as a transitivity-constrained version of the thresholding rule.
Figure (ref) plots the number of distinct grades that result from minimizing $\mathcal{R}(d;\lambda)$ along with the Discordance Rate of those grades as a function of the parameter $\lambda$. As expected, the number of grades tends to increase with $\lambda$ as does the $DR$. In the absence of industry effects, setting $\lambda=0.25$ yields three groups and an unconditional DR of roughly 3.9%. Introducing industry effects yields four groups and decreases the DR to 5.2%.
Figure (ref) illustrates the empirical tradeoff between the information content of our grades, quantified by the expected rank correlation $\bar \tau$, and their reliability, as quantified by the Discordance Rate. Without industry effects, setting $\lambda=1$ yields $\bar \tau=0.46$ and a Discordance Rate of 0.27. Including industry effects increases the $\bar \tau$ of the Condorcet ranks to 0.51\ and lowers their DR to 0.24. In contrast, ranking naively on $\hat \theta_{i}$ yields both a higher Discordance Rate and lower $\bar \tau$ than the Condorcet ranks, indicating such an approach is both less informative and less reliable. Interestingly, ranking based upon the EB posterior means yields a $\bar \tau$ and DR essentially equivalent to the Condorcet ranks.\footnote{While ranking based upon posterior means is known to possess certain optimality properties when $G$ is normal and the normal noise is homoscedastic portnoy1982maximizing, our environment features both heteroscedasticity and a decidedly non-normal mixing distribution $\hat G$.}
To improve the reliability of the Condorcet ranks, we set $\lambda=0.25$. In the absence of transitivity violations, this choice of $\lambda$ requires a posterior threshold of at least 80% to make pairwise ranking decisions. As depicted in Figure (ref), however, numerous transitivity violations emerge in this example. Resolving these violations raises the required posterior certainty above 80% in most instances, yielding a Discordance Rate of only 3.9 % in the baseline specification without industry effects and 5.2 % in the hierarchical specification with industry effects. Fortunately, the resulting grades remain highly informative: $\bar \tau$ is 0.21 \ in our baseline specification and 0.32 \ when industry effects are included.
The report cards generated by our grading procedure provide a concise, low-dimensional summary of differences in discrimination across firms. This can be seen in Figure (ref), which displays the report card results for the baseline specification without industry effects. In addition to the report card grades, the Figure plots a posterior mean estimate of each firm's bias $\theta_i$ along with 95% credible intervals, which are constructed by connecting the posterior 2.5th percentile of $\theta_i$ to the posterior 97.5th percentile. The lower limit of each credible interval is positive as a result of our support restriction ruling out bias against white applicants. The firms are ordered by their Condorcet ranks (i.e., their grades under $\lambda=1$). In this draft we have replaced each firm's name with the rank of its bias estimate $\hat \theta_i$, which allows us to compare firm orderings across loss functions. Firms that are federal contractors, and hence subject to higher regulatory standards regarding equal opportunity laws, have been listed in black, while those that are not contractors are listed in gray. After the paper has undergone peer review, we intend to label the report card with firms' actual names.
Setting $\lambda=0.25$ generates a report card with three grades, represented in Figure (ref) by a number of $\star$'s between one (the worst grade) and three (the best). The shading of credible intervals reflects the grade assigned to each firm. Most firms receive the middle grade of $\star\star$, which reflects both the noise in our estimates and the shape of the estimated distribution $\hat G$. By contrast, only two firms out of 97 are assigned the grade of $\star$, suggesting they are the heaviest discriminators against Black names. Fourteen firms are assigned the score of $\star\star\star$, which indicates that this group is the least-biased against Black applicants.
While the Condorcet ranks of the posterior means are highly correlated with the ranks of the bias estimates, heteroscedasticity makes the correlation less than perfect. For example, the firm with the sixth largest point estimate of bias (AKA firm \#6) has a Condorcet rank of 1 and the largest posterior mean, while the firm with the largest bias point estimate (AKA firm \#1) has a Condorcet rank of 2 and the second largest posterior mean. This rank reversal reflects that firm \#6 has a larger standard error than firm \#1, which leads the EB posterior mean to apply more shrinkage of the point estimate towards the center of the distribution.
Appendix Figure (ref) depicts the relationship between report card grades and firm-specific bias estimates and standard errors. Firms assigned the best grade of $\star\star\star$ tend to have both small contact gap estimates and standard errors, while firms assigned the grade $\star\star$ range widely in their standard errors but have modest contact gap estimates falling uniformly below 0.2. Firms assigned the worst grade of $\star$ exhibit very large contact gap estimates and widely varying standard errors. Appendix Figure (ref) depicts the grade assignments that result from different choices of $\lambda$.
Though we have used stars to represent the firm ranks, it is important to remember that these grades were designed to convey ordinal rather than cardinal information. One of us gu2023comment has recently cautioned against focusing excessively on rankings without also considering absolute standards of conduct. There is nothing in our integer linear programming problem that guarantees a grade of $\star$ implies a particularly egregious level of discrimination. Conversely, there is nothing that guarantees firms assigned a grade of $\star\star\star$ exhibit no bias against Black names.
As it turns out, however, the grades assigned by our procedure yield groups of firms with large cardinal differences in contact gaps. The firms assigned the grade of $\star\star\star$ have an average posterior mean value of $\theta_i$ of 0.03, while the two firms assigned the worst grade exhibit posterior means indicating a 24% penalty against Black names on average. Notably, both of these heavily discriminating firms are federal contractors.
Our past work kline2021systemic found that federal contractors, who are subject to monitoring by OFCCP for compliance with equal employment laws, tend to be substantially less biased against Black names on average, which is consistent with a variety of other evidence on the causal effects of affirmative action provisions on hiring behavior mccrary2007effect,kurtulus2016impact,miller2017persistent. Indeed, an early audit study of federal contractors by newman1978discrimination found evidence of a systematic preference for Black over white applicants among such firms. It is somewhat surprising then that the Condorcet ranks indicate that the four most heavily discriminating firms are all federal contractors. This finding is, to some extent, a reflection of the fact that the vast majority of the firms in our sample of large employers are contractors (63 of 97). The mean Condorcet rank of federal contractors is 54 (with rank 1 showing the most bias against Black applicants) while the mean Condorcet rank of non-contractors is 42.
Although a legal precedent for audit studies has yet to be established, a commonly applied standard in discrimination cases is the so called “four fifths rule,” described in the Uniform Guidelines on Employee Selection equal1978department which state that
Our estimates suggest the contact rates for fictitious applicants in our experiment may have violated this standard.
As a result of the substantial variation in contact penalties across industries, a report card that incorporates industry information is substantially more informative. This can be seen in Figure (ref), which displays discrimination report card results based on a model with industry effects. Adding industry information while maintaining the preference parameter $\lambda$ at 0.25 yields a report card with four grades rather than three. The number of firms assigned the worst grade of $\star$ increases from two to five, while nine firms are now assigned the second-worst grade $\star\star$. Eleven firms are assigned the best grade $\star\star\star\star$. Appendix Figure (ref) depicts the grade assignments that result from different choices of $\lambda$.
The average value of the posterior mean $\bar \theta_i$ among the firms assigned the grade $\star$ is 0.22, indicating an expected bias against Black names in this group of 22%. In contrast, the average value of $\bar \theta_i$ among the eleven firms assigned grade $\star\star\star\star$ is 0.03, suggesting a negligible effect of race on callback outcomes in this group. This finding indicates that many large firms are nearly unbiased, an important possibility result for companies seeking to improve the fairness of their recruiting process.
The small number of grades generated by our report card procedure explain a substantial portion of the total variation in discrimination across employers, especially when we incorporate industry. To summarize the explanatory power of the grades, we again utilize the grade-average posterior means as detailed in the Appendix. The variance estimate is weighted by the number of firms per grade, so that the ratio of between-grade to total variance has an $R^2$ interpretation. The estimated between-grade standard deviation in contact penalties is 0.034 \ for the three grades reported in Figure (ref), implying an $R^2$ of roughly 19%. Adding industry boosts the $R^2$ to 47%. In other words, the four categories displayed in Figure (ref) explain nearly half of the variance in discrimination across the 97 companies in our experiment.
Our ranking procedure allows us to grade the conduct of entire industries in addition to individual firms. Figure (ref) plots posterior estimates $\mathbb{E}[\eta_k|Y=y]n_k^{-1}\sum_{i:k(i)=k}{s_i^{\hat \beta}}$ of industry mean contact penalties. The most biased industry is estimated to be SIC 55, “Auto dealers / services,” with a posterior mean contact penalty of 25%. In an industry grading scheme with $\lambda=0.25$ (which yields three total grades), SIC 55 is assigned its own unique grade of $\star$, indicating that this industry can be distinguished as the most biased in the experiment with high confidence. A group of six industries receive the best grade of $\star\star\star$, all of which have posterior mean contact gaps of roughly 5%. The role of common industry-level practices in generating the stark differences between these low- and high-performers is an interesting topic for further inquiry.
(ref) details the firm rankings derived from minimizing our square-weighted notion of risk $\mathcal{R}^2(d;\lambda)$. Under this grading scheme, misrankings involving small differences in discriminatory conduct between firms yield negligible losses, which makes a more aggressive partitioning of the firms optimal. Choosing $\lambda=0.25$ yields a six grade ranking without industry effects and an eight grade ranking when industry affiliation is taken into account. Because more grades are employed under square-weighting than under our default scheme, the variance in contact penalties explained by the grades is somewhat larger than for the grades reported in Figures (ref) and (ref). As explained in the Appendix, the square-weighted grades nevertheless maintain tight control over the expected probability of mistakes involving misrankings of firms with large absolute differences in discriminatory conduct.
In addition to conveying substantial information about discrimination, our report card system limits the ranking errors generated by the resulting classification. Figure (ref) assesses the reliability of report card grades by reporting the lower-triangular matrix of estimated between-grade DRs in our baseline model that omits industry effects. Panel (a), for example, shows that 11% of the firm comparisons across grades $\star$ and $\star\star$ are expected to be misordered. The Discordance Rate naturally declines when comparing non-adjacent grades. The expected share of misordered comparisons across grades $\star$ and $\star\star\star$ is below 1%. Adjacent grades have $DR$'s between 11 and 14%, implying Bayes Factors between 7 and 6, respectively. The discordance rate for non-adjacent grades of 0.8% implies a Bayes Factor exceeding 11.
Our preferred report card with industry effects limits misrankings between firms selected as high- and low-performers. Panel (b) of Figure (ref) summarizes the reliability of the grades obtained when conditioning on industry effects. Discordance Rates between adjacent grades range from 15% to 17%. DRs for grades separated by two categories are less than 4%, and the DR between the worst grade ($\star$) and the best grade ($\star\star\star\star$) is 0.4%. Combined with the results from Section 6.1, this implies that a comparison of the best- and worst-performers in Figure (ref) isolates firms with large differences in discriminatory conduct while yielding a negligible chance of misclassification.
We have proposed a new Empirical Bayes method for ranking units based upon noisy measurements and used it to grade the discriminatory conduct of firms towards distinctively Black names in a large-scale correspondence experiment. The experiment is shown to contain a wealth of information about the relative conduct of firms: our most granular (Condorcet) grades taking into account industry affiliation yield an expected correlation with the true firm ranks of 0.51. These grades are noisy, however, resulting in (expected) mistakes in nearly one quarter of the $\binom{97}{2}=4,656$ possible pairwise firm comparisons.
A generalization of the Condorcet scheme based on a desired 80% posterior certainty threshold for pairwise contrasts yields a report card with only four grades. These coarse grades turn out to be substantially more reliable than the Condorcet ranks, lowering the chances that a randomly sampled pair of firms is misordered to 5.2%. The four grades are also highly informative, offering an expected correlation with the true firm ranks of 0.32. In addition to conveying information about the ranking of firm conduct, the grades capture important differences in conduct levels. Firms assigned the worst grade favor white applicants over Black applicants by 23% while those assigned the best grade favor white applicants by only 3%.
The finding of negligible contact gaps in a large group of firms provides a possibility result for employers seeking to improve the fairness of their hiring processes. Recent research points towards centralization of hiring processes as a possible means of dampening bias in large organizations berson2020outsourcing,challeeffect, a conjecture that aligns with findings in behavioral economics that snap judgments by individuals are especially susceptible to bias agan2023automating. Further corroboration of this view comes from miller2017persistent's finding that temporary exposure to the heightened scrutiny over HR practices accompanying federal contractor status has persistent effects on the composition of firm hires.
We plan to eventually release the information in this report card to the public along with the identities of the companies in the experiment. Our hope is that this information will prompt corporate leaders to rethink whether their inclusive hiring goals are being met. Much work remains to establish which sorts of reforms to organizational practices can improve the fairness and efficiency of corporate recruiting efforts. Releasing these data for use by other researchers will hopefully accelerate the pace of research into strategies for subduing hiring discrimination.
\FloatBarrier
\FloatBarrier
\FloatBarrier \setcounter{figure}{0} \setcounter{table}{0} \setcounter{section}{0}