Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
39,715 characters · 10 sections · 75 citation commands
Reasonable uncertainty: Confidence intervals in empirical Bayes discrimination detection
\onehalfspacing
Empirical Bayes robbins1956empirical, efron2019bayes methods are increasingly popular in applied research. Prominent examples covering a diverse array of applications include studies by rozema2019good on police behavior, wernerfelt2022estimating on advertising treatment effects, gu2022ranking on journal ratings, NBERw31192 on managerial productivity effects, and coey2022empirical on online controlled experiments. The growing importance of empirical Bayes methods is further highlighted by walters2024empirical, who surveys applications in labor economics. A distinguishing feature of these and other empirical Bayes analyses is that they rarely include uncertainty quantification for posterior estimands.
In this paper, we revisit this methodological shortcoming in the context of a compelling application of empirical Bayes described in kline2021reasonable to discrimination detection based on correspondence experiments. Their analysis emphasizes the role of uncertainty stemming from partial identification, but, with a few notable exceptions that we describe further below in Section (ref), kline2021reasonable primarily report point estimates for their posterior estimands without further incorporating sampling uncertainty. Our discussion, by contrast, highlights the way in which partial identification and sampling uncertainty for posterior estimands are intertwined and naturally addressed in concert. In the course of doing so, we draw attention to some new results on confidence intervals for empirical Bayes analyses and introduce some novel methods exploiting inference methods recently developed for other problems.
We apply these methods to data from one of the three correspondence experiments analyzed by kline2021reasonable, specifically the study by arceo-gomez2014race of race and gender discrimination in Mexico City. After doing so, we find that some of the empirical conclusions in kline2021reasonable concerning this data prove substantially more robust than others. In this way, our analysis demonstrates the importance of accounting for sampling uncertainty in empirical Bayes discrimination analysis. We further show that this remains true for an alternative estimand based on an odds ratio that we argue may be preferred relative to the one considered in kline2021reasonable for several different reasons. Through these contributions, we hope to make it routine to report confidence intervals alongside point estimates for empirical Bayes estimands and, as called for by imbens2022comment, to encourage further research on statistical inference accompanying empirical Bayes analyses.
The remainder of our paper is organized as follows. In Section (ref), we first review the key ingredients of an empirical Bayes analysis and then specialize our discussion to the setting in kline2021reasonable. In Section (ref), we discuss partial identification and sampling uncertainty through the lens of four different optimization problems. This discussion motivates a particular approach to account for uncertainty described in Section (ref) based on ignatiadis2022confidence. Other methods to account for uncertainty are described in Section (ref), including a novel application of fang2023inference. Along the way, we apply each of these methods to re-assess some empirical conclusions in kline2021reasonable. Finally, in Section (ref), we introduce and discuss our alternative estimand, and apply these methods to it as well.
A canonical empirical Bayes analysis robbins1956empirical, efron2019bayes, ignatiadis2022confidence consists of three ingredients:
The empirical Bayesian has no knowledge of the ensemble $G$, yet seeks to mimic the oracle decision maker by learning from indirect evidence, i.e., by using the observed $Z_1,\dotsc,Z_n$ to learn about $G$. The connection between the observed data and the unknown ensemble is established through the marginal density of the $Z_i$,
In the setting considered by kline2021reasonable,
kline2021reasonable demonstrate using data from three different correspondence experiments how empirical Bayes provides a principled way to detect discrimination patterns across jobs in this type of setting. There are, however, two important sources of uncertainty in such an analysis. First, in some empirical Bayes problems, such as the bivariate binomial model in (ref) described above, even if we had precise knowledge of $f_G$, we could not recover $G$ uniquely. In other words, $G$ is only partially identified. Second, in practice, we do not know $f_G$ either and must estimate it from data.
Before proceeding, we note that we confine our analysis below to data from one of the three correspondence experiments analyzed by kline2021reasonable, specifically the study by arceo-gomez2014race of gender discrimination, but the same considerations apply equally well to the other two correspondence experiments they analyze, namely bertrand2004are and nunley2015racial.
In this section, we discuss four different optimization problems that will help us not only explain the way in which kline2021reasonable address the partial identification issue, but also how sampling variability and partial identification are intertwined. The optimization problems each seek to minimize the estimand $\theta(\widetilde{G};z)$ over all distributions $\widetilde{G}$, but are distinguished by different additional constraints.\footnote{The reader should keep the estimand (ref) in mind, but the discussion applies to any estimand $\theta(G;z)$.} To solve each optimization problem, we discretize \smash{$\widetilde{G}$} on a two-dimensional grid with $K^2$ points; we defer a more detailed discussion of computational issues to Section (ref).
Below, we will explain each of these optimization problems in turn and their constraints, defining all required quantities along the way.
Optimization problem (i) represents an idealized benchmark: if we knew the true marginal density $f_G$, what would be the smallest possible value of $\theta(\widetilde{G};z)$ among all $\widetilde{G}$ that are consistent with this density? Though one could also maximize, we focus on the minimum as, for the discrimination probability $\theta^{\text{discr}}(G;z)$, it represents the most conservative value that is compatible with the true marginal density. Optimization problem (i) captures the fundamental partial identification challenge. We next turn to problems (ii)--(iv) that also capture issues that stem from the fact that $f_G$ is not known and must be estimated.
In optimization problems (ii) and (iii) the true density $f_G$ is replaced by an estimate. Problem (ii) uses the empirical frequencies $\bar{f}(z) = \sum_{i=1}^n \mathds{1}(Z_i = z)/n$, which provide a natural estimate for discrete data.\footnote{For continuous data, estimation of $f_G$ would require more sophisticated density estimation techniques.} kline2021reasonable pursue precisely this approach to compute estimates of lower bounds in their application to the bertrand2004are experiment. Problem (ii) may, however, be infeasible: there may be no distribution \smash{$\widetilde{G}$} whose implied marginal density exactly matches $\bar{f}$. This situation can occur due to sampling variability in $\bar{f}$, or due to misspecification of the bivariate binomial model. Indeed, infeasibility occurs in two of the three empirical examples in kline2021reasonable and is a well understood phenomenon in the related univariate binomial problem wood1999binomial.
To address infeasibility, kline2021reasonable introduce the following generalized method of moments (GMM) problem:
where \smash{$\widehat{W}$} is a weighting matrix computed in a first-stage GMM step. Let \smash{$G^{\mathrm{proj}}$} be the solution to (ref), $\bar{f}^{\mathrm{proj}} = f_{G^{\mathrm{proj}}}$ the implied marginal density, and $J_n^{\mathrm{opt}} = J_n(\bar{f}^{\mathrm{proj}}, \bar{f})$ the optimal value of (ref). Optimization problem (iii) then replaces $\bar{f}$ in the constraint for problem (ii) with $\bar{f}^{\text{proj}}$. Unlike problem (ii), problem (iii) is always feasible by construction, but still ignores sampling variability.
Following common practice for empirical Bayes analyses (as discussed in the introduction), kline2021reasonable only report point estimates of their lower bounds for the posterior discrimination estimand in (ref) without further incorporating sampling uncertainty, that is, they report the objective value of optimization problem (ref)(iii) (or (ii) when feasible). We emphasize, however, that kline2021reasonable are aware of sampling uncertainty more generally and address it in some other parts of their analysis. They propose, for example, a shape-constrained bootstrap scheme, following chernozhukov2023constrained (CNS) for goodness-of-fit testing based on the distribution of $J_n^{\text{opt}}$ (the minimum value of the GMM statistic). Beyond goodness-of-fit testing, kline2021reasonable also adapt the bootstrap scheme of CNS to test null hypotheses such as $\mathbb P_G[p_a \neq p_b] = 0$ or $\mathbb P_G[p_a > p_b]=0$.
Figure (ref) shows the bootstrap distribution\footnote{ We generate the bootstrap samples by directly rerunning the reproduction code of kline2021reasonable. } of $J_n^{\text{opt}}$ for the study of gender discrimination by arceo-gomez2014race (AGCV), where $J_n^{\text{opt}} \approx 2.65$. Under the null hypothesis that the true probabilities are consistent with the bivariate binomial mixture model, the bootstrap distribution suggests that much larger values than $J_n^{\text{opt}}$ can be realized under the null---its 95% quantile is 9.9.
Building on their bootstrap analysis, we propose optimization problem (iv) to illustrate how uncertainty affects their bounds by allowing all distributions with $J_n(f_{\widetilde{G}}, \bar{f}) \leq \kappa$ for some choice of $\kappa >0$. For $\kappa < J_n^{\text{opt}}$, the problem is infeasible. When $\kappa = J_n^{\text{opt}}$ and assuming uniqueness of the minimizer, problems (iii) and (iv) yield identical optimal values. For $\kappa > J_n^{\text{opt}}$, problem (iv)'s optimal value can be strictly smaller, reflecting the incorporated additional uncertainty.
Figure (ref) shows the results of solving optimization problem (iv) for two callback patterns in the AGCV dataset and for different values of $\kappa$. In panel a) for $\theta^{\text{discr}}(G, (1,0))$ and in panel b) for $\theta^{\text{discr}}(G, (4,0))$. For each pattern, we present results using three different discretization strategies corresponding to $K=50$, $150$, or $300$. All lower bound curves begin at $\kappa = J_n^{\text{opt}}$, with finer discretizations yielding slightly smaller values of $J_n^{\text{opt}}$. While discretization choices substantially impact the lower bounds when $\kappa$ is close to $J_n^{\text{opt}}$, these effects become less pronounced as $\kappa$ increases.
The analysis reveals stark differences in robustness across discrimination estimands for different callback patterns. kline2021reasonable' estimate that “an employer that calls back a single woman and no men has at least a 74% chance of discriminating against men” proves highly sensitive to uncertainty---the lower bound rapidly drops as soon as we relax $\kappa$ beyond $J_n^{\text{opt}}$, falling to just 2% at $\kappa = 9.9$ (the 95% quantile of the bootstrap distribution of $J_n^{\text{opt}}$). By contrast, their estimate that “at least 97% of the jobs that call back four women and no men are estimated to discriminate against men” demonstrates greater robustness, maintaining a lower bound of 88% even at $\kappa = 9.9$. As explained in Section (ref) below, this particular choice of $\kappa$ is a special case of the $F$-localization approach of ignatiadis2022confidence and ensures that the lower bound is a valid 95% lower confidence bound on $\theta(G;z)$. Table (ref) records this lower confidence bound as well as lower confidence bounds from two other methods that are described in detail in Section (ref). Collectively, these results demonstrate the importance of accounting for sampling uncertainty in empirical Bayes discrimination analysis, as some findings prove substantially more robust than others.
As mentioned previously, setting $\kappa$ equal to the 95% quantile of the bootstrap distribution of $J_n^{\mathrm{opt}}$ is a special case of the $F$-localization approach of ignatiadis2022confidence and ensures that the lower bound obtained in this way is a valid lower confidence bound on $\theta(G;z)$. To see why, suppose that we choose a potentially data-driven $\widehat{\kappa}$ such that $\mathbb P_G[J_n(f_{G}, \bar{f}) \leq \widehat{\kappa}] \geq 1-\alpha$. Fix an estimand of interest $\theta(G;z)$ and solve optimization problem (ref)(iv) at $\kappa = \widehat{\kappa}$ calling the optimal value $\underline{\theta}(\widehat{\kappa})$. It follows that $$ \mathbb P_G[ \theta(G;z) \geq \underline{\theta}(\widehat{\kappa})] \geq \mathbb P_G[J_n(f_{G}, \bar{f}) \leq \widehat{\kappa}] \geq 1-\alpha, $$ where the first inequality follows since on the event $\{J_n(f_{G}, \bar{f}) \leq \widehat{\kappa}\}$, $G$ is a feasible solution in optimization problem (ref)(iv).
The above construction is a special case of the $F$-localization principle ignatiadis2022confidence. The key idea is to construct a $(1-\alpha)$-confidence set of marginal distributions $\mathcal{F}(\alpha)$ such that $\mathbb P_G[F_G \in \mathcal{F}(\alpha)] \geq 1-\alpha$, where $F_G$ denotes the marginal distribution of $Z$ under $G$, i.e., the distribution with density $f_G$ defined in (ref). From this confidence set, one can construct confidence intervals for any functional of interest $\theta(G;z)$ by optimizing over all distributions $G$ whose marginals lie in $\mathcal{F}(\alpha)$, similar to the optimization problem in (ref)(iv).\footnote{ It is possible that there is no $\widetilde{G} \in \mathcal{G}$ such that $F_{\widetilde{G}} \in \mathcal{F}(\alpha)$. If $\mathcal{F}(\alpha)$ is a valid $F$-localization, then this can happen for two reasons: we are on the low probability event that $F_G \notin \mathcal{F}(\alpha)$ or the model is misspecified. Thus the $F$-Localization approach includes an embedded specification test, similar to e.g., romano2008inference and stoye2009more.}
In this way, the $F$-localization approach translates statements concerning the uncertainty about the distribution of observables $F_G$ into statements concerning the uncertainty about the latent distribution $G$ and functionals thereof. The projection idea underlying the $F$-Localization approach traces back to the fundamental ideas of scheffe1953method and anderson1969confidence. There are several ways of constructing $F$-localizations. For instance, one generic approach that works for any univariate $Z_i$ is to use the Dvoretzky-Kiefer-Wolfowitz inequality with Massart's massart1990tight tight constant. In other cases, more specialized and refined constructions can work instead; for instance, ignatiadis2022confidence construct a $F$-localization in the Gaussian empirical Bayes problem by considering an $L_{\infty}$ neighborhood of the marginal density $f_G$. For discrete problems, such as the one considered by kline2021reasonable, methods using a $\chi^2$-based $F$-localization were already developed by lord1975empirical,lord1976interval.
An important feature of the $F$-localization principle is that it can be used to construct confidence intervals for any functional of the distribution $G$, provided that the resulting optimization problem is tractable. Moreover, all these confidence intervals have simultaneous $1-\alpha$ coverage: if $F_G \in \mathcal{F}(\alpha)$, which has probability at least $1-\alpha$ if $\mathcal{F}(\alpha)$ is a valid $F$-localization, then all the resulting confidence intervals will cover the true value of the functional with at least the desired probability. Simultaneity is a desirable property when the empirical Bayes analysis is highly exploratory as in kline2021reasonable: therein the authors consider all kinds of estimands: $\theta^{\text{discr}}(G;z)$ for different callback patterns $z$; alternative definitions of discrimination, e.g., $\mathbb P_G[p_a \neq p_b \mid Z=z]$ (again, for different $z$); unconditional discrimination probabilities such as $\mathbb P_G[p_a > p_b]$ and so forth. The $F$-localization principle allows one to construct confidence intervals for all of these estimands with simultaneous coverage.
It is common in empirical Bayes problems to discretize the space of distributions $\mathcal{G}$ koenker2014convex. For the discrimination detection problem, where $\mathcal{G}$ consists of distributions over $[0,1]^2$, we introduce a grid \smash{$\mathcal{D}_K = \{\theta_{\ell}\,:\,\ell=1,\dotsc,K^2\} \subset [0,1]^2$} (following kline2021reasonable) and represent distributions as \smash{$\widetilde{G} = \sum_{\ell=1}^{K^2} \pi_{\ell} \delta_{\theta_{\ell}}$} where $\delta_{\theta_\ell}$ denotes the Dirac measure at $\theta_\ell \in [0,1]^2$, and the weights $\pi_{\ell}$ satisfy $\pi_{\ell} \geq 0$ and $\sum_{\ell} \pi_{\ell} = 1$. Ideally, one should use as fine a grid as computationally feasible. In Figure (ref), we use, like kline2021reasonable, the above discretization scheme with varying grid sizes corresponding to $K=50$, $150$, $300$ to assess sensitivity to discretization.
Following kline2021reasonable, let us explain how this discretization turns optimization problems (i)--(iii) into linear programs. Common empirical Bayes estimands, including the discrimination estimand $\theta^{\text{discr}}(G;z)$, take the form,
for a function $h(\cdot)$, e.g., $h(\theta) = \mathds{1}(p_a > p_b)$. For such estimands, after discretization, one can solve optimization problem (ref)(iii) (and analogously, (i) and (ii)) by linear programming: $$
$$ Observe that the objective and the constraints are linear in the $\pi_{\ell}$.\footnote{ Note that $\bar{f}^{proj}$ is computed in a first step in a separate optimization problem and is treated as fixed in the linear program; by doing so, the ratio objective becomes linear in the optimization variables. The fractional programming techniques we describe below allow directly solving optimization problems with a ratio objective. } Analogously, after discretization, optimization problem~\eqref{eq:J_functional} is also a convex program that can be solved by second order conic programming (SOCP).\footnote{ Note that the first stage GMM matrix $\widehat{W}$ in~\eqref{eq:J_functional} and the bootstrap distribution of $J_n^{\mathrm{opt}}$ shown in Figure~\ref{fig:bootstrap}, also depend on the discretization. We ignore this dependence for simplicity and compute these quantities only under the $K=150$ grid.}
It turns out that we can substantially extend both the class of estimands and the constraints (beyond linear) and still maintain computational tractability that facilitates the construction of $F$-localization based confidence intervals. Concretely, consider any estimand that may be written as a ratio of linear functionals of $G$,
with $N(G;z)$ and $D(G;z)$ linear functionals of $G$.\footnote{For instance, the estimand in (ref) can be written in this way by setting $N(G;z) = \int h(\theta) p(z \mid \theta) \mathrm{d} G(\theta)$ and $D(G;z) = f_G(z)$.} Then, the optimization problem (ref)(iv) can also be solved as a SOCP using techniques from fractional programming charnes1962programming; see ignatiadis2022confidence for details. Hence, e.g., computing the lower bounds in Figure (ref) for the discrimination estimand $\theta^{\text{discr}}(G;z)$ is computationally fast even for the grid with $300^2$ points.
In some situations, because it achieves simultaneous coverage over all possible empirical Bayes estimands, $F$-localization may be overly conservative. Some recent innovations permit construction of confidence intervals that have nominal coverage for a specific estimand of interest, and so, in some cases, can be substantially shorter than $F$-localization intervals. Here, we describe two approaches, both of which account for both sources of uncertainty, partial identification and sampling variability. Discretization considerations for these methods are similar to those for $F$-localization.
This method developed in ignatiadis2022confidence provides confidence intervals for any ratio estimand of the form in (ref). The starting point is to test for each $c$ whether $\theta^{\text{ratio}}(G;z)=c$ and to obtain a confidence interval by inversion. By an Anderson-Rubin-type argument, it thus suffices to test whether $L(G;c):= N(G;z) - c D(G;z) = 0$, where $L$ is a linear functional of $G$. Given this reduction, the method proceeds by bias-aware inference using the affine minimax approach of donoho1994statistical and armstrong2018optimal carefully tailored to the empirical Bayes setting. AMARI requires a pilot $F$-Localization, and here we use the $F$-Localization implied by the constraint $J_n(f_{\widetilde{G}}, \bar{f}) \leq 13.2$ (where $\kappa=13.2$ is the $99\%$ quantile of the bootstrap distribution of $J_n^{\mathrm{opt}}$). We refer to ignatiadis2022confidence for more details.
This method was not developed for the empirical Bayes setting per say, yet here we observe that it is applicable to discrete empirical Bayes problems, such as the one here with a bivariate binomial likelihood. Using the same Anderson-Rubin-type argument as above, the confidence interval for $\theta^{\text{ratio}}(G; z)$ is the collection of values of $c$ for which the null hypothesis that $L(G;c) = 0$ can be not rejected. Since $L(G;c)$ is a linear functional of $G$, after discretization as described above, this null hypothesis can be restated as \[ \exists \pi \in \mathbb R^d_+ \text{ such that } A\pi = \beta \text{ and } a'\pi = 0~, \] where $d = K^2$, $\beta = f_G$, $A$ is a $d \times p$-dimensional matrix that encodes the bivariate binomial likelihood function, evaluated at different $z \in \mathcal{Z}$ and the grid of $K^2$ elements $(p_a,p_b) \in [0,1]^2$, and $a'\pi$ encodes the restriction that $L(G;c) = 0$. fang2023inference develop a general approach to testing such a null hypothesis. See also bai2024generalizedstrata for related applications of this methodology in causal inference.
There are further potential ways in which one can form confidence intervals, for instance, by pursuing the Anderson-Rubin-type argument above and test inversion. For instance, we could use CNS again (which above we used to facilitate $F$-Localization) to test the null hypothesis $L(G;c)=0$, see chernozhukov2023constrained. Yet another alternative (that however, in general, lacks distribution-uniform coverage) is given in d2017measuring.
Building on our previous analysis of uncertainty quantification, we now turn to a fundamental question that underlies the entire empirical Bayes approach to discrimination detection: the choice of estimand itself. The discrimination estimand in (ref) presents two challenges that intertwine with our previous discussion of uncertainty. First, the estimand $\theta^{\text{discr}}(G;z)$ is discontinuous with respect to weak convergence of measures. This discontinuity complicates interpretation as small perturbations in the ensemble $G$ can lead to large changes in this discrimination estimand.\footnote{ A similar critique also applies to common multiple testing analyses. For instance, the critique applies to the local false discovery rate $\theta^{\text{lfdr}}(G;z) := \mathbb P_G[\theta = 0 \mid Z=z]$ in the Gaussian empirical Bayes problem with $\theta \sim G$, $Z \mid \theta \sim \mathrm{N}(\theta, 1)$ mccullagh2018statistical,xiang2024interpretation. } Second, it does not reflect the magnitude of discrimination, which, in practical policy applications, is often relevant for resource allocation and enforcement prioritization. To illustrate these concerns, consider a distribution $G$ where $p_a = p_b + 10^{-10}$ almost surely. Then $\theta^{\text{discr}}(G;z) = 1$ for all $z$, suggesting complete discrimination against group $b$, even though such a small difference would not lead to any observable differences in hiring patterns. Moreover, if we slightly perturb $G$ such that $p_a = p_b$ almost surely, then $\theta^{\text{discr}}(G;z) = 0$ for all $z$. We note that while the exact-zero discrimination threshold creates technical challenges for the estimand, this binary framing aligns with certain legal frameworks such as the Civil Rights Act, where discrimination of any magnitude is in violation of the law.
Motivated by such concerns, kline2021reasonable propose the logit estimand,\footnote{kline2021reasonable also incorporate applicant quality in their estimand definition, which we omit.} $$\theta^{\text{logit}}(G; z) := \mathbb E_G\left[\Lambda\left(\Lambda^{-1}(p_a) - \Lambda^{-1}(p_b) \right) \mid Z=z\right],\quad \Lambda(p):=\frac{\exp(p)}{1+\exp(p)}.$$ The logit estimand captures differences between group callback probabilities. Building upon this foundation, we propose a complementary counterfactual estimand that offers an alternative perspective on measuring discrimination magnitude. Our estimand answers the following question: “If we were to send additional applications to an employer with callback pattern $z=(c_a,c_b)$, what is the relative probability of observing strictly more callbacks for group $a$ versus group $b$?” Formally, for employer $i$ with observed callback pattern $(C_a,C_b)=(c_a,c_b)$, consider the counterfactual experiment of sending $L'$ additional applications from each group, resulting in callbacks $C_a'$ and $C_b'$. Note that $L'$ could be different from $L$ sent in the original experiment. To define counterfactual probabilities, we assume that conditional on $\theta = (p_a, p_b)$, $C_a'$ and $C_b'$ are independent of $C_a$, $C_b$, and follow the bivariate binomial in (ref) with $L'$ trials. In this sense, we assume that our new experiment is a perfect replication of the original one except for a potentially different number of applications. See yang2024largescale for a related notion of an idealized replication experiment. With this setup, we define the “posterior callback odds ratio” given $z=(c_a,c_b)$ as
This estimand represents the odds ratio of callbacks for group $a$ versus group $b$ in a counterfactual experiment that the experimenter could actually implement. It has a natural betting interpretation: it quantifies the odds one would accept when wagering that group $a$ will receive strictly more callbacks than group $b$ (rather than strictly fewer) in a new experiment, given the observed callback pattern. For instance, if $\theta^{\text{odds}}(G; z, L') = 3$, a rational decision-maker would be willing to bet up to 3:1 odds on group $a$ receiving more callbacks than group $b$ in a counterfactual experiment with $L'$ applications per group. This provides a meaningful quantification of discrimination that is tied to outcomes.
This estimand in (ref) has several desirable properties, as we next document (see Appendix (ref) for a proof).
An important property of $\theta^{\text{odds}}(G; z, L')$ is that it can be expressed as a ratio of linear functionals of $G$:
Hence, this estimand is amenable to uncertainty quantification methods including $F$-localization described in Section (ref) and other methods described in Section (ref). Table (ref) presents confidence intervals for this estimand using $L'=4$ on the AGCV dataset. For the pattern $(C_a,C_b)=(1,0)$, all three inference methods yield intervals containing 1, suggesting insufficient evidence of systematic discrimination. In contrast, for $(C_a,C_b)=(4,0)$, all methods yield intervals strictly above 1, with AMARI providing a tighter lower bound of 17. The interpretation is operationally clear: if we send 4 more applications, we are much more likely to observe a callback pattern favoring the first applicant group.
Lastly, we note that the posterior estimand is often used to inform judgments and policy decisions regarding firms with a specific callback pattern (e.g., deciding which firms to audit or sanction). For further discussion, see the work on firm discrimination in kline2024discrimination, as well as related practices in teacher value-added models gilraine2020new and medical facility rankings gu2023invidious. The common empirical Bayes estimand typically takes the form of a posterior expectation, which ensures, for a suitable choice of loss function, that decisions are of high-quality on average. In the specific setting of kline2021reasonable, suppose $\theta^{\text{discr}}(G; z) = 0.99$, then if a policy maker were to audit all employers (or a random subset thereof) having this callback pattern $z$, only 1% of the resources would be spent on non-discriminating firms. However, for decisions of particularly high stakes, e.g., imposing sanctions on individual firms, the above criterion may not be sufficiently conservative and one may prefer a frequentist approach that provides error control for individual firms without relying on the exchangeability of all firms. See, e.g., the methods developed in mogstad2024inference and the related discussion in mogstad2022comment. Even so, the empirical Bayes approach may provide useful preliminary evidence. In such use, as emphasized in our discussion above, it is additionally important to account for uncertainty from partial identification and sampling.