The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
90,638 characters
Uncertainty in Compound Decisions
\title{Uncertainty in Compound Decisions\thanks{I thank Isaiah Andrews and Anna Mikusheva for their generous guidance and support. This paper also benefited from conversations with Alberto Abadie, Nikhil Agarwal, Victor Chernozhukov, Nathaniel Hendren, Chenxi Jiang, Andrew Komo, Haruki Kono, Soonwoo Kwon, Ricky Li, Jos\'e Luis Montiel Olea, Sarah Moon, Ashesh Rambachan, Azeem Shaikh, Yucheng Sheng, Steven Shi, Ot\'avio Tecchio, Santiago Torres, and Vod Vilfort. I additionally thank participants in the MIT econometrics lunch, third-year lunch, and second-year paper workshop for helpful comments.}}
\author{Andreas Petrou-Zeniou
\thanks{Department of Economics, MIT.
\url{[email removed]}}
}
\maketitle
\begin{spacing}{1.0}
\begin{abstract}
{\fontsize{10}{12}\selectfont
Researchers and policymakers frequently make a large number of decisions simultaneously over many units using noisy estimates of unit-level effects. Researchers often assume that unit-level effects are drawn from a common effect distribution, allowing for optimal decision making. In applications, however, this effect distribution is unknown to the econometrician, introducing uncertainty in the ultimate decision making problem. This paper considers the problem of quantifying this uncertainty. For instance, researchers may seek to understand how an oracle decision-maker with knowledge of the effect distribution would compare different feasible decisions, or understand which decisions an oracle would plausibly make. Analyzing these questions from the perspective of an oracle decision-maker involves comparing the conditional means of unit-level effects given estimates thereof, suggesting a sufficient inferential objective of uniform coverage of these conditional means. Based on an equivalence between uniform coverage of conditional means and multiple testing, I develop methods that satisfy this coverage. Finally, I consider an application to the problem of identifying high-economic-mobility neighborhoods. My methods sharpen inference relative to existing methods and identify units that can be confidently included in oracle selections.}
\end{abstract}
\end{spacing}
\onehalfspacing
\section{Introduction}
Many decision problems in economics are compound decisions (\citealt{Robbins1956}), in that they involve comparing or classifying a large number of units or populations, such as schools or neighborhoods, according to some noisily-observed feature of each unit, such as value-added. In studies of teacher or school value-added, for example, researchers seek to differentiate high-performing schools from low-performing schools (\citealt{MorgansteinWasserstein2014}, \citealt{Angristetal2025}). Relatedly, in healthcare, policymakers may seek to encourage patients to select high value-add nursing homes (\citealt{EFM2025}). In studies of employer discrimination, policymakers may seek to select a set of firms to audit for civil rights violations (\citealt{KlineWalters2021}, \citealt{Klineetal2022}, \citealt{Klineetal2024}). In problems of place-based policy-targeting, researchers may seek to distinguish between high and low economic mobility neighborhoods (\citealt{Chettyetal2014}, \citealt{Chettyetal2018aer}, \citealt{Chettyetal2026}, \citealt{ChettyandHendren2018}). To make such decisions, researchers often assume that these features are drawn independently from a common distribution, providing optimal decision rules. In many applications, this distribution of unit-level features or effects is unknown to the econometrician. As a result, the optimal decision described above is infeasible, and can be thought of as the decision made by an oracle with knowledge of the unknown distribution of effects. This article considers a range of inferential exercises that arise given uncertainty around an oracle decision-maker's actions and beliefs.
To fix ideas, consider a recent experiment due to \citealt{Bergmanetal2023}, who seek to identify high economic-mobility neighborhoods in Seattle and thus nudge low-income housing voucher recipients to move to these areas. While true economic mobility is not observed, policymakers have access to noisy estimates thereof (see e.g. \citealt{Chettyetal2018aer}, \citealt{ChettyandHendren2018}). In using these estimates to differentiate between low and high-mobility tracts, the decision-maker faces a key dilemma between selecting tracts with high and noisy estimates of mobility and tracts with lower mobility and more precise estimates. To navigate this dilemma, the oracle decision maker obtains conditional mean economic mobility at each neighborhood given these estimates and the distribution of mobility across neighborhoods. This distribution is unknown to the econometrician, and different choices of this distribution can lead to dramatically different selections of high-mobility neighborhoods. Given this uncertainty, this paper considers questions such as: can policymakers be confident that neighborhoods with noisily estimated, high mobility should be included in an oracle decision-makers selection of tracts? Given a realized selection, can we be confident that decision-making could or could not be improved by including more, or fewer, tracts with low and precisely estimated mobility? How do different ways of navigating the planner's dilemma compare according to an oracle decision-maker?
To answer the above questions, I first observe that for many decision problems considered in applied work in economics, the oracle decision-maker's actions depend on the distribution of effects only through the conditional means of unit-level effects given the realized, noisy estimates thereof. Consequently, this paper advances a notion of coverage of conditional means which holds simultaneously over potential realizations of the data. To understand why this notion of coverage suffices to address the above exercises, notice that these exercises frequently involve comparing conditional mean effects across many units. For example, analyzing the dilemma described above involves comparing many high-noise tracts with high estimated mobility to many lower-noise tracts with lower estimated mobility. Consequently, coverage of conditional means of unit-level effects given individual, potential realizations of the data, which I refer to as marginal inference, does not suffice for our objectives; coverage of individual conditional means does not guarantee simultaneous validity of these comparisons. \citealt{IgnatiadisandWager2021} develop methods satisfying this desired marginal coverage, which I extend to allow for heteroskedasticity and which I demonstrate via simulation can dramatically undercover in the exercises above.
This paper's main contribution is to develop methods that uniformly cover conditional means over potential realizations of the data in order to guard against such false discoveries. Although such uniform confidence bands can be constructed from a confidence set for the effect distribution, they should focus power on its decision-relevant features. In service of this objective, I make two observations. First, I show that the problem of constructing such uniform confidence bands is equivalent to a multiple testing problem in which the econometrician seeks to control the familywise error rate for a particular class of null hypotheses. To demonstrate the utility of this first observation, I next observe that the null hypotheses defining this multiple testing problem can be written as zero restrictions on a class of linear functionals of the effect distribution.
Combining these observations has three important consequences. First, the observations identify the decision-relevant features of the effect distribution; namely the linear functionals of the effect distribution corresponding to the null hypotheses considered. Second, the linearity of these null hypotheses on the effect distribution allows me to draw on a large literature on minimax estimation and inference for linear functionals in nonparametric settings (see e.g. \citealt{Donoho1994}, \citealt{IgnatiadisandWager2021}, \citealt{ArmstrongKolesar2018}) that adapt to the estimation difficulty of each linear functional of interest. Finally, due to my multiple testing equivalence, I build on canonical multiple-testing methods (see e.g. \citealt{RomanoWolf2005}) to obtain a step-down approach that accounts for the bias of the minimax test statistics used in the inferential procedure. I provide a theoretical analysis of this step-down procedure, and demonstrate in theorem \ref{thm:bootstrap_validity} the uniform asymptotic consistency in level of the procedure using multiplier bootstrap critical values. The key technical difficulty in obtaining this result is that when allowing for studentization, the class of minimax test statistics in this paper can grow increasingly complex in large sample sizes (see e.g. \citealt{CCK_2014}, \citealt{CCK_2016}, \citealt{CCK_ac_2014}).
My methods build on recent work due to \citealt{IgnatiadisandWager2021}, who study inference for the conditional means considered in this paper.\footnote{Work due to \citealt{Guetal2025} also considers the problem of inference on the conditional means considered in this paper. They review the methods due to \citealt{IgnatiadisandWager2021}, in addition to providing a method based on \citealt{FSST2021}.} \citealt{IgnatiadisandWager2021} provide two approaches to inference, one which satisfies uniform coverage as considered in this article, and the other which satisfies only marginal coverage. The former approach uses a particular confidence set for the effect distribution to construct uniform confidence bands, which can fail to focus power on decision-relevant features of the effect distribution. The latter, marginal, approach is based on a similar test inversion to that considered in this paper. However, the inversion in \citealt{IgnatiadisandWager2021} is based on tests that do not control the familywise error rate, and therefore does not provide the uniform guarantees desired in this paper. Nevertheless, \citealt{IgnatiadisandWager2021} derive test statistics for their null hypotheses, and I extend their arguments to allow for the precision of the estimated unit-level effects to vary across units in deriving my test statistics. Finally, the multiple testing equivalence established in this paper builds on a similar equivalence between set inference and multiple testing derived in \citealt{RomanoShaikh2010}. In fact, I establish a further equivalence between uniform confidence band construction to set inference as considered in \citealt{ChernozhukovHongTamer2007} and \citealt{RomanoShaikh2010}, which I discuss in remark \ref{rem:set-inference-connection}.
To establish the practical relevance of my methods, I consider an application to the policy-targeting task of \citealt{Bergmanetal2023}. I consider two main exercises: which neighborhoods can policymakers be confident that an oracle decision-maker would select? How would an oracle decision-maker compare different realized selections? In my application, my methods deliver substantially tighter confidence sets than existing methods due to \citealt{IgnatiadisandWager2021}, shifting economic conclusions and suggesting the step-down procedure developed in this paper is a worthwhile methodological investment for applied work.
In my first application, I consider the problem of differentiating between low and high-mobility census tracts as in \citealt{Bergmanetal2023}. For each tract, I take as data local economic mobility estimates due to \citealt{Chettyetal2018aer}, which measure the conditional mean earnings percentile in adulthood of low income children growing up in each given census tract. Given these mobility estimates, my findings suggest that many Seattle neighborhoods can be confidently included and excluded (9\% and 44\%, respectively) from an oracle selection of high-mobility tracts. In a second exercise, I show that the gaps in conditional mean mobility between feasibly-classified low and high-mobility tracts are on the order of five percentile points in adulthood earnings, at a lower confidence bound. Interpreted causally, my analyses suggest that researchers can be confident that an oracle decision-maker would expect economically large gains in the adulthood earnings of children moving to most of the high-mobility tracts selected by \citealt{Bergmanetal2023}.
My analyses are complementary to recent work studying the informativeness of noisy mobility estimates for the decision-making objectives of \citealt{Bergmanetal2023}. Existing exercises (\citealt{andrewsetal2023}, \citealt{Mogstadetal2023}, \citealt{petrouzeniou_and_shaikh_2025}) effectively compare to studying a fully-informed oracle with knowledge of true economic mobility at each tract, rather than the distribution of mobility across tracts. Building on the analysis of \citealt{AndrewsandChen2025}, the exercises considered in this paper can be viewed as providing evaluations and certifications for feasible decisions according to an oracle decision-maker with knowledge of the effect distribution, whereas existing work provides evaluations and certifications for the fully-informed oracle. The researcher's choice of inferential exercise therefore depends on the notion of an oracle decision-maker of interest to the policymaker. Given recent work providing methods to approximate the distributional oracle decision-maker in the \citealt{Bergmanetal2023} policy-targeting exercise (see e.g. \citealt{chen2024}, \citealt{barnhard2026}), the exercises in this paper complement existing inferential exercises studying the fully-informed oracle. In particular, existing work (\citealt{Mogstadetal2023}, \citealt{petrouzeniou_and_shaikh_2025}) highlights that economic mobility estimates are too noisy to draw informative conclusions on the actions of a fully-informed oracle decision-maker. In contrast, my results suggest that in policy-targeting tasks using mobility estimates, researchers can instead draw far more informative conclusions when considering an oracle decision-maker with knowledge of the distribution of mobility across geographies.
The remainder of the paper proceeds as follows. Section \ref{sec:setup} introduces the compound decision problem via the selection problem of \citealt{Bergmanetal2023}, my inferential objectives, and a set of further examples. Section \ref{sec:equivalence_results} discusses a key simplification of my inferential objectives by drawing a connection to set inference and multiple testing. Section \ref{sec:methods} leverages this connection to develop a step-down approach to uniform confidence band construction. Section \ref{sec:economic-mobility-application} applies my methods to the setting of \citealt{Bergmanetal2023}. Section \ref{sec:conclude} concludes. The appendix contains proofs, details on applications, and an additional application to the setting of \citealt{Klineetal2022}.
\section{Setup, Notation, and Motivating Examples}
\label{sec:setup}
This section introduces the compound decision problems considered in this paper. I will use as a running example the selection problem of \citealt{Bergmanetal2023} described in the previous section, though I conclude with a range of further examples to highlight the range of compound decision problems considered in this paper. In general, I consider units $i=1,\ldots,n$, each associated with a unit-level feature or effect $\theta_i$. These unit-level effects are unobserved to the econometrician, but may be observed with noise. I denote by $\hat\theta_i$ this estimate, and I denote by $\sigma_i$ the precision with which $\hat\theta_i$ is estimated. Both $\hat\theta_i$ and $\sigma_i$ are known. Formally, $\hat\theta_i|\theta_i,\sigma_i$ is drawn from a Gaussian centered at $\theta_i$ with variance $\sigma_i^2$. In the context of the targeting exercise of \citealt{Bergmanetal2023}, these $\theta_i$ denote economic mobility at each tract -- the conditional earnings percentile of low-income children growing up in tract $i$ -- and the corresponding $\hat\theta_i$ denote estimates thereof.
To relate unit-level effects across units, I assume that $\theta_i$ are drawn iid across units from a common distribution of unit-level effects $G$, which can be thought of as the distribution of economic mobility across census tracts. Consequently, the data are sampled according to the following model:
\begin{equation}
\label{eq:EB_DGP}
\hat\theta_i|\theta_i,\sigma_i
\overset{ind}{\sim}\mathcal{N}(\theta_i,\sigma_i^2),~
(\theta_i,\sigma_i)\overset{iid}{\sim}G
\end{equation}
In what follows, I will assume that $G$ lies in a convex class $\mathcal{G}$. I will also assume that the convex class $\mathcal{G}$ is dominated by the Lebesgue measure,\footnote{More generally, the methods developed in this paper allow $\mathcal{G}$ to be dominated by some known measure $\nu$.} such that the density of each $G$ with respect to the Lebesgue measure is given by $g_G$. The normality of effect estimates $\hat\theta_i$ can be motivated by the central limit theorem, since the $\hat\theta_i$ are often distributed as sample means.\footnote{When the data are exchangeable, the existence of a common distribution generating latent unit-level effects can be justified by de Finetti's theorem. The data can be treated as exchangeable when restricting attention to permutation equivariant decision rules. See \citealt{KoenkerGuForthcoming} for further discussion.} I additionally allow for dependence between unit-level effects and the precision with which these effects are estimated.\footnote{For instance, low-income families may be less likely to reside in high-mobility areas, meaning that on average, estimates of mobility in high-mobility areas are noisier than in low-mobility areas. See \citealt{chen2024} for further discussion. The dependence between $\theta$ and $\sigma$ is restricted by $\mathcal{G}$, and conditions must be imposed for bootstrap validity to hold.}
Policymakers often seek to make unit-level decisions for each $i$. For example, a policymaker may seek to label a subset of the $n$ tracts as high-mobility. To this end, define a decision $\delta_i$ lying in $\{0,1\}$, where $\delta_i = 1$ denotes selecting tract $i$. The policymaker can only select one-third of tracts, so I assume that the vector of classifications $\delta_i$ satisfies $\sum_i \delta_i = \lfloor n/3\rfloor$. As in \citealt{Bergmanetal2023}, the policymaker may seek to maximize a welfare function consisting of total economic mobility among the selected tracts:
\begin{equation}
\label{eq:util_max_welfare_defn}
W(\boldsymbol{\delta},\boldsymbol{\theta}) := \sum_i \delta_i\theta_i~.
\end{equation}
This welfare function depends on tract-level mobility, which are unknown to the econometrician. Consequently, the decision-maker computes the average welfare by taking expectations over tract-level mobility. The decision-maker's goal is therefore to choose a map from data $(\boldsymbol{\hat\theta},\boldsymbol{\sigma})$ to tract-level selections in $\{0,1\}^n$ that maximizes the total mobility among selected tracts, on average:
\begin{equation}
\label{eq:bayesian_decision_problem}
\boldsymbol{\delta}^*(\cdot) = \operatorname*{argmax}_{\boldsymbol{\delta}} \mathbb{E}_G W(\boldsymbol{\delta},\boldsymbol{\theta})~.
\end{equation}
To solve \eqref{eq:bayesian_decision_problem}, it suffices to maximize conditional expected welfare given the realized data.\footnote{This is a consequence of the law of iterated expectations.} Namely, for the welfare defined in \eqref{eq:util_max_welfare_defn}:
\begin{equation}
\label{eq:optimal_decision_selection}
\boldsymbol{\delta}^*(\boldsymbol{\hat\theta},\boldsymbol{\sigma}) = \operatorname*{argmax}_{\substack{\boldsymbol{\delta}\in\{0,1\}^n\\ \sum_i\delta_i = \lfloor n/3\rfloor}}\mathbb{E}_G \left(
W(\boldsymbol{\delta},\boldsymbol{\theta}) |\boldsymbol{\hat\theta},\boldsymbol{\sigma}
\right) =\operatorname*{argmax}_{\substack{\boldsymbol{\delta}\in\{0,1\}^n\\ \sum_i\delta_i = \lfloor n/3\rfloor}}\sum_i\delta_i\mathbb{E}_G \left( \theta_i|\hat\theta_i,\sigma_i\right) ~.
\end{equation}
Towards interpretation, the above provides that to analyze the decision problem \eqref{eq:bayesian_decision_problem}, it suffices to consider the total conditional mean economic mobility across selected tracts. Moreover, because the total conditional mean economic mobility across selected tracts is additively separable in the conditional mean mobility at each tract, it suffices to consider the conditional mean economic mobility at each individual tract. Therefore, the problem \eqref{eq:optimal_decision_selection} admits a simple solution: it is optimal to select the top-third of tracts according to their conditional mean economic mobility.
The optimal selection of census tracts described in \eqref{eq:optimal_decision_selection} depends on the distribution of latent effects $G$, which is unknown to the econometrician. In this sense, the decision $\boldsymbol{\delta}^*$ described in \eqref{eq:optimal_decision_selection} is that of an oracle decision-maker with knowledge of the effect distribution $G$. Given uncertainty surrounding $G$, how should researchers evaluate uncertainty surrounding the downstream, oracle decision-maker's objectives?
\subsection*{Consequences of Effect Distribution Uncertainty}
To illustrate the ambiguity relating to decision-making that arises due to uncertainty in $G$, I notice that the optimal selection of tracts can vary substantially with the true latent distribution of effects. To demonstrate this, recall the planner's dilemma: should a policymaker select noisy tracts with high estimated mobility, or less noisy tracts with lower estimated mobility? More precisely, consider two tracts $h$ and $\ell$, such that estimated mobility at tract $h$ exceeds that at tract $\ell$, namely $\hat\theta_h>\hat\theta_\ell$, but estimation precision at tract $h$ is lower than at $\ell$, namely $\sigma_h > \sigma_\ell$. Whether the oracle decision-maker selects tract $h$ over tract $\ell$ depends on whether the conditional mean mobility $\mathbb{E}_G(\theta_h|\hat\theta_h,\sigma_h)$ at tract $h$ exceeds that at tract $\ell$ given by $\mathbb{E}_G(\theta_\ell|\hat\theta_\ell,\sigma_\ell)$. The oracle decision-maker may naturally penalize the estimate $\hat\theta_h$ more than $\hat\theta_\ell$, since it is more noisily estimated and thus less informative of true mobility $\theta_h$. In this regard, the oracle decision-maker's selection depends on the magnitude of the gap $\hat\theta_h-\hat\theta_\ell$. Moreover, mobility may be positively correlated with estimation precision in a phenomenon \citealt{chen2024} refers to as precision dependence: fewer low-income children grow up in high-mobility tracts, and therefore economic mobility is more noisily-estimated in high-mobiltiy neighborhoods. This introduces the opposite effect to the above: if noisily estimated tracts tend to be higher-mobility, the oracle decision-maker may prefer to select tract $h$ over tract $\ell$.
While the oracle decision can vary widely with the distribution of effects generating the data, the joint distribution of the observed data $(\hat\theta_i, \sigma_i)$ contains information on the joint distribution $G$ of unit-level effects and estimation precision $(\theta_i,\sigma_i)$. For example, suppose that $\sigma_i = \sigma$ for all units $i$. In this case, the variance of estimated mobility $\hat\theta_i$ above the variance of the estimation error $\sigma_i$ provides information on the variance, or dispersion, of the distribution of mobility. Similarly, allowing $\sigma_i$ to vary across units, the correlation between estimated mobility $\hat\theta_i$ and estimation precision $\sigma_i$ is informative of the correlation between true mobility and estimation precision. Consequently, some optimal selections of tracts are more plausible than others given the observed data.
To provide examples of two plausible effect distributions, consider figure \ref{fig:two_consistent_priors}. Given an effect distribution $G$, let $F_G$ denote the joint distribution of the observed data $(\hat\theta_i,\sigma_i)$ assuming the data are generated by $G$. While the true joint distribution of the observed data is unknown, its empirical counterpart is known, from which a confidence set for the true $F_G$ can be constructed. I use the economic mobility estimates due to \citealt{Chettyetal2018aer} to obtain this empirical joint distribution and corresponding confidence set for $F_G$. In figure \ref{fig:two_consistent_priors}, I plot the densities of two effect distributions, $G_1$ and $G_2$, for which $F_{G_i}$ lies in this confidence set. In this sense, the mobility distributions $G_1$ and $G_2$ are both consistent with the observed data. Visual inspection of figure \ref{fig:two_consistent_priors} suggests that different patterns of precision dependence (i.e. correlation between mobility and estimation precision) can be consistent with the data. $G_1$, plotted in the left panel, displays positive dependence, consistent with the intuition described above. On the other hand, $G_2$ displays negative dependence.
\begin{figure}[H]
\centering
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[
width=\linewidth,
height=4.7cm,
keepaspectratio
]{figs/oa_illustrative_density_G4.png}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[
width=\linewidth,
height=4.7cm,
keepaspectratio
]{figs/oa_illustrative_density_G5.png}
\end{minipage}
\caption{Two distributions $G_1$ and $G_2$ consistent with the empirical distribution of the observed data. The left panel plots the density of $G_1$, and the right panel plots the density of $G_2$.}
\label{fig:two_consistent_priors}
\end{figure}
These differing dependence patterns in the distribution of economic mobility can yield different optimal decisions. To see this, I plot the resulting optimal selection of tracts treating both $G_1$ and $G_2$ as the true effect distributions in figure \ref{fig:two_consistent_decisions}. Visual inspection of figure \ref{fig:two_consistent_decisions} validates the intuition relating precision dependence to downstream decisions. When the effect distribution displays positive dependence between mobility and estimation precision, the resulting oracle decision-maker is more likely to select high-noise, high-mobility tracts, and vice versa. Comparing panels 1 and 2 of figure \ref{fig:two_consistent_decisions} suggests that different mobility distributions consistent with the mobility estimates of \citealt{Chettyetal2018aer} can yield drastically different decisions. Ambiguity around the distribution generating economic mobility across neighborhoods leads to ambiguity about the downstream decision.
\begin{figure}[H]
\centering
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[
width=\linewidth,
height=4.7cm,
keepaspectratio
]{figs/oa_illustrative_selection_G4.png}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[
width=\linewidth,
height=4.7cm,
keepaspectratio
]{figs/oa_illustrative_selection_G5.png}
\end{minipage}
\caption{Optimal decisions given $G_1$ and $G_2$. The left panel displays the optimal decision given $G_1$, and the right panel displays the optimal decision given $G_2$. Seattle census tracts are plotted according to their $(\hat\theta_i, \log\sigma_i)$. Selected tracts are colored in teal, and are otherwise colored in copper.}
\label{fig:two_consistent_decisions}
\end{figure}
\subsection*{Appropriate Uncertainty Quantification}
Uncertainty around $G$ translates into uncertainty around the oracle decision-makers beliefs and actions. For example, can researchers be confident that an oracle decision-maker would select a given tract with high-estimated mobility over other tracts? Relatedly, which tracts can researchers be confident that an oracle decision-maker would select? Can researchers be confident that an oracle decision-maker prefers high-mobility low-precision tracts to low-mobility, high-precision tracts, and if so, by how much? For any two feasible decisions, can we be confident that an oracle decision-maker would prefer one over the other?
To address such questions, this paper proposes a notion of uniform coverage of conditional means. Let $\mu_G(t,s) := \mathbb{E}_G(\theta_i|\hat\theta_i = t, \sigma_i=s)$. Formally, we seek $\widehat{CS}(\cdot,\cdot)$ which satisfies:
\begin{equation}
\label{eq:uniform_coverage}
\liminf_{n\to\infty}\inf_{G\in\mathcal{G}} P_{G}\left(
\mu_G(t,s) \in \widehat{CS}(t,s) ~ \text{for all}~ t,s
\right)\geq 1-\alpha~.
\end{equation}
To see why confidence sets satisfying \eqref{eq:uniform_coverage} suffice to address the inferential questions above, notice that such exercises involve pairwise comparisons of conditional mean mobility across many tracts. For example, determining the set of tracts in the oracle optimal selection involves determining those tracts for which conditional mean economic mobility is relatively large compared to other tracts. Evaluating the oracle decision-maker's dilemma between selecting noisy tracts with high estimated mobility and lower-mobility, lower-noise tracts involves comparing pairs of many such tracts. Comparing decisions involves comparing the conditional mean mobility at tracts selected by one decision but not the other. I therefore propose \eqref{eq:uniform_coverage} as a method that guarantees simultaneous validity of these comparisons.
It is instructive to compare confidence sets $\widehat{CS}(\cdot,\cdot)$ satisfying \eqref{eq:uniform_coverage} to those satisfying marginal coverage:
\begin{equation}
\label{eq:marginal_coverage}
\liminf_{n\to\infty}\inf_{G\in\mathcal{G}} P_{G}\left(
\mu_G(t,s) \in \widehat{CS}(t,s)
\right)\geq 1-\alpha~ \text{for all}~ t,s~.
\end{equation}
While \eqref{eq:marginal_coverage} guarantees coverage of individual conditional means, it can fail to cover the many comparisons of conditional means that allow researchers to evaluate decisions. In simulations calibrated to the economic mobility estimates of \citealt{Chettyetal2018aer}, I compare the coverage properties of an approach providing marginal coverage,\footnote{Which extends methods developed in \citealt{IgnatiadisandWager2021} to allow for dependence between the $\theta_i$ and $\sigma_i$.} with the methods providing simultaneous coverage developed in this paper. Figure \ref{fig:coverage_sims} provides the results from this simulation. The left panel compares simultaneous coverage (the expression on the left-hand side of \eqref{eq:uniform_coverage}) for both approaches. In the right panel, I use confidence sets for conditional mean mobility to obtain a confidence set for a top-third selection according to conditional means over a grid of representative $\hat\theta_i, \sigma_i$ pairs. This latter confidence set reports the tracts which researchers can confidently include or exclude in an oracle optimal selection. The coverage results for this latter exercise are displayed in the right panel of figure \ref{fig:coverage_sims}.
\begin{figure}[H]
\centering
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[
width=\linewidth,
height=4.7cm,
keepaspectratio
]{figs/oa_sims_simult_coverage_posterior_means.png}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[
width=\linewidth,
height=4.7cm,
keepaspectratio
]{figs/oa_sims_simult_coverage_top_third_grid.png}
\end{minipage}
\caption{Simulation study evidence for coverage for methods satisfying \eqref{eq:uniform_coverage}, plotted in teal, and \eqref{eq:marginal_coverage}, plotted in copper. The left panel plots simultaneous coverage of the conditional means $\mu_G(\hat\theta_i,\sigma_i)$, while the right panel plots coverage of the subset of tracts lying in the top-third according to their $\mu_G(\hat\theta_i,\sigma_i)$, i.e. the oracle optimal selection. }
\label{fig:coverage_sims}
\end{figure}
Crucially, visual inspection of figure \ref{fig:coverage_sims} suggests that, not only do marginal confidence sets satisfying \eqref{eq:marginal_coverage} dramatically undercover when targeting simultaneous coverage of conditional means, but moreover dramatically undercover when used to make statements about the oracle decision-maker's problem. These are the exact statements of interest in this article. Consequently, this paper considers the problem of constructing confidence sets satisfying \eqref{eq:uniform_coverage}.
\subsection*{Further Examples}
\label{subsec:further_exmp}
To demonstrate the broad applicability of the setting described in this section, I list a few decision problems distinct from the selection problem of \citealt{Bergmanetal2023} considered thus far that arise in applied work in economics. In each of the examples described below, the oracle optimal decision depends on $G$ only through the conditional means of effects, or functions of the effects, given the observed data $\hat\theta_i$ and $\sigma_i$. Consequently, the same uncertainty quantification exercises considered above can be considered in each of the applications below using the method satisfying \eqref{eq:uniform_coverage} developed in this paper.
\begin{exmp}
\label{ex:selection_exmp_1}
\textbf{(Selection)} The selection problem described above is not limited to the economic mobility application of \citealt{Bergmanetal2023}. For instance, \citealt{Angristetal2025} consider the problem of selecting the highest value-added school, given a choice set. \citealt{EFM2025} suggest that nudging patients to move from low to high value-added nursing homes within their market could improve patients' outcomes in underserved, rural areas.
In each of these problems, the welfare in \eqref{eq:util_max_welfare_defn} characterizes the decision-maker's objective of maximizing total value-added at selected neighborhoods, schools, or nursing homes. To capture different constraints on the decision-makers selection (i.e. selection of the best school, or the highest value-add nursing home in a given market), one can impose different constraints on the decision rule $\boldsymbol{\delta}$.
\end{exmp}
\begin{exmp}
\label{ex:util_max_1}
\textbf{(Utility Maximization)} A related problem to the above is that of utility, or welfare-maximization. For example, \citealt{Klineetal2022} are interested in selecting discriminatory employers $i$ with black-white callback differentials $\theta_i$, subject to firm-specific audit costs $K_i$. A positive callback differential $\theta_i$ corresponds to discrimination against black applicants, and vice-versa. Restricting the unit-level decision $\delta_i\in\{0,1\}$, I obtain the following welfare:\footnote{See section 11 in \citealt{Klineetal2022}, or \citealt{Chenetal2025} for further discussion of utility maximization in this setting.}
\begin{equation*}
\label{eq:util_max_loss}
W^{(util)} (\boldsymbol{\delta}, \boldsymbol{\theta}) = \sum_{i=1}^n \delta_i(\theta_i-K_i)~.
\end{equation*}
Similar welfare functions, albeit with different action spaces and constraints, arise naturally in the methodological literature on empirical welfare analysis as discussed in \citealt{Moon2025}, who studies welfare maximization in the context of the Marginal Values of Public Funds (MVPF) of \citealt{HendrenandSprung_Keyser2020}. For a welfare more tailored to discrimination applications, we consider a welfare where $\delta_i \in \{0,1\}^2$:
\begin{equation*}
\label{eq:util_max_loss_ts}
W^{(util2)} (\boldsymbol{\delta}, \boldsymbol{\theta}) = \sum_{i=1}^n \delta_{i,1}(\theta_i-K_i) + \sum_{i=1}^n \delta_{i,2}(-\theta_i-K_i)~.
\end{equation*}
This two-sided welfare corresponds to auditing for discrimination against either black or white applicants. Under the welfare above, the Bayes-optimal decision consists of the thresholding rules $\delta_{i,1}^{OB}(G) := \mathbb{1}(\mathbb{E}_G(\theta_i|\hat\theta_i,\sigma_i) \geq K_i)$ and $\delta_{i,2}^{OB}(G) := \mathbb{1}(\mathbb{E}_G(-\theta_i|\hat\theta_i,\sigma_i) \geq K_i)$.
\end{exmp}
\begin{exmp}
\label{ex:high_dim_test_1}
\textbf{(High-Dimensional Testing)} Local false discovery rate control can be interpreted as a compound decision (see \citealt{SunandCai2007}, \citealt{xie2011optimal}). The discrimination correspondence study of \citealt{Klineetal2022}, considers the problem of detecting discriminatory firms. Restricting $\delta_i\in\{0,1\}$, one can interpret $\delta_i=1$ as labeling firm $i$ as discriminatory against black applicants. The following welfare for $\lambda\in(0,1)$:
\begin{equation*}
\label{eq:test_loss}
W^{(test)}(\boldsymbol{\delta},\boldsymbol{\theta}) = -\left\{\sum_{i=1}^n \underbrace{\lambda \delta_i\mathbb{1}(\theta_i\le 0)}_{\text{False Discovery}} + \underbrace{(1-\lambda)(1-\delta_i)\mathbb{1}(\theta_i>0)}_{\text{False Non-Discovery}}\right\}~,
` \end{equation*}
corresponds to trading off false discoveries against false non-discoveries. In addition to the application of \citealt{Klineetal2022}, \citealt{GuKoenker2023} consider a similar welfare function to the above, derive the Bayes-optimal decision rule, and apply their methods to the problem of ranking U.S. dialysis centers. Under the welfare above, the Bayes-optimal decision follows a thresholding rule $\delta_i := \mathbb{1}(P_G(\theta_i\geq 0|\hat\theta_i,\sigma_i) \geq \lambda)$. Here, the posterior probability of discrimination against black applicants characterizes the Bayes-optimal decision. The methods in this paper extend to address cases when the conditional means of interest are $\mathbb{E}_G( b(\theta_i)|\hat\theta_i,\sigma_i)$ for arbitrary $b$, meaning that decisions based on conditional probabilities can be addressed using my methods.
\end{exmp}
\section{Equivalence to Multiple Testing}
\label{sec:equivalence_results}
Our goal is to obtain confidence sets satisfying \eqref{eq:uniform_coverage}. Before introducing the key observations that underlie the approach to inference developed in this paper, I first note that any confidence set for the effect distribution $G$ suffices to obtain confidence sets satisfying \eqref{eq:uniform_coverage}. However, such confidence sets for $G$ also provide simultaneous confidence bounds for any feature of $G$, not only the conditional means of interest to the oracle decision-maker of study in this paper. Consequently, naive constructions of such confidence sets for $G$ may be conservative for the problem \eqref{eq:uniform_coverage}. The desideratum motivating the approach to inference developed in this paper, informally stated, is to obtain confidence bands satisfying \eqref{eq:uniform_coverage} while focusing power on the conditional means of interest to the oracle decision-maker.
In service of this objective, I provide two key observations that allow a tractable approach to inference satisfying \eqref{eq:uniform_coverage}. I first establish an equivalence between \eqref{eq:uniform_coverage} and multiple testing, where the econometrician seeks to control the familywise error rate for a particular class of null hypotheses. I then obtain a characterization of these null hypotheses in terms of linear functionals of the effect distribution $G$, and conclude by remarking on the utility of these two observations.
To establish the first equivalence, I first define for each $t$ in $\mathbb{R}$ corresponding to a realized unit-level effect estimate, for each $s > 0$ corresponding to realized unit-level estimate precision, and for each $c$ in $\mathbb{R}$ corresponding to a potential value of conditional mean economic mobility, the following hypothesis:
\begin{equation}
\label{eq:null_hypotheses}
H_{c,t,s}:\mu_G(t,s) = c~.
\end{equation}
This hypothesis corresponds to the statement that the conditional mean effect $\theta_i$ at a tract satisfying $\hat\theta_i = t$ and $\sigma_i=s$ equals $c$. Consider the problem of testing the family of null hypotheses $H_{c,t,s}$ while controlling the familywise error rate:
\begin{equation*}
\text{FWER}_G := P_G\left(
\left\{
\text{Reject at least one $(c,t,s)$ such that $H_{c,t,s}$ holds.}
\right\}
\right)~.
\end{equation*}
In particular, the following proposition establishes equivalence between the problem of obtaining a testing procedure satisfying asymptotic control of the familywise error rate:
\begin{equation}
\label{eq:fwer_control}
\limsup_{n\to\infty}\sup_{G\in\mathcal{G}}\text{FWER}_G \le \alpha~,
\end{equation}
and of obtaining a confidence set satisfying \eqref{eq:uniform_coverage}.
\begin{prop}
\label{prop:MHT_equivalence}
Given any confidence band satisfying \eqref{eq:uniform_coverage}, a testing procedure satisfying \eqref{eq:fwer_control} can be constructed. Similarly, given any testing procedure satisfying \eqref{eq:fwer_control}, a confidence band satisfying \eqref{eq:uniform_coverage} can be constructed. In either case, the coverage probability of the confidence band equals one minus the familywise error rate of the corresponding test:
\begin{equation*}
P_{G}\left(
\mu_G(t,s) \in \widehat{CS}(t,s) ~ \text{for all}~ t,s
\right) = 1-\text{FWER}_G~.
\end{equation*}
\end{prop}
\begin{proof}
To see this, for any confidence band $\widehat{CS}(\cdot,\cdot)$, define $\phi_{c,t,s}$ as follows:
\begin{equation}
\label{eq:construct_phi_from_cs}
\phi_{c,t,s} := \mathbb{1}(c\not\in \widehat{CS}(t,s))~.
\end{equation}
Similarly, \eqref{eq:construct_phi_from_cs} can be used to construct $\widehat{CS}(\cdot,\cdot)$ from $\phi_{c,t,s}$. The following equivalence of events:
\begin{align*}
\{\exists (t,s):\mu_G(t,s) \not\in \widehat{CS}(t,s)\} &= \{
\exists (t,s):\phi_{\mu_G(t,s),t,s} = 1
\}\\
& = \{
\exists (c,t,s): \text{$H_{c,t,s}$ is true, $\phi_{c,t,s}$ rejects}
\}
\end{align*}
yields the proposition.
\end{proof}
Proposition \ref{prop:MHT_equivalence} is based on a construction where, for a family of tests of the hypotheses \eqref{eq:null_hypotheses}, the surviving nulls correspond to inclusions in a uniform confidence band, and vice versa. Consequently, the event that the uniform confidence band fails to cover conditional mean effects at some potential realization of the data corresponds to a false discovery in the corresponding multiple testing problem. Figure \ref{fig:MHT_equivalence} visualizes this intuition:
\begin{figure}[H]
\centering
\begingroup
\definecolor{fdCopper}{RGB}{182,116,61}
\begin{tikzpicture}[
x=1cm,
y=1cm,
line cap=round,
line join=round,
>={Stealth[length=4pt,width=3.3pt]},
declare function={
fdmean(\x)=0.19+0.55/(1+exp(-4.2*(\x-0.48)));
fdwlo(\x)=0.080+0.030*(2*\x-1)^2;
fdwup(\x)=0.045+0.018*(2*\x-1)^2;
fdmove(\x)=0.15*exp(-((\x-0.64)/0.29)^2);
fdlo(\x)=fdmean(\x)-fdwlo(\x)-fdmove(\x);
fdup(\x)=fdmean(\x)+fdwup(\x)-fdmove(\x);
}
]
\begin{scope}[
shift={(1.05,0.55)},
x=7.90cm,
y=4.65cm
]
\path[fill=black!12]
plot[domain=0:1,samples=180,variable=\x]
(\x,{fdlo(\x)})
--
plot[domain=1:0,samples=180,variable=\x]
(\x,{fdup(\x)})
-- cycle;
\draw[black!35,dashed,line width=.55pt]
plot[domain=0:1,samples=180,variable=\x]
(\x,{fdlo(\x)});
\draw[black!35,dashed,line width=.55pt]
plot[domain=0:1,samples=180,variable=\x]
(\x,{fdup(\x)});
\draw[->,black!80,line width=.65pt]
(0,0)--(1.06,0)
node[
anchor=north west,
font=\fontsize{9}{11}\selectfont
] {$t$};
\draw[->,black!80,line width=.65pt]
(0,0)--(0,.92)
node[
anchor=south,
font=\fontsize{9}{11}\selectfont
] {$c$};
\draw[black!25,densely dotted,line width=.55pt]
(0.64,0)--(0.64,{fdmean(0.64)});
\draw[black!65,line width=.55pt]
(0.64,-.010)--(0.64,.010);
\node[
anchor=north,
yshift=-2pt,
font=\fontsize{9}{11}\selectfont
]
at (0.64,0) {$t_0$};
\draw[teal,line width=1.25pt]
plot[domain=0:1,samples=180,variable=\x]
(\x,{fdmean(\x)});
\node[
anchor=west,
text=black,
xshift=4pt,
font=\fontsize{10}{12}\selectfont\mdseries
]
at (1,{fdmean(1)})
{$\mu_G(t,s_0)$};
\node[
anchor=south,
align=center,
text=black,
inner sep=2pt,
font=\fontsize{10}{12}\selectfont\mdseries
]
(fdfalse) at (0.64,.79)
{\textcolor{fdCopper}{False discovery}\\[-.3ex]
Reject $H_{\mu_G(t_0,s_0),\,t_0,\,s_0}$};
\draw[
->,
line width=.85pt,
shorten <=2pt,
shorten >=1.5pt
]
(fdfalse.south)--(0.64,{fdmean(0.64)});
\end{scope}
\end{tikzpicture}
\endgroup
\caption{A false rejection of
$H_{\mu_G(t_0,s_0),t_0,s_0}$ corresponds to failure of the confidence
band to cover the true conditional mean at $(t_0,s_0)$.}
\label{fig:MHT_equivalence}
\end{figure}
\subsection*{Simplifying the Null Hypotheses}
The above proposition \ref{prop:MHT_equivalence} establishes equivalence between the problem of interest in this paper, \eqref{eq:uniform_coverage}, and the classical problem of multiple hypothesis testing with familywise error rate control (see e.g. chapter 9 of \citealt{lehmann_romano_2022}). While this is a useful characterization, it is unclear how the representation afforded by proposition \ref{prop:MHT_equivalence} helps obtain an inferential procedure satisfying both \eqref{eq:uniform_coverage} and the inferential desideratum. In what follows, I obtain a characterization of the null hypotheses $H_{c,t,s}$ defining this multiple testing problem in terms of linear functionals of $G$. This characterization identifies the decision-relevant features of $G$, and thus helps address the inferential desideratum introduced above.
Extending arguments due to \citealt{IgnatiadisandWager2021} provides a convenient characterization of the null hypotheses $H_{c,t,s}$ as restrictions on linear functionals of the effect distribution $G$. To this end, notice that conditional mean effects can be written:
\begin{equation}
\label{eq:fractional_characterization}
\mu_G(t,s) = \frac{\int \theta\varphi\left(
\frac{t-\theta}{s}
\right)g_G(s,\theta)d\theta}{\int \varphi\left(
\frac{t-\theta}{s}
\right)g_G(s,\theta)d\theta}~.
\end{equation}
To understand the consequences of \eqref{eq:fractional_characterization}, notice that the numerator and denominator of the right hand side of the above are both linear functionals of the effect distribution $G$. Consequently, the hypotheses $H_{c,t,s}$ can be rewritten:
\begin{equation*}
H_{c,t,s}:L_{(c,t,s)}(G):=\int \left(\theta-c\right)\varphi\left(
\frac{t-\theta}{s}
\right)g_G(s,\theta)d\theta = 0~.
\end{equation*}
Observe that $L_{(c,t,s)}(G)$ is a linear functional of $G$. To simplify notation, let $\gamma$ denote the triple $(c,t,s)$, and rewrite the above family of hypotheses as $H_\gamma:L_\gamma(G) = 0$. Let $\Gamma$ denote $\mathbb{R}^2\times\mathbb{R}_+$, which contains each $\gamma$, and let $\Gamma(G)$ denote the collection of true nulls $\{\gamma\in\Gamma:H_\gamma\text{ holds}\}$.
Consequently, the problem \eqref{eq:uniform_coverage} is not only equivalent to a multiple testing problem, but to a multiple testing problem where the null hypotheses entail restrictions on linear functionals of the effect distribution $G$. First, notice that this characterization isolates the features of the effect distribution $G$ to which the econometrician should focus power: the linear functionals $L_\gamma(G)$. Moreover, this characterization identifies the exact features of $G$ that an inferential procedure must cover: the $L_\gamma(G)$ such that $L_\gamma(G) = 0$. That is, to obtain a testing procedure satisfying \eqref{eq:fwer_control}, it is necessary and sufficient to cover the $L_\gamma(G)$ for which $H_\gamma$ holds. To build on this intuition, section \ref{sec:methods} develops a step-down approach to inference that seeks to protect exactly these null linear functionals, while focusing power on the collection of functionals $(L_\gamma(G))_{\gamma\in \Gamma}$.
\begin{rem}
\label{rem:set-inference-connection}
This paper is not the first to remark on equivalence between problems of uniform coverage as in \eqref{eq:uniform_coverage} and multiple hypothesis testing. Work due to \citealt{RomanoShaikh2010} establishes a similar equivalence between inference for the partially identified set defined by criterion functions (see e.g. \citealt{ChernozhukovHongTamer2007}). Our proposition \ref{prop:MHT_equivalence} is closely related to these previous equivalence results. In particular, \eqref{eq:uniform_coverage} can be alternatively viewed as a special case of the problem of inference for the partially identified set: coverage of the conditional mean function as in \eqref{eq:uniform_coverage} is equivalent to coverage of the zero-level set of $L_\gamma(G)$ as functions of $\gamma$, which coincides with the set of true nulls $\Gamma(G)$. Motivated by their equivalence results, \citealt{RomanoShaikh2010} obtain a step-down approach to inference for the identified set, providing further motivation for the step-down methods developed in section \ref{sec:methods}.
\end{rem}
\section{Methods}
\label{sec:methods}
This section develops methods satisfying \eqref{eq:uniform_coverage}, applying the equivalence results developed in section \ref{sec:equivalence_results}. In particular, the results of section~\ref{sec:equivalence_results} suggest that it is necessary and sufficient to construct a familywise-error-rate-controlling family of tests of the hypotheses $\left\{H_\gamma:L_\gamma(G)=0\right\}_{\gamma\in\Gamma}$. My methods draw on a classical literature on multiple testing \citep{RomanoWolf2005, RomanoShaikh2010}. The present setting differs from the classical settings in the key respect that estimators of the $L_\gamma(G)$ need not be unbiased.\footnote{\citealt{Pensky2015} demonstrates that in the homoskedastic version of the model considered in \eqref{eq:EB_DGP}, the minimax rate of estimation for many linear functionals of $G$ is not $n^{-1/2}$. Unbiased estimation therefore entails estimators with variance that grows with $n$. It therefore may be desirable to introduce bias in estimation to allow for reduced variance in estimation.}
To address this departure from the classical setting, I develop a bias-aware, step-down procedure for inference that takes arbitrary estimators of the $L_\gamma(G)$ and allows for biased estimation in subsection \ref{sec:step_down_details}. To obtain test statistics, I build on work due to \citealt{Donoho1994} and \citealt{IgnatiadisandWager2021} to derive minimax estimators which, for each functional $L_\gamma(G)$, minimize worst-case bias subject to a constraint on variance. I provide a high-level description of these test statistics in remark \ref{rem:choice-of-t-stat}, and provide an in-depth description and derivation of these test statistics in appendix \ref{app:minimax_details}. Given a bias-aware step-down procedure, I turn to asymptotics. Proposition \ref{prop:step_down_validity_est_cv} shows that this step-down procedure is uniformly asymptotically valid under certain validity and monotonicity conditions on critical values, stated in assumption \ref{assu:cv_validity_monotonicity}. Theorem \ref{thm:bootstrap_validity} verifies these conditions for multiplier bootstrap critical values. Verifying these conditions is complicated by the fact that the minimax test statistics need not be Donsker, or even VC-type, after studentization.
\subsection{Step-down Procedure for Inference}
\label{sec:step_down_details}
By proposition \ref{prop:MHT_equivalence} and the subsequent characterization of the null hypotheses $H_\gamma$, it suffices to obtain a collection of tests $\{\phi_\gamma\}_{\gamma\in\Gamma}$ satisfying \eqref{eq:fwer_control}. To this end, first assume that, for every $\gamma\in\Gamma$, there exists an estimator $\widehat L_{n,\gamma}$ of the functional $L_\gamma(G)$ that can be written as a sample mean:
\begin{equation*}
\widehat L_{n,\gamma}
:=
\frac{1}{n}
\sum_{i=1}^n
Q_\gamma(\widehat\theta_i,\sigma_i)~,
\end{equation*}
where $Q_\gamma$ is some function from $\mathbb{R}\times\mathbb{R}_+$, the space of potential realizations of $(\hat\theta_i,\sigma_i)$ to the real numbers $\mathbb{R}$. Before describing my approach to inference, I define the bias of $\hat{L}_{n,\gamma}$ and variance of $Q_\gamma$ as follows:
\begin{equation*}
B_\gamma(G) := \mathbb{E}_G\hat{L}_{n,\gamma} - L_\gamma(G)~,\quad \sigma_G^2(Q_\gamma) = \operatorname{Var}_G(Q_\gamma) = \mathbb{E}_G\left(Q_\gamma(\hat\theta_i,\sigma_i)^2\right) - \mathbb{E}_G\left(Q_\gamma(\hat\theta_i,\sigma_i)\right)^2~.
\end{equation*}
The bias of $\hat L_{n,\gamma}$ is typically unknown, and I therefore denote by $\bar{B}_\gamma := \sup_{G\in\mathcal{G}}|B_\gamma(G)|$ the worst-case bias over the class $\mathcal{G}$. Finally, for any $\Gamma' \subseteq\Gamma$, let $J(x,\Gamma',G)$ denote the distribution of the supremum of the estimation error of the $\hat{L}_{n,\gamma}$ over $\Gamma'$:
\begin{equation*}
\label{eq:oracle-cdf}
J_n(x,\Gamma',G)
:=
P_G\!\left(
\sup_{\gamma\in\Gamma'}
\left|
\hat T_{n,\gamma}
\right|
\leq x
\right)~, \quad \hat T_{n,\gamma} :=\frac{
\sqrt n\left(
\widehat L_{n,\gamma}
-
\mathbb E_G\widehat L_{n,\gamma}
\right)
}{
\sigma_G(Q_\gamma)
}~.
\end{equation*}
Finally, I define the corresponding $1-\alpha$ critical value by $c(1-\alpha,\Gamma',G)$, defined as:
\begin{equation*}
c(1-\alpha,\Gamma',G):=
\inf\left\{
x\geq0:
J(x,\Gamma',G)\geq1-\alpha
\right\}~.
\end{equation*}
Ideally, given the equivalence outlined in section \ref{sec:equivalence_results}, hypotheses $H_\gamma$ should be rejected according to critical values $c(1-\alpha,\Gamma(G),G)$. However, because these critical values depend on the true nulls $\Gamma(G)$, which is the exact object of inferential interest, such approaches are not feasible. To obtain a feasible approach to inference that approximates this ideal benchmark, I build on and extend work due to \citealt{RomanoWolf2005} and \citealt{RomanoShaikh2010} to obtain a step-down, bias-aware procedure that iteratively rejects hypotheses $H_\gamma$. Crucially, this procedure highlights that it is not necessary to control for false discoveries among already-rejected hypotheses, targeting coverage of the functionals $L_\gamma(G)$ for which $H_\gamma$ holds.
\begin{algorithm}[Step-Down Procedure (Known Critical Values)]
\label{alg:step-down-procedure-known-cv}
\medskip
\noindent\textbf{Initialization.}
Set $\Gamma_0 := \Gamma$.
\medskip
\noindent\textbf{Iteration.}
For $k=1,2,\ldots$, set
\[
\Gamma_k
:=
\left\{
\gamma\in\Gamma_{k-1} :
\left|\widehat L_{n,\gamma}\right|
\leq
\overline B_\gamma
+
c(1-\alpha,\Gamma_{k-1},G)
\frac{\sigma_G(Q_\gamma)}{\sqrt n}
\right\}~.
\]
Stop at the first $k$ such that $\Gamma_k=\Gamma_{k-1}$,
and return $\Gamma_k$.
\end{algorithm}
Where $\Gamma_{\hat k}^{sd}$ denotes the outcome of this step-down procedure, define $\widetilde{CS}^{sd}(\cdot,\cdot)$ as follows:
\begin{equation}
\label{eq:step_down_definition}
\widetilde{CS}^{sd}(t,s) := \{c : (c,t,s) \in \Gamma_{\hat{k}}\}~.
\end{equation}
\begin{prop}
\label{prop:step_down_finite_sample}
The step-down uniform confidence set \eqref{eq:step_down_definition} satisfies:
\begin{equation*}
P_G\left(
\mu_G(t,s) \in \widetilde{CS}^{sd}(t,s)\text{ for all }t,s
\right)\ge 1-\alpha~,
\end{equation*}
for all $G$ in $\mathcal{G}$.
\end{prop}
The proof of proposition \ref{prop:step_down_finite_sample} is included in appendix \ref{app:proofs_of_main_results}. At this point, I comment on four features of the procedure. First, the set of true nulls $\Gamma(G)$ encodes shape restrictions on the conditional mean function that are implied by the hierarchical sampling model \eqref{eq:EB_DGP}. The step-down procedure treats $\Gamma(G)$ only as the unknown set of true null hypotheses and does not otherwise exploit this structure. Procedures that exploit the shape restrictions implied by the \eqref{eq:EB_DGP} may therefore be more informative, although constructing such procedures is non-obvious and an interesting direction for future research. Second, the step-down construction eliminates the need to know $\Gamma(G)$ when determining the collection of hypotheses over which critical values are computed. The oracle critical values nevertheless continue to depend on the unknown distribution $G$, because the distribution of $\widehat L_{n,\gamma}$ depends on the latent distribution generating the data. Subsection~\ref{sec:asymptotic_validity} demonstrates how these distributions and critical values can be approximated using a multiplier bootstrap. Third, the construction above uses simultaneous, two-sided, bias-aware confidence sets for the functionals $L_\gamma(G)$. For ease of exposition, the maximum-bias and sampling-error components enter the confidence bounds additively. As discussed by \citet{ArmstrongKolesar2018}, this construction implicitly permits the bias to be chosen adversarially for each realization of the sampling error. One can address this source of conservatism by developing simultaneous analogues of the folded-normal critical values of \citet{ArmstrongKolesar2018}. Finally, the procedure takes the estimators $\widehat L_{n,\gamma}$, or equivalently the functions $Q_\gamma$, as given. As discussed previously, the choice of estimator poses a crucial tradeoff between bias and variance. In Appendix \ref{app:minimax_details}, I extend the methods of \citet{Donoho1994}, \citet{ArmstrongKolesar2018}, and \citet{IgnatiadisandWager2021} to construct minimax estimators that navigate this tradeoff in a principled manner. These are the estimators used in my bootstrap-validity analysis and ultimately implemented in the applications. Remark \ref{rem:choice-of-t-stat} details this choice of estimator.
\begin{rem}[Choice of Test Statistic]
\label{rem:choice-of-t-stat}
So far, the step-down methods provided allow for an arbitrary choice of test-statistic. I build on work due to \citet{Donoho1994}, \citet{ArmstrongKolesar2018}, and \citet{IgnatiadisandWager2021} to provide a principled choice of test-statistic. In Appendix \ref{app:minimax_details}, I derive a class of minimax estimators that allow for (i) rich forms of dependence between $\theta_i$ and $\sigma_i$; and (ii) a method for trading off worst-case bias and variance. Let $\widehat{\operatorname{Var}}(Q_\gamma)$ denote an estimator of the variance of $Q_\gamma$. In appendix \ref{app:minimax_details}, I show how to obtain $Q_\gamma$ which solve, for each $\gamma$ in $\Gamma$, the following optimization problem:
\begin{equation}
\label{eq:minimax_program_intuition}
\min_{Q_\gamma}\sup_{G\in\mathcal{G}} B_\gamma(G)^2~\text{s.t.}~\frac{1}{n}\widehat{\operatorname{Var}}(Q) \leq V_n~,
\end{equation}
where $V_n$ denotes an upper constraint on the variance of $\hat{L}_{n,\gamma}$. Solutions to the program in \eqref{eq:minimax_program_intuition} minimize maximum bias over $\mathcal{G}$, subject to a variance constraint, and are in this sense minimax. Alongside characterizing $Q_\gamma$ in \ref{app:minimax_details}, I discuss how to choose $V_n$ to trade off bias and variance for the purposes of the step-down procedure considered in this paper. Finally, similar minimax estimators are derived in \citealt{IgnatiadisandWager2021}, and are used to obtain marginal coverage of individual conditional means. I emphasize that the affine minimax estimators developed and used in this paper differ from those of \citet{IgnatiadisandWager2021}, in that my affine minimax estimators allow for heteroskedasticity, and in particular dependence between the $\theta_i$ and $\sigma_i$. Finally, I emphasize that choosing minimax test-statistics for each functional $L_\gamma(G)$ allows researchers to focus power on these decision-relevant features of $G$.
\end{rem}
\subsection{Asymptotic Validity}
\label{sec:asymptotic_validity}
The above approach for constructing confidence sets satisfying \eqref{eq:uniform_coverage} assumes that the critical values $c(1-\alpha,\Gamma',G)$ are known. In practice, these critical values must be estimated. This section formalizes this estimation problem and provides a multiplier bootstrap algorithm to do so. Similarly, the $\sigma_G(Q_\gamma)$ are unknown, and are assumed to be estimated by $\hat\sigma_n(Q_\gamma)$. I first establish general sufficient conditions for estimated critical values to deliver an asymptotically valid approach to inference.
\begin{assu}
\label{assu:cv_validity_monotonicity}
$\hat{c}_n$ satisfy validity and monotonicity, in the following sense:
\noindent (i) $\hat{c}_n$ are valid in the sense that:
\begin{equation*}
\label{eq:cv_validity}
\limsup_{n\to\infty}\sup_{G\in\mathcal{G}} P_G\left(
\sup_{\substack{\gamma\in \Gamma(G)}}
\left|
\frac{
\sqrt n\left(
\widehat L_{n,\gamma}
-
\mathbb E_G\widehat L_{n,\gamma}
\right)
}{
\hat\sigma_n(Q_\gamma)
}
\right| >\hat{c}_n(1-\alpha,\Gamma(G))
\right)\leq \alpha~.
\end{equation*}
\noindent (ii) $\hat{c}_n$ are monotone, in the sense that whenever $\Gamma_1\subseteq\Gamma_2\subseteq\Gamma$:
\begin{equation*}
\label{eq:cv_monotonicity_1}
\hat{c}_n(1-\alpha,\Gamma_2)\geq \hat{c}_n(1-\alpha,\Gamma_1)~.
\end{equation*}
\end{assu}
Towards interpretation, part (i) of the above assumption suggests that the critical values $\hat{c}_n$ need not be valid for every possible subset of $\Gamma$, but rather only for $\Gamma(G)$. In order to correct for the fact that critical values containing $\Gamma(G)$ may not be valid, part (ii) of the above assumption imposes monotonicity of these critical values in the sense of set inclusion. Proposition \ref{prop:step_down_validity_est_cv} verifies that these assumptions are sufficient for the uniform validity of this paper's methods.
Under this assumption, proposition \ref{prop:step_down_validity_est_cv}, stated below, ensures the asymptotic validity of the following algorithm:
\begin{algorithm}[Step-Down Procedure]
\label{alg:step-down-procedure}
\medskip
\noindent\textbf{Initialization.}
Set $\Gamma_0 := \Gamma$.
\medskip
\noindent\textbf{Iteration.}
For $k=1,2,\ldots$, set
\[
\Gamma_k
:=
\left\{
\gamma\in\Gamma_{k-1} :
\left|\widehat L_{n,\gamma}\right|
\leq
\overline B_\gamma
+
\hat{c}_n(1-\alpha,\Gamma_{k-1})
\frac{\hat\sigma_n(Q_\gamma)}{\sqrt n}
\right\}~.
\]
Stop at the first $k$ such that $\Gamma_k=\Gamma_{k-1}$,
and return $\Gamma_k$.
\end{algorithm}
Where $\Gamma_{\hat k}^{sd}$ denotes the outcome of this step-down procedure, define $\widehat{CS}^{sd}(\cdot,\cdot)$ as follows:
\begin{equation}
\label{eq:step_down_definition_asymp}
\widehat{CS}^{sd}(t,s) := \{c : (c,t,s) \in \Gamma_{\hat{k}}^{sd}\}~.
\end{equation}
\begin{prop}
\label{prop:step_down_validity_est_cv}
Suppose that assumption \ref{assu:cv_validity_monotonicity} holds. Then $\widehat{CS}^{sd}(\cdot,\cdot)$ satisfies \eqref{eq:uniform_coverage}.
\end{prop}
The proof of proposition \ref{prop:step_down_validity_est_cv} is included in appendix \ref{app:proofs_of_main_results}.
To obtain critical values satisfying \eqref{assu:cv_validity_monotonicity}, I proceed via the multiplier bootstrap. Let $\xi_i$ be i.i.d. standard normal random variables for $i=1,\ldots,n$, and denote by $\xi$ the collection $(\xi_i)_{i=1}^n$. For some realization of $\xi$, I define $\hat{L}_{n,\gamma}(\xi)$ as follows:
\begin{equation*}
\hat{L}_{n,\gamma}(\xi) := \frac{1}{n}\sum_{i=1}^n \xi_i \left(Q_\gamma(\hat\theta_i,\sigma_i) -\hat{L}_{n,\gamma}\right)~.
\end{equation*}
Let $J_{n}^{MB}(x,\Gamma)$ denote the distribution of the multiplier bootstrap statistic, holding $\hat\theta_i$ and $\sigma_i$ fixed, and integrating over the law of $\xi$:
\begin{equation*}
J_{n}^{MB}(x,\Gamma) = P_\xi\left(
\sup_{\gamma\in\Gamma}
\left|
\frac{\sqrt{n}\hat{L}_{n,\gamma}(\xi)}{\hat\sigma_n(Q_\gamma)}
\right|\le x
\right)~.
\end{equation*}
I define the multiplier bootstrap critical values $\hat{c}_n(1-\alpha,\Gamma)$ as follows:
\begin{equation*}
\hat{c}_n(1-\alpha,\Gamma) := \inf\left\{
x\ge 0: J_{n}^{MB}(x,\Gamma)\geq 1-\alpha
\right\}~.
\end{equation*}
In appendix \ref{app:bootstrap_theory}, I demonstrate that these multiplier bootstrap critical values satisfy assumption \ref{assu:cv_validity_monotonicity}. The following theorem is proven therein.
\begin{theorem}
\label{thm:bootstrap_validity}
Let $ n \to\infty$, and let assumptions \ref{assu:smoothness}, \ref{assu:modulus_binding}, and \ref{assu:pilot_independence} hold. Then the multiplier bootstrap critical values $\hat{c}_n(1-\alpha,\Gamma)$ satisfy assumption \ref{assu:cv_validity_monotonicity}, and $\widehat{CS}^{sd}(\cdot,\cdot)$ using the bootstrap critical values satisfies \eqref{eq:uniform_coverage}.
\end{theorem}
The proof of theorem \ref{thm:bootstrap_validity} is contained in appendix \ref{app:bootstrap_theory}.
The assumptions in theorem \ref{thm:bootstrap_validity} are largely standard. Assumption \ref{assu:smoothness} establishes bounds on the support of $\sigma$ and the H\"older smoothness of the marginal distribution of $\sigma$, and of the conditional distribution of $\theta$ given $\sigma$, as a function of $\sigma$. Intuitively, assumption \ref{assu:smoothness} ensures that the conditional law of the effects $\theta$ given precision $\sigma$ cannot vary too much with small changes in $\sigma$. Assumption \ref{assu:smoothness} is similar to assumptions 3 and 4 of \citealt{chen2024}. Assumption \ref{assu:modulus_binding} asserts that the variance constraint defining the minimax problem \eqref{eq:minimax_program_intuition} binds, and additionally imposes the same condition as assumption A in theorem 2 in \citealt{IgnatiadisandWager2021}. Assumption \ref{assu:pilot_independence} is identical to assumption E in theorem 2 in \citealt{IgnatiadisandWager2021}. Finally, while not needed for validity, assumption \ref{assu:variance_uniform_estimability} can be used to show that estimation error in the $\hat\sigma_n$ is negligible, as shown in theorem \ref{thm:cv_assumptions_verified_var_est}. Assumption \ref{assu:variance_uniform_estimability} is a standard consistency condition on the variance estimators $\hat\sigma_n(Q_\gamma)$, which follows condition H4 of \citealt{CCK_2016}.
The proof of theorem \ref{thm:bootstrap_validity} is contained in appendix \ref{app:bootstrap_theory}. The key challenge in proving theorem \ref{thm:bootstrap_validity} is that the collection of studentized statistics $\hat T_{n,\gamma}$ need not be Donsker, or even VC-type, since variances are not bounded away from zero.\footnote{Similar challenges with studentization arise in \citealt{chetverikov2012}.} The proof proceeds in three steps. I first verify the conditions needed to obtain a Gaussian coupling for the statistic $\hat T_{n,\gamma}$ via theorem 2.1 of \citealt{CCK_2014}. I then obtain an analogous Gaussian coupling for the corresponding multiplier bootstrap statistic, extending arguments due to \citealt{CCK_2016} to allow for the fact that the studentized test-statistics are not VC-type. Finally, application of anti-concentration results due to \citealt{CCK_2014} and \citealt{CCK_ac_2014} yields the necessary convergence of distribution functions for the bootstrap critical values to satisfy the conditions in assumption \ref{assu:cv_validity_monotonicity}.
\section{Application: Economic Mobility Revisited}
\label{sec:economic-mobility-application}
Recent work asks studies the geography of opportunity in the United States \citep{Chettyetal2014,ChettyandHendren2018,Chettyetal2018aer}. Most recently, \citealt{Chettyetal2018aer} derive and report estimates of economic mobility in US commuting zones and neighborhoods, and advocate the use of these estimates for downstream policy targeting tasks. For example, in his testimony before the congressional budget committee \citep{Chetty2014MobilityTestimony}, Chetty stated:
\begin{quote}
Perhaps the most cost-effective way to improve mobility may be to publicize local statistics on economic mobility and other related outcomes. Simply drawing attention to the areas that need improvement can motivate local policy makers to take action.
\end{quote}
Subsequent work has applied such local statistics to downstream policy tasks. In this section, I revisit a particular example of such an exercise, namely the policy targeting exercise of \citealt{Bergmanetal2023} considered throughout this paper. In particular, \citealt{Bergmanetal2023} seek to draw the attention of low-income housing voucher recipients to high-mobility neighborhoods. As described in section \ref{sec:setup}, they select a subset of high-mobility neighborhoods in Seattle and inform treated voucher recipients, as part of the Creating Moves to Opportunity program, that these neighborhoods are high mobility.
A recent debate highlights that the economic mobility estimates may not be sufficiently informative for informative decision making in the spirit of \citet{Bergmanetal2023}. For example, \citet{Mogstadetal2023} find that the economic mobility estimates of \citet{Chettyetal2018aer} may contain more noise than signal, and that one cannot confidently conclude that the tracts selected in the spirit of \citet{Bergmanetal2023} lie in the true top third according to true economic mobility. Similarly, \citet{petrouzeniou_and_shaikh_2025} find similar results even when relaxing the notion of type-I-error control to allow researchers to focus power on tracts of interest. Motivated by these findings, my applications study the effectiveness of tract-level mobility estimates according to an oracle decision-maker with knowledge of the distribution of economic mobility across tracts.
In order to shed light on the informativeness of these economic mobility estimates for decision-making, I consider two exercises. In the first exercise, I ask which tracts researchers can confidently conclude an oracle decision maker would select as high mobility. While the results of this exercise can be suggestive of differences in mobility across tracts, according to an oracle decision-maker, the magnitudes of these differences may be small. To rule out this possibility, I ask in my second exercise how large of an improvement in economic mobility researchers can confidently conclude an oracle decision maker would associate with moving from a low- to a high-mobility neighborhood. Large gaps in conditional mean mobility between low and high estimated mobility tracts suggest that an oracle decision-maker believes that tracts with low estimated mobility are, on average, substantially lower-opportunity than those with high estimated mobility. The remainder of this section provides results from these two exercises.
\subsection*{Inference on the oracle selection of tracts}
\label{subsec:mobility-oracle-selection}
Consider the first exercise: obtaining a confidence set for the oracle decision. Recall that the optimal, oracle selection of tracts selects the top-third of census tracts according to their conditional mean mobility $\mu_G(\hat\theta_i,\sigma_i)$. To construct a confidence set for this decision using the confidence bands derived in this paper, let $\widehat{CS}_i^{sd}$ be a confidence set for conditional mean mobility at tract $i$. In particular, given $\widehat{CS}^{sd}(\cdot,\cdot)$, take $\widehat{CS}_i^{sd} := \widehat{CS}^{sd}(\hat\theta_i,\sigma_i)$. Confidence bounds for pairwise differences in conditional means can be constructed by comparing the infimum of $\widehat{CS}_i$ to the supremum of $\widehat{CS}_j$, for $i\ne j$, and vice versa. Consequently, one can construct simultaneous confidence sets for the ranks of tracts according to conditional mean mobility.\footnote{See \citealt{Mogstadetal2023} and \citealt{Kleinetal2020} for further discussion of such constructions.} In particular, I obtain confidence sets $\widehat{CS}_{\mathrm{in}}^{sd}$ and $\widehat{CS}_{\mathrm{exc}}^{sd}$, which denote the set of tracts confidently included and excluded in the oracle optimal selection, respectively. With asymptotic probability of at least $1-\alpha$, these confidence sets satisfy:
\begin{equation*}
\widehat{CS}_{\mathrm{in}}^{sd} \subseteq \left\{
i: \delta_i^* = 1
\right\} \subseteq \left( \widehat{CS}_{\mathrm{exc}}^{sd}\right)^c ~.
\end{equation*}
Figure~\ref{fig:confidence_sets_inner_outer_pts} plots the inner and outer confidence sets for the top-third Bayes-optimal selection using both the step-down method developed in this paper and the $F$-localization method of \citet{IgnatiadisandWager2021} derived formally in appendix \ref{app:Floc}. Tracts are plotted according to their $\hat\theta_i$ and $\sigma_i$. Tracts that researchers can confidently include in the oracle decision are highlighted in teal, whereas tracts that researchers can confidently exclude from the oracle decision are highlighted in copper. Tracts in neither set, corresponding to those tracts for which a researcher is uncertain about the oracle classification, are highlighted in gray.
\begin{figure}[H]
\begin{subfigure}{0.45\textwidth}
\centering
\includegraphics[width=\linewidth]{figs/hfloc_inner_outer_theta_logsigma.png}
\caption{$F$-localization}
\end{subfigure}
\hfill
\begin{subfigure}{0.45\textwidth}
\centering
\includegraphics[width=\linewidth]{figs/unif_boot_inner_outer_theta_logsigma.png}
\caption{Step-Down}
\end{subfigure}
\caption{Inner and outer confidence sets for top-third Bayes-optimal selection, in $(\hat\theta,\log(\sigma))$ space.}
\label{fig:confidence_sets_inner_outer_pts}
\end{figure}
Visual inspection of Figure~\ref{fig:confidence_sets_inner_outer_pts} yields two conclusions. First, the step-down procedure is substantially more informative than $F$-localizations. Using the step-down procedure, we can confidently conclude that 80 tracts are selected and 399 tracts are not selected. Relative to the 25 confident inclusions and 241 confident exclusions under $F$-localizations, this represents a $220\%$ increase in confident inclusions and a $66\%$ increase in confident exclusions. Consequently, I conclude that the set of oracle decisions that are consistent with the observed data is relatively narrow. Second, my methods allow researchers to place greater confidence on the classifications of low-noise tracts. This not only matches the intuition around confidence sets for the latent $\theta_i$ themselves, but also provides the key intuition that the data are more informative about the oracle decision for tracts with a higher signal-to-noise ratio. In particular, there is little researchers can conclude about the conditional mean of $\theta_i$ when the $\sigma_i$ are sufficiently large as to ensure that information about the conditional law of $\theta_i$ given $\sigma_i$ is limited.
In appendix \ref{app:KDE_results_mob}, I compare my step-down methods to alternative implementations of the $F$-localizations of \citealt{IgnatiadisandWager2021}, and show that the performance gains afforded by my methods are robust to choices in implementation of alternative methods. In appendix \ref{app:additional_figs_mob}, I provide additional results from this exercise. I plot in appendix figure \ref{fig:sea_maps} a map of the inner and outer confidence sets for the Bayes-optimal selection using both my methods and using $F$-localizations. My methods are most-informative in densely-populated, urban Seattle tracts. Appendix figure \ref{fig:sea_maps_urban} provides these inner and outer confidence set maps for urban Seattle tracts. Appendix figures \ref{fig:rank_LCBs} and \ref{fig:rank_UCBs} plot lower and upper confidence sets, respectively, for the ranks of Seattle census tracts on a map. Appendix figures \ref{fig:urban_rank_LCBs} and \ref{fig:urban_rank_UCBs} repeat this exercise for urban Seattle tracts.
\subsection*{Welfare implications of a feasible decision rule}
\label{subsec:mobility-welfare}
While the previous exercise allows us to quantify uncertainty around the oracle decision, it does not allow researchers to draw conclusions about the welfare of feasible decision rules. For instance, while the tracts in $\widehat{CS}_{\mathrm{in}}^{sd}$ and $\widehat{CS}_{\mathrm{exc}}^{sd}$ may differ in terms of conditional mean mobility, the gaps in mobility between these confidently included and excluded tracts may be small. In my second exercise, I consider the problem of comparing low and high-mobility tracts as selected by a feasible decision rule that selects tracts according to estimated conditional means. Given this feasible rule, I obtain confidence bounds on the gains in conditional mean economic mobility associated with moving from an arbitrary low-mobility tract to an arbitrary high-mobility tract. For each pair consisting of a feasibly-labeled low-mobility tract $i$ and a feasibly-labeled high-mobility tract $j$, define $\Delta_{ij}(G)
:=
\mu_G(\widehat\theta_j,\sigma_j)
-
\mu_G(\widehat\theta_i,\sigma_i)$. In the following exercise, I use the step-down confidence bands developed in this paper to obtain simultaneous lower confidence bounds on $\Delta_{ij}(G)$ across all such pairs.
Figure~\ref{fig:LCB_welfare_gains_diff_selections} presents a histogram of these lower confidence bounds across all pairs of low- and high-mobility tracts. As above, I present results using both the step-down method developed in this paper (plotted in teal) and the $F$-localization method (plotted in copper) of \citet{IgnatiadisandWager2021}. Values lying to the right of zero correspond to pairs for which we can confidently conclude that the conditional mean gain in economic mobility is positive. The larger the area of the histogram to the right of zero, the larger the proportion of tract pairs for which researchers can conclude that there is a positive mobility gain. The vertical dotted lines represent the average lower confidence bound across pairs of low- and high-mobility tracts using each method. The farther to the right these dotted lines, the larger the magnitudes of gains in conditional mean economic mobility associated with moving from low to high-mobility tracts,
\begin{figure}[H]
\centering
\includegraphics[width=0.5\linewidth]{figs/hist_lcb_eb_top3_minus_ucb_eb_bottom3_overlay_unnormalized_only.png}
\caption{Lower confidence bounds on conditional mean gains in mobility associated with moving from low to high-mobility tracts, as selected by an EB decision rule.}
\label{fig:LCB_welfare_gains_diff_selections}
\end{figure}
A visual inspection of these histograms yields two conclusions. First, for the vast majority of pairs of tracts labeled low and high mobility by the feasible decision maker, an oracle decision-maker would believe that moving from the low- to the high-mobility tract generates a positive gain in conditional mean mobility. The average lower confidence bound corresponds to a five percentile-point improvement in economic mobility. The step-down method yields positive lower confidence bounds for $91.9\%$ of these tract pairs. Second, the methods developed in this paper are suggestive of substantially larger gains across pairs of tracts than $F$-localizations. $F$-localizations analogously conclude that only $73.7\%$ of tract pairs lead to a positive gain in conditional mean economic mobility.
\subsection*{Implications for the classification of high-opportunity neighborhoods}
\label{subsec:mobility-implications}
Given the uncertainty around the distribution of economic mobility across Seattle tracts, one may naturally ask whether an oracle decision-maker would believe that the CMTO program actually directs families to high-mobility neighborhoods. Assuming that the correlational estimates of tract-level mobility used in my exercises are causal, would an oracle DM ascribe positive conditional mean welfare gains to the CMTO program?
First, as noted in \citet{Mogstadetal2023}, the answer to this question depends on which assumptions the researcher is willing to make on movement patterns between low- and high-mobility neighborhoods. In this vein, notice that in the exercises undertaken in this paper, we cannot reject the possibility that a small fraction of high-noise tracts selected as high mobility by a feasible decision rule are among the lowest-mobility tracts in Seattle according to oracle conditional mean mobility. Consequently, under an extreme thought experiment wherein all treated individuals move to these tracts, we cannot reject the possibility that the oracle decision-maker would conclude that the CMTO program is welfare decreasing. However, the set of tracts for which this is true is a small fraction of the feasibly-selected tracts. In particular, for the majority of feasibly-selected tracts, my exercises suggest that researchers can be confident that the oracle decision-maker would place these tracts among the top half, and in many cases among the top third, according to conditional mean economic mobility.
However, these extreme cases are instructive in one sense that is informative for decision-making. In particular, there is little one can say about the oracle conditional mean mobility at tracts where economic mobility is measured with high noise. In these low signal-to-noise regimes, one can learn little about the distribution of economic mobility relevant for the conditional mean, and as a result one can say little about the optimal decision. As a result, under the rich forms of heteroskedasticity considered in this paper, it is unclear how researchers should navigate the planner's dilemma. In particular, researchers cannot be confident in feasible solutions to the problem of trading off selection of high-mobility, high-noise tracts with selection of lower-mobility, lower-noise tracts.
Given the above conclusions on the informativeness of the economic mobility estimates for optimal decision-making, a natural related question asks how well an oracle decision maker would believe feasible rules perform according to their objectives. In particular, while an oracle decision maker can conclude that some tracts are higher mobility than others according to their conditional means, the magnitudes of the conditional mean gains between low- and high-mobility tracts could nevertheless be small. In particular, this would be the case when the distribution of mobility is fairly concentrated, meaning that tracts are qualitatively indistinguishable according to true economic mobility. In the setting of the \citet{Bergmanetal2023} experiment, however, the results above show that this is not the case. Indeed, for the vast majority of pairs of low- and high-mobility tracts, the gains in economic mobility expected by an oracle decision maker are economically large.
The exercises considered in this application suggest two results. First, researchers can be confident that an oracle decision-maker would rank the majority of feasibly selected tracts in the top half, and many in the top third, according to conditional mean mobility. As a result, the movement patterns needed to generate losses in conditional mean economic mobility due to the CMTO experiment are necessarily fairly extreme. Moreover, in comparing low- to high-mobility tracts as selected by a feasible decision maker who approximates the oracle, my exercises suggest that an oracle decision maker would ascribe qualitatively large gains in economic mobility associated with moving from low- to high-mobility neighborhoods. This also suggests wide dispersion in economic mobility across neighborhoods. That said, I cannot reject the possibility that an oracle decision maker makes mistakes according to true mobility at each neighborhood, in line with results from existing work.
\subsection*{Comparison to Analyses of Latent Effects}
Given the above discussion, it is worth comparing the exercises considered above to existing exercises conducting inference on the latent effects $\theta_i$ themselves. While the exercises considered in this paper consider the problem of an oracle decision maker with knowledge of the distribution of mobility across tracts, existing exercises effectively compare to analyzing the problem of a fully-informed oracle decision maker with knowledge of true mobility at each tract. In this sense, existing exercises admit similar economic interpretation to those considered in my application to the \citealt{Bergmanetal2023} selection exercise.
In particular, observe that my first exercise -- obtaining inner and outer confidence sets for the oracle optimal selection -- is analogous to exercises considered in \citealt{Mogstadetal2023}, who consider the problem of obtaining confidence sets for the top-third selection of tracts according to the true $\theta_i$. This latter selection would indeed be the optimal selection for an oracle decision-maker with knowledge of the $\theta_i$. Similarly, my second exercise -- obtaining confidence bounds on the gains in conditional mean mobility between low and high-mobility tracts -- is analogous to exercises considered in \citealt{petrouzeniou_and_shaikh_2025}, who consider the problem of obtaining confidence bounds on the gains in true mobility between low and high-mobility tracts. Again, these latter quantities correspond exactly to the gains in welfare ascribed to moving from low to high-mobility tracts by a fully-informed oracle. Consequently, the exercises considered in this paper admit similar economic interpretations to existing work.
Fundamentally, interest across the exercises considered in this paper and in existing papers is in evaluating and characterizing mistakes relative to an oracle decision maker, as discussed in \citealt{AndrewsandChen2025}. For instance, my first exercise, and consequently the exercises of \citealt{Mogstadetal2023}, allow researchers to evaluate which selections would not be classified as mistakes by an oracle decision maker. My second exercise, and thus the exercises of \citealt{petrouzeniou_and_shaikh_2025}, allow researchers to evaluate the magnitudes of these mistakes in terms of welfare. Therefore, the exercises considered in this paper, and those considered in previous work, differ in the sense of the oracle decision maker defining these mistakes.
The choice of oracle considered in this paper is natural given existing work. \citealt{chen2024} explicitly considers the problem of approximating an oracle decision maker with knowledge of the effect distribution $G$. In the original decision problem of \citealt{Bergmanetal2023}, selections were made using shrunk estimates of economic mobility, thus estimating conditional mean mobility and implicitly approximating the same oracle decision maker with knowledge of the effect distribution $G$. Given the theoretical and applied interest, therefore, in evaluating and approximating the decisions of an oracle decision maker with knowledge of the distribution of mobility across tracts, I view the exercises considered in this paper as complementary to existing exercises considering the fully-informed oracle.
Results due to \citealt{Mogstadetal2023} and \citealt{petrouzeniou_and_shaikh_2025} suggest that evaluating the fully-informed oracle decision maker is challenging in the setting of \citealt{Bergmanetal2023}. Economic mobility estimates are too numerous and too noisy to draw informative conclusions on the problem of a fully-informed oracle decision maker. My exercises complement this existing work by considering an alternative oracle decision-maker, and yield qualitatively different conclusions. I obtain positive results, and can draw informative conclusions on the problem of the oracle decision considered in this paper. Of course, my analyses do not rule out the possibility that an oracle with knowledge of $G$ makes mistakes relative to a fully-informed oracle. In fact, existing work suggests that this possibility cannot be rejected. Nevertheless, the framework developed in this paper, alongside the methodological advances developed herein, allow for new, economically substantive insights on place-based policy targeting using local statistics on mobility.
\section{Conclusion}
\label{sec:conclude}
This paper develops a framework and methods for evaluating uncertainty in a broad range of compound decision problems. I show that such compound decisions are ubiquitous in applied work in economics, and define a notion of an oracle decision maker in these problems. I argue that this oracle decision maker's problem is subject to substantial uncertainty in finite-samples. I provide tools to evaluate this uncertainty. Applying my tools to a canonical compound decision problem in economics -- the problem of differentiating low and high-mobility areas -- shows that local mobility statistics may be more informative for place-based policy targeting than previously understood.
\begin{appendix}
\bibliography{refs}
\clearpage
\startcontents[app]