The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
84,682 characters
Compound Selection Decisions: An Almost SURE Approach
\maketitle
\begin{abstract}
This paper proposes methods for producing \emph{compound selection decisions} in a
Gaussian sequence model. Given unknown, fixed parameters $\mu_ {1:n}$ and known
$\sigma_{1:n}$ with observations $Y_i \sim \Norm(\mu_i, \sigma_i^2)$, the decision maker
would like to select a subset of indices $S$ so as to maximize utility
$\frac{1}{n}\sum_{i\in S} (\mu_i - K_i)$, for known costs $K_i$. Inspired by Stein's
unbiased risk estimate (SURE), we introduce an \emph{almost} unbiased estimator, called
{\footnotesize \textsc{ASSURE}}, for the expected utility of a proposed decision rule.
{\footnotesize \textsc{ASSURE}} allows a user to choose a welfare-maximizing rule from a
pre-specified class by optimizing the estimated welfare, thereby producing selection
decisions that borrow strength across noisy estimates. We show that {\footnotesize
\textsc{ASSURE}} produces decision rules that are asymptotically no worse than the
optimal but infeasible decision rule in the pre-specified class. We apply {\footnotesize \textsc{ASSURE}} to the selection of
Census tracts for economic opportunity, the identification of discriminating firms, and the analysis of $p$-value decision procedures in A/B testing.
\end{abstract}
\pagestyle{plain}
\newpage
\section{Introduction}
In settings with parallel noisy estimates for many parameters, researchers are often
interested in \emph{selecting} units with high values of the parameter. For
these selection decisions, it is popular to screen based on shrunken estimates. Yet the
objectives for shrinkage and for selection are misaligned: it is not obvious that good
selection decisions must rely on good estimates, nor that good estimates lead to good
selections \citep{manski2021econometrics}. This paper directly studies these \emph{compound selection} problems\footnote{In this
paper, we
refer to \emph{compound decision}---often used interchangeably with ``empirical Bayes''
and ``shrinkage'' in the literature---specifically as decision-theoretic frameworks in
which (i) the true parameters are fixed or conditioned upon and (ii) the loss function
averages over units.
\emph{Empirical Bayes} additionally posits a random effect model for the distribution of
the true parameters, and evaluates decisions by integrating over the distribution of the
parameters. } and proposes methods that optimize selection decisions while still
``borrowing strength'' across different noisy estimates \citep
{robbins1985asymptotically}.
To introduce this procedure, we consider a standard setup for empirical Bayes and
compound decision \citep[see, e.g.,][]
{jiang2009general,efron2012large,jiang2020general,walters2024empirical,soloff2024multivariate,chen2022empirical}.
Researchers observe noisy estimates $Y_ {1:n} = (Y_1,\ldots, Y_n)'$ and standard errors
$\sigma_{1:n}$ for unknown \emph {parameters} $\mu_{1:n}$. Motivated by central limit
theorems applied to the procedure generating $Y_i$, we model the signals as Gaussian
with known variance: $ Y_i \mid \mu_i, \sigma_i, X_i \sim \Norm
(\mu_i, \sigma_i^2).$ Here, $X_i$ are covariates that do not predict the noise
$Y_i - \mu_i$. We seek binary decisions $a_i
\in \br{0,1}$ so as to maximize the compound utility
\begin{equation}
\label{eq:welfare}
\frac{1}{n} \sum_{i=1}^n a_i \left (\mu_i - K_i
\right).
\end{equation} The objective \eqref{eq:welfare} is the average payoff of selection
decisions over units $1,\ldots,n$, where the payoff of selecting unit $i$ is $\mu_i -
K_i$, for some known cost $K_i$. It can be viewed as a compound version of the
treatment choice problem \citep{manski2004statistical}, where normality is motivated by
local asymptotic approximations
\citep{hirano2009asymptotics,https://doi.org/10.3982/ECTA20364}.
For concreteness,\footnote{This type of problem also appears in other economic
applications such as the meta-analysis of experiments \citep {azevedo2020b}, teacher
value-added \citep{chetty2014measuring,kwon2023optimal,cheng2025optimal}, identifying
discrimination \citep {kline2022systemic}, and treatment choice
\citep{kitagawa2018should, athey2021policy,moon2025optimalpolicychoicesuncertainty}. This
problem also appears in the statistical literature under the name \emph{empirical Bayes
testing with linear loss} \citep{liang1988convergence, liang2000empirical,
liang2004optimal, karunamuni1996optimal}.} in \citet{bergman2024creating}, $Y_i$ is the
estimated economic mobility of a Census tract, estimated on Census microdata. $\mu_i$
is true economic mobility, defined as the population economic outcome of poor children
in adulthood in the tract. We may imagine that the costs $K_i$ are equal to some value
$K$ that represents the economic mobility level for which a social planner is
indifferent between incentivizing a low-income family to move to a tract with mobility
$K$ and not doing so. The objective \eqref {eq:welfare} then rewards selecting an
above-breakeven tract, punishes selecting a below-breakeven tract, and weighs the
rewards and penalties according to the distance to $K$. Since $Y_i$ are fixed effect
estimates, their normality---similarly invoked in \citet
{armstrong2022robust,mogstad2024inference,andrews2024inference, chen2022empirical}---is
motivated by the central limit theorem applied to regression specifications that generate
them \citep{chetty2018opportunity}.
Because the Gaussian family has monotone likelihood ratio \citep
{karlin1956theory}, admissible selection decisions
are increasing in $Y_i$.\footnote{See \cref{lemma:monotone} for a formal result.} We thus identify
selection decisions with \emph{threshold rules} $\delta_i(Y_{-i})$ such that $a_i =
\one(Y_i > \delta_i)$.\footnote{The threshold rules depend on $i$ through the contextual
information $Z_i = (\sigma_i, K_i, X_i)$, which we suppress in the notation.} For a given
threshold rule $\delta_ {1:n} (Y_ {1:n})$, define the \emph{compound welfare} \[
W(\delta_{1:n}, \mu_{1:n}) =
\frac{1}{n} \sum_{i=1}^n \E_{Y_i \sim
\Norm(\mu_i,
\sigma_i^2)} \bk{
\one(Y_i > \delta_i(Y_{-i})) \left (\mu_i - K_i
\right) } \label{eq:welfare_expected}
\numberthis
\] as the expected utility, integrating solely over $Y_{1:n}$. This is the compound
decision analogue of frequentist risk \citep{wald1950statistical}. Optimal decisions
should maximize
\eqref{eq:welfare_expected}, but doing so is infeasible since welfare depends on the
unknown parameters $\mu_{1:n}$. For estimation problems interested in minimizing MSE,
one feasible approach is \emph{Stein's unbiased risk estimate} \citep[SURE,][]
{stein1981estimation}. SURE \emph{estimates} $\frac{1} {n}\sum_{i=1}^n \E_
{Y_i \sim \Norm(\mu_i,
\sigma_i^2)}[(\delta(Y_i) - \mu_i)^2]$ and produces decisions by optimizing the
estimated objective over a parametrized class of decisions
\citep{xie2012sure,kwon2023optimal}.
Inspired by SURE, we construct a novel estimator $\hat W(\delta_{1:n})$ of $W(\delta_
{1:n})$. Unlike with MSE, it turns out that an exactly unbiased estimator for
\eqref{eq:welfare_expected} does not exist, but an \emph{almost} unbiased one---whose
bias decays exponentially in a tuning parameter---does. We thus term our approach
{\footnotesize \textsc{ASSURE}}, for \textbf{A}lmo\textbf{s}t \textbf{SURE}. This
almost unbiasedness is an appealing and nontrivial property. One could instead
estimate
\eqref{eq:welfare_expected} by (synthetically) sample splitting the estimates
\citep{oliveira2024unbiased,chen2022empirical,ignatiadis2025empirical}.\footnote{That
is, $Y_{i+} = Y_i + \epsilon W_i$ and $Y_{i-} = Y_i - \frac{1}{\epsilon} W_i$ for
independent $W_i
\sim \Norm(0,\sigma_i^2)$ satisfy $Y_{i+} \indep Y_{i-} \mid \mu_i,
\sigma_i^2$. Proposition 1 in \citet{chen2022empirical} uses this \emph{coupled
bootstrap} idea to estimate
\eqref{eq:welfare}
for decisions that depend on $Y_{i+}$.} Doing so
incurs
bias that does not vanish as quickly as {\footnotesize \textsc{ASSURE}}, and thus we can view {\footnotesize \textsc{ASSURE}}{} as a
further debiased version of this approach.
Like SURE, we then optimize this estimated welfare over a user-chosen class of decision
rules $\delta \in \mathcal D$. For instance, $\mathcal D$ may be the class of decisions
that screens
on linear shrinkage rules \[\mathcal D_{\textsc{close-gauss}} =\br{\delta_i :
\one (y >
\delta_i)
= \one\br{a(\sigma_i, X_i,
K_i;
\beta) y + b (\sigma_i, X_i, K_i ; \beta) > K_i}}\] for functions $a(\cdot), b
(\cdot)$, parametrized by a finite-dimensional $\beta \in B \subset \R^d$, that
depends on contextual information $ (\sigma_i, X_i, K_i)$ for some covariates
$X_i$ \citep{weinstein2018group,chen2022empirical}. Such restrictions are necessitated
by familiar bias-variance concerns---optimizing over too large a class of decisions
risks overfitting to the noise $\hat W - W$. In practice, these restrictions can come
from external preference for simple decisions, similar to
\citet{kitagawa2018should,sudijono2024optimizing,crippa2025regret}, or from a benchmark
(empirical) Bayesian model on the parameters $\mu_{1:n}$, similar to \citet{kwon2023optimal} and \citet
{cheng2025optimal}. $\mathcal D_{\textsc{close-gauss}}$, for
instance, can be motivated either by simplicity preferences or by a correlated random
effects model in which $\mu_i
\mid \sigma_i, K_i, X_i
\sim \Norm (m(\sigma_i, K_i, X_i; \beta), s^2(\sigma_i, K_i, X_i; \beta))$, under which
posterior means $\mu_i \mid Y_i, \sigma_i, K_i, X_i$ are linear in $Y_i$.\footnote{\citet{chen2022empirical} refers to this random effects model as \textsc{close-gauss}.}
Decisions selected by {\footnotesize \textsc{ASSURE}}{} enjoy strong optimality
guarantees. As the number of parameters diverges ($n \to\infty$), we show that the
performance gap between the estimated optimal decision and the optimizer of \eqref
{eq:welfare_expected} within $\mathcal D$---i.e.,
\emph{regret}---converges to zero at minimax optimal rates, up to log factors. Both the upper and lower
bounds for regret are novel analyses. Applied to $\mathcal D_{\textsc{close-gauss}}$,
since many popular shrinkage methods are nested within $\mathcal
D_{\textsc{close-gauss}}$, decisions tuned by {\footnotesize \textsc{ASSURE}} would
asymptotically improve over screening on these status-quo shrinkage methods. The bulk of
the paper concerns Gaussian estimates, but similar arguments pertain to Poisson $Y_i$ as
well; we present these related results as an extension.
Importantly, our analysis differs from and complements \emph{empirical Bayes} analyses
\citep{jiang2020general,gu2023invidious,walters2024empirical,kline2024discrimination,chen2022empirical}
by focusing on the compound objective \eqref{eq:welfare_expected} rather than its
empirical Bayesian counterpart \[W_{\text{EB}}(\delta_{1:n}) = \E_{\mu_{1:n} \sim
P}[W(\delta_ {1:n},
\mu_{1:n})] \text{ for } \mu_{1:n} \mid \sigma_{1:n}, K_{1:n}, X_{1:n} \sim P.\]
Empirical Bayes analyses treat $\mu_ {1:n}$ as random effects and integrates over their
distribution; in contrast, our compound perspective is analogous to fixed effects in
panel data \citep{dano2025binary}. The benefit of treating $\mu_{1:n} \sim P$ as random
is that the optimal selection decision screens on the posterior mean $\E_{P}
[\mu_i \mid Y_i]$: Under $P$ and $W_{\text{EB}}$, there is no misalignment between
estimation and selection decisions.
That benefit comes with two costs that our framework avoids. First, empirical Bayesians
have to carefully model $P$. Failing to model this distribution well---either
misspecifying how $\mu_i$ is predicted by the contextual information
\citep{chen2022empirical} or how $\mu_i$ correlates with $\mu_j$
\citep{bonhomme2024estimating}---could harm the selection decision. The compound decision
framework avoids these problems entirely by conditioning on $\mu_{1:n}$. Second,
optimality for $W_ {\text{EB}}$ is with respect to a new draw of units $\mu_i \sim P$. In
\citet{bergman2024creating}, for instance, empirical Bayes evaluates performance over a
randomly drawn new Census tract, whereas optimality with respect to \eqref{eq:welfare}
keeps fixed the sample of Census tracts, and only imagines alternative draws of the
estimation error in economic mobility.
To be sure, a SURE-based compound decision is limited to optimality within a class
of decision
rules $\mathcal D$. Empirical Bayes methods can match the performance of the optimal
decision under $P$, without constraining to a class $\mathcal D$. That shortcoming of
compound decisions can be overcome by selecting $\mathcal D$ via some working
empirical Bayes model. When a model of $P$ implies a parsimonious class of decisions rules
through $a(Y_i) =
\one (\E_P [\mu_i \mid Y_i] > 0)$,\footnote{Posterior means with Gaussian $Y$ are
monotone in $Y$, and thus this threshold rule can be equivalently written as $Y_i >
\delta_i$.} optimizing among this class
using {\footnotesize \textsc{ASSURE}} weakly
improves over the empirical Bayes decisions under an estimated prior $\hat P$. Thus, our
method can be combined with (parametric) empirical Bayesian modeling: it \emph{fine-tunes}
such a model without harming performance.
We illustrate \assure{} in three empirical applications: selecting census tracts to
maximize economic mobility \citep{chetty2018opportunity,bergman2024creating}, picking
innovations in A/B testing \citep{azevedo2020b}, and identifying discrimination in large
firms \citep{kline2022systemic, kline2024discrimination}. These applications show that
\assure{} offers improvements and robustness over common plug-in empirical Bayes methods.
The \assure-estimated welfare $\hat W$ additionally assesses whether a status-quo decision
rule has reasonable welfare performance under assumptions on decision costs $K_i$. It also
enables researchers to infer $K_i$ under an assumption that a compound decision-maker
performs optimally.
This paper is structured as follows. \Cref{sec:unbiased_welfare_estimate} defines the
{\footnotesize \textsc{ASSURE}} estimator and discusses its bias and variance.
\Cref{sec:theory} presents several theoretical guarantees on selecting a decision using
{\footnotesize \textsc{ASSURE}}, showing upper and lower bounds on regret. We next
highlight extensions to other observation distributions like the Poisson and to
complicated decision procedures in \cref{sec:extensions}. Simulation results and empirical
applications are discussed in \cref{sec:simulations} and \cref{sec:empirical_application}.
\section{Setup \& Methodology}
\label{sec:unbiased_welfare_estimate}
Let $Y_i \sim \Norm(\mu_i,\sigma_i^2), i \in [n] = \br{1,\ldots,n}$ with known $\sigma_i$,
known costs
$K_i$, with potentially related known covariates $
X_i$. The contextual information $Z_i
:=
\left(X_i, K_i, \sigma_i \right)$ and the parameters $\mu_i$ are treated as fixed, and all
probability and expectation statements are over the joint distribution $Y_i \sim \Norm
(\mu_i,\sigma_i^2)$. For simplicity, we assume that $Y_i$ mutually independent across $i$.\footnote{If $
(Y_1,\ldots, Y_n) \sim \Norm((\theta_1,\ldots,\theta_n)', \Sigma)$ for some known
$\Sigma$, then our arguments generalizable by studying the conditional distributions
$Y_i \mid Y_{-i}$.} We also restrict to \emph{separable} thresholds of the form
\[
\delta_i(Y_{-i}) =\delta(Z_i; \beta).
\]
in all classes $\mathcal D$ that we consider. That is, the threshold for unit $i$ does not depend on information for unit $j \neq i$ (except
through estimation of $\beta$).
Having parametrized thresholds with $\beta$, we can rewrite \eqref{eq:welfare_expected} as
\begin{align}
W(\beta) &:= \frac{1}{n}\sum_{i=1}^n (\mu_i - K_i) \P\left(Y_i \geq \delta
(Z_i;\beta)
\right) = \frac1n \sum_{i=1}^n (\mu_i - K_i) \Phi\left( \frac{ \mu_i - \delta(Z_i;\beta)}
{\sigma_i} \right) \label{eq:general_welfare}
\\
u(\beta) &:= \frac{1}{n} \sum_{i=1}^n \one(Y_i > \delta(Z_i;\beta)) \cdot (\mu_i - K_i).
\end{align}
Optimizing \eqref{eq:general_welfare} directly is infeasible since it depends on the
unknown $\mu_{1:n}$. However, if we had access to an estimator $\hat W_n(\beta)$, we
can optimize $\hat W_n$ over a class of threshold rules $\mathcal D = \br{\delta
(\cdot;\beta):
\beta \in \mathcal
B \subset \R^d}$ (e.g., $\mathcal D = \mathcal D_{\textsc{close-gauss}}$), yielding:
\[
\hat\beta \in \argmax_{\beta \in \mathcal B} \hat W_n(\beta).\numberthis
\label{eq:sure_type}
\]
Our subsequent theoretical results control the difference, or \emph{regret}, between the
welfare of the best decision rule in $\mathcal D$ and the expected utility achieved by
$\hat\beta$: \[
\regret_n = \sup_{\beta \in \mathcal B} W(\beta) - \E_{\mu_{1:n}}[u(\hat\beta)].
\numberthis \label{eq:regret}
\] Intuitively, for $\regret_n$ to be small, it is critical that our estimator $\hat W_n$
for welfare is accurate. A good welfare estimator for the compound selection problem is
the focus of this paper.
\subsection{The \assure{} estimator for welfare}
We propose the following estimator for \eqref{eq:general_welfare} with good regret
properties, building
on the deconvolution literature
\citep{kolmogorov1950unbiased,tate1959unbiased,liang2000empirical,pensky2017minimax,zhou2019fourier}. Define the sinc,
sine integral, and cumulative sinc functions as follows:
\[
\sinc(x) = \frac{\sin x}{\pi x} \qquad \Si(x) = \int_0^x \frac{\sin t}{t} dt \qquad
\Csinc(x) = \int_{-\infty}^x \sinc(t)\,dt = \frac{1}{2} + \frac{1}{\pi} \Si(x).
\]
\begin{definition}[{\footnotesize \textsc{ASSURE}}] For a bandwidth parameter $h$,
define
\begin{align}
\label{eq:assure_term}
w_h(Y_i; Z_i, \beta) & := (Y_i - K_i) \Csinc\pr{
\frac{Y_i - \delta(Z_i;\beta)}{\sigma_i h}
} - \frac{\sigma_i}{h} \sinc\pr{
\frac{Y_i - \delta(Z_i;\beta)}{\sigma_i h}
}.
\end{align}
Then the {\footnotesize \textsc{ASSURE}} estimator is given by
\begin{equation}
\label{eq:assure}
\widehat{W}_n(\beta) := \frac{1}{n}\sum_{i=1}^n w_{1/\sqrt{2\log n}} (Y_i; Z_i, \beta).
\end{equation}
We will define the scaling parameter $\lambda_n = \sqrt{2 \log n}$ and set the bandwidth
as $h = 1/\lambda_n$. With this choice of $h$, we will usually omit dependence of $w$ on $h$.
\end{definition}
\begin{figure}[h]
\centering
\includegraphics[width=0.9\linewidth]{Figures/function_plot.pdf}
\caption{A plot of the function $\frac{y}{2} + \frac{y}{\pi} \Si((y-C)/h) - \frac{1}{h} \sinc((y-C)/h)$, for $C = 1$ and various values of the bandwidth $h$.}
\label{fig:assure_function}
\end{figure}
\cref{fig:assure_function} visualizes the function $w_h$ in \eqref{eq:assure_term} for
$K=0$ and $\sigma = 1$. The
estimator $w_h$ is chosen due to its excellent bias properties: The following
proposition shows that its bias decays
exponentially in $1/h$.
\begin{proposition}[{\footnotesize \textsc{ASSURE}} has low bias]
\label{thm:assure_bias}
Fix $\mu, K \in \R$.
Let $Y \sim \Norm(\mu, \sigma^2)$ and $\delta(Z_i;\beta) = \beta$, then $
\lim_{h\to 0}\E_\mu w_h(Y; (\sigma, K), \beta) = (\mu - K) \Phi\pr{\frac{\mu - \beta}
{\sigma}}.
$ The bias attains the following bound: For all $h > 0$, \[
\abs{\E_\mu w_h(Y; (\sigma, K), \beta) - (\mu - K) \Phi\pr{\frac{\mu - \beta}
{\sigma}}} \le |\mu - K| h^2 e^{-\frac{1}{2h^2}}.
\]
Thus, with $h := h_n := \lambda_n^{-1} := 1/\sqrt{2\log n}$,\footnote{We upper bound
bias with $h^2 e^{-Ch^2}$ and variance with $n^{-1} \pr{h^{-4} C_\sigma + h^{-2} C_
{\mu-K}}$ for constants $C$ that depends on $(\sigma_i, \mu_i, K_i)$. Minimization of
corresponding MSE bounds with respect to $h$ shows that $h_n^{-1} \rateeq
\sqrt{C_1\log n - \log C_{2,\sigma} - C_3 \log
\log n} \rateeq \sqrt{C_1 \log n}$. The leading constant $C_1$ does not depend on
$\sigma_i^2$. } the bias is bounded by $C|\mu - K| \frac{1}{n \log n}.$ Under this
choice, pointwise in $\beta$,
\begin{align*}
\E\bk{
\pr{
\hat W_n (\beta) - W(\beta)
}^2
} & \lesssim \frac{(\log n)^2}{n} \left(\frac{1}{n} \sum_{i=1}^n \sigma_i^2 \right) + \frac{\log n}{n} \left(\frac{1}{n} \sum_{i=1}^n (\delta(Z_i;\beta) - K_i)^2 \right).
\numberthis \label{eq:pointwise_mse}
\end{align*}
\end{proposition}
Unlike SURE, {\footnotesize \textsc{ASSURE}} is biased. The bandwidth term $h_n$ is chosen
to ensure the bias has negligible contribution towards mean squared error for estimating
$W(\beta)$. For $W(\beta)$, it turns out that unbiased estimators (with reasonable
growth behavior) do not exist: we formalize in \cref{prop:unbiased_stefanski} an argument
in
\citet{stefanski1989unbiased}.
The next subsection provides heuristic motivation for the functional form
\eqref{eq:assure_term} in {\footnotesize \textsc{ASSURE}} and connects it to SURE, sample splitting, and to a literature
on estimation using Fourier transforms. The subsection after compares our SURE-type
analysis \eqref{eq:sure_type} to other approaches to compound selection problems, such
as empirical Bayes.
\subsection{Intuition for {\footnotesize \textsc{ASSURE}}}\label{sec:heuristic}
Inspecting \eqref{eq:general_welfare}, estimating $W(\beta)$ can be reduced to a Gaussian
estimation problem: Given $Y \sim \Norm(\mu, \sigma^2)$ and $K, \delta \in \R$, we would
like to estimate the parameter $(\mu-K) \Phi\pr{\frac{\mu-\delta}{\sigma}}$. Inspired by
SURE, a natural starting point is \emph{Stein's identity}: For differentiable $F(Y)$ with
derivative $f(Y)$, \[
\E_{Y\sim \Norm(\mu, \sigma^2)}[(Y-\mu) F(Y)] = \sigma^2 \E_{Y\sim \Norm(\mu, \sigma^2)}[f(Y)].
\]
We can rearrange to obtain \[
\E[(Y-K) F(Y) - \sigma^2 f(Y)] = (\mu-K) \E[F(Y)]. \numberthis
\label{eq:stein_identity}
\]
Thus, $(Y-K) F(Y) - \sigma^2 f(Y)$ is an unbiased estimator for $(\mu-K) \E[F(Y)]$. To
obtain a low-bias estimator for $(\mu-K) \Phi\pr{\frac{\mu-\delta}{\sigma}}$, we could
find some differentiable $F$ whose expectation $\E[F(Y)]$ is approximately $\Phi
\pr{\frac{\mu-\delta}{\sigma}} = \E[\one(Y > \delta)].$
Having reduced the problem in this way, by rescaling if necessary, we may assume $\sigma =
1, \delta = 0$ without loss of generality. A natural idea is to smooth the Heaviside
function $H(y) := \one(y > 0)$ with some kernel $k_h(y) = \frac{1}{h} k\pr{\frac{y}
{h}}$---for some kernel $k(\cdot)$ and bandwidth $h$---so that the resulting function is
differentiable. That is, we may consider convolving $H$ with the kernel: \[ F_h(y) := (H
\star k_h)(y) = \int_{-\infty}^{y/h} k(t) \,dt.
\]
Which kernel should we choose? A natural but suboptimal idea is to use the Gaussian kernel $k
(\cdot) = \varphi (\cdot)$.
Doing so yields an estimator that has $O(h^2)$-bias \[
\E\bk{(Y-K) \Phi\pr{\frac{Y}{h}} - \varphi\pr{\frac{Y}{h}}} = (\mu-K) \Phi(\mu) + O
(h^2). \numberthis \label{eq:coupled_bootstrap_form}
\]
Interestingly, this choice of kernel has a natural interpretation as \emph {coupled
bootstrap}
\citep{oliveira2024unbiased,leiner2023data,ignatiadis2025empirical,chen2022empirical}. For
$Y \sim \Norm (\mu, 1)$, we can construct two independent samples by adding and
subtracting an independent Gaussian noise $Q$ \[ Y_1 = Y + h Q \quad Y_2 = Y -
\frac{1}{h} Q \quad Q \sim \Norm(0,1)
\implies Y_1 \indep Y_2.
\numberthis \label{eq:cb}
\] This \emph{coupled bootstrap} procedure is a synthetic version of
sample-splitting.\footnote{If $Y$ is a sample mean of Normally distributed micro-data, $Y
= \frac
{1}{m} \sum_{j=1}^m Y_ {(j)}$ for $Y_ {(j)} \sim \Norm (\mu, m)$, then
sample-split means over $Y_{(1)},\ldots, Y_{(m)}$ can be represented in the form
\eqref{eq:cb}.}
With \eqref{eq:cb}, the welfare of selection decisions based on $Y_1 > 0$ can be
unbiasedly
estimated by $(Y_2 - K)\one(Y_1 > 0)$, since $Y_2$ acts as fresh ``testing data'' that is
independent of the
``training data'' $Y_1$. The estimator \eqref{eq:coupled_bootstrap_form} is exactly the
Rao--Blackwellization of coupled bootstrap: \[
\E[(Y_2 - K)\one(Y_1 > 0) \mid Y] = (Y-K) \Phi\pr{\frac{Y}{h}} - \varphi\pr{\frac{Y}{h}}.
\]
The $O(h^2)$ bias, however, means that the regret rate for coupled bootstrap is $\tilde O(n^
{-4/5})$ (\cref{thm:cb_regret_bound}).\footnote{Throughout, we use $\tilde O(\cdot)$ to
denote the analogue of big-O notation but ignoring logarithmic terms in $n$. }
Instead of a Gaussian kernel, the \assure{} estimator uses the sinc kernel for improved
bias \citep{davis1975mean,
Tsybakov2009IntroNonparametricEstimation}.
Observe that the expectation $\E[F_h]$ convolves $F_h$ with a Gaussian kernel: \[
\E_{Y \sim \Norm(\mu,1)}[F_h(Y)] = (H \star k_h \star \varphi) (\mu),
\]
whereas the target parameter is a convolution without $k_h$, $ \Phi(\mu) = (H\star
\varphi) (\mu). $ Thus, a low-bias kernel is one that barely smoothes the Gaussian
density: $k_h \star \varphi \approx \varphi$. The sinc kernel is motivated by this
approximation in frequency space:\footnote{A large literature on Gaussian estimation and
deconvolution follows similar ideas
\citep{kolmogorov1950unbiased,zhou2019fourier,pensky2017minimax,tate1959unbiased,stefanski1989unbiased}.}
By taking the Fourier transform of both sides, we would like a kernel for which \[
\hat k_h e^{-\omega^2/2} \approx e^{-\omega^2/2} \text{, for } \hat k_h(\omega) :=
\int_\R
k_h (t)
e^{-it\omega}\,dt.
\]
The sinc kernel has Fourier transform $\one(|\omega| < 1/h)$, which only truncates the
high frequency signals in $\varphi$. Since $\hat\varphi(\cdot)$ has Gaussian tails, this
truncation leaves it essentially unchanged---allowing the bias to decay exponentially in
$1/h$ (\cref{thm:assure_bias}).
The identity \eqref{eq:stein_identity} is also suggestive of \assure{}'s robustness to
viewing $Y$ as approximately Gaussian---though we leave a formal analysis to future work.
Let $Z \sim \Norm (\mu, \sigma^2)$ and suppose $Y$ has mean $\mu$ and variance $\sigma^2$,
but is not necessarily Gaussian. Then
\begin{align*} &\E[(Y-K)F(Y) - \sigma^2 f(Y)] \\ &= (\mu-K)\E[F(Y)] + \E[(Y-\mu)F(Y) -
\sigma^2 f(Y)]\\ &= (\mu-K)\E[F(Z)] + (\mu-K)\underbrace{\br{\E[F(Y)]-\E[F(Z)]}}_{A} +
\underbrace{\E[ (Y-\mu)F (Y) - \sigma^2 f (Y)]}_B
\end{align*} The term $A$ measures the discrepancy between the distributions of $Y$ and
$Z$ in terms of $\E[F(\cdot)]$. This term is controlled with various weak convergence
arguments, such as controlling the Wasserstein-1 distance between $Y$ and $Z$. The term
$B$ measures the extent to which $Y$ does not obey Stein's identity. It often controlled
directly by central limit theorems using Stein's method. As long as $F_h$ is
sufficiently well-behaved so that $A$ and $B$ are small by the approximate Gaussianity
of $Y$, we would like $\E[F_h(Z)] \approx \Phi(\mu)$, for which the sinc kernel is
optimizing. Simulation studies in Sections \ref{sec:simulations} and \ref
{sec:empirical_application} show robustness to the Gaussian assumption.
\subsection{Discussion}
We now contextualize our approach in the SURE literature and compare it against a few
alternatives. We also disambiguate our regret notion \eqref{eq:regret} from the regret
notions in these literatures.
\smallskip\noindent\textbf{Stein's unbiased risk estimate.} Our approach exactly mimics
SURE: For $Y \sim \Norm (\mu, \sigma^2)$ and a differentiable estimator $\delta(Y)$, the
squared error risk of $\delta$ is unbiasedly estimable by $T (Y; \delta, \sigma) = (\delta
(Y) - Y)^2 + 2 \sigma^2 \delta'(Y) -
\sigma^2$ \citep{stein1981estimation}, so that
\[
\E_{\mu}\bk{
T(Y; \delta, \sigma)
} = \E_{\mu}[(\delta(Y) - \mu)^2]. \tag{SURE}
\]
Thus, for compound \emph{estimation} problems---where one is interested in
\emph{predicting} $\mu_i$ using $\delta(Y_i; Z_i, \beta)$---one could form a risk
estimator $\hat R_ {\mathrm{SURE}}(\beta) := \frac{1} {n} \sum_{i=1}^n T(Y; \delta(\cdot,
\beta) , \sigma_i)$. This is the strategy pursued in
\citet{xie2012sure,kwon2023optimal,cheng2025optimal}. This research \citep[e.g.,
in][]{kwon2023optimal,cheng2025optimal} often takes selection problems as motivation and
illustrates that resultant shrinkage estimators $\hat\delta(Y)$ for $\mu$ lead to
different selection decisions from naive selection rules. Yet, these shrinkage estimators
are motivated by estimation---it is not obvious that they are simultaneously good for
selection \citep{manski2021econometrics}. Our contribution is to directly target the
selection problem.
\smallskip\noindent\textbf{Empirical Bayes.} A popular framework for compound decision
problems is
\emph{empirical Bayes} \citep{efron2012large}, which models $\mu_i$ as random effects.
This additional distributional structure characterizes optimal decision
rules as posterior quantities involving the estimable distribution of $\mu_i$ \citep{liang1988convergence, liang2000empirical, liang2004optimal,
karunamuni1996optimal,gupta2005empirical,gu2023invidious,kline2024discrimination}.
However, the additional structure hinges on modeling and estimating the
distribution of $\mu_i$ well, which can make subsequent procedures and guarantees less
robust.\footnote{A similar concern---about the robustness of correlated random effect
models---motivates the literature of fixed effect nonlinear panels
\citep{dano2025binary}.} In particular, our procedure can be viewed as a compound
analogue of
\citet{liang2000empirical}, which robustifies empirical Bayes selection.
Our procedure is also complementary to empirical Bayes.
Empirical Bayes-style modeling for $\mu_i$ is powerful at generating classes of decision
rules $\mathcal D$ that are reasonable---indeed optimal if the model on $\mu_i$ happens
to be correct. Choosing a member of $\mathcal D$ with {\footnotesize \textsc
{ASSURE}} provides guarantees for the compound selection problem directly---thus not
requiring researchers to take the random effects model fully seriously. At the same time,
if the random effect model is correct, tuning via {\footnotesize
\textsc{ASSURE}} does little harm.\footnote{For concreteness, a model for $\mu_i$ that
performs well in the empirical exercise in \citet{chen2022empirical} is \[
\mu_i \mid Z_i \sim \Norm(m_0(Z_i; \beta), s_0^2(Z_i; \beta)),
\]
where $m_0, s_0$ are flexibly parametrized by $\beta$.
Under this prior, \begin{align*}
\E[\mu_i \mid Y_i, Z_i] &= m_0(Z_i;\beta) + \frac{s_0^2(Z_i; \beta)}{\sigma_i^2 + s_0^2(Z_i; \beta)}(Y_i
- m_0 (Z_i;\beta))
\end{align*}
This motivates using {\footnotesize \textsc{ASSURE}} to choose among the following threshold decision rules,
derived from inverting $\E[\mu_i \mid Y_i, Z_i] > K_i$ in $Y_i$ :
\begin{equation}
\label{eq:close_gauss_decision_class}
\mathcal D_{\textsc{close-gauss}} = \br{\delta(Z_i;\beta) = K_i + \frac{\sigma_i^2}
{s_0^2(Z_i; \beta)}(K_i - m_0(Z_i;\beta)): \beta \in \mathcal B}.
\end{equation}}
From a formal perspective, the empirical Bayes literature
\citep{jiang2020general,soloff2024multivariate,chen2022empirical} often considers the
difference in $W_{\mathrm{EB}}$ between the oracle Bayes decision rule and the empirical
Bayes decision rule:\[
\mathsf{EBRegret}_n = \E_{\substack{\mu_{1:n} \sim P \\ Y_i \mid \mu_i, Z_i \sim \Norm
(\mu_i,
\sigma_i^2)}} \bk{\frac{1} {n}\sum_ {i=1}^n \max(0, \E_P[\mu_i \mid Y_i, Z_i] - K_i)
- u(\hat P)
},
\]
where $u(\hat P) = \frac{1}{n} \sum_{i=1}^n \one(\E_{\hat P}[\mu_i \mid Y_i, Z_i] > K_i)
(\mu_i - K_i)$. Relative to the regret criterion \eqref{eq:regret} that we subsequently
study, $ \mathsf{EBRegret}_n$ (i) takes another expectation under $\mu_{1:n} \mid Z_{1:n}
\sim P$, (ii) chooses the benchmark as the oracle decision rule $\one(\E_P[\mu_i \mid Y_i,
Z_i] > K_i)$, and (iii) typically considers the class of posterior decision rules indexed
by an estimate of $P$. When the class of decision rules $\mathcal D$ is exactly the class
of posterior thresholding decisions, controlling
\eqref{eq:regret} automatically controls $\mathsf{EBRegret}_n$: \[
\mathsf{EBRegret}_n = \E_{\substack{\mu_{1:n} \sim P \\ Y_i \mid \mu_i, Z_i \sim \Norm
(\mu_i,
\sigma_i^2)}}\bk{W(P) - u(\hat P)} \le \E_{\mu_{1:n}\sim P}\bk{\regret_n} = \E_{\mu_{1:n}\sim P}\bk{\eqref{eq:regret}}.
\]
\smallskip\noindent\textbf{Treatment choice and policy learning.} The decision problem
\eqref{eq:welfare} is a compound version of the ``testing an innovation'' decision problem
in \citet{manski2009identification}, where $\mu_i$ represents the average treatment effect
of a new treatment relative to a known status quo. To connect to the treatment choice
literature
\citep{manski2004statistical,stoye2012minimax,kitagawa2018should,athey2021policy}, the
compound aspect may represent $n$ parallel innovations that the decision-maker is
simultaneously entertaining, or it may represent discrete covariates taking on $n$ values,
wherein $\mu_i$ is the conditional average treatment effect on covariate value
$i$.\footnote{Though, from this perspective, the objective function
\eqref{eq:welfare} assumes that each covariate cell is equally probable.
} The normality
assumption on $Y_i$ corresponds to the limit experiment in \citet{hirano2009asymptotics},
though rigorously justifying this approximation through Le Cam--Hajek-style arguments
remains important future work \citep{https://doi.org/10.3982/ECTA20364}.
Our statistical perspective is distinct from the treatment-choice literature. First, we
aim for decision rules whose regret \eqref{eq:regret} converges to zero as $n\to\infty$,
uniformly over sequences $(\mu_i, Z_i)_{i=1}^n$. These decision rules are not necessarily
minimax-$\mathsf{FreqRegret}$ for any finite $n$, for $\mathsf{FreqRegret}$ defined as
follows \citep [e.g., in][] {ishihara2022shrinkage}:
\[
\mathsf{FreqRegret}_n = \E_{\mu_{1:n}}\bk{
\frac{1}{n} \sum_{i=1}^n \max(0, \mu_i - K_i) - u(\hat\beta)
}.
\]
This means that \assure{} may suffer from higher worst-case $ \mathsf{FreqRegret}$ for any
finite $n$. Conversely, minimax-$\mathsf{FreqRegret}$ decision rules, like the empirical
success rule $a(Y_i) =
\one(Y_i > 0)$ \citep[Proposition 1,][]{stoye2012minimax}, have non-vanishing worst-case
regret $\sup_ {\mu_ {1:n}} \regret_n = \Omega (1)$---meaning that these decisions can be
substantially improved from the perspective of \eqref{eq:regret}. See
\cref{rmk:freqregret} for further comparisons and intuition.
Second, our asymptotic regime is different from the policy learning literature
\citep{kitagawa2018should,athey2021policy,mbakop2021model}. If a unit $i$ is taken to be a
covariate cell, we consider a regime where the number of covariate cells grow large,
perhaps due to finer discretization of continuous covariates. In contrast, the standard
asymptotic regime in the policy learning literature considers increasing sample size
within each covariate cell, which can be approximately written as $\sigma_i^2 \rateeq
c_i/m$ for $m \to \infty$. Thus the regret rates we obtain are not directly comparable to
the regret rates in \citet{kitagawa2018should}.
\begin{remark}
\label{rmk:freqregret}
The comparison between \assure-based decision rules and minimiax-$ \mathsf {FreqRegret}$
decision rules is analogous to the fact that, in estimation settings, the maximum
likelihood estimator $\delta (Y_i) = Y_i$ is a minimax estimator for $\mu_ {1:n}$ in any
dimension, but its squared error risk is dominated by the James--Stein estimator for $n
\ge 3$ \citep{lehmann1998theory}. Indeed, the James--Stein estimator is a minimizer of
SURE under homoskedasticity, up to a degree-of-freedom adjustment. Related examples, for
the advantages of compound decisions, are found in
\citet{robbins1985asymptotically,sun2007oracle,koenker2024empirical}.
To intuitively understand how {\footnotesize \textsc{ASSURE}} improves over the empirical
success rule $\one(Y_i \ge K_i)$, note that certain aggregate statistics like $m :=
\frac{1} {n} \sum_ {i=1}^n
(\mu_i - K_i)$ are precisely estimable from the data. Estimates of these statistics are
informative. For example, if our estimate for $m$ is greater than zero, we expect to
perform better by considering less conservative decision rules such as $\delta_i =
\mathbf{1}\set{Y_i \geq K_i - \e}.$ If our estimate $m$ were significantly less than zero,
we would prefer a more conservative decision such as $\delta_i =
\mathbf{1}\set{Y_i \geq K_i + \e}$, for appropriate choices of $\e$. {\footnotesize
\textsc{ASSURE}} thus adapts the selection decision to features of the data, implicitly
betting on the estimates for $m$ to be accurate. Fixed decision rules---often motivated by
concerns about the estimation error for $m$---lack this adaptivity.
\end{remark}
\section{Theoretical assurances}
\label{sec:theory}
This section presents statistical guarantees on {\footnotesize \textsc{ASSURE}}. The main
results are upper and matching lower bounds for expected {regret} \eqref{eq:regret}. We
first present general upper and lower bounds for $\tilde O(1/\sqrt{n})$ regret. Additional
assumptions on $\mu_{1:n}$ allow for a faster rate by exploiting an analogue of the margin
condition \citep{audibert2005fast}.
\subsection{Main regret bound}
Our main result is the following $(1/\sqrt{n})$-regret bound.
\begin{restatable}[$\tilde O(1/\sqrt{n})$-regret for {\footnotesize \textsc{ASSURE}}]{theorem}{thmmain}
\label{thm:main_regret_bound}
Define the quantities
\begin{align*}
s_k := \left(\frac{1}{n}\sum_{i=1}^n \sigma_i^k \right)^{1/k} \quad
m_k := \left(\frac{1}{n}\sum_{i=1}^n |\mu_i - K_i|^k \right)^{1/k} \quad
\nu_k := \left(\frac{1}{n}\sum_{i=1}^n \frac{|\mu_i - K_i|^k }{\sigma_i^k}\right)^{1/k}.
\end{align*}
Suppose that the class of decision rules $\delta(z;\beta)$ forms a VC subgraph class with index $V(\cl{D})$ and envelope function $D$. Let $\widehat{\beta} = \argmax_\beta \widehat{W}(\beta)$ be the optimal decision chosen by {\footnotesize \textsc{ASSURE}}. Then the regret satisfies
\[
\regret_n(\widehat{\beta}) \lesssim \frac{m_1}{n \log n} + M\frac{\log n}{\sqrt{n}}
\]
with $M := \sqrt{V(\cl{D})} (s_2 + m_2 + (1+\nu_4) (\E \norm{D}_n + (\E \norm{D}_{4,n}^2)^{1/2}) ).$
\end{restatable}
Thus, assuming that $s_2,m_2,\nu_4$ are bounded and the envelope function $D$ is
controlled, the procedure achieves $\frac{\log n}{\sqrt{n}}$ regret in general. Constants
in the regret rate depend on various norms of the parameters. If all parameters are
bounded, we have the following corollary, showing a simple $\log n /\sqrt{n}$ rate.
\begin{corollary}
\label{cor:bounded_assure_corollary}
Assume that $\mu_{1:n}, \sigma_{1:n}, \sigma_{1:n}^{-1}, K_{1:n}$ are bounded by an absolute constant $B_1$, not depending on $n$. Let $\cl{D}$ be VC subgraph with envelope function $D$ uniformly bounded by a constant $B_2$. Then
\[
\sup_{\mu_{1:n}} \regret_n(\widehat{\beta}) = O\left( \frac{\log n}{\sqrt{n}}\right).
\]
\end{corollary}
\Cref{thm:main_regret_bound} builds on a standard argument in empirical risk
minimization:\footnote{ See, e.g., 8.4.3. in \citet{vershynin2009high}. Regret guarantees
\eqref{eq:regret} are related to, but are distinct from, excess risk control in empirical
risk minimization. Standard empirical risk minimization (e.g., Theorem 8.4.4 in
\citet{vershynin2009high}) controls---in our notation---the welfare gap $W (\beta^*) -
\E[W(\hat\beta)]$. This is a different quantity than \eqref{eq:regret} since $\E[W
(\hat\beta)]\neq \E[u(\hat\beta)]$. $\E[W (\hat\beta)]$ imagines evaluating $\hat\beta$ on
a new draw of $Y_i$, whereas $\E[u (\hat\beta)]$ evaluates $\hat\beta$ on the same sample
of draws $Y_{1:n}$.} from the optimality of
$\beta^*$ and $\hat\beta$,
\begin{align}
\regret_n & \le 2 \E\bk{\sup_\beta |\hat W(\beta) - u(\beta)|}
\label{eq:regret_decompose_main_bound1} \\
& \le 2\E\bk{\sup_\beta
|\hat W(\beta) - W(\beta)|} + 2\E\bk{\sup_\beta
|u(\beta) - W(\beta)|}.
\label{eq:regret_decompose_main_bound}
\end{align}
The $1/\sqrt{n}$-rate follows from analyzing the empirical processes on the right-hand
side---in particular, $\beta \mapsto \hat W (\beta) - W(\beta)$. \Cref{thm:assure_bias}
shows that $|\hat W (\beta) - W(\beta)| = O_P(\sqrt{\log n} / n)$ pointwise. Empirical
process arguments control this uniformly at the price of an additional $\sqrt{\log n}$
factor.\footnote{The VC subgraph assumption is standard but not crucial, as long as an
appropriate covering number of the decision class is controlled. We state our results in
terms of the VC subgraph dimension for simplicity.} We modify standard empirical process
theory to accommodate independent but not identically distributed data in our setting.
\subsection{Examples}
\label{sec:examples}
We now give several examples of function classes with finite VC
dimension.
\begin{example}[Simple truncation rules]
\label{ex:truncation_class}
Consider decision rules \[\cl{D}_{\text{threshold}} = \br{\delta(z;\beta) = k +
\beta : \beta \in [-M,M]}.\]
Let $\beta$ be restricted to a compact set $[-M,M].$ $\cl{D}_{\text{threshold}}$ is VC
subgraph with a uniform envelope function $k + M$, satisfying the requirements of
\cref{thm:main_regret_bound,cor:bounded_assure_corollary}. $\cl{D}_{\text{threshold}}$ is
common; see \citet{crippa2025regret} for several examples.
\end{example}
\begin{example}[Thresholding $t$-statistics]
\label{ex:t_stats}
Consider decision rules \[\cl{D}_{\text{$t$-stat}} = \br{
\delta(z;\beta) = k + \beta \sigma : \beta \in [-M, M ]}
\] Such rules nest selection via
one-sided $p$-values, common in digital experimentation \citep{sudijono2024optimizing}
and medicine \citep{manski2019treatment}.\footnote{These decision rules can be shown to be
Bayes optimal for a class of exponentially tilted improper priors.} For $\sigma_i$
bounded, $\cl{D}$ is VC subgraph with a uniformly bounded envelope function.
\end{example}
\begin{example}[Finite Classes of Decision Rules] Suppose now a discrete decision class $\cl{D}_{\text{finite}}$ consisting of $p$ different decision thresholds
\[
\set{\delta^{(1)}(z), \dots, \delta^{(p)}(z)}.
\]
A finite class of functions is VC subgraph. If $\sup_{i=1,\dots,p} \delta^{(i)}(z)$ is
uniformly bounded, then \cref{thm:main_regret_bound,cor:bounded_assure_corollary} apply.
By standard arguments, convex combinations of these decision rules are also VC subgraph.
\end{example}
\begin{example}[Gaussian Empirical Bayes Models]
\label{ex:close_gauss_example}
Recall in Eq. \eqref{eq:close_gauss_decision_class} the decision rules obtained by imposing the prior
\[
\mu_i \mid Z_i \sim \Norm(m_0(Z_i; \beta), s_0^2(Z_i; \beta)),
\]
and selecting a unit if its posterior mean is greater than $K_i$. Rearranging this yields
\[
\mathcal D_{\textsc{close-gauss}} = \br{\delta(Z_i;\beta) = K_i + \frac{\sigma_i^2}
{s_0^2(Z_i,\beta)}(K_i - m_0(Z_i,\beta)): \beta \in \mathcal B}.
\]
Particular choices of $m_0,s_0^2$ yield important examples. For example, taking both $m_0
= \mu_0,s_0^2 = \tau^2$ to be constants independent on $Z_i$ generates \textit{linear
shrinkage} decision rules where the parameters are $\beta = (\mu_0,\tau):$
\begin{equation}
\label{eq:linear_shrinkage_decisions}
\cl{D}_{\text{linear-shrink}} := \set{\delta(Z_i;\beta) = K_i + \frac{\sigma_i^2}{\tau^2}(K_i - \mu_0): \beta \in \bR^2}.
\end{equation}
Next, taking $s_0^2$ to be a constant and $m_0$ to be the linear model $X_i^\top \beta$ where $X_i$ are auxiliary covariates yields the well-known \textit{Fay-Herriot model} \citep{fay1979estimates}:
\begin{equation}
\label{eq:fay_herriot}
\cl{D}_{\text{Fay-Herriot}} := \set{K_i + \frac{\sigma_i^2}{A}(K_i - X_i^\top \beta): (A,\beta) \in \bR^{1+p}}.
\end{equation}
For either model, empirical Bayes procedures typically plug in estimates of $\mu_0,\tau^2,
A$
using the method of moments or maximum likelihood. See \cite{ignatiadis2019covariate} and
\cite{luo2023empirical} for other empirical Bayes methods which utilize covariates or side
information.
Finally, \cite{chen2022empirical} discusses fitting models $m_0(\sigma_i),s_0^2(\sigma_i)$
only depending on $\sigma_i$. \citet{chen2022empirical} considers fitting $m_0,s_0^2$
nonparametrically and also
parametrically using the model
\begin{align}
\label{eq:parametric_close_gauss_decisions}
\mathcal D_{\textsc{close-gauss}} & = \br{K_i + \frac{\sigma_i^2}
{s_0^2(Z_i; \beta)}(K_i - m_0(Z_i;\beta)): \beta \in \mathcal B} \\
m_0(Z_i;\beta) & = a_1 + a_2\sigma_i\\
s_0^2(Z_i;\beta) & = \exp(b_1 + b_2\log \sigma_i),
\end{align}
with $\beta = (a_1,a_2,b_1,b_2) \in \bR^4$. {\footnotesize \textsc{ASSURE}} can be used to
tune the parameters $\beta$ for the parametric specification. We also discuss how to use
{\footnotesize \textsc{ASSURE}} to improve on the nonparametric method in Section
\ref{sec:extensions}.
These decision threshold classes are VC subgraph: the thresholds for linear shrinkage
\eqref{eq:linear_shrinkage_decisions} and the Fay-Herriot model \eqref{eq:fay_herriot} can
be reparametrized to be linear. \cref{prop:vc_subgraphness_examples}, via model theory,
shows that parametric {\footnotesize \textsc{CLOSE-GAUSS}} is VC-subgraph.
\end{example}
\subsection{Lower bound}
\label{sec:lower_bound}
\Cref{thm:main_regret_bound} is unimprovable, up to log factors, in the worst case.
\begin{restatable}[Matching Regret Lower Bounds]{theorem}{thmmatchinglb}
\label{thm:matching_lb}
Fix $\sigma_i = 1$ and $K_i = 0$ for all $i$. Without loss of generality, assume $W
(\beta; \mu_{1:n})$ is defined with respect to the thresholds $\delta(Z_i;\beta) = \beta
\in \bR$. Let $M > 0$, then
over \emph{all} decision rules $a_i(Y_1,\ldots, Y_n) \in [0,1]$,
\[
\inf_{a(\cdot) : a_i \in [0,1]} \sup_{\mu_{1:n} \in [-1,1]^n} \regret_n(a_{1:n}) \ge \frac{C}{\sqrt{n}},
\]
where
\begin{equation}
\label{eq:gen_regret}
\regret_n(a_{1:n}) := \sup_{|\beta| \le
M}
W (\beta; \mu_
{1:n}) -
\frac{1}{n}\sum_{i=1}^n \mu_i \E_{\mu_{1:n}}[a_i(Y_1,\ldots,
Y_n)].
\end{equation}
Here, $C$ is a constant that only depends on $M$.
\end{restatable}
Here, the regret notion in \cref{thm:matching_lb} compares the welfare of an arbitrary
decision $a_i(Y_1,\dots,Y_n)$ to the expected welfare of the best decision in the
thresholding class restricted to $|\beta| \leq M$. Lower bounds for this quantity thus
imply lower bounds when we allow the oracle to be more flexible than using $\delta =
\beta$.\footnote{Since the setting in \cref{thm:matching_lb} does not have contextual
information, the optimal separable decision rule does take the threshold form. An
analogous regret notion was considered by \cite{polyanskiy2021sharp} in their Eq. (12) for
the mean square case.}
\cref{thm:matching_lb} is derived by considering cases in which, for some $h > 0$, either
$\mu_ {i} = h/ \sqrt{n}$ for all $i$ or $\mu_i = -h/\sqrt{n}$ for all $i$. In this case,
the optimal $\beta^*$ is either $+M$ or $-M$, meaning that $W(\cdot)$ is maximized on the
boundary instead of at a local maximum. In fact, when $\beta^*$ is a local maximum, the
upper and lower bounds can be improved to $\tilde O(1/n)$.
\subsection{Fast Rates for {\footnotesize \textsc{ASSURE}}}
\label{sec:fast_rates}
In particular settings, the performance for {\footnotesize \textsc{ASSURE}} can be better than the
$\tilde{O}(n^{-1/2})$ rate predicts. Technically, this is due to the fact that the
derivatives of {\footnotesize \textsc{ASSURE}} also estimate the derivatives of welfare
\eqref{eq:welfare_expected} as shown in \cref{thm:derivative_bias}. This analysis shares similarities with the kernel estimator introduced by
\cite{liang2000empirical} for the random effect case.
The assumptions needed for the improved rate are reminiscent of the \textit{margin
condition} used to achieve fast rates in classification and treatment choice
\citep{audibert2005fast,audibert2007fast,kitagawa2018should,ponomarev2024lower,crippa2025regret}.
There, the margin condition bounds the density of difficult examples near the optimal
classification boundary. Intuitively, with many units near the classification boundary, it
is difficult to distinguish between different candidate parameter values, meaning that the
objective function lacks a cleanly separated maximizer. This is exactly the failure mode
that \cref{thm:matching_lb} exploits for the lower bound, which we rule out in
\cref{as:fast_rate_assn}.
For simplicity in the proof, we will
restrict to the $d=1$ case which covers a range of examples. We leave the multidimensional
extension to future work. Suppose the following conditions hold.
\begin{as}
\label{as:fast_rate_assn}
\begin{enumerate}[label=(A\arabic*), ref=(A\arabic*)]
\item \label{assmp:boundedness} (Boundedness). Suppose that $\mu_i,K_i,\sigma_i,\sigma_i^{-1}$ are all bounded in absolute value by a constant $B$. Furthermore, suppose that the decision rules $\delta(\cdot,Z_i)$ are smooth and $L_i$-Lipschitz for each $i$. Further, assume that the decision rules and all derivatives are uniformly bounded by a uniform constant $B$.
\item \label{assmp:unimodal} (Unimodality). There exists a global maximizer $\beta^*$ at which $W(\beta^*) > 0;$ moreover $\beta^*$ is the unique local and global maximizer.
\item \label{assmp:curvature} (Curvature). $W''(\beta^*) < -\kappa$ for some positive $\kappa > 0$, not depending on $\mu_1,\dots,\mu_n$.
\item \label{assmp:well_sep_max} (Well-separated Maximum). There exists $\xi > 0$ not depending on $\mu_1,\dots,\mu_n$ such that
\[
\sup_{|\beta - \beta^*| > \kappa/B} W(\beta) < W(\beta^*) - \xi.
\]
\end{enumerate}
\end{as}
For convenience in the proof, we additional conditions on the decision rules and search space.
\begin{restatable}[Fast Rates for {\footnotesize \textsc{ASSURE}}]{theorem}{thmfastrates}
\label{thm:fast_rates}
Suppose \cref{as:fast_rate_assn} holds. Furthermore, suppose that $\beta$ is restricted to
a compact space $\Theta$ of finite volume, which
includes $\beta^*.$ Lastly, suppose that the
derivatives $\delta'(\beta,z), \delta''(\beta,z)$ form VC subgraph function classes with
index $V(\cl{D}'), V(\cl{D}'')$. Then the regret of {\footnotesize \textsc{ASSURE}} satisfies
\[
\E\left[W(\beta^*) - u(\widehat{\beta})\right] = O\left( \frac{(\log n)^5}{n}\right).
\]
\end{restatable}
\Cref{ex:fast_rate_class} in \cref{sec:example_fast_rate_class} illustrates a class of
decision rules and lower-level assumptions on $\mu_{1:n}$ for which \cref
{thm:fast_rates} applies. A heuristic argument for \cref{thm:fast_rates} is as follows.
If $\hat\beta$ is a local maximum of $\hat W$, then from a first-order Taylor expansion,
\begin{align*}
0 = \hat W'(\hat\beta) & \approx \hat W' (\beta^*) + \hat W''(\beta^*) (\hat\beta -
\beta^*). \\
& = \hat W' (\beta^*) - W'(\beta^*) + \hat W''(\beta^*) (\hat\beta -
\beta^*).
\end{align*}
It can be shown that $\hat W'$ estimates $W'$ uniformly at rate $\tilde O(1/\sqrt{n})$,
from which it follows that $\hat\beta - \beta^* = \tilde O_P(1/\sqrt{n})$. Another Taylor
expansion $W (\hat\beta)\approx W (\beta^*) + W''(\beta^*)(\hat\beta - \beta^*)^2$ shows
that the welfare gap is $\tilde O (1/n)$: $W (\hat\beta) - W(\beta^*)= \tilde O_P(1/n)$.
Finally, we consider a leave-one-out stability argument to control $\E[W(\hat\beta) -
u(\hat\beta)]$.\footnote{The log factors are likely not tight and removable with a more
refined application of empirical process theory. }
Finally, we present a matching lower bound for the fast rate.
\begin{restatable}[Fast Regret Lower Bounds]{theorem}{propfastratelb}
\label{prop:gaussian_matching_lb_fast_rate}
Consider the homoskedastic $\sigma_i = 1$ costless case with the decision rules $\set{Y_i \geq \beta}$ ranging over $\beta \in \bR$. Let $\Theta = \Theta_{n,\kappa,\xi} \subseteq \bR^n$ be the space of $(\mu_1,\dots,\mu_n)$ satisfying the assumptions \ref{assmp:boundedness}-\ref{assmp:well_sep_max} for a fixed $\xi, \kappa$. Then there exists choices of $\xi$ and $\kappa$ such that
\[
\inf_{a(\cdot) : a_i \in \set{0,1}} \sup_{\mu_{1:n} \in \Theta^n} \regret_n(a_{1:n}) = \Omega(n^{-1}),
\]
with $\regret_n(a_{1:n})$ defined as in \eqref{eq:gen_regret}.
\end{restatable}
The intuition for \cref{prop:gaussian_matching_lb_fast_rate} is to lower bound the
compound regret by the Bayes regret for a well-chosen prior and then further lower bound
the Bayes regret by Le Cam's two point argument.\footnote{Section 3.1 of
\citet{liang2004optimal} states a $\Omega(n^{-1})$ lower bound for a related but distinct
regret quantity, where the decision rule is learned from data points $Y_1,\dots,Y_n$ and
evaluated on a newly drawn decision problem $(\mu_{n+1},Y_{n+1})$. We are concerned with
an ``in-sample'' version of \citet{liang2004optimal}'s regret, motivating a distinct proof
strategy.} We will show that the Bayes regret for this problem can be interpreted as a
type of weighted classification loss, building on an insight by
\citet{polyanskiy2021sharp}. As a result, \cref{prop:gaussian_matching_lb_fast_rate} may
be of independent interest for empirical Bayes testing with linear loss.
\section{Extensions}
\label{sec:extensions}
\subsection{Other observation distributions}
\label{sec:extensions_other_distributions}
The approach---finding an estimator for $W$ and optimizing this estimator over
decisions---can be extended to settings where the observation distribution is
non-Gaussian. For certain exponential families like the Poisson distribution and
exponential distribution, one can find an exactly unbiased estimate for the welfare $W$.
The intuition for these cases is to exploit a formula akin to
Tweedie's formula. We focus on the Poisson setting as it
frequently appears in empirical Bayes and compound decisions.\footnote{See Chapter 6 of
\cite{efron2021computer} for an overview of classic applications of the Poisson model to
insurance claims, the missing species problems, and medical applications. See
\cite{jana2022optimal, jana2023empirical} for modern methods to this problem and also
\citet{montiel2021empirical} for econometric applications.}
In this setting, let $\mu_i \geq 0$ and $Y_i \sim \Poi(\mu_i)$. Again, associated with
each decision problem is a cost $K_i$ and auxiliary information $Z_i$.\footnote{For
example, analogous to $\sigma_i$ in the Gaussian case, $Z_i$ might include the known
length $t_i$ of the observation window for the Poisson outcome $Y_i$. If the unknown
parameter $\mu_i$ denotes the Poisson rate per unit time, then $Y_i \sim \Poi(\mu_i t_i)$.
The decision rule might incorporate the observation window $t_i$. } Consider any
integer-valued decision rule $\delta(Z_i;\beta)$ parametrized by $\beta$. Then the
estimator
\begin{equation}
\label{eq:sure_poisson}
\widehat{W}_{\mathsf{Poi}}(\beta) := \frac{1}{n}\sum_{i=1}^n \left( Y_i \mathbf{1}\set{Y_i \geq \delta(Z_i;\beta) + 1} - K_i\mathbf{1}\set{Y_i \geq \delta(Z_i;\beta)} \right)
\end{equation}
is the analog for {\footnotesize \textsc{ASSURE}}.
\begin{restatable}{proposition}{proppoissonestimator}
\label{prop:poisson_estimator}
The Poisson {\footnotesize \textsc{ASSURE}} estimator $\widehat{W}_{\mathsf{Poi}}(\beta)$ is unbiased for the welfare
\begin{equation}
\label{eq:poisson_welfare}
W(\beta;\mu_{1:n}) := W(\beta) := \frac{1}{n}\sum_{i=1}^n \left(\mu_i - K_i \right) \Prob(Y_i \geq \delta(Z_i;\beta)).
\end{equation}
\end{restatable}
Under mild conditions, {\footnotesize \textsc{ASSURE}} achieves $1/\sqrt{n}$ regret, which
is optimal. In contrast to the Gaussian case, the discreteness of the Poisson distribution
suggests that there is no fast-rate regime.
\begin{restatable}[Regret upper bounds for the Poisson Case]{theorem}{thmpoissonub}
\label{thm:poisson_ub}
Let $m_k := \left(\frac{1}{n}\sum_{i=1}^n \mu_i^k \right)^{1/k}.$ Suppose the class of decision rules $\cl{D}$ is a VC subgraph class with index $V(\cl{D})$. Then {\footnotesize \textsc{ASSURE}} for Poisson observations has regret at most an absolute constant times
\[
\frac{1}{\sqrt{n}} \sqrt{V(\cl{D})} (m_2 + \sqrt{m_1}).
\]
\end{restatable}
We prove the upper bound by appealing to the similar empirical process theory arguments as
in the Gaussian case. We suspect some of the assumptions of \cref{thm:poisson_ub} may be
further relaxed by. Mirroring the argument of the Gaussian case, we have a matching lower
bound. Proofs of these results are in \cref{sec:proofs_for_sec_extensions}.
\begin{restatable}[Matching Regret Lower Bounds]{theorem}{thmmatchingpoissonlb}
\label{thm:matching_lb_pois}
Fix costs $K_i = K > 0$, and let $\Theta$ be some compact interval containing any neighborhood of $K$. We have
\[
\inf_{a(\cdot) : a_i \in \set{0,1}} \sup_{\mu_{1:n} \in \Theta^n} \regret_n(a_{1:n}) = \Omega(n^{-1/2}),
\]
where $\regret_n(a_{1:n})$ is defined analogously to \eqref{eq:gen_regret} except with
$\mu_i$ replaced with $\mu_i - K$.
\end{restatable}
\begin{remark}
Consider the case $K_i = K$ and the threshold class of decisions $Y_i \geq \delta, \delta \in \bR$. An alternative empirical Bayes approach to the Poisson selection problem is to threshold the \textit{Robbins estimator} given by
\begin{equation}
\widehat{\mu}(y) := (y+1) \frac{N_{y+1}}{N_y}
\end{equation}
where $N_y := \#\set{i : Y_i = y}$. The estimator can also be used for selection in the
strict compound case. That is, select all indices $i$ for which the Robbins estimator is
greater than $K$. By rearranging $\widehat{\mu}(Y_i) \geq K$, one can see that truncating
the Robbins estimator bears a passing resemblance to the {\footnotesize \textsc{ASSURE}}
decision. However, truncating the Robbins estimator does not respect monotonicity. The
estimator \eqref{eq:sure_poisson} regularizes and enforces this monotonicity, giving a
more stable and intuitive alternative to decisions based on the Robbins estimator
\citep{brown2013poisson, jana2022optimal}.
\end{remark}
\subsection{Complex Decisions}
\label{sec:loo_fitted_decisions}
We discuss how to extend the {\footnotesize \textsc{ASSURE}} framework to accommodate
complex, non-separable decision rules $\delta(\beta,Z_{1:n},Y_{-i})$ which may depend on
the outcomes and auxiliary information of other units. For example, an analyst might
consider making decisions using empirical Bayes posterior mean estimates
$\E_{\hat{G}}[\mu_i \mid \sigma_i, Y_i]$ and flexible machine-learning models
$\hat{f}(X_i)$ that estimate the regression $\E[Y_i\mid X_i]$. In both of these cases,
it would be desirable to estimate $\hat{G}$ and model $\hat{f}$ using the data
itself.
By a similar argument to \cref{thm:assure_bias}, one can create a near unbiased
estimator of the welfare of this decision rule using a leave-one-out cross fitting
construction. Redefine the {\footnotesize \textsc{ASSURE}} summand \eqref{eq:assure_term}
with the decision rule $\delta(\beta,Z_{1:n},Y_{-i})$, so that
\[
w_h(Y_i; \beta,Z_i,Y_{-i}) :=
(Y_i - K_i) \Csinc\pr{
\frac{Y_i - \delta(\beta,Z_{1:n},Y_{-i})}{\sigma_i h}
} - \frac{\sigma_i}{h} \sinc\pr{
\frac{Y_i - \delta(\beta,Z_{1:n},Y_{-i})}{\sigma_i h}
}.
\]
Then,
\begin{align*}
\frac{1}{n} \sum_{i=1}^n \E w_h(Y_i; \beta,Z_i,Y_{-i}) & =\frac{1}{n} \sum_{i=1}^n \E\left[\E\left[w_h(Y_i; \beta,Z_i,Y_{-i}) \ | \ Y_{-i} \right] \right] \\
& \approx \frac{1}{n} \sum_{i=1}^n \mu_i \E\left[ \Phi\left(\mu_i - \delta(\beta,Z_{1:n},Y_{-i}) \right) \right],
\end{align*}
which can be interpreted as the average welfare of using the non-separable decision rule.
\Cref{thm:assure_bias} shows that the bias in the above approximation is small.
Given this observation, a simple construction is the following. Let $m_{\hat
G_1}^{(-i)}(y,\sigma)$ be a posterior mean function obtained by applying an empirical
Bayes method on the data $(Y_{-i},\sigma_{-i})$. Let $m_{\hat G_2}^{(-i)}(y,\sigma)$
denote the same for a different empirical Bayes method. Finally, let $\widehat{f}^{(-i)}$
denote a machine learning model trained using the data $(Y_{-i},X_{-i})$. Consider the
class of decision rules which selects a unit whenever the ensemble prediction of these
models is greater than the implementation cost:
\begin{equation}
\label{eq:ensemble_class}
b_1 m_{\hat G_1}^{(-i)}(Y_i,\sigma_i) + b_2 m_{\hat G_2}^{(-i)}(Y_i,\sigma_i) + b_3 \widehat{f}^{(-i)}(X_i) \geq K_i
\end{equation}
where the parameters $\beta = (b_1,b_2,b_3)$ are restricted to be in the unit simplex.
Extensions to multiple empirical Bayes posterior mean models or machine learning models is
straightforward. By the monotonicity of the posterior mean function, the left hand side of
\eqref{eq:ensemble_class} is increasing in $Y_i$: the decision is equivalent to $Y_i \geq
\delta(\beta,Z_{1:n},Y_{-i})$ for some function $\delta$. As a result, {\footnotesize
\textsc{ASSURE}} can be used to tune the parameter $\beta$. Under mild stability
assumptions and for large enough samples, {\footnotesize \textsc{ASSURE}} is expected to
perform no worse than the constituent models since the decision class nests each model
individually. This ensemble class thus immediately enables {\footnotesize \textsc{ASSURE}}
to improve on any fixed empirical Bayes decision method and illustrates the utility of the
framework. We defer a detailed theoretical analysis of the ensembling method for future
work.
\section{Simulation Results}
\label{sec:simulations}
This section conducts two calibrated simulations where the $\mu_i$'s are constructed by
sampling from some estimated empirical Bayes model. In each draw of the simulation, we
then sample $Y_i \sim \Norm(\mu_i ,\sigma_i^2)$. Code to replicate these simulations and
the empirical applications in the next section may be found
\href{https://github.com/tsudijon/CompoundWelfareMaximization/tree/main}{on Github}.
\subsection{Calibrated Opportunity Atlas Simulation}
\label{sec:oa_synthetic_application}
We first consider a calibrated simulation based on the Opportunity Atlas (OA) dataset from
\citep{chetty2018opportunity,bergman2024creating}. See \cref{sec:oa_application} for
further background on the underlying dataset. The data for this simulation comes from a
simulation exercise of \cite{chen2022empirical} and is generated by sampling $\mu_{1:n}, n
\approx 10^4,$ from a Monte Carlo sample of an empirical Bayes prior fitted on the OA
dataset. We compare several classes of methods by evaluating the in-sample welfare
obtained by the chosen decision using the known $\mu_i$. In particular, we compare (a) the
linear shrinkage class in Eq. \eqref{eq:linear_shrinkage_decisions}, (b) the Fay--Herriot
class of Eq. \eqref{eq:fay_herriot}, and (c) the {\footnotesize \textsc{CLOSE-GAUSS}}
decision class of Eq. \eqref{eq:close_gauss_decision_class}. In each of these three
classes, we compare (i) empirical Bayes with plug-in estimates of prior parameters against
the decisions selected by (ii) {\footnotesize \textsc{ASSURE}} and (iii) coupled
bootstrap
\eqref{eq:coupled_bootstrap_form}. In addition we include two {\footnotesize
\textsc{NPMLE}} methods, and finally an ensemble tuned using {\footnotesize
\textsc{ASSURE}} as in \cref{sec:extensions}. Complete details on the simulation setting
are given in \cref{sec:additional_numerical_results}, alongside further numerical results.
\begin{figure}[htb]
\centering
\includegraphics[width=\linewidth]{Figures/oa/OA_comparison_normal_mc_runs40_cost0.361_goodX.pdf}
\caption{Semisynthetic simulation comparison on Opportunity Atlas dataset. Box and whisker plots summarize 40 Monte Carlo runs where only $Y_i$ are redrawn. Constants costs are taken with $K = 0.361;$ the rationale behind this choice is discussed in \cref{sec:oa_application}.
}
\label{fig:oa_semisynthetic_well_specified_goodX}
\end{figure}
\cref{fig:oa_semisynthetic_well_specified_goodX} presents our results. For the linear
shrinkage and Fay-Herriot classes, {\footnotesize \textsc{ASSURE}} consistently produces
decisions that improve over the respective empirical Bayes baselines. These improvements
highlight that \assure{} robustifies empirical Bayes procedures, improving performance
when empirical Bayes models are misspecified. Indeed, these respective empirical Bayes
methods assume that the $\mu_i$ are drawn from Gaussian priors with a fixed constant
variance. This assumption is questionable in the present dataset, as can be seen in
\cref{fig:oa_viz}. When the model is misspecified, {\footnotesize \textsc{ASSURE}} selects
a better-performing set of parameters within the class. This gain can be significant, as
seen in the linear shrinkage class.
For empirical Bayes models which are fairly well-specified, {\footnotesize
\textsc{ASSURE}} does not do harm and performs comparably to the respective empirical
Bayes method, as shown by the comparison between methods
{\footnotesize\textsc{CLOSE-GAUSS}} and the {\footnotesize \textsc{ASSURE}}-chosen
decision in the class. As expected, ensembling produces decisions which outperform
{\footnotesize\textsc{CLOSE-GAUSS}} and {\footnotesize\textsc{CLOSE-NPMLE}}. Since the
covariates in this setting are predictive of the true effects, we expect the ensemble
method to strictly outperform the empirical Bayes counterpart. Across these simulations,
{\footnotesize \textsc{ASSURE}} and coupled bootstrap have similar performance. Though the
theory for {\footnotesize \textsc{ASSURE}} suggests a better rate of $\tilde O(n^{-0.5})$
as opposed to $\tilde O(n^{-0.4})$, the difference appears negligible in practice unless
$n$ is very large. However, a small difference might be economically significant,
especially for large $n$. The performance of our methods appear robust to the normality
assumption, as shown in \cref{fig:oa_semisynthetic_misspecified_purenoise}.
\cref{sec:additional_numerical_results} details additional specifications where both
normality is misspecified and the covariates are uninformative.
\subsection{Calibrated Experimentation Program Dataset}
\label{sec:synthetic_experimentation_program_application}
Next, we present results on a semisynthetic dataset derived from the experimentation
program dataset of Section \cref{sec:experimentation_program_application}. The
specification is somewhat challenging due to smaller size of the dataset ($n \approx 330$)
and the large heterogeneity in the $\sigma_i$, as seen in
\cref{fig:experimentation_program_viz}. On this dataset, we evaluate three classes of
decision rules: (a) the $t$-statistic/$p$-value threshold rules given
in \cref{ex:t_stats}, (b) linear shrinkage rules, and (c) the Fay-Herriot decision rules.
We also consider the performance of standard nonparametric maximum likelihood
\citep{jiang2020general}.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figures/experimentation_programs/exp_program_comparison_informativeX.pdf}
\caption{Experimentation program semisynthetic simulation, with $X_i = \mu_i + \sigma_i t_{10}, (\rho = 0.7)$. Boxplots show 100 Monte Carlo comparisons where $Y_i$ is rerandomized.}
\label{fig:experimentation_semisynthetic}
\end{figure}
\Cref{fig:experimentation_semisynthetic} shows the results. Importantly, both
{\footnotesize \textsc{ASSURE}} and the coupled bootstrap method consistently identify a
better choice of $t$-statistic threshold than the fixed decision rule corresponding to $p
< 0.05$. The performance of both {\footnotesize \textsc{ASSURE}} and coupled bootstrap are
comparable to the corresponding empirical Bayes methods in the linear shrinkage class and
Fay--Herriot class, but somewhat more variable due to the smaller size of this dataset. In
contrast to the data from the Opportunity Atlas in \cref{sec:oa_application}, the
empirical Bayes assumptions underlying the plug-in methods are not unreasonable for this
dataset, and thus {\footnotesize \textsc{ASSURE}} does not provide improvements.
Nevertheless, even with a good empirical Bayes model, applying {\footnotesize
\textsc{ASSURE}} does not significantly harm performance.
\section{Empirical Applications}
\label{sec:empirical_application}
\subsection{The Opportunity Atlas.}
\label{sec:oa_application}
The Opportunity Atlas \citep{chetty2018opportunity} provides census-tract level estimates
of economic mobility as measured by a suite of children's outcomes in adulthood. For each
tract, multiple economic indicators such as earnings and incarceration rates are estimated
by strata such as parental income, race, and sex. Building on the Opportunity Atlas
dataset, \cite{bergman2024creating} conducted the randomized trial \emph{Creating
Moves to Opportunity} (CMTO), offering low-income families housing vouchers and
assistance to move to neighborhoods with historically high upward mobility. Implicit in
defining highly upward-mobile tracts is a compound selection problem where the goal is to
select the top 1/3 upwardly-mobile tracts. \citet{bergman2024creating} and \citet
{chen2022empirical} approach this selection problem using empirical Bayes.
We consider the related selection task as defined in \eqref{eq:welfare} on a particular
economic mobility outcome pictured in \cref{fig:oa_viz}, where the mobility measure is the
average adult income rank of children whose parents are at the 25th percentile of income.
For each Census tract $i$, we decide whether or not to select it as high-mobility.
\begin{figure}[H]
\centering
\includegraphics[width=\linewidth]{Figures/oa/oa_viz.pdf}
\caption{Visualization of Opportunity Atlas estimates from \citep{chetty2018opportunity, bergman2024creating}. Each point represents a census tract within the largest 20 Commuting Zones, measuring the household income rank in adulthood for Black children, with genders pooled, whose parents were at the 25th percentile of income. }
\label{fig:oa_viz}
\end{figure}
We consider the class of linear shrinkage decision rules given in
\eqref{eq:linear_shrinkage_decisions} with specifications of $K_i$ to be discussed below.
Recall that this class of decisions can be motivated by assuming that $\mu_i \sim
\mathcal{N}(m_0, s_0^2)$, popular in many empirical economic contexts
\citep{walters2024empirical}, which motivates the decision class:\footnote{With constant
costs, the linear shrinkage decision class simplifies to this form.}
\begin{equation}
\label{eq:linear_shrinkage_constant_costs}
\cl{D}'_{\text{linear-shrink}} = \set{\delta(Z_i;\beta) = K + \beta \sigma_i^2: \beta \in \bR}.
\end{equation}
We compare fitting the plug-in empirical Bayes estimator discussed in
\cref{ex:close_gauss_example} to the {\footnotesize \textsc{ASSURE}} chosen decision in
the class \eqref{eq:linear_shrinkage_constant_costs}. We consider three settings for costs
$K_i$, which will be assumed constant for simplicity, given by $\set{0.2, 0.361, 0.369}$.
The first specification corresponds roughly to the treatment cost in
\citep{bergman2024creating}. The latter two choices are different specifications which
emulate the top 1/3 selection exercise in Bergman. Details are given in
\cref{sec:additional_numerical_results}.
\begin{figure}[H]
\centering
\includegraphics[width=\linewidth]{Figures/oa/opportunity_atlas_analysis.pdf}
\caption{{\footnotesize \textsc{ASSURE}} estimate curves for linear shrinkage class with constant costs on the OA dataset of Figure \ref{fig:oa_viz}. See text for an explanation of the different cost regimes. Grey lines indicate 1.96 times the standard errors of the {\footnotesize \textsc{ASSURE}} estimate, calculated pointwise.}
\label{fig:oa_analysis}
\end{figure}
\Cref{fig:oa_analysis} shows the results. For $K = 0.2$, the majority of census tracts
likely have a parameter $\mu_i$ which is greater than the cost $K = 0.2$, so the optimal
decision is to select more tracts. In this case, the empirical Bayes plug-in and
{\footnotesize \textsc{ASSURE}} chosen decision are comparable in performance. In the two
other specifications $K \in \set{0.361, 0.369},$ the cost tradeoffs are more meaningful.
The {\footnotesize \textsc{ASSURE}} welfare estimate suggests that the empirical Bayes
plug-in is meaningfully suboptimal compared to the optimal decision, which takes $\beta$
slightly less than zero. \cref{fig:oa_analysis_decision0.361} visualizes the difference,
where {\footnotesize \textsc{ASSURE}} selects more aggressively.
\begin{figure}[H]
\centering
\includegraphics[width=0.8\linewidth]{Figures/oa/oa_decision_viz_cost0.361.pdf}
\caption{{\footnotesize \textsc{ASSURE}} decisions and empirical Bayes decisions overlayed on Opportunity Atlas data. $K = 0.361,$ matching the middle panel of \cref{fig:oa_analysis}.}
\label{fig:oa_analysis_decision0.361}
\end{figure}
This example shows that the {\footnotesize \textsc{ASSURE}} estimator has utility beyond
just selecting a near-optimal decision through a black box. Consider again the left-hand
side of \cref{fig:oa_analysis}. The estimated curve suggests that decisions where $\beta
\leq 0$ are all roughly equivalent in terms of expected welfare. Policy-makers can then
choose a decision amongst this near-optimal subset on the basis of other criteria, such as to maximize another metric, or for interpretability.
{\footnotesize
\textsc{ASSURE}} therefore enables decision-makers to certify a status-quo
decision if the estimated welfare is near-optimal. In other settings, it may also reveal
that a status quo decision might be far from optimal, as discussed in the next empirical
application.
\subsection{Experimentation Programs.}
\label{sec:experimentation_program_application}
In the technology industry, experimentation programs are large collections of related A/B
tests where the goal is to move a metric of interest \citep{azevedo2020b,
sudijono2024optimizing, chou2025evaluating}. By explicit randomization, the treatment
effect estimates $Y_i$ are plausibly normally distributed around their true unknown
average treatment effects $\mu_i$ \citep{goldstein2007l1,li2017general}. $\mu_i$ are measured in some metric of practical business interest.
We take a dataset of anonymized treatment effects from Netflix
\citep{sudijono2024optimizing}, consisting of treatment effects and standard deviations
for $331$ feature experiments. \Cref{fig:experimentation_program_viz} shows some
descriptive statistics for the data.
Our goal is to select a subset $S$ of innovations $i$ to implement in the platform.
\cite{azevedo2020b} explicitly introduce the problem of choosing a subset of indices $i$
in order to maximize the welfare \eqref{eq:welfare} in the Bayes setting where $\mu_{1:n}
\iid G$. Past work typically takes the empirical Bayes approach where $G$ is estimated
from large repositories of past A/B tests. In some cases, the Bayes assumption may be too strong. For example, innovations are often created as variants of the same idea,
suggesting correlation between the returns to innovations. In new experimentation
programs, no past data may be available to estimate $G.$ In the absence of a trustworthy
prior, the compound decision formulation is a compelling alternative.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figures/experimentation_programs/experimentation_program_viz.pdf}
\caption{Left: Treatment effect estimate vs. standard deviation estimate for a dataset of 330 feature experiments from a large online tech company. The orange line and orange dots highlight tests which pass the traditional $p < 0.05$ criteria for statistical significance. Right: histogram of the treatment effects estimates. Both treatment effects and standard errors were multiplied with different random factors to preserve confidentiality.}
\label{fig:experimentation_program_viz}
\end{figure}
The status quo decision procedures in the industry is the $t$-statistic decision class of
\cref{ex:t_stats}. Traditionally, the chosen threshold corresponds to a one-sided
$p$-value of $0.05$. We use {\footnotesize \textsc{ASSURE}} to select the threshold
$\beta$ that maximizes the expected welfare of the decision $W(\beta) := \sum_{i=1}^n
(\mu_i - K_i) \Prob(Y_i \geq \beta\sigma_i)$ with $K_i = 0.$
\cref{fig:welfare_curve_example_experimentation_programs} shows the {\footnotesize
\textsc{ASSURE}} estimate $\widehat{W}(\beta)$ as a function of the curve $C$. By
maximizing the curve $\widehat{W}(\beta)$, {\footnotesize \textsc{ASSURE}} suggests to use
a decision corresponding to $C \approx 0$.
The data in this application are anonymized: both the treatment effect estimates and
standard deviations are obfuscated by random multiplicative factors. Thus, the optimal
decision cannot directly be interpreted, though we hope this section serves as a template
for the reader to conduct similar exercises on their own datasets. Despite this,
the shape of the estimated welfare curve suggests that a lenient threshold is more
warranted than a conservative one, questioning the optimality of the industry-standard $p
< 0.05$ decision rule. Other papers in the experimentation literature find similar
conclusions under different frameworks \cite{azevedo2020b,
berman2022false, sudijono2024optimizing}.
\begin{figure}
\centering
\includegraphics[width=0.75\linewidth]{Figures/experimentation_programs/experimentation_program_welfare_curve.pdf}
\caption{
Blue line shows welfare estimate $\widehat{W}(\beta)$ for decisions of the form $Y_i \geq \beta\sigma_i$. Anonymized data on treatment effects from a single experimentation program. As such, the $y$-axis scale is obfuscated. }
\label{fig:welfare_curve_example_experimentation_programs}
\end{figure}
As reflected in the literature, the suboptimality of $p < 0.05$ in the compound setting is
not surprising when $K=0$. In general, measuring costs for A/B tests is a difficult
firm-specific exercise. Often costs are implicit, reflecting a host of factors including
upfront costs of
implementing and increasing future complexity of the
platform.
A useful exercise is to back out the implicit costs that justify the $p < 0.05$ decision
rule and to use this as a heuristic benchmark when choosing a $p$-value threshold in
practice. As a start, we assume that costs associated with each decision are given by
$\kappa$ times the median treatment effect size across the dataset, for various levels of
$\kappa$. The left panel shows the estimates $\widehat{W}(\beta)$ for various levels of
$\kappa$. As $\kappa$ grows, the optimal decision generally becomes more conservative. The
right panel of \cref{fig:example_exp_program_cost_analysis} shows the {\footnotesize
\textsc{ASSURE}}-optimal $t$-statistic threshold as a function of the cost multiplier. The
true optimal threshold is generally non-decreasing.\footnote{The estimated threshold in
\cref{fig:example_exp_program_cost_analysis} is decreasing due to finite-sample effects
and the form of the {\footnotesize \textsc{ASSURE}} estimator. The discontinuous jumps are
however also present for the true optimal thresholds.} Crucially,
\cref{fig:example_exp_program_cost_analysis} suggests that the $p < 0.05$ can be
rationalized by costs which are roughly twice the median treatment effect.
\begin{figure}
\centering
\includegraphics[width=0.99\linewidth]{Figures/experimentation_programs/experimentation_program_welfare_costs.pdf}
\caption{Left: {\footnotesize \textsc{ASSURE}} estimated welfare curves for $t$-statistic decision class $Y_i \geq K + \beta \sigma_i$ with constant costs $K$ corresponding to multiples of the median absolute value of treatment effect. Right: optimal $t$-statistic decision threshold chosen by {\footnotesize \textsc{ASSURE}}. Grey dashed line corresponds to the $p < 0.05$ decision. }
\label{fig:example_exp_program_cost_analysis}
\end{figure}
\subsection{Discrimination in Large Firms.}
In this empirical illustration, we demonstrate how selection decisions at the individual
level can differ when using ASSURE versus thresholding on nonparametric EB estimates.
\cite{kline2022systemic} conducted a large correspondence experiment that sent up to
1,000 job applications to each of 108 firms. Here $\mu_i$ is the firm contact gap and
$Y_i$ is the average difference in callback across pairs of resumes within a firm.
Specifically, each firm in the sample had approximately 125 job listings throughout the
experiment. For each job, applications were sent in pairs, one with a distinctly
white-sounding name and the other with a distinctly black-sounding name. Other attributes
in the resume were randomized. The difference in callback rates between races is then
averaged over jobs to create $Y_i$. Due to the explicit paired randomization, the
assumption of normality---similarly made in \citet{kline2022systemic}---is fairly
justified.
Based on an auditor with an ``extensive margin'' preference, \cite{kline2022systemic} found 23 firms to
discriminate against black applicants, controlling false discovery rates to the 5\% level. To adapt their setting for our illustration, we instead consider an auditor with an ``intensive margin'' preference, whose objective is to maximize the expected welfare gain from an investigation selection rule $S$ given by $\sum_{i=1}^n
(\mu_i -K) \Prob\set{i \in S}$ where $K$ is the cost of an investigation. Using the
nonparametric EB estimates of \cite{kline2022systemic}, adapted from \cite{efron2016empirical}, a Bayesian interpretation implies setting $K=0.025$ to match with the selection of 23 firms,
To demonstrate how ASSURE can make different selections in practice, we consider the linear shrinkage class with constant cost $K=0.025$ as specified in Eq. \eqref{eq:linear_shrinkage_constant_costs}.
\begin{figure}[ht]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/firm/firm_gaps_white_objective_K_0025.pdf}
\caption{Estimated Average Welfare}
\label{fig:firm_risk_white}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/firm/firm_gaps_white_estimates_vs_se_K_0025.pdf}
\caption{Selection}
\label{fig:firm_risk_male}
\end{subfigure}
\caption{{\footnotesize \textsc{ASSURE}} and linear shrinkage rule on data from \cite{kline2022systemic}}\label{fig:discrimination_white_assure}
\end{figure}
As shown in Figure~\ref{fig:discrimination_white_assure}(b), comparing with the nonparametric EB estimates of \cite{kline2022systemic}, the selections by {\footnotesize \textsc{ASSURE}} are largely concordant, including a few more but excluding a firm with noisy estimate. As argued before, the linear shrinkage class can also be motivated with a parametric normal prior. Figure~\ref{fig:discrimination_white_assure}(a) shows that directly plugging in estimates for $(m_0, s_0)$ would give a higher threshold and select fewer firms than {\footnotesize \textsc{ASSURE}}.
\section{Conclusion}
\label{sec:discussion}
In this work, we introduce {\footnotesize \textsc{ASSURE}}, a data-driven approach for
compound selection decisions. By using a near-unbiased estimate of the welfare,
{\footnotesize \textsc{ASSURE}} targets an instance-specific optimal decision while making
only mild assumptions on the unknown parameters $\mu_{1:n}$. This robustifies empirical
Bayes procedures where it may be difficult to justify distributional assumptions on
$\mu_i$. Section \ref{sec:empirical_application} discusses the
pertinence of the Bayes assumption in our empirical applications. Another strength of
{\footnotesize \textsc{ASSURE}} is its flexibility and ability to validate the performance
of a wide range of decision procedures which may involve covariates and other auxiliary
information. The construction of Section \ref{sec:extensions} allows decision-makers to
improve on any black-box empirical Bayes decision by ensembling. In applications with large $n$, the {\footnotesize
\textsc{ASSURE}}-optimal decision procedure consistently improves over plug-in empirical
Bayes estimates. At the same time, the {\footnotesize \textsc{ASSURE}} estimator is also a useful tool for
heuristic decision making: the nearly-unbiased welfare estimates produced by
{\footnotesize \textsc{ASSURE}} allows a policy-maker to screen out clearly
poorly-performing decision procedures or validate a fixed decision rule. For these theoretical and empirical reasons, we envision {\footnotesize \textsc{ASSURE}} to be a useful tool for selection decisions in practice.
\newpage
\bibliography{main}
\bibliographystyle{ecca}
\newpage