Using Prior Studies to Design Experiments: An Empirical Bayes Approach

-0.5 in Using Prior Studies to Design Experiments: An Empirical Bayes Approach

Keywords: Experimental design, meta-analysis, empirical bayes, statistical decision theory, stratified randomized experiments

JEL Classification: C11, C93, C90, C44

Introduction

Researchers often design new randomized controlled trials (RCTs) in settings where related evidence already exists. That evidence may come from earlier RCTs, quasi-experiments, or observational studies that estimate conceptually similar parameters in different contexts---such as labor supply elasticities, returns to schooling, or treatment effects of related interventions. How can results from prior studies be used ex ante to make a new experiment more informative for the decisions it is meant to support? This paper develops an empirical Bayes (EB) approach to optimal experimental design that turns collections of prior-study estimates into a data-driven prior and then optimizes the new design under that prior.

The core idea is to treat study-level parameters as draws from an unknown cross-study distribution $G$. A collection of published estimates---often only point estimates and standard errors---contains information about $G$, and hence about the parameter in the next setting. Empirical Bayes provides a natural way to estimate $G$ and obtain a predictive prior for the new study. The designer can then choose sampling and assignment rules to maximize expected downstream performance under a specified objective. For example, a planner running a stratified RCT could use prior subgroup-effect estimates to shift experimental effort toward groups where external evidence indicates higher uncertainty or larger expected gains.

Our starting point is the classical Bayesian experimental design framework of Lindley1972, which formulates design as a statistical decision problem: the researcher first chooses a design; then observes experimental data generated under the chosen design; and finally takes an action (such as an estimator, a treatment rule, or a policy recommendation) to maximize a welfare function that depends on both the action and the unknown parameter. A design is valuable when it tends to generate data that lead to better downstream actions given uncertainty about the study parameter.

Implementing this framework hinges on the prior distribution that represents that uncertainty. In applied work, researchers often want to avoid subjective priors, yet they also do not want to ignore the information embedded in a growing body of related studies. We therefore learn the prior from prior-study results. Rather than specifying a hyperprior over $G$ as in fully Bayesian hierarchical meta-analysis, we estimate $G$ from prior evidence and treat the resulting $\hat G_n$ as the prior for design and analysis of the new experiment. We consider two practical routes: a parametric Gaussian specification and a nonparametric prior estimator given by the nonparametric maximum likelihood estimator (NPMLE) for Gaussian location mixtures KieferWolfowitz1956.

To make the approach concrete, we study propensity-score designs for stratified randomized experiments in Section (ref). A key motivation is that prior studies often report treatment effect estimates of common subgroups---such as race, income, or gender---and the new experiment is designed as a stratified RCT that adopts those subgroups as its randomization strata. The experimenter chooses stratum-specific treatment propensities subject to feasibility constraints (e.g., a fixed overall budget), creating tradeoffs across strata. We characterize EB-optimal propensity designs under three canonical objectives: (i) minimizing Bayes risk for estimating a target parameter such as the average treatment effect, (ii) maximizing in-experiment welfare when treatment is beneficial CariaEtAl2024, and (iii) maximizing the expected value of a post-experiment policy adoption decision KasySautmann2021. Across objectives, the EB prior affects design through different channels---prior variances under estimation objectives, prior means under welfare objectives, and both means and variances under policy objectives---yielding distinct assignment rules.

We then provide theoretical guarantees for the plug-in EB design in Section (ref). We first establish a finite-sample oracle inequality: the design regret relative to the oracle that knows $G$ is bounded by twice the uniform error in the value functional induced by replacing $G$ with $\hat G_n$. Regret consistency then follows from the weaker, verifiable primitive that $\hat G_n$ converges weakly to $G$: under mild continuity and envelop conditions, weak convergence of the prior estimator implies that the uniform value error $\Delta_n\to 0$ in probability, which in turn drives regret to zero. For a broad class of decision problems in which welfare is affine in the payoff-relevant state, we derive first-order regret rates that mirror the accuracy of the prior estimator: $\operatorname{Reg}_n=O_{p}(n^{-1/2})$ under Gaussian EB and $\operatorname{Reg}_n=O_{p}((\log n)^{(d+\max(d/2,4))/2}/\sqrt{n})$ under NPMLE. Under additional smoothness and curvature conditions on the design objective, regret becomes a second-order effect and the corresponding rates improve to $O_{p}(n^{-1})$ and $O_{p}((\log n)^{d+\max(d/2,4)}/n)$, respectively. We also characterize design comparisons in the Gaussian experiment, where one observes a normal observation with design-dependent noise covariance. In particular, we show that the prior is irrelevant for design choice only in the genuinely univariate parameter case; with multivariate parameters, the optimal design generally depends on the prior, and this is precisely where empirical Bayes can improve design.

In Section (ref), we present two empirical applications that illustrate the practical implementation of the propensity-score design developed in Section (ref). The first application uses oncology drug trial results from an online clinical-trials database to construct an EB prior for the design of a new immunotherapy trial. Here, the main practical challenge is assembling a usable prior-study dataset from heterogeneous clinical studies with varying estimands and subgroup definitions. The second application treats each site in the Tennessee Project STAR class-size experiment as a prior study, and uses the cross-site distribution of stratum-level treatment effects to design a stratified RCT for a new site. This application highlights how different objectives lead to qualitatively different EB-optimal designs, as each objective channels prior information through a distinct feature of the prior distribution.

This paper relates to several strands of literature. First, the paper relates to the Bayesian optimal experimental design literature, which formalizes design as expected utility maximization under a prior Lindley1972, ChalonerVerdinelli1995,RainforthEtAl2024. Most of this work takes the prior as given and focuses on approximating and optimizing expected utilities in complex models. Our focus is complementary: we study how the prior used for design can be learned from a growing archive of prior-study estimates via empirical Bayes, and we provide regret guarantees that quantify the impact of prior estimation on design performance.

Second, it connects to Bayesian evidence synthesis and the construction of informative priors from historical studies. Bayesian meta-analysis uses cross-study heterogeneity to form predictive distributions for effects in new settings SuttonAbrams2001,Meager2019,Meager2022,CrostaEtAl2024,SchorfheideYou2025, and the clinical-trials literature develops explicit “prior-from-history” tools such as meta-analytic-predictive (MAP) priors and robust MAP mixtures NeuenschwanderEtAl2010,SchmidliEtAl2014, as well as commensurate and power-prior formulations HobbsEtAl2011,IbrahimChen2000,IbrahimEtAl2015. LinEtAl2022 study how hierarchical meta-analysis can inform phase I trial design. Relative to this literature, our contribution is to (i) embed the predictive distribution from prior studies directly inside an explicit optimal design problem; (ii) use empirical Bayes to estimate the cross-study distribution, avoiding hyperprior specification while still borrowing strength; and (iii) analyze a general objective class that includes not only estimation accuracy and testing but also welfare- and policy-oriented criteria.

Third, the paper is related to work on incorporating prior information into experimental practice. IacovoneEtAl2025 illustrate how informative priors---elicited from experts and stakeholders---can sharpen Bayesian impact evaluation in small-sample field experiments. FinanPouzo2026 develop a Bayesian model-averaging approach that combines multiple prior studies with uncertain external validity to improve learning about treatment effects, and FinanPouzo2024 study how multiple prior sources can be incorporated into the design of adaptive experiments. We differ by focusing on the design of a new one-shot experiment when the researcher has access to a growing collection of prior-study estimates, and by providing oracle-style regret guarantees for the resulting empirical Bayes design.

Fourth, the paper relates to the empirical Bayes literature. Foundational work includes Robbins1964 and KieferWolfowitz1956, with modern results on rates and efficiency JiangZhang2009,SahaGuntuboyina2020. Empirical Bayes and related shrinkage ideas are widely used in economics to improve precision in high-dimensional estimation problems (for example, in teacher value-added and neighborhood effect measurement; see, e.g., ChettyFriedmanRockoff2014,ChettyHendren2018, and in experimental settings with many treatment effects to estimate; see AzevedoEtAl2019, AdusumilliEtAl2025). Unlike the classical compound-decision view, we use empirical Bayes to learn the prior entering the Bayesian experimental design objective. We then study how plugging in this estimated prior affects the performance of the resulting design choice, via regret relative to the oracle design that knows the true prior.

The remainder of the paper is organized as follows. Section (ref) presents the general decision-theoretic framework. Section (ref) discusses how to construct the EB prior $\hat G_n$ from prior-study summaries under parametric Gaussian and nonparametric NPMLE approaches. Section (ref) studies EB propensity-score designs in stratified randomized experiments under several canonical objectives. Section (ref) develops regret bounds and rates for the EB approach, and characterizes the comparison of Gaussian experiments. Section (ref) presents two empirical applications on drug-trials and Project STAR.

General Setup

This section sets up (i) how prior studies provide information about study-level parameters, (ii) a decision-theoretic criterion for evaluating designs, and (iii) an empirical Bayes procedure that estimates the cross-study distribution and then chooses a design optimal under the estimated prior.

\noindentPrior studies and a cross-study distribution. A researcher observes results from $n$ prior independent studies, indexed by $i=1,\ldots,n$, and wishes to design a new experiment (study $n{+}1$) that targets a parameter $\theta_{n+1}\in\mathbb R^d$. Prior study $i$ reports an estimate $\hat\theta_i$ and a consistent covariance estimate $\Sigma_i\in\mathbb R^{d\times d}$, which we treat as known and condition on. We use the standard asymptotic normal approximation \[ \hat{\theta}_i \mid \theta_i, \Sigma_i \ \sim \ \mathcal N(\theta_i,\Sigma_i). \] Let $\mathcal D_n=\{(\hat\theta_i,\Sigma_i)\}_{i=1}^n$ denote the historical evidence.

To connect studies, we impose exchangeability: $\theta_i \stackrel{iid}{\sim} G$ for $i=1,\ldots,n{+}1$. This assumption captures the idea that studies estimate conceptually similar but context-specific parameters, with heterogeneity summarized by $G$. A natural extension allows conditional exchangeability given study features $x_i$, i.e.\ $\theta_i\mid x_i\sim G_{x_i}$; for clarity we work with the unconditional case.

A leading parametric specification is the parametric normal model $\theta_i \mid \tau,\nu \stackrel{iid}{\sim}\mathcal N(\tau,\nu)$. Here, $\nu$ captures the degree of external validity SchorfheideYou2025: when $\nu$ is small, historical studies are tightly clustered and are informative about the next study; when $\nu$ is large, learning from history is weak and the optimal design reverts toward a prior-free benchmark.

\noindentDesign as a decision problem. Following Lindley1972, design is chosen before data are observed to maximize ex-ante expected welfare. The timing of the researcher is: (1) observe previous evidence $\mathcal D_n$ and choose design $\eta\in\mathcal H$; (2) observe experimental data $Y\in\mathcal Y$ generated under likelihood $p(y\mid\theta_{n+1};\eta)$; (3) choose an action $a\in\mathcal A$ based on $(\mathcal D_n,Y)$ to maximize welfare $W(a,\theta_{n+1})$. The design affects what the researcher learns about $\theta_{n+1}$, not $\theta_{n+1}$ itself. We assume that the new data $Y$ are independent of past evidence $\mathcal D_n$ conditional on $\theta_{n+1}$.

Given a prior $\Pi(\cdot\mid\mathcal D_n)$ over $\theta_{n+1}$, the ex-ante value of design $\eta$ is

where $\lambda$ is a dominating measure on $\mathcal Y$, and the second equality follows from the conditional independence assumption. Intuitively, a design is valuable if it tends to produce data that move posterior beliefs in directions that matter for downstream decisions. The exact form of $\eta$ is application-specific (treatment assignment rules, sampling algorithm, etc.); its only role here is to determine $p(\cdot\mid\theta;\eta)$.

\noindentEmpirical Bayes prior construction and EB-optimal design. A fully Bayesian approach would place a hyperprior on $G$ and update twice--first using evidence from previous studies and then using data from the new experiment. The EB approach instead estimates $G$ from $\mathcal D_n$ and plugs the estimated distribution into the design problem. Let $\hat G_n$ be an estimator of $G$ based on $\mathcal D_n$. The EB posterior after observing $Y=y$ is \[ \hat\Pi_{\eta}(d\theta\mid \mathcal D_n,y) := \frac{p(y\mid \theta;\eta)\,\hat G_n(d\theta)} {\int_{\Theta} p(y\mid \vartheta;\eta)\,\hat G_n(d\vartheta)}. \] Define the plug-in EB design objective by replacing $\Pi(\cdot\mid\mathcal D_n)$ with $\hat G_n$ in (ref): \[ U_{\hat G_n}(\eta) = \int_{\mathcal Y} \left[ \sup_{a\in\mathcal A} \int_{\Theta} W(a,\theta)\,p(y\mid \theta;\eta)\,\hat G_n(d\theta) \right]\lambda(dy). \] The EB-optimal design is any maximizer $\eta^{EB}\in\operatorname*{argmax}_{\eta\in\mathcal H}U_{\hat G_n}(\eta)$.

Estimating the Prior from Previous Studies

The EB design approach treats the cross-study distribution $G$ as an object that can be learned from a collection of prior studies. This section briefly discusses how $G$ can be estimated in practice from the study-level summaries that are typically available in applied work. The goal is not to introduce new estimation methods, but rather to clarify the two common estimation approaches and their trade-offs.

Gaussian Prior

A widely-used working model in meta-analysis is the parametric Gaussian specification

combined with $\hat\theta_i\mid \theta_i \sim \mathcal N(\theta_i,\Sigma_i)$. Integrating out $\theta_i$ yields the marginal model

The Gaussian EB estimator $(\hat\tau,\hat V)$ is typically defined as a maximizer of the profile marginal likelihood induced by (ref), possibly under structure on $V$ (diagonal, block-diagonal, or low-rank) to improve interpretability and stability.

A convenient feature of (ref) is that, for fixed $V$, the mean $\tau$ has a closed-form generalized least squares (GLS) estimator:

Plugging $\hat\tau(V)$ into the log marginal likelihood yields a profile objective in $V$:

and $\hat V$ can be obtained by numerical maximization of (ref). Given $\hat V$, the EB prior for the new study is $\hat G_n=\mathcal N(\hat\tau(\hat V),\hat V)$.

Nonparametric Prior via NPMLE

When the Gaussian working model is too restrictive, a natural alternative is to estimate $G$ nonparametrically. Under $\hat\theta_i\mid\theta_i\sim \mathcal N(\theta_i,\Sigma_i)$, the marginal density of $\hat\theta_i$ is the Gaussian location mixture $f_{G,i}(x)=\int \varphi_{\Sigma_i}(x-\vartheta)\,dG(\vartheta)$, where $\varphi_{\Sigma}$ denotes a mean-zero multivariate normal density with covariance $\Sigma$. The Kiefer--Wolfowitz nonparametric MLE (NPMLE) estimates $G$ by maximizing the mixture likelihood over all probability measures on $\mathbb R^d$ KieferWolfowitz1956:

The optimization problem (ref) is infinite-dimensional but convex in $G$.

\noindentComputation in multidimensional $\theta$. In moderate dimension, a standard approach is to approximate (ref) by restricting $G$ to a finite grid $\{\vartheta_k\}_{k\le K}\subset\mathbb R^d$ and optimizing over weights $(w_k)_{k\le K}$ on that grid: \[ \max_{w\in\Delta_K}\ \sum_{i=1}^n \log\!\Big(\sum_{k=1}^K w_k\,\varphi_{\Sigma_i}(\hat\theta_i-\vartheta_k)\Big), \qquad \Delta_K:=\{w\ge 0:\sum_k w_k=1\}. \] This yields a finite-dimensional convex program (often solved by interior-point methods or EM-type fixed point iterations) and is the basis of widely-used implementations in the univariate case KoenkerMizera2014, KoenkerGu2017. For multivariate and heteroscedastic Gaussian location mixtures, FengDicker2018 propose an approximate NPMLE based on convex optimization, and SoloffGuntuboyinaSen2025 develop tractable approximations with oracle properties for the resulting empirical Bayes posterior means. Complementary theoretical results for the multivariate Gaussian location-mixture NPMLE are developed by SahaGuntuboyina2020. Scalable computation for large-scale mixture NPMLEs is discussed by ZhangCuiSenToh2024.

\noindentTradeoffs. Relative to the parametric Gaussian specification, the NPMLE allows heavy tails, multimodality, and other shapes in $G$ that are empirically common in cross-study heterogeneity. The tradeoff is that nonparametric estimation is computationally harder, and the resulting regret rates in Section (ref) are typically slower than parametric rates.

Propensity Score Designs in Stratified Randomized Experiments

A recurring feature of applied empirical work is that prior studies report heterogeneous treatment effects broken down by a common set of subgroups---defined by characteristics such as race, income, gender, or age. When a researcher plans a new experiment in a related setting, a natural design choice is to use exactly those reported subgroups as randomization strata, running a stratified RCT. This section specializes the EB framework to that setting. The central design problem is that resource constraints force tradeoffs across strata: increasing precision in one stratum mechanically reduces precision elsewhere. We show that an EB prior constructed from previous studies can materially change optimal treatment assignment.

Design Class and Design-Induced Likelihood

Let $(Y(0),Y(1),X)$ denote potential outcomes and pre-treatment covariates in the target population. The experimenter observes $X$ prior to assignment and chooses a covariate-dependent propensity score \[ e(x)\equiv \Pr(D=1\mid X=x), \qquad e:\mathcal X \to [\underline e,\,1-\underline e], \] where $\underline e\in(0,1/2)$ is an overlap bound. Conditional on $X$, treatment is assigned independently as $D\mid X\sim \mathrm{Bernoulli}(e(X))$, and the realized outcome is $Y = DY(1)+(1-D)Y(0)$.

\noindentDesign class. Stratified randomization is implemented through a fixed partition of the covariate space that is shared with the prior studies. Let $S=s(X)\in\{1,\dots,G\}$ denote the pre-specified stratification and let $\pi_g := \Pr(S=g)$. A propensity-score design is a vector $e=(e_1,\dots,e_G)'\in[\underline e,1-\underline e]^G$ such that \[ \Pr(D=1\mid S=g)=e_g,\qquad g=1,\dots,G. \] We impose a standard resource constraint

and we write $\mathcal H\subset[\underline e,1-\underline e]^G$ for the resulting feasible set.

\noindentParameter of interest. Define the design-relevant parameter as the stratum-specific average treatment effects (ATE) \[ \tau_g := \mathbb E\!\left[Y(1)-Y(0)\mid S=g\right],\qquad g=1,\dots,G, \] and collect them in $\theta:=(\tau_1,\dots,\tau_G)'\in\mathbb R^G$. From $\theta$ one can recover common estimands in applied work: the population ATE is $\tau^{ATE}=\sum_g \pi_g \tau_g=\pi'\theta$, while subgroup effects correspond to selecting coordinates of $\theta$.

\noindentGaussian likelihood induced by the design. Let $\hat\theta=(\hat\tau_1,\dots,\hat\tau_G)'$ denote the within-stratum difference-in-means estimators computed from the new experiment. We adopt the following normal approximation, which is standard in large samples and convenient for design analysis.

Under within-stratum difference-in-means, a canonical expression is

with $N_g$ the stratum sample size (typically $N_g\approx N\pi_g$). The nuisance quantities $\sigma_{dg}^2$ can be estimated from pilot data, administrative sources, or related experiments. In what follows we treat them as fixed for design purposes; extending the EB prior to include them is conceptually straightforward but notationally burdensome.

Prior-Study Information

The EB prior summarizes what related studies reveal about treatment effect heterogeneity in the new setting. For the design analysis below, it is convenient to work with a harmonized parameterization: each prior study delivers an estimate of the same stratum-effect vector.

Formally, let $\theta_i\in\mathbb R^G$ denote the study-$i$ analogue of the stratum-effect vector (defined using the same stratification map $s(\cdot)$ after harmonization), and suppose study $i$ reports $(\hat\theta_i,\Sigma_i).$ We maintain the same large-sample approximation and exchangeability assumptions as before. Denote the historical evidence as $\mathcal D_n:=\{(\hat\theta_i,\Sigma_i)\}_{i=1}^n$, and the empirical Bayes prior for the new study is $ Q^{EB}:=\hat G_n, $ where $\hat G_n$ is constructed from $\mathcal D_n$ using the Gaussian or NPMLE procedures in Section (ref).

In practice, prior studies rarely report a common, harmonized vector of subgroup effects. Instead they often report noisy estimates of linear functionals of $\theta_i$, which can be written in the generic form \[ \hat\psi_i \mid \theta_i \ \stackrel{approx}{\sim}\ \mathcal N(R_i\theta_i,\Sigma_i), \] for a known reporting operator $R_i$ and reported covariance $\Sigma_i$. This covers the main reporting regimes in applied work:

The analysis in the main text assumes the harmonized case to keep the design problem transparent. The estimation strategies in Section (ref) extend to the above setting by replacing each study's likelihood contribution with the corresponding density of $\hat\psi_i$ under the assumed mixing distribution. For Gaussian random effects, the marginal model becomes $\hat\psi_i\sim \mathcal N(R_i\tau,\Sigma_i+R_i V R_i')$. For the NPMLE, the mixture likelihood contribution becomes $\int \varphi_{\Sigma_i}(\hat\psi_i-R_i\vartheta)\,dG(\vartheta)$.

EB Designs and No-Information Benchmarks

For a given prior $Q$ on $\theta$ and a given objective, define the ex-ante value of a design $e\in\mathcal H$ as

where $\mathcal A$ is the post-experiment action space and $W$ is the corresponding payoff. An optimal design under $Q$ is $$e^*(Q)\in\arg\max_{e\in\mathcal H}U_Q(e),$$ and the optimal EB design is $e^{EB}\equiv e^*(Q^{EB}).$

To make the comparison “EB versus no information” exact, fix a reference prior $Q^0$ that does not depend on $\mathcal D_n$ (for example, a diffuse Gaussian prior, or a symmetric benchmark prior). The corresponding no-information design is thus $e^{NI}\equiv e^*(Q^0).$ The difference between $e^{EB}$ and $e^{NI}$ isolates how external evidence changes assignment probabilities, holding fixed the feasible design class.

Optimal EB Designs Under Different Objectives

We now characterize the optimal design under three canonical objectives that arise in applied experimental work. Across objectives, the EB prior enters through different channels, and these differences are what generate distinct assignment rules.

\noindentObjective I: estimation under quadratic loss. Let the post-experiment action be an estimate $a\in\mathbb R^K$ of a target linear functional $L\theta$, where $L\in\mathbb R^{K\times G}$ is fixed (e.g., $L=\pi'$ for the ATE, or $L=I_G$ for the full heterogeneity profile). Consider quadratic payoff \[ W(a,\theta)=-(a-L\theta)'\Lambda(a-L\theta),\qquad \Lambda\succeq 0, \] so maximizing $U_Q(e)$ is equivalent to minimizing Bayes risk.

Proposition (ref) makes the role of EB transparent: with quadratic loss, design quality is entirely governed by the posterior covariance, and the prior affects the optimizer only through $V$. When $\mathcal H$ imposes a binding constraint across strata, the optimizer must decide where to place sampling precision; EB shifts precision toward strata whose effects remain externally uncertain.

To see the wedge relative to $e^{NI}$ in a simple case, suppose $V=\operatorname*{diag}(v_1,\dots,v_G)$ and $L=I_G$ so the goal is to learn the heterogeneity profile. Then the criterion is separable: \[ \min_{e\in\mathcal H}\ \sum_{g=1}^G \lambda_g \cdot \frac{v_g\,s_g^2(e_g)}{v_g+s_g^2(e_g)}, \qquad \Lambda=\operatorname*{diag}(\lambda_1,\dots,\lambda_G). \] Under the cost constraint (ref), any interior optimum satisfies the KKT system

with $\lambda$ chosen so that $\sum_g \pi_g c_g e_g=B$ when the cost constraint binds, and truncation to $[\underline e,1-\underline e]$ applied when the interior solution violates overlap. In the no-information limit $v_g\to\infty$, the factor $v_g^2/(v_g+s_g^2)^2\to 1$ and (ref) collapses to the classical prior-free condition. When EB yields heterogeneous $v_g$, the factor downweights strata with strong external evidence (small $v_g$), so the constrained optimizer reallocates informativeness toward strata where the prior leaves more to learn.

\noindentObjective II: in-experiment welfare. Some experiments are implemented as part of a real program rollout, so the designer values outcomes realized during the experiment. Holding the population shares $\pi_g$ fixed, expected in-experiment net welfare is \[ W^{exp}(e,\delta)=\sum_{g=1}^G \pi_g e_g \tau_g. \] Objective II is a one-stage design problem: the designer chooses $e$ ex ante and welfare accrues during the experiment itself; the experimental data are not used for any subsequent action. Since this objective depends on $\delta$ only through its mean, EB affects the design exclusively through predictive means $m_g:=\mathbb E_Q[\tau_g]$.

This threshold rule reflects a basic economic force: because welfare is linear in treatment probabilities, the designer targets treatment toward strata with higher predicted net gains until the aggregate constraint binds. A symmetric no-information benchmark with $\mathbb E_{Q^0}[\tau_g]\equiv 0$ yields a flat objective in $e$, so any feasible non-targeted assignment is optimal; a canonical benchmark sets a common propensity $e_g\equiv \bar e$ chosen to satisfy $\sum_g \pi_g c_g \bar e = B$ (truncated to $[\underline e,1-\underline e]$ if needed). EB makes the objective nondegenerate by translating historical evidence into heterogeneous $m_g$, which in turn induces systematic targeting.

\noindentObjective III: post-experiment policy choice. A common purpose of experimentation is to inform a subsequent policy that can target treatment by covariates. We model the post-experiment action as a stratum-specific adoption rule $a=(a_1,\dots,a_G)'\in\{0,1\}^G$, where $a_g=1$ means treat stratum $g$ in the post-experiment rollout. Define $\kappa_g$ as the implementation cost of policy and $\delta_g=\tau_g-\kappa_g$ as net effects. Write welfare as \[ W^{pol}(a,\delta)=\sum_{g=1}^G \pi_g a_g \delta_g. \] Given experimental data $D(e)$, the Bayes-optimal adoption rule under prior $Q$ is $a_{g,Q}^*(D(e))=\mathbf 1\{\mathbb E_{Q}[\delta_g\mid D(e)]>0\}$. Under Assumption (ref), $D(e)$ enters only through $\hat\delta_g$ and its sampling variance $s_g^2(e_g)$, so we write equivalently $a_{g,Q}^*(\hat\delta_g;\,e_g)=\mathbf 1\{\mathbb E_{Q}[\delta_g\mid \hat\delta_g;\,e_g]>0\}$. To obtain a closed-form design criterion, we assume a Gaussian prior $\delta_g \sim \mathcal N(m_g,v_g).$

The form (ref) highlights an exploration--exploitation logic inside a randomized design. The factor $\phi(m_g/\sigma_{B,g}(e_g))$ in (ref) is largest when $m_g$ is close to zero relative to posterior-mean dispersion, so (holding costs fixed) experimentation concentrates where the adoption decision is most uncertain given prior evidence. EB matters twice: $m_g$ determines how close the decision is to the threshold, and $v_g$ determines how responsive posterior beliefs are to additional data.

A useful benchmark is the symmetric no-information case with $m_g= 0$ and $v_g= \bar{v}$. Then $\Gamma_g(e_g)=\sigma_{B,g}(e_g)\phi(0)$ is strictly increasing in informativeness, so $e^{NI}$ coincides with a variance-minimizing (information-maximizing) allocation subject to $\mathcal H$. Under EB, heterogeneity in $(m_g,v_g)$ creates a wedge relative to $e^{NI}$: the designer explores more in strata where the policy decision is near the boundary and external evidence is imprecise, and explores less where prior evidence already makes the sign clear (large $|m_g|$) or externally precise (small $v_g$).

\noindentTakeaway. Across objectives, EB affects the propensity design through distinct features of the EB prior. Under quadratic-loss estimation, EB enters through the prior dispersion $V$ in the posterior variance formula (ref), reshaping constrained allocations by downweighting strata that are already well learned from external evidence. Under in-experiment welfare, EB enters only through prior means and induces targeting toward strata with higher predicted treatment effects. Under post-experiment policy choice, EB enters through both means and variances via (ref), concentrating information where it is most likely to change the sign of the adoption decision.

Theoretical Analysis

This section develops theoretical guarantees for EB design. The organizing idea is simple: a design is chosen by maximizing a value functional that depends on the prior. If we estimate the prior and then optimize, performance depends on how estimation error propagates through the optimization step.

Regret Bounds and Gains over No Prior Evidence

For any probability measure $Q$ on $\mathbb R^d$, define the ex-ante value of design $\eta$ under prior $Q$ as

where we write $p_\eta(y|\,\theta)$ for the likelihood under design $\eta$. The oracle design $\eta^O \in \operatorname*{argmax}_{\eta\in\mathcal H} U_G(\eta)$ maximizes value under the true prior $G$. The EB design $\eta^{EB}\in \operatorname*{argmax}_{\eta\in\mathcal H} U_{\hat G_n}(\eta)$ maximizes value under the estimated prior $\hat G_n$.

The first step is to ensure $U_Q(\eta)$ is well-defined and uniformly controlled over the design class. We thus impose the following assumption.

Assumption (ref) rules out pathological cases where the Bayes act fails to exist or the value functional is not integrable. In the leading examples, it is satisfied by simple moment conditions (e.g.\ bounded $W$, or quadratic loss combined with finite second moments under $Q$).

Our first result is a finite-sample oracle inequality. It shows that the welfare loss from optimizing under $\hat G_n$ rather than $G$ is controlled by how well $U_{\hat G_n}(\eta)$ approximates $U_G(\eta)$ uniformly over $\eta$.

The intuition for Theorem (ref) is geometric. Think of $\eta\mapsto U_G(\eta)$ and $\eta\mapsto U_{\hat G_n}(\eta)$ as two objective surfaces. If the surfaces are uniformly close, then maximizing the estimated surface cannot lead to a design that performs much worse under the true surface.

To relate EB to a benchmark that ignores previous evidence, fix a baseline prior $G_0$ independent of $\mathcal D_n$ and define the no-information design $\eta^{NI}\in\operatorname*{argmax}_{\eta\in\mathcal H}U_{G_0}(\eta)$.

Corollary (ref) provides a simple sufficient condition for comparing the EB design with the no-information benchmark, expressed in terms of the uniform deviation $\Delta_n$. Because $\Delta_n$ is a worst-case measure of the maximum discrepancy between the two objective surfaces over the entire design class, the resulting sufficient condition can be conservative in applications.

To illustrate the comparison to $\eta^{NI}$ in a setting where the oracle gain $\delta_G$ and the event $\{U_G(\eta^{EB})\ge U_G(\eta^{NI})\}$ admit closed-form characterizations, we consider the following two-stratum quadratic-loss allocation problem, which is a simple special case of the quadratic-loss criterion in Proposition (ref).

In this simple setting, EB beats the no-information design as long as it (i) gets the direction of the cross-stratum prior precision gap right (equivalently, which stratum is more externally uncertain), and (ii) does not exaggerate that gap by more than a factor of two. Appendix (ref) provides the full details.

Regret Consistency

The finite-sample oracle bound reduces the problem of regret consistency to showing $\Delta_n\to 0$. The next theorem makes that reduction explicit.

To show $\Delta_n\to 0$, a natural starting point is weak convergence $\hat G_n\Rightarrow G$. But weak convergence alone does not automatically imply uniform convergence of the value functional. The next assumption imposes continuity and moment conditions that allow one to pass limits through the integrals defining $U_Q(\eta)$ uniformly over $\eta$.

Assumption (ref) is a standard “uniform dominated convergence” device: continuity gives pointwise convergence when priors converge weakly, the envelope $\bar p$ controls likelihood variation across designs, and the moment condition controls the tails that matter when welfare grows with $\|\theta\|$.

The following Lemma (ref) provides the main bridge: any prior estimator that is weakly consistent (and satisfies the moment conditions) delivers regret consistency via Theorem (ref).

In Gaussian parametric EB, parameter convergence implies weak convergence and all required moments exist automatically, so the following corollary is a direct application of Lemma (ref).

A useful special case of Corollary (ref) arises under a quadratic loss objective in a Gaussian–Gaussian experiment, where both the prior and the likelihood are Gaussian. In this case, the value admits a closed-form expression, and continuity in Assumption (ref) can be verified directly.

The key intuition is that with quadratic loss, ex-ante value depends on the design only through posterior covariance, and posterior covariance depends continuously on $V$ as long as designs do not produce degenerate covariance matrices.

For the NPMLE, we impose two standard regularity conditions: compact support of the mixing distribution $G$ and uniform eigenvalue bounds on the reported covariance matrices. Both conditions are also used in SoloffGuntuboyinaSen2025 for proving the rate of convergence of prior estimation.

The proof follows analogous logic to that used for establishing consistency of the NPMLE under i.i.d. mixture models (in our setting, all $\Sigma_i$s are equal), as reviewed in Chen2017. Specifically, we verify the condition in the “trivialized Wald theorem” Chen2017 by showing that outside any weak neighborhood of $G$ the average expected log-likelihood is uniformly smaller. In the i.i.d.\ mixture case, Chen verifies this condition using Pfanzagl-type inequalities Pfanzagl1988; in our heteroskedastic error setting we establish the same condition uniformly over the covariance class implied by Assumption (ref), and then replace the strong law of large numbers for i.i.d. variables with Kolmogorov’s strong law for independent, non-identically distributed variables to accommodate varying $\Sigma_i$.

Rates of Convergence

The consistency results above reduce EB optimality to a single quantitative object: the uniform value-approximation error \[ \Delta_n=\sup_{\eta\in\mathcal H}\big|U_{\hat G_n}(\eta)-U_G(\eta)\big|. \] By the finite-sample oracle inequality (Theorem (ref)), $\operatorname{Reg}_n\le 2\Delta_n$, so rate statements for regret follow once we control $\Delta_n$.

\noindentGaussian limit experiments. In many randomized experiments, the raw outcome vector is high-dimensional while the payoff-relevant parameter is low-dimensional. In such settings, it is often natural to work with sufficient statistics (e.g., regression coefficients) or with limit-experiment approximations. A theoretical justification is that, under regularity conditions, a broad class of experiments can be approximated by Gaussian limit experiments uniformly over design classes Higbee2024. Accordingly, to obtain explicit regret rates, we work with the Gaussian limit experiment induced by the low-dimensional statistic that is used for downstream decision-making.

\noindentClass of decision problems. The rate results in this section are stated for a broad class of decision problems characterized by Assumption (ref). This class encompasses many widely used settings, including point estimation, hypothesis testing, portfolio choice, binary treatment adoption, and best-arm selection. Table (ref) summarizes these canonical examples and their corresponding continuation values.

The following lemma characterizes a defining feature of this class of decision problems: the welfare is affine in the payoff-relevant state $\theta$, so the effect of the prior can be summarized through posterior means.

Lemma (ref) makes clear what must be controlled to obtain rates: we need bounds on how $\Psi(\mu_{Q,\eta}(Y))$ changes when $Q$ is replaced by $\hat G_n$, uniformly over $\eta\in\mathcal H$.

\noindentFirst-order rates. First-order regret bounds treat the design problem as a plug-in maximization problem: once $\hat G_n$ is close to $G$ and the objective is stable to small prior perturbations, the uniform deviation $\Delta_n$ is small and $\operatorname{Reg}_n\le 2\Delta_n$ inherits the same rate. We present this logic for (i) Gaussian parametric EB and (ii) nonparametric EB via NPMLE.

Assumption (ref) is a stability condition on the decision problem. It allows $\Psi$ to be non-smooth (e.g.\ threshold rules) and to be unbounded (e.g.\ quadratic loss), while still ensuring that changes in posterior means translate into controlled changes in value. In Appendix (ref), we verify that the continuation values associated with the five decision problems summarized in Table (ref) all satisfy Assumption (ref).

Theorem (ref) shows that, for this class of decision problems, parametric EB design regret inherits the parametric prior-estimation rate. The reason is that under Gaussian conjugacy, posterior means (and predictive distributions) depend smoothly on $(\tau,V)$, so a $\sqrt{n}$ perturbation in the estimated hyperparameters produces a uniform $O_{p}(n^{-1/2})$ perturbation of the value functional.

We next state the corresponding first-order bound for nonparametric EB based on NPMLE.

The following theorem establish the first-order rate of regret under NPMLE.

Theorem (ref) shows that EB remains oracle-optimal with a nonparametric prior estimator, but the regret convergence rate is slower (up to logs) because estimating an entire mixing distribution is intrinsically harder than estimating finitely many hyperparameters. As a result, if the number of prior studies is small and a Gaussian distribution is a plausible approximation for the distribution of study-level parameters, then a Gaussian EB prior is a sensible choice.

\noindentSecond-order rates. The first-order bounds above control regret through the uniform value error $\Delta_n$, and therefore apply to a wide range of (possibly non-smooth) decision problems covered by Assumptions (ref) and (ref). In many smooth design problems, regret is locally quadratic: if the oracle objective $U_G$ has curvature at its maximizer, then using a design that is close to $\eta^O$ induces only a second-order welfare loss. The next two assumptions formalize this refinement.

Assumption (ref) imposes differentiability and curvature of $U_G$ on the feasible design set, yielding a unique interior maximizer and a quadratic upper bound on value losses as a function of $\|\eta-\eta^O\|_2$. Assumption (ref) replaces uniform value control by a uniform bound on the gradient error induced by using $\hat G_n$; in regular parametric settings this gradient error typically inherits the $n^{-1/2}$ rate of hyperparameter estimation. These conditions are most natural for smooth objectives (e.g.\ quadratic-loss estimation or mean--variance criteria under smooth propensity-score or allocation design classes). In contrast, for maximum or threshold objectives (binary adoption, best-alternative selection, and hypothesis testing), the induced objective can be non-differentiable and optimizers often occur at corners, so second-order guarantees are generally inappropriate.

\noindentExample (Cont.): two-stratum quadratic loss. In Example (ref), both Assumptions (ref) and (ref) can be verified explicitly. In Appendix (ref), we derive a closed-form expression for $U_G(\eta)$ under the two-stratum Gaussian quadratic-loss allocation problem and show that $U_G$ is strongly concave on the feasible allocation set $\mathcal H=\{(N_1,N_2)\in\mathbb R_+^2:\ N_1+N_2=N\}$, with a unique interior maximizer under the maintained interior condition from Example (ref). Moreover, if the stratum prior variances $v_g$ admit EB estimates $\hat v_g$ satisfying $|\hat v_g - v_g|=O_p(n^{-1/2})$, then the induced plug-in objective satisfies the uniform gradient approximation condition $\sup_{\eta\in\mathcal H}\|\nabla U_{\hat G_n}(\eta)-\nabla U_G(\eta)\|_2=O_{p}(n^{-1/2})$.

Theorem (ref) isolates the mechanism behind faster rates: curvature $m$ converts gradient errors into quadratic value losses.

As an immediate consequence of Theorem (ref), we obtain the second-order rates for Gaussian EB and NPMLE EB.

Characterization of Gaussian Experiments

This section compares Gaussian experiments, i.e., designs $\eta$ that generate a normal observation with design-dependent noise covariance. We emphasize three points. First, Loewner improvements in sampling noise deliver prior- and objective-robust dominance (Blackwell dominance) and hence leave no scope for EB. Second, the only benchmark in which all feasible Gaussian designs are completely ordered---so the prior never affects the design choice---is the genuinely univariate case $d=1$. Third, even when $d>1$, a tractable and widely used class of problems arises when welfare depends on $\theta$ only through a scalar linear index $\omega=\alpha'\theta$, and the prior is Gaussian. In this Gaussian--Gaussian environment, every scalar-index decision problem ranks designs by the posterior variance of $\omega$; hence the optimal design can be characterized by minimizing that posterior variance. Unlike the $d=1$ benchmark, this criterion depends on the true prior covariance and therefore is exactly where EB can matter.

Consider a Gaussian experiment: \[ Y_\eta = \theta_{n+1} + \varepsilon_\eta, \qquad \varepsilon_\eta \sim \mathcal N\!\big(0,\Sigma(\eta)\big), \qquad \Sigma(\eta) \succ 0 . \]

Theorem (ref) is an immediate consequence of Blackwell’s informative theorem Blackwell1951. In Gaussian experiments, the corresponding comparison via Loewner ordering of covariance matrices is established in HansenTorgersen1974.

Theorem (ref) isolates a robust notion of “better design.” If one design reduces sampling noise in the Loewner sense, it is uniformly better for every objective and every prior, so there is no role for EB in choosing between them. When $\Sigma(\eta_1)$ and $\Sigma(\eta_2)$ are not comparable in Loewner order---the generic case in multivariate design problems, where increasing precision in one direction typically requires sacrificing precision in another---Theorem (ref) is silent: neither design uniformly dominates the other. The remainder of this subsection discusses two useful simplifications and their implications for when the prior can affect the optimal design.

\noindentUnivariate $\theta$. When $d=1$, Loewner order reduces to the usual scalar order on variances: $\Sigma(\eta_1)\le \Sigma(\eta_2)$. Hence, in scalar Gaussian experiments, designs are completely Blackwell-ordered by their sampling variances, so the most informative feasible experiment is optimal regardless of the prior. This is the only setting in which the prior is irrelevant for all design comparisons: there is no cross-direction trade-off for the prior to resolve.

\noindentScalar payoff-relevant state. The univariate benchmark is special because feasible Gaussian designs are completely ordered. With $d>1$, that complete order typically fails. A central intermediate case is when payoffs depend on $\theta$ only through a scalar index $\omega=\alpha'\theta$ (e.g., an ATE or a portfolio value), and the prior is Gaussian. In the Gaussian--Gaussian model, the induced posterior for $\omega$ is normal with a deterministic variance, which implies that every scalar-index decision problem ranks designs by that posterior variance. Importantly, unlike the $d=1$ benchmark, the posterior variance depends on the true prior covariance $V$, so EB can change the optimal design through its estimate of $V$ even though the form of the objective function is irrelevant within this class.

The next lemma and corollary make this reduction explicit by constructing a scalar sufficient statistic $Z_\eta$ that is Blackwell equivalent to the multivariate Gaussian signal for learning the payoff-relevant index $\omega$. Related dynamic, continuous-time versions of Corollary (ref) appear in LiangEtAl2022.

Corollary (ref) follows because scalar normal experiments are completely Blackwell-ordered by precision, and $\tau^2(\eta)$ in Lemma (ref) is a monotonic transformation of $\mathbb V(\omega\mid Y_{\eta})$. The key message is that, in scalar-index problems, the objective does not affect the optimal design choice, but the prior typically does through $V$. In contrast, when welfare depends on multiple components of $\theta$ (as in Objective I with $L=I_G$), no such complete ordering exists and the objective and prior jointly determine how the designer trades off precision across directions.

Empirical Applications

This section provides two empirical applications with complementary lessons about how empirical Bayes design works in practice. The first uses oncology drug trials drawn from a public online database of clinical research studies, where the main challenge is constructing a usable prior-study dataset: aggregating heterogeneous studies requires choices about poolability (which estimands are comparable enough to combine), harmonizing subgroup definitions, and handling multiple eligible estimates per study. The second is a multisite RCT setting in economics, where treating sites as “studies” makes the evidence base more naturally comparable and easier to aggregate; the focus then shifts to design choice itself, illustrating how different objectives can imply different optimal treatment allocations even under the same EB prior.

Oncology Drug Trials

This section presents our first empirical application based on oncology trial results from ClinicalTrials.gov, an online database of clinical research studies and information about their results. We consider a researcher who is designing a new immunotherapy drug trial and wants to use published results from similar drugs to inform the design. We focus on PD-L1 (programmed death-ligand 1) expression as the key subgroup variable. PD-L1 is a biomarker that predicts response to cancer immunotherapy drugs, with higher expression usually associated with greater benefit. Because treatment effects differ across PD-L1 levels, trial designers routinely pre-specify subgroup analyses---and sometimes stratify randomization---by PD-L1 level (see, e.g., ImEtAl2024, MilesEtAl2021). We use PD-L1 subgroup effect estimates from prior trials to construct an empirical Bayes prior that informs how to allocate experimental effort across PD-L1 subgroups in the new trial.

\noindentData construction and poolability. We query ClinicalTrials.gov API using a broad oncology therapy basket, keep studies with posted results, and extract subgroup analyses that contain both an estimate and standard error. This yields 821 PD-L1 subgroup estimates from 87 studies. Each estimate corresponds to a treatment effect for a particular outcome measure. We classify outcomes into four broad families: overall survival (OS), progression-free survival (PFS), tumor response, and other efficacy outcomes (see Table (ref)). As a basic poolability requirement, we restrict attention to estimates within the same outcome family.

Three practical issues arise when aggregating prior studies. First, subgroup definitions are not standardized across studies. PD-L1 can be reported using different scoring systems: positive versus negative, high versus low, combined positive score (CPS) thresholds, tumor proportion score thresholds, or tumor-cell and immune-cell staining classes. We therefore harmonize reported labels into two sides, high-expression and low-expression, with one guiding idea: preserve a monotone ranking of PD-L1 expression so that “more expression” maps to high-expression and “less or no expression” maps to low-expression, regardless of the original language. We provide concrete mappings and examples in the Online Appendix.

Second, studies often report multiple subgroup effects within the same trial record. This happens for three recurring reasons: multiple PD-L1 cutoff classes on the same side (for example, CPS $\ge 1$ and CPS $\ge 10$), multiple treatment arms, and repeated estimates under different follow-up periods or sample definitions. These within-study estimates are not independent because they are computed from overlapping participants; keeping all of them without taking into account their correlation would overweight those trials in prior estimation. We therefore enforce one estimate per study. If a study reports multiple cutoff-defined classes on one side, we keep the class closest to a cross-study reference cutoff within the same scoring system. In other cases, we randomly select one estimate. Full implementation details are reported in the Online Appendix.

Third, metric mixing is substantial across outcome families. Outside survival outcomes, reported effects combine incompatible scales (for example, odds ratios, least-square mean differences, and percentage-point differences), so direct pooling is not meaningful. We therefore narrow to the survival family (OS and PFS), where hazard ratios have a common interpretation across studies. A hazard ratio below one means treatment lowers the hazard rate and is therefore favorable for survival outcomes. We map hazard ratio to treatment effect $\mathrm{TE}=1-\mathrm{HR}$, so $\mathrm{TE}=0$ is the no-effect benchmark and larger $\mathrm{TE}$ implies better survival.

In total, the prior-study dataset for the OS outcome family includes 71 estimates from 64 studies (7 studies report both high and low, 52 report high only, and 5 report low only). We then represent the historical evidence as \[ \hat\psi_i \mid \theta_i \ \stackrel{approx}{\sim}\ \mathcal N(R_i\theta_i,\Sigma_i),\qquad \theta_i=

, \] with $R_i=I_2$ when both sides are reported and $R_i=[1\ 0]$ or $[0\ 1]$ when only one side is reported. We focus on the OS family; details and results for the PFS family are provided in the Online Appendix.

\noindentResults. We estimate four EB priors: Gaussian-joint, NPMLE-joint, Gaussian-independent, and NPMLE-independent. “Joint” means estimating a bivariate prior on $(\theta_{\text{high}},\theta_{\text{low}})$ that allows dependence between the two subgroup effects. “Independent” means fitting each subgroup marginal prior separately and taking their product, which imposes cross-subgroup independence. We report joint-prior results here; independent-prior results are reported in the Online Appendix.

Figure (ref) summarizes the estimated OS prior marginals under joint Gaussian EB and joint NPMLE EB. Both estimators imply that subgroup effects are centered above zero, with larger average effects in the high-expression subgroup than in the low-expression subgroup, consistent with the clinical motivation for PD-L1 stratification that higher expression predicts greater benefit from immunotherapy. Quantitatively, under both EB approaches the estimated prior mean is about twice as large in the high-expression subgroup as in the low-expression subgroup, while the low-expression subgroup has the higher prior variance.

For the propensity-score design of the new drug trial, we focus on Objective I, which minimizes posterior quadratic loss for $(\theta_{\text{new,high}},\theta_{\text{new,low}})$. For our numerical illustration, we set subgroup shares to $\pi=(0.5,0.5)$ and arm-specific outcome variances to $\sigma_{1,\text{high}}^2=\sigma_{0,\text{high}}^2=\sigma_{1,\text{low}}^2=\sigma_{0,\text{low}}^2=1$. We impose overlap bounds $e_g\in[0.05,0.95]$ for $g\in\{\text{high},\text{low}\}$, treatment costs $c_{\text{high}}=c_{\text{low}}=2$, and a budget constraint $\sum_g \pi_g c_g e_g\le B$ with $B=0.5$. As a comparison, the no-information benchmark uses a diffuse Gaussian prior (formally $V^{-1}\to 0$), so the design is driven only by sampling precision and constraints. Under the symmetric inputs above and equal costs, this benchmark yields the balanced feasible design $(e_{\text{high}},e_{\text{low}})=(0.25,0.25)$.

Table (ref) reports the optimal designs under the estimated EB priors. Both joint priors shift assignment toward the low-expression subgroup: $(0.229,0.271)$ under Gaussian EB and $(0.231,0.269)$ under NPMLE EB. Gaussian and NPMLE can imply different optimal designs because Gaussian imposes a single elliptical prior shape, while NPMLE allows discrete mass and non-Gaussian tail behavior; when those shape differences matter for posterior-risk gradients, the implied designs can diverge more. In OS, the two joint priors are close in means and second moments, so the resulting designs are also close.

Table (ref) reports the Objective I optimal designs under the estimated EB priors. The two joint-prior designs are similar: $(0.229,0.271)$ under Gaussian-joint EB and $(0.231,0.269)$ under NPMLE-joint EB. Both shift assignment toward the low-expression subgroup relative to the no-information benchmark because, under both priors, the low-expression effect has higher prior variance than the high-expression effect; the posterior-risk objective therefore places more value on reducing uncertainty about $\theta_{\text{new,low}}$. OS independent-subgroup robustness and all PFS results are reported in the Online Appendix.

Project STAR

Our second application uses Project STAR (Student to Teacher Achievement Ratio), the Tennessee class-size RCT, to illustrate our EB design approach in a setting with rich historical evidence. We treat each STAR school as one prior “study” because random assignment to class types was carried out within schools and the treatment contrast and outcome measurement follow a common protocol across sites Krueger1999,KruegerWhitmore2001, so each school yields an internally valid but noisy vector of stratum-level treatment-effect estimates. Cross-school differences in student composition and implementation generate cross-site treatment-effect heterogeneity, and the resulting external-validity problem motivates borrowing strength across sites when forecasting effects in a new setting Menzel2025,AdjahoChristensen2025. We use empirical Bayes to aggregate the collection of school-level estimates to learn the cross-school distribution and form a predictive prior for a future, STAR-like experiment. We then use this predictive prior to guide the design of the next experiment by choosing stratum-specific treatment propensities: in the new experiment, each student is randomized with a known assignment probability that can vary by stratum, subject to overlap and budget constraints.

\noindentData. Project STAR randomly assigned students (and teachers) within each participating school to one of three class types: small classes, regular-size classes, and regular-size classes with a full-time teacher aide. Let $D=1$ denote assignment to a small class and $D=0$ denote assignment to either a regular class or a regular+aide class. We pool the latter two arms because achievement outcomes are similar across them in STAR Krueger1999,KruegerWhitmore2001. We focus on end-of-kindergarten reading scores as the outcome variable. For the new experiment, we define four randomization strata using the $2\times 2$ cross of race and free-lunch eligibility (a proxy for low family income): \[

\] Among baseline covariates, race and free-lunch status are the two strongest single predictors of kindergarten reading in our sample, and the partition is directly policy-relevant for targeted class-size policies. We keep students with non-missing outcomes and covariates, leaving $N=5{,}772$ students in $79$ schools. The resulting stratum shares in the STAR samples are $(26.1\%,6.5\%,22.2\%,45.2\%)$ for $(S_1,S_2,S_3,S_4)$.

Each school is treated as one prior study $i$. For each stratum $g$, we construct the school-level stratum-specific treatment-effect estimate from the raw data as the within-school difference in mean outcomes, \[ \hat\theta_{ig}=\bar Y_{ig}(1)-\bar Y_{ig}(0),\qquad \widehat{\mathrm{SE}}_{ig}^2=\frac{s_{1ig}^2}{n_{1ig}}+\frac{s_{0ig}^2}{n_{0ig}}, \] where $\bar Y_{ig}(d)$ is the average reading score among students in school $i$, stratum $g$, assigned to arm $D=d$, and $(n_{dig},s_{dig}^2)$ are the corresponding sample sizes and variances. If a school has fewer than two observations in either arm for stratum $g$, we treat that component as missing and represent the observed components with a selector matrix $R_i$. Formally, prior information enters as $\hat\psi_i\sim \mathcal N(R_i\theta_i,\Sigma_i)$. This keeps all $79$ schools in prior estimation: $10$ schools contribute one stratum estimate, $53$ contribute two, $10$ contribute three, and $6$ contribute all four. Figure (ref) plots these school-by-stratum estimates with 95% confidence intervals; the substantial dispersion across schools within each stratum provides direct evidence of cross-site treatment-effect heterogeneity, motivating EB aggregation across schools.

\noindentResults. As in the first application, we estimate four EB priors for the stratum-specific treatment-effect vector: Gaussian and NPMLE, each under independent-subgroup and joint specifications. Figure (ref) summarizes the joint priors used in the main text. Despite the different functional forms, the two joint estimators tell a consistent story: expected gains from small classes are largest for the more disadvantaged strata $S1$ and smallest for the most advantaged stratum $S4$, and the amount of prior uncertainty is substantial---which mirrors the wide dispersion of school-level estimates shown in Figure (ref). Quantitatively, the implied prior means span roughly $3$--$6$ points across strata, while prior variances are on the order of $10^2$ and are notably larger for the disadvantaged groups (especially $S_2$), indicating that historical evidence is informative but far from decisive. The joint priors also imply positive cross-stratum dependence, so additional precision in one subgroup can spill over to others.

We then solve for new-site treatment propensities $e=(e_1,\dots,e_4)$ under overlap bounds $e_g\in[0.05,0.95]$ and an average treatment budget $\sum_{g=1}^4 \pi_g e_g\le 0.30$, using the pooled STAR stratum shares $\pi$ from the data section. Because the new-site outcome variances $(\sigma_{1g}^2,\sigma_{0g}^2)$ are unknown, we use the estimated variance from pooled historical STAR data: \[ \sigma_{1g}^2=\widehat{\operatorname{Var}}(Y\mid D=1,S=g),\qquad \sigma_{0g}^2=\widehat{\operatorname{Var}}(Y\mid D=0,S=g). \] Figure (ref) reports the resulting optimal designs under Gaussian-joint and NPMLE-joint priors. The two EB estimators yield very similar allocations across all three objectives, so the qualitative design implications are robust to the choice of joint-prior family.

Under Objective I (estimation), optimal propensities are interior. A useful benchmark is the within-stratum sampling-variance--minimizing split absent the budget constraint, $ e_g^{\text{bal}}=\sigma_{1g}/(\sigma_{1g}+\sigma_{0g}), $ which is close to one-half in all strata here (in particular, $e_2^{\text{bal}}= 0.48$). Consistent with this benchmark, the no-information design already places $e_2$ close to $e_2^{\text{bal}}$ (about $0.46$), and the joint-EB designs keep $e_2$ in the same range (about $0.42$--$0.45$). Thus, even though $S_2$ is the most prior-uncertain component, EB does not “solve” this by pushing $e_2$ further upward; instead it mainly reallocates probability mass away from $S_4$ and raises $e_1$ toward $e_2$. This matches Proposition (ref): posterior-risk reduction depends not only on each stratum's prior variance but also on the covariance between strata. Here, $S_2$ is a small-share stratum ($\pi_2=6.5\%$ versus $\pi_1=26.1\%$), so the total information available within $S_2$ is limited once the treated/control split is already near $e_2^{\text{bal}}$. At the same time, the joint EB prior links the two disadvantaged strata: $S_1$ is most correlated with $S_2$ (Corr$(S_1,S_2)=0.79$ under Gaussian-joint and $0.31$ under NPMLE-joint), so improving precision in the larger stratum $S_1$ also tightens inference about $S_2$. This combination---small $\pi_2$ and positive cross-stratum correlation---explains why the Objective I solution keeps $e_1$ close to $e_2$ rather than concentrating experimentation exclusively in $S_2$.

Under Objective II (in-experiment welfare), the design becomes primarily mean-targeting. Both joint priors push the highest-mean stratum to the upper overlap bound ($e_1=0.95$) and the lowest-mean strata to the lower bound ($e_3=e_4=0.05$), illustrating the exploitation force characterized in Proposition (ref); the remaining stratum is set at an interior value $0.28$ determined by the budget and overlap constraints.

Under Objective III (post-policy choice), Proposition (ref) implies that policy-relevant uncertainty is highest where the posterior mean is close to the adoption boundary and posterior uncertainty remains large. In our data, $S_2$ is the most policy-relevant uncertain stratum because it combines moderate prior mean with the largest prior variance. Moreover, its target share is small ($\pi_2=6.5\%$), so the design-stage sampling noise term $s^2_g(e)$ is large for a given $e$, making extra allocation especially valuable for policy learning. This is why $e_2$ rises sharply relative to no-information (from $0.149$ to $0.331$--$0.341$). By contrast, $S_4$ has a lower mean but also substantially lower uncertainty and much larger target share ($\pi_4=45.2\%$), so posterior uncertainty is reduced more quickly even without heavy reallocation; correspondingly, optimal designs reduce $e_4$ (from $0.317$ to $0.265$--$0.271$).

In the Online Appendix, we report additional STAR results, including independent-subgroup EB designs and comparisons of optimal designs across prior estimation schemes under each objective.

Conclusion

This paper develops an empirical Bayes approach to experimental design that leverages information from prior related studies. The framework combines (i) prior estimation from collections of related studies and (ii) a decision-theoretic criterion for choosing designs. The central message is that external evidence can be translated into a predictive prior for the next study, and that this prior can reshape optimal assignment decisions in constrained designs.

The analysis of propensity score design highlights three channels through which EB affects optimal designs: prior dispersion reshapes precision allocation under quadratic-loss estimation; prior means drive targeting under in-experiment welfare; and both means and variances matter under post-experiment policy choice, concentrating information where adoption decisions are most uncertain. The theoretical analysis links design performance to prior-estimation accuracy through a finite-sample oracle inequality and comparison bounds against a no-information benchmark that ignores external evidence; it also shows that plug-in EB designs are asymptotically oracle-optimal under weak convergence of the estimated prior, derives explicit first- and second-order regret rates for Gaussian EB and NPMLE EB, and clarifies when the prior is irrelevant in Gaussian design comparisons (only in the genuinely univariate case). The two empirical applications illustrate that our approach can deliver interpretable reallocations of experimental effort in practice.

Several extensions worth further investigation. First, while the current framework assumes exchangeability across studies, richer models could incorporate study-level covariates to allow for conditional exchangeability; one natural approach is to adopt covariate-powered empirical Bayes methods as in IgnatiadisWager2019. Second, the framework could be extended to sequential experimental design, in which the empirical Bayes prior is updated as new experimental evidence accumulates over time. Third, complementary work could develop inference procedures for the design choice itself that properly account for uncertainty arising from empirical Bayes prior estimation, following the ideas in AndrewsChen2025.

\setcounter{page}{1}