Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
51,144 characters · 17 sections · 21 citation commands
Prior-Free Sample Size Design for Test-and-Roll Experiments
\setstretch{1.25}
\setstretch{1.5}
Randomized controlled trials (RCTs) are the gold standard in causal inference, but choosing the experimental sample size remains a central theoretical and practical challenge. Standard approaches based on Type I and Type II error control have little consequence for the welfare objectives of the decision-maker. This paper studies sample-size design from a welfare perspective, focusing on the exploration--exploitation tradeoff that arises when an RCT is conducted within a fixed finite population.
We consider a “test-and-roll” setting with a finite population of size $N$. A decision-maker first conducts an RCT of size $m$, assigning half of the experimental units to treatment and half to control. Based on the observed outcomes, the decision-maker assigns the remaining $N-m$ units to the treatment that performed better in the experiment. The objective is to maximize expected welfare over the entire finite population, $\mathbb{E}\left[\sum_{i=1}^N Y_i\right]$, where the $N$ units include both the experimental and rollout units. The key design question is how large the experimental sample size $m$ should be.
This problem involves a fundamental exploration--exploitation tradeoff. A larger experiment improves the accuracy of the rollout decision, reducing the risk of assigning the remaining population to the inferior treatment. At the same time, because half of the experimental units receive the inferior treatment ex post, increasing the experimental sample size directly imposes welfare losses on the experimental population. Thus, excessive exploration is costly, while insufficient exploration can lead to an erroneous rollout decision and welfare losses for the rollout population.
This tradeoff arises in many applications, including clinical trials comparing alternative medical treatments wang2013clopidogrel, test-and-roll A/B testing in digital marketing feit2019test_and_roll, and field experiments on workplace incentives friebel2017team.
A central challenge in optimizing this tradeoff is that the true treatment effects are unknown ex ante. Existing approaches to experimental sample-size design typically address this uncertainty by placing Bayesian prior distributions on the unknown parameters. Such priors can be difficult to specify properly, and the resulting design can be sensitive to the chosen specification. This difficulty is especially acute in our setting because the experimental sample size must be chosen before any data are observed. Thus, a practically useful design rule should provide guidance before learning takes place, without relying on subjective priors or arbitrary hyperparameters.
We address this challenge through a prior-free minimax analysis of the experimental sample-size choice. A natural starting point is the standard absolute minimax regret criterion stoye2009minimax_regret_finite_samples, which evaluates the worst-case welfare loss relative to the infeasible oracle decision. Although Bayesian test-and-roll designs have been studied feit2019test_and_roll, the implications of this purely minimax benchmark are less well understood. We show that, in finite-population test-and-roll problems, the standard absolute minimax regret criterion can lead to implausibly small experimental sample. This issue arises because the criterion is overly sensitive to extreme cases. The worst-case adversary concentrates on parameter values that maximize the absolute treatment effect, thereby over-penalizing exploration.
To address this limitation, we propose a new robust criterion for sample-size design: the Worst-case Marginal Benefit (WMB) rule. The key idea is to evaluate experimentation at the margin rather than through worst-case total regret. The WMB rule compares the worst-case marginal benefit of adding one more matched pair to the experiment with the corresponding marginal exploration cost, and stops experimentation once the former no longer exceeds the latter. We study the optimal experimental sample size under this criterion for binary and Gaussian outcomes.
This paper establishes a simple rule-of-thirds benchmark. For Bernoulli outcomes, after excluding a boundary rare-event pathology, we show that the WMB criterion yields the optimal experimental sample size, $m^{\ast}\approx N/3$, through a Gaussian approximation. For Gaussian outcomes with a known common variance, the same benchmark arises exactly. Because this benchmark depends only on the total population size $N$, it can be implemented directly in practice and provides a prior-free guide for sample-size design without requiring subjective priors, prior variances, or other tuning parameters.
This paper contributes to the statistical literature on sample-size selection and experimental design. Conventional sample-size calculations are typically based on power analysis, which chooses the experimental sample size to control Type I and Type II error probabilities Lachin1981, DufloGlennersterKremer2007. While useful for hypothesis testing, this approach does not directly account for the welfare consequences of experimentation. Our paper departs from the classical power-based framework by choosing the experimental sample size to optimize a finite-population welfare objective that accounts for both the welfare cost imposed on experimental units and the welfare consequences of the rollout decision.
Several studies examine optimal sample-size selection from a welfare-based perspective, typically using Bayesian approaches to test-and-roll designs. cheng2003choosing_sample_size_decision_analysis study the optimal first-stage sample size for a finite population under Beta priors, and stallard2017optimal_sample_size_population_size extend this decision-theoretic framework to more general models. feit2019test_and_roll formulate A/B testing as a finite-population test-and-roll problem and derive optimal sample sizes under normal priors. These papers share our welfare-based perspective, but their methods depend on prior distributions or parametric assumptions. By contrast, we take a prior-free approach by studying minimax criteria and proposing the Worst-case Marginal Benefit rule.
Our work is also related to the literature on statistical treatment choice. Starting with manski2004statistical_treatment_rules, this literature studies how experimental or observational data should be used to choose treatments so as to maximize welfare; see also stoye2009minimax_regret_finite_samples, kitagawa2018who_should_be_treated, and athey2021policy_learning. These studies typically take the sample size as given and focus on the treatment assignment rule after data have been collected. In contrast, we study an ex ante design problem in which the decision-maker must choose the experimental sample size before observing any data. Thus, our analysis complements the statistical treatment choice literature by incorporating the cost of experimentation into the sample-size decision itself.
Our framework is also related to the multi-armed bandit literature, especially explore-then-commit strategies garivier2016etc, perchet2016batched, LattimoreSzepesvari2020BanditAlgorithms. While bandit algorithms address exploration--exploitation tradeoffs through sequential treatment assignment, such adjustment is often infeasible in social-science, medical, and marketing experiments because outcomes are delayed, implementation is costly, or frequent reassignment raises ethical concerns. We therefore focus on test-and-roll designs, where the experimental sample size must be fixed in advance.
Our paper also relates to recent work incorporating economic and welfare considerations into experimental design azevedo2020ab_testing_fat_tails, azevedo2023ab_testing_gaussian_priors,narita2021incorporating, hu2024minimax_regret_sample_selection. Relative to these studies, our contribution is to isolate the finite-population test-and-roll sample-size problem and derive a simple prior-free benchmark based on a marginal minimax criterion.
The remainder of the paper is organized as follows. Section (ref) formulates the Bernoulli test-and-roll problem and analyzes absolute minimax regret. Section (ref) introduces the WMB rule. Section (ref) examines its finite-sample behavior through numerical examples. Section (ref) establishes the rule-of-thirds benchmark. Section (ref) extends the analysis to Gaussian outcomes with a common variance. Section (ref) concludes. Proofs and additional discussions are provided in the appendix.
This section introduces the finite-population test-and-roll problem with binary outcomes. We first define the test-and-roll experiment and introduce the empirical success rule of stoye2009minimax_regret_finite_samples, which is minimax-regret optimal for the corresponding superpopulation problem under a matched-pairs design. We then define a finite-population regret criterion for sample-size choice and show why the standard absolute minimax regret objective can recommend implausibly small experiments.
Suppose that $N$ is even and that, for each individual $i=1,\dots,N$, the potential-outcome pair $(Y_i(1),Y_i(0))$ is drawn i.i.d. across individuals with Bernoulli marginals: $$Y_i(1)\sim \mathrm{Bernoulli}(\mu_1), \qquad Y_i(0)\sim \mathrm{Bernoulli}(\mu_0).$$
Let $T_i\in\{0,1\}$ denote the treatment assignment. The observed outcome is $Y_i=T_iY_i(1)+(1-T_i)Y_i(0)$. We write $$ \tau:=\mu_1-\mu_0 $$ for the average treatment effect.
Suppose that the first $m$ individuals are assigned to the experiment, where $m$ is even. We consider a matched-pairs design in which $m/2$ individuals are assigned to treatment and the remaining $m/2$ individuals are assigned to control. Without loss of generality, we index the treated individuals by $1,\dots,m/2$ and the control individuals by $m/2+1,\dots,m$. The sample mean outcomes for the treatment and control groups are $$ \hat\mu_1:=\frac{2}{m}\sum_{i=1}^{m/2}Y_i, \qquad \hat\mu_0:=\frac{2}{m}\sum_{i=m/2+1}^{m}Y_i. $$
Following stoye2009minimax_regret_finite_samples, let $\omega=\{(T_i,Y_i)\}_{i=1}^{m}$ denote the realized experimental data. A statistical treatment rule is a mapping $ \delta:(\{0,1\} \times \{0,1\})^m\to[0,1], $ where $\delta(\omega)$ is interpreted as the probability of assigning treatment $1$ to an individual in the rollout population after observing $\omega$. Let $\mathcal{D}$ denote the class of all such rules.
For decision-making under uncertainty, it may be tempting to choose a treatment rule that maximizes worst-case welfare, \[ \max_{\delta\in\mathcal D}\min_{(\mu_1,\mu_0)\in[0,1]^2} u(\delta,\mu_1,\mu_0). \] However, this maximin welfare criterion is uninformative because an adversarial nature would choose the worst possible expected outcomes, $\mu_1=\mu_0=0$, in which case expected welfare is zero for any treatment rule $\delta$.
A common alternative is to evaluate a treatment rule using worst-case superpopulation regret. Superpopulation regret is defined as the shortfall relative to an oracle that knows the true state ex ante: \[ \mathrm{SReg}(\delta,\mu_1,\mu_0) := \max\{\mu_1,\mu_0\}-u(\delta,\mu_1,\mu_0). \] The minimax-regret approach then selects a treatment rule that minimizes the worst-case superpopulation regret, \[ \max_{(\mu_1,\mu_0)\in[0,1]^2} \mathrm{SReg}(\delta,\mu_1,\mu_0). \]
stoye2009minimax_regret_finite_samples shows that the empirical success rule defined below is minimax optimal for worst-case superpopulation regret.
The superpopulation problem does not account for the exploration--exploitation tradeoff that arises in a finite population of size $N$. Nevertheless, Proposition (ref) remains a key building block for the finite-population framework developed below.
We refer to a pair $(m,\delta)$, consisting of the experimental sample size and the treatment rule, as a policy. For a given policy $(m,\delta)$, the expected finite-population welfare is $$
$$ Under the i.i.d.\ assumption across both experimental and rollout units, $$ g(m,\delta,\mu_1,\mu_0) = \frac{m}{2}(\mu_1+\mu_0) + (N-m)\Bigl[\mu_0\{1-\mathbb{E}\left[\delta(\omega)\right]\}+\mu_1\mathbb{E}\left[\delta(\omega)\right]\Bigr]. $$
As in the superpopulation case, directly maximizing the worst-case finite-population welfare is uninformative because the adversary would simply choose $\mu_1 = \mu_0 = 0$, yielding exactly zero welfare for any policy $(m, \delta)$.
We therefore evaluate policies based on their shortfall relative to an oracle benchmark. Consider an oracle that knows $(\mu_1,\mu_0)$ ex ante. The oracle does not need to experiment, so $(m^{\mathrm{oracle}},\delta^{\mathrm{oracle}})$ with $m^{\mathrm{oracle}}=0$ and $\delta^{\mathrm{oracle}}=\mathbf{1}\{\mu_1>\mu_0\}$ is the first-best policy, and its exptected welfare is $$ g^{\mathrm{oracle}} = N\max\{\mu_1,\mu_0\}. $$ For any policy $(m,\delta)$, we define the finite-population regret by $$ \mathrm{Reg}(m,\delta,\mu_1,\mu_0) := g^{\mathrm{oracle}}-g(m,\delta,\mu_1,\mu_0). $$
Under the empirical success rule $\hat\delta$, regret decomposes naturally into an exploration cost and an exploitation risk:
where the exploration cost is $$ C(m,\mu_1,\mu_0):=\frac{m}{2}|\tau| $$ and the exploitation risk is $$ R(m,\mu_1,\mu_0):=(N-m)|\tau|\,e(m,\mu_1,\mu_0). $$ Here
is the probability that the empirical success rule fails to select the better arm.
The exploration cost $C(m,\mu_1,\mu_0)$ represents the welfare loss from assigning, ex post, half of the experimental sample to the inferior arm. The exploitation risk $R(m,\mu_1,\mu_0)$ represents the welfare loss from making an erroneous rollout decision and assigning the inferior arm to the remaining $N-m$ individuals.
This decomposition implies that the choice of the experimental sample size $m$ involves a tradeoff between the exploration cost and the exploitation risk. A larger $m$ increases the exploration cost but reduces the exploitation risk, while a smaller $m$ does the reverse. A sensible choice of $m$ should therefore balance these two components.
We first consider selecting a policy $(m,\delta)$ by minimizing worst-case regret.
Previous work, such as feit2019test_and_roll, typically chooses sample sizes by placing priors on $(\mu_1,\mu_0)$, taking the treatment rule as given. Such priors can be difficult to specify, and the resulting design can be sensitive to misspecification. By contrast, the minimax regret criterion selects policies based on worst-case performance, thereby avoiding the arbitrariness associated with prior selection.
We next ask whether the empirical success rule $\hat{\delta}$ remains minimax-regret optimal in our test-and-roll setting. The following proposition shows that, for any fixed experimental sample size $m$, it remains globally minimax-regret optimal, extending the result of stoye2009minimax_regret_finite_samples to the finite-population setting.
This result justifies our restriction to the empirical success rule $\hat{\delta}$ and reduces the design problem to the choice of the experimental sample size $m$.
This section illustrates sample-size selection under the standard absolute minimax regret criterion through a numerical study. We show that, in the finite-population setting, this criterion can recommend experimental sample sizes that are implausibly small for practical applications.
We first examine the behavior of $\hat m^{\mathrm{MR}}$ across different population sizes. Table (ref) reports the minimax-regret sample size $\hat m^{\mathrm{MR}}$ and the corresponding experimental fraction $\hat m^{\mathrm{MR}}/N$ for several values of $N$. The criterion recommends very small experiments: the optimal experimental fraction is about $9\%$ when $N=200$ and falls to only $2.3\%$ when $N=10{,}000$.
Figure (ref) illustrates the adversary's strategy and the resulting minimax regret for $N=500$. Panel (a) shows the worst-case parameter values as a function of $m$. When $m$ is small, the adversary seeks to maximize exploitation risk: because the rollout population, $N-m$, carries much greater weight than the experimental sample, the adversary favors interior parameter values with $\mu_1\approx\mu_0$, where the better arm is difficult to identify. When $m$ is large, the adversary instead maximizes exploration cost by choosing extreme parameter values, such as $(\mu_1,\mu_0)=(1,0)$, that maximize $|\tau|$. Panel (b) shows the resulting worst-case regret for each value of $m$.
Although these two adversarial regimes reflect the exploration--exploitation tradeoff, the standard absolute minimax regret criterion suppresses exploration to an excessive degree. To see this, consider the regret decomposition \[ \mathrm{Reg}(m,\hat\delta,\mu_1,\mu_0) = \frac{m}{2}|\tau| + (N-m)|\tau|\,e(m,\mu_1,\mu_0), \] where the first term is the exploration cost and the second term is the exploitation risk. The exploitation risk is scaled by the error probability $e(m,\mu_1,\mu_0)$, which is at most one. Hence, for large $m$, the adversary can generate large regret simply by choosing parameter values with a large treatment-effect magnitude $|\tau|$, such as $(\mu_1,\mu_0)=(1,0)$, even if the probability of a mistaken rollout decision is small. As a result, the maximum regret is often driven by the exploration-cost component, $(m/2)|\tau|$. The criterion therefore penalizes experimentation too severely and pushes the optimal sample size toward very small values.
A similar issue arises under other common criteria, including relative regret, $$ \mathrm{RReg}(m,\hat\delta,\mu_1,\mu_0) := \frac{\mathrm{Reg}(m,\hat\delta,\mu_1,\mu_0)}{g^{\mathrm{oracle}}}, $$ which normalizes regret by the oracle gain. Appendix (ref) shows that this criterion also fails to yield a meaningful interior optimum.
These limitations suggest that total-regret criteria may not provide an appropriate balance between exploration and exploitation. We therefore turn to a criterion that evaluates this tradeoff at the margin, as developed in the next section.
The undesirable behavior of absolute minimax regret stems from an imbalance in the exploration--exploitation tradeoff: the exploration cost receives too much weight relative to the exploitation risk. We therefore propose a criterion that evaluates this tradeoff directly at the margin.
As the experimental sample size increases, the exploration cost rises while the exploitation risk falls. For small $m$, additional experimentation is typically worthwhile because the resulting reduction in exploitation risk exceeds the incremental exploration cost. Since information exhibits diminishing marginal returns, however, there should be a stopping point at which this inequality reverses.
We formalize this idea by comparing the marginal reduction in exploitation risk from adding one matched pair to the experiment with the corresponding increase in exploration cost.
The ratio $\eta(m,\mu_1,\mu_0)$ measures the marginal reduction in exploitation risk from adding one matched pair to the experiment, normalized by the corresponding marginal increase in exploration cost. When $\eta(m,\mu_1,\mu_0)>1$, this additional experimentation is worthwhile in expectation; when $\eta(m,\mu_1,\mu_0)\le 1$, it is not justified because the expected marginal benefit no longer exceeds the expected marginal cost.
Because $ C(m+2,\mu_1,\mu_0)-C(m,\mu_1,\mu_0)=|\tau|, $ this criterion can be written equivalently as
where $(N-m)e(m,\mu_1,\mu_0)$ represents the expected number of individuals who receive the inferior arm during the rollout phase if the experiment stops at size $m$. Thus the expression ((ref)) measures how many individuals are saved from the inferior treatment in the rollout by adding one more matched pair to the experiment.
We now define the optimal experimental sample size by applying this marginal cost--benefit comparison in a worst-case manner over all possible states of nature.
Under this criterion, the adversary no longer maximizes total regret. Instead, it chooses the parameter values that maximize the marginal benefit of continued experimentation. Conceptually, the WMB rule stops as soon as the decision-maker can guarantee that switching from exploration to exploitation would not entail a marginal welfare loss from stopping too early. In this sense, the rule guards against under-experimentation: there is no possible scenario in which one more matched pair would have reduced exploitation losses by more than its marginal exploration cost.
This section illustrates the behavior of the Bernoulli WMB rule through a numerical study, which motivates the theoretical analysis in Section (ref). We highlight three key findings. First, when $N$ is small, the least favorable states lie on the boundary, leading to an optimal sample size $\hat m^{\mathrm{WMB}}$ close to $N/2$. Second, as $N$ grows, the least favorable parameters move away from the boundary and localize near the diagonal. Third, within this interior regime, the WMB sample size approaches the rule-of-thirds benchmark, $\hat m^{\mathrm{WMB}}\approx N/3$.
For the numerical computations, we calculate the WMB sample size over discrete grids with $\epsilon=0.01$ and $\epsilon=0.005$: $$ \Theta_{\epsilon}^{\mathrm{grid}} = \left\{ (\mu_1,\mu_0)\in[0,1]^2: \mu_1,\mu_0\in\{0,\epsilon,2\epsilon,\ldots,1\}, \ |\mu_1-\mu_0|\ge \epsilon \right\}. $$ Specifically, we compute $$ \hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N) = \min\left\{ m\in\{0,2,\ldots,N-2\}: \max_{(\mu_1,\mu_0)\in\Theta_{\epsilon}^{\mathrm{grid}}} \eta(m,\mu_1,\mu_0)\le 1 \right\}. $$
Figure (ref) shows the maximized marginal cost--benefit ratio and illustrates where it crosses one for $N=500$ and $\epsilon=0.01$. The shape of the curve indicates that the marginal benefit of exploration declines as the experimental sample size increases.
Table (ref) reports the calculated WMB sample size $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N)$ for each population size $N$, together with the corresponding least favorable parameters $(\mu_0,\mu_1)$. When $N$ is small relative to $1/\epsilon$, the required experimental share is close to $1/2$ because the least favorable states lie on the boundary, for example with one parameter equal to zero. As $N$ increases, the calculated experimental share decreases and stabilizes near the $N/3$ benchmark, while the least favorable parameters move to the interior and localize near the diagonal.
These findings suggest that the pathological result $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N) \approx N/2$ is driven by least favorable states on the boundary. Such boundary cases can arise for any population size $N$ unless the state space $\Theta$ is restricted by the grid parameter $\epsilon$. After excluding these cases, the optimal sample size $\hat m^{\mathrm{WMB}}$ approaches the $N/3$ benchmark as $N$ becomes sufficiently large.
This section formalizes the numerical findings from Section (ref). We first show that the pathological result $\hat m^{\mathrm{WMB}}\approx N/2$ arises from a boundary rare-event detection problem, which is conceptually distinct from the comparison of two nondegenerate treatments. We then prove that, under a regular growth regime, the least favorable Bernoulli states localize near the diagonal. This localization allows the exact Bernoulli WMB ratio to be uniformly approximated by an asymptotic Gaussian limit, which leads to our main result: the rule-of-thirds benchmark, $\hat m^{\mathrm{WMB}}\approx N/3$.
Before turning to our main analysis, it is useful to clarify why the exact Bernoulli WMB problem over the unrestricted parameter space $[0,1]^2$ can demand $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N) \approx N/2$. This extreme sample size requirement arises not from comparing two well-behaved distributions, but from the pathological nature of the boundary, where one arm has zero variance and the other's success probability vanishes at the rate of $O(1/N)$.
Example (ref) shows that the near-$N/2$ behavior is generated by a pathological rare-event detection mechanism.
We threfore proceed by excluding this pathological case. To rule out the boundary pathology, the remainder of the analysis uses the centered parametrization $$ \mu:=\frac{\mu_1+\mu_0}{2}, \qquad \delta:=\frac{\mu_1-\mu_0}{2}, $$ and imposes the interior midpoint restriction $\mu\in[\kappa,1-\kappa]$ for some fixed $\kappa\in(0,1/2)$. In this notation, $\delta$ denotes the half-gap between the two arms.
As a foundation for the subsequent analysis, we first examine the least favorable state of nature that maximizes the marginal benefit ratio $\eta(m,\mu_1,\mu_0)$. The next result formalizes the simulation finding that, as the experimental sample size grows with the population size, the least favorable Bernoulli configuration moves toward the diagonal.
Proposition (ref) identifies the relevant asymptotic behavior of the least favorable states: the least favorable Bernoulli parameters concentrate near the diagonal. As shown in the next subsection, this localization places the problem on a scale on which a Gaussian approximation becomes uniformly valid.
Building on Proposition (ref), this subsection establishes a uniform Gaussian approximation on the relevant interior region and solves the resulting asymptotic Gaussian objective. This analysis yields the analytical rule-of-thirds benchmark.
For the remainder of this subsection, we parameterize the treatment and control means by their midpoint $\mu$ and half-gap $\delta$: $$ \mu_1 = \mu+\delta, \qquad \mu_0 = \mu-\delta, $$ and write $P_{\mu,\delta}$ for probabilities evaluated under this state. The exclusion of the pathological boundary case discussed in Section (ref), together with the near-diagonal localization result in Proposition (ref), motivates us to restrict attention to the following local parameter space: $$ \mathcal A_N(\kappa,K) = \left\{ (\mu,\delta): \mu\in[\kappa,1-\kappa],\ 0<\delta\le K\sqrt{\frac{\log N}{N}},\ \mu\pm\delta\in[0,1] \right\}. $$ This class rules out the pathological boundary case by imposing $\mu\in[\kappa,1-\kappa]$ and captures the near-diagonal behavior by allowing the half-gap $\delta$ to shrink with $N$.
Define the Gaussian limit function as $$ f_k(t):=2\Phi(-t)+kt\phi(t), $$ where $\Phi$ and $\phi$ denote the standard normal distribution and density functions. For any even sequence of sample sizes $\{m_N\}$, set $$ k_N:=\frac{N-m_N}{m_N}, \qquad q:=\mu(1-\mu), \qquad t_N(\mu,\delta):=\frac{\sqrt{m_N}\,\delta}{\sqrt{q-\delta^2}}. $$ To analyze the tie-breaking event, let $S_{m_N}$ denote the difference in the total number of successes between the treatment and control arms: $$ S_{m_N} := \sum_{i=1}^{m_N/2} Y_i(1) - \sum_{i=m_N/2+1}^{m_N} Y_i(0). $$
The next theorem shows that, uniformly over the full interior region, the exact Bernoulli marginal ratio is asymptotically governed by the Gaussian limit function $f_{k_N}$.
Theorem (ref) provides a bridge between the exact Bernoulli problem and a continuous asymptotic analysis. Once the least favorable sequence is interior in the midpoint and local in the gap, the exact Bernoulli WMB problem reduces asymptotically to a Gaussian approximation problem, with the marginal ratio governed by the Gaussian limit function $f_{k_N}\bigl(t_N(\mu,\delta)\bigr)$.
This reduction allows us to characterize the asymptotic behavior of the Bernoulli WMB sample size $\hat m^{\mathrm{WMB}}$. The resulting normal-approximation objective admits a closed-form solution, as shown in the following lemma.
Together with Proposition (ref) and Theorem (ref), this theorem establishes the rule-of-thirds benchmark, $\hat m^{\mathrm{NA}}\approx N/3$, as the asymptotically optimal experimental sample size under the WMB criterion for Bernoulli outcomes. This conclusion applies once the boundary rare-event pathology described in Example (ref) has been ruled out.
The following theorem formalizes the rule-of-thirds benchmark as an asymptotic stopping threshold.
This section examines the experimental design problem under Gaussian outcomes with a common variance. We show that the two main conclusions from the Bernoulli analysis continue to hold in a non-asymptotic Gaussian setting. Unlike in the Bernoulli case, the WMB criterion yields the rule of thirds without excluding the boundary pathology discussed in Section (ref).
We suppose that the potential outcomes follow Gaussian distributions: \[ Y_i(1) \sim N(\mu_1,\sigma^2), \qquad Y_i(0) \sim N(\mu_0,\sigma^2), \] where $(\mu_1,\mu_0)\in\mathbb{R}^2$ and the variance $\sigma^2>0$ is common across arms. As in the Bernoulli setting, the superpopulation regret is $$ \mathrm{SReg}(\delta,\mu_1,\mu_0) = \max\{\mu_1,\mu_0\} - \Bigl[ \mu_0\{1-\mathbb{E}\left[\delta(\omega)\right]\} + \mu_1\mathbb{E}\left[\delta(\omega)\right] \Bigr]. $$
The following proposition shows that the empirical success rule remains minimax optimal for the Gaussian superpopulation problem.
We next consider sample-size selection under the absolute minimax regret criterion. For a fixed policy $(m,\delta)$, the expected finite-population gain is $$ g(m,\delta,\mu_1,\mu_0) = \frac{m}{2}(\mu_1+\mu_0) + (N-m)\Bigl[ \mu_0\{1-\mathbb{E}\left[\delta(\omega)\right]\} + \mu_1\mathbb{E}\left[\delta(\omega)\right] \Bigr]. $$ The oracle benchmark is $ g^{\mathrm{oracle}}=N\max\{\mu_1,\mu_0\}, $ and the corresponding regret is $$ \mathrm{Reg}(m,\delta,\mu_1,\mu_0) = \frac{m}{2}|\tau|+(N-m)\mathrm{SReg}(\delta,\mu_1,\mu_0). $$ Unlike in the Bernoulli model, the Gaussian parameter space is unbounded. Consequently, for every fixed $m$ and every rule $\delta$, $$ \sup_{(\mu_1,\mu_0)\in\mathbb{R}^2} \mathrm{Reg}(m,\delta,\mu_1,\mu_0)=\infty. $$ Thus, without an arbitrary compact restriction on the parameter space, the absolute minimax regret criterion provides no meaningful guidance in the Gaussian model. This highlights the usefulness of the WMB criterion in the present setting.
We now turn to sample-size selection under the WMB criterion. In the Gaussian model, the rollout error probability can be derived exactly in closed form.
To facilitate an analysis, we define continuous relaxations of the cost, risk, and marginal ratio introduced in the Bernoulli setting.
Using this continuous relaxation, the marginal ratio can be evaluated explicitly. The following theorem shows that, in the Gaussian model, the rule of thirds emerges as the exact WMB threshold without requiring any asymptotic approximation or restriction of the parameter space.
Theorem (ref) provides the exact Gaussian analogue of the Bernoulli rule-of-thirds benchmark. In contrast to the Bernoulli model, no local approximation is required: the function $f_k$ characterizes the marginal ratio exactly, and the WMB threshold is obtained directly from its worst-case behavior.
This paper studies welfare-aware sample-size selection for test-and-roll experiments. In this setting, increasing the experimental sample size improves the accuracy of the rollout decision, but also exposes more experimental units to the inferior treatment ex post. The design problem therefore involves an exploration--exploitation tradeoff that is not captured by conventional power-based sample-size calculations.
We show that the standard absolute minimax regret criterion can recommend implausibly small experiments because its worst-case objective over-penalizes exploration at extreme parameter values. To address this limitation, we propose the Worst-case Marginal Benefit (WMB) rule, which compares the worst-case marginal benefit of adding one more matched pair with the corresponding marginal exploration cost.
Our main result is a simple rule-of-thirds benchmark. For Bernoulli outcomes, after excluding a boundary rare-event pathology, the WMB criterion yields $\hat m^{\mathrm{NA}} \approx N/3$ through a Gaussian approximation. For Gaussian outcomes with a common variance, the same benchmark arises exactly in the continuous-relaxation problem. These results provide a prior-free and practically implementable guide for welfare-based sample-size design.