The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
51,112 characters
Prior-Free Sample Size Design for Test-and-Roll Experiments
\setstretch{1.25}
\title{
Prior-Free Sample Size Design\\ for Test-and-Roll Experiments\thanks{The authors gratefully acknowledge financial support from JSPS KAKENHI Grant (number 24K16342).}
}
\author{Kentaro Kawato\thanks{Graduate School of Economics, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan. Email:~[email removed].} \ and \ Shosei Sakaguchi\thanks{Faculty of Economics, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan. Email:~[email removed].}}
\date{\today}
\vspace{-0.8cm}
\maketitle
\begin{abstract}
This paper studies sample-size design for finite-population test-and-roll experiments, where a decision-maker first conducts an experiment on \(m\) units and then assigns the remaining \(N-m\) units to the treatment that performs better in the experiment.
We consider welfare-aware sample-size choice, which involves an exploration--exploitation tradeoff: larger experiments improve the rollout decision but impose welfare losses on experimental units assigned to the inferior treatment.
We show that the standard absolute minimax regret criterion can lead to implausibly small experiments by over-penalizing exploration in its worst-case objective.
To address this limitation, we propose the Worst-case Marginal Benefit (WMB) rule, which compares the worst-case marginal benefit of adding one more matched pair to the experiment with the corresponding marginal exploration cost.
We establish a simple rule-of-thirds benchmark. For Bernoulli outcomes, after excluding pathological cases, the WMB criterion yields the optimal sample size of \(m\approx N/3\) through a Gaussian approximation. For Gaussian outcomes with a known common variance, the same benchmark arises exactly. These results provide a prior-free and practically implementable guide for welfare-based sample-size design.
\bigskip
\medskip
\begin{spacing}{1.2}
\noindent
\textbf{Keywords:} Experimental design; test-and-roll experiments; sample-size design; minimax regret; exploration--exploitation tradeoff. \\
\end{spacing}
\end{abstract}
\setstretch{1.5}
\newpage
\section{Introduction}
Randomized controlled trials (RCTs) are the gold standard in causal inference, but choosing the experimental sample size remains a central theoretical and practical challenge. Standard approaches based on Type I and Type II error control have little consequence for the welfare objectives of the decision-maker. This paper studies sample-size design from a welfare perspective, focusing on the exploration--exploitation tradeoff that arises when an RCT is conducted within a fixed finite population.
We consider a ``test-and-roll'' setting with a finite population of size $N$. A decision-maker first conducts an RCT of size $m$, assigning half of the experimental units to treatment and half to control. Based on the observed outcomes, the decision-maker assigns the remaining $N-m$ units to the treatment that performed better in the experiment. The objective is to maximize expected welfare over the entire finite population,
$\mathbb{E}\left[\sum_{i=1}^N Y_i\right]$, where the $N$ units include both the experimental and rollout units. The key design question is how large the experimental sample size $m$ should be.
This problem involves a fundamental exploration--exploitation tradeoff. A larger experiment improves the accuracy of the rollout decision, reducing the risk of assigning the remaining population to the inferior treatment. At the same time, because half of the experimental units receive the inferior treatment ex post, increasing the experimental sample size directly imposes welfare losses on the experimental population. Thus, excessive exploration is costly, while insufficient exploration can lead to an erroneous rollout decision and welfare losses for the rollout population.
This tradeoff arises in many applications, including clinical trials comparing alternative medical treatments \citep[e.g.,][]{wang2013clopidogrel}, test-and-roll A/B testing in digital marketing \citep{feit2019test_and_roll}, and field experiments on workplace incentives \citep{friebel2017team}.
A central challenge in optimizing this tradeoff is that the true treatment effects are unknown ex ante. Existing approaches to experimental sample-size design typically address this uncertainty by placing Bayesian prior distributions on the unknown parameters. Such priors can be difficult to specify properly, and the resulting design can be sensitive to the chosen specification. This difficulty is especially acute in our setting because the experimental sample size must be chosen before any data are observed. Thus, a practically useful design rule should provide guidance before learning takes place, without relying on subjective priors or arbitrary hyperparameters.
We address this challenge through a prior-free minimax analysis of the experimental sample-size choice. A natural starting point is the standard absolute minimax regret criterion \cite[e.g.,][]{stoye2009minimax_regret_finite_samples}, which evaluates the worst-case welfare loss relative to the infeasible oracle decision. Although Bayesian test-and-roll designs have been studied \citep[e.g.,][]{feit2019test_and_roll}, the implications of this purely minimax benchmark are less well understood. We show that, in finite-population test-and-roll problems, the standard absolute minimax regret criterion can lead to implausibly small experimental sample. This issue arises because the criterion is overly sensitive to extreme cases. The worst-case adversary concentrates on parameter values that maximize the absolute treatment effect, thereby over-penalizing exploration.
To address this limitation, we propose a new robust criterion for sample-size design: the \emph{Worst-case Marginal Benefit} (WMB) rule. The key idea is to evaluate experimentation at the margin rather than through worst-case total regret. The WMB rule compares the worst-case marginal benefit of adding one more matched pair to the experiment with the corresponding marginal exploration cost, and stops experimentation once the former no longer exceeds the latter. We study the optimal experimental sample size under this criterion for binary and Gaussian outcomes.
This paper establishes a simple rule-of-thirds benchmark. For Bernoulli outcomes, after excluding a boundary rare-event pathology, we show that the WMB criterion yields the optimal experimental sample size, $m^{\ast}\approx N/3$, through a Gaussian approximation. For Gaussian outcomes with a known common variance, the same benchmark arises exactly. Because this benchmark depends only on the total population size $N$, it can be implemented directly in practice and provides a prior-free guide for sample-size design without requiring subjective priors, prior variances, or other tuning parameters.
\subsection{Related Literature}
This paper contributes to the statistical literature on sample-size selection and experimental design. Conventional sample-size calculations are typically based on power analysis, which chooses the experimental sample size to control Type I and Type II error probabilities \citep[e.g.,][]{Lachin1981, DufloGlennersterKremer2007}. While useful for hypothesis testing, this approach does not directly account for the welfare consequences of experimentation.
Our paper departs from the classical power-based framework by choosing the experimental sample size to optimize a finite-population welfare objective that accounts for both the welfare cost imposed on experimental units and the welfare consequences of the rollout decision.
Several studies examine optimal sample-size selection from a welfare-based perspective, typically using Bayesian approaches to test-and-roll designs. \citet{cheng2003choosing_sample_size_decision_analysis} study the optimal first-stage sample size for a finite population under Beta priors, and \citet{stallard2017optimal_sample_size_population_size} extend this decision-theoretic framework to more general models. \citet{feit2019test_and_roll} formulate A/B testing as a finite-population test-and-roll problem and derive optimal sample sizes under normal priors. These papers share our welfare-based perspective, but their methods depend on prior distributions or parametric assumptions. By contrast, we take a prior-free approach by studying minimax criteria and proposing the Worst-case Marginal Benefit rule.
Our work is also related to the literature on statistical treatment choice. Starting with \citet{manski2004statistical_treatment_rules}, this literature studies how experimental or observational data should be used to choose treatments so as to maximize welfare; see also \citet{stoye2009minimax_regret_finite_samples}, \citet{kitagawa2018who_should_be_treated}, and \citet{athey2021policy_learning}. These studies typically take the sample size as given and focus on the treatment assignment rule after data have been collected. In contrast, we study an ex ante design problem in which the decision-maker must choose the experimental sample size before observing any data. Thus, our analysis complements the statistical treatment choice literature by incorporating the cost of experimentation into the sample-size decision itself.
Our framework is also related to the multi-armed bandit literature, especially explore-then-commit strategies \citep{garivier2016etc, perchet2016batched, LattimoreSzepesvari2020BanditAlgorithms}. While bandit algorithms address exploration--exploitation tradeoffs through sequential treatment assignment, such adjustment is often infeasible in social-science, medical, and marketing experiments because outcomes are delayed, implementation is costly, or frequent reassignment raises ethical concerns. We therefore focus on test-and-roll designs, where the experimental sample size must be fixed in advance.
Our paper also relates to recent work incorporating economic and welfare considerations into experimental design \citep{azevedo2020ab_testing_fat_tails, azevedo2023ab_testing_gaussian_priors,narita2021incorporating, hu2024minimax_regret_sample_selection}. Relative to these studies, our contribution is to isolate the finite-population test-and-roll sample-size problem and derive a simple prior-free benchmark based on a marginal minimax criterion.
\subsection{Structure of the Paper}
The remainder of the paper is organized as follows. Section~\ref{sec:minimax-regret} formulates the Bernoulli test-and-roll problem and analyzes absolute minimax regret. Section~\ref{sec:wmb} introduces the WMB rule. Section~\ref{sec:simulation} examines its finite-sample behavior through numerical examples. Section~\ref{sec:theoretical-results} establishes the rule-of-thirds benchmark. Section~\ref{sec:gaussian-known-variance} extends the analysis to Gaussian outcomes with a common variance. Section~\ref{sec:conclusion} concludes. Proofs and additional discussions are provided in the appendix.
\section{The Minimax Regret Criterion and Its Limitations}\label{sec:minimax-regret}
This section introduces the finite-population test-and-roll problem with binary outcomes. We first define the test-and-roll experiment and introduce the empirical success rule of \citet{stoye2009minimax_regret_finite_samples}, which is minimax-regret optimal for the corresponding superpopulation problem under a matched-pairs design. We then define a finite-population regret criterion for sample-size choice and show why the standard absolute minimax regret objective can recommend implausibly small experiments.
\subsection{Setup and Treatment Rule}
Suppose that $N$ is even and that, for each individual $i=1,\dots,N$, the potential-outcome pair $(Y_i(1),Y_i(0))$ is drawn i.i.d. across individuals with Bernoulli marginals:
$$Y_i(1)\sim \mathrm{Bernoulli}(\mu_1), \qquad Y_i(0)\sim \mathrm{Bernoulli}(\mu_0).$$
Let $T_i\in\{0,1\}$ denote the treatment assignment. The observed outcome is $Y_i=T_iY_i(1)+(1-T_i)Y_i(0)$.
We write
$$
\tau:=\mu_1-\mu_0
$$
for the average treatment effect.
Suppose that the first $m$ individuals are assigned to the experiment, where $m$ is even. We consider a matched-pairs design in which $m/2$ individuals are assigned to treatment and the remaining $m/2$ individuals are assigned to control. Without loss of generality, we index the treated individuals by $1,\dots,m/2$ and the control individuals by $m/2+1,\dots,m$. The sample mean outcomes for the treatment and control groups are
$$
\hat\mu_1:=\frac{2}{m}\sum_{i=1}^{m/2}Y_i,
\qquad
\hat\mu_0:=\frac{2}{m}\sum_{i=m/2+1}^{m}Y_i.
$$
Following \citet{stoye2009minimax_regret_finite_samples}, let $\omega=\{(T_i,Y_i)\}_{i=1}^{m}$ denote the realized experimental data. A statistical treatment rule is a mapping
$
\delta:(\{0,1\} \times \{0,1\})^m\to[0,1],
$
where $\delta(\omega)$ is interpreted as the probability of assigning treatment $1$ to an individual in the rollout population after observing $\omega$. Let $\mathcal{D}$ denote the class of all such rules.
For decision-making under uncertainty, it may be tempting to choose a treatment rule that maximizes worst-case welfare,
\[
\max_{\delta\in\mathcal D}\min_{(\mu_1,\mu_0)\in[0,1]^2}
u(\delta,\mu_1,\mu_0).
\]
However, this maximin welfare criterion is uninformative because an adversarial nature would choose the worst possible expected outcomes, $\mu_1=\mu_0=0$, in which case expected welfare is zero for any treatment rule $\delta$.
A common alternative is to evaluate a treatment rule using worst-case superpopulation regret. Superpopulation regret is defined as the shortfall relative to an oracle that knows the true state ex ante:
\[
\mathrm{SReg}(\delta,\mu_1,\mu_0)
:=
\max\{\mu_1,\mu_0\}-u(\delta,\mu_1,\mu_0).
\]
The minimax-regret approach then selects a treatment rule that minimizes the worst-case superpopulation regret,
\[
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\mathrm{SReg}(\delta,\mu_1,\mu_0).
\]
\citet{stoye2009minimax_regret_finite_samples} shows that the empirical success rule defined below is minimax optimal for worst-case superpopulation regret.
\medskip
\begin{proposition}[Superpopulation minimax optimality of the empirical success rule; \citealp{stoye2009minimax_regret_finite_samples}]\label{prop:stoye-optimal-treatment}
Under the matched-pairs design, the minimax-regret optimal treatment rule is the empirical success rule
\begin{align}
\hat\delta(\omega)
:=
\begin{cases}
0, & \text{if } \hat\mu_1<\hat\mu_0,\\[3pt]
\frac12, & \text{if } \hat\mu_1=\hat\mu_0,\\[3pt]
1, & \text{if } \hat\mu_1>\hat\mu_0.
\end{cases}
\end{align}
That is,
$$
\hat\delta
=
\operatorname*{arg\,min}_{\delta\in\mathcal{D}}
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\mathrm{SReg}(\delta,\mu_1,\mu_0).
$$
\end{proposition}
\medskip
The superpopulation problem does not account for the exploration--exploitation tradeoff that arises in a finite population of size $N$. Nevertheless, Proposition~\ref{prop:stoye-optimal-treatment} remains a key building block for the finite-population framework developed below.
\subsection{Finite-Population Regret}
We refer to a pair $(m,\delta)$, consisting of the experimental sample size and the treatment rule, as a \emph{policy}. For a given policy $(m,\delta)$, the expected finite-population welfare is
$$
\begin{aligned}
g(m,\delta,\mu_1,\mu_0)
&:=
\mathbb{E}\left[\sum_{i=1}^{m/2}Y_i \mathbf{1}\{T_i=1\} + \sum_{i=m/2+1}^{m}Y_i \mathbf{1}\{T_i=0 \} + \sum_{i=m+1}^{N}Y_i \mathbf{1}\{T_i=\delta(\omega)\} \right] \\
&=
\mathbb{E}\left[
\sum_{i=1}^{m/2}Y_i(1)
+
\sum_{i=m/2+1}^{m}Y_i(0)
+
\sum_{i=m+1}^{N}
\bigl(
\delta(\omega)Y_i(1)+\{1-\delta(\omega)\}Y_i(0)
\bigr)
\right].
\end{aligned}
$$
Under the i.i.d.\ assumption across both experimental and rollout units,
$$
g(m,\delta,\mu_1,\mu_0)
=
\frac{m}{2}(\mu_1+\mu_0)
+
(N-m)\Bigl[\mu_0\{1-\mathbb{E}\left[\delta(\omega)\right]\}+\mu_1\mathbb{E}\left[\delta(\omega)\right]\Bigr].
$$
As in the superpopulation case, directly maximizing the worst-case finite-population welfare is uninformative because the adversary would simply choose $\mu_1 = \mu_0 = 0$, yielding exactly zero welfare for any policy $(m, \delta)$.
We therefore evaluate policies based on their shortfall relative to an oracle benchmark. Consider an oracle that knows $(\mu_1,\mu_0)$ ex ante. The oracle does not need to experiment, so $(m^{\mathrm{oracle}},\delta^{\mathrm{oracle}})$ with
$m^{\mathrm{oracle}}=0$ and $\delta^{\mathrm{oracle}}=\mathbf{1}\{\mu_1>\mu_0\}$ is the first-best policy, and its exptected welfare is
$$
g^{\mathrm{oracle}}
=
N\max\{\mu_1,\mu_0\}.
$$
For any policy $(m,\delta)$, we define the finite-population regret by
$$
\mathrm{Reg}(m,\delta,\mu_1,\mu_0)
:=
g^{\mathrm{oracle}}-g(m,\delta,\mu_1,\mu_0).
$$
Under the empirical success rule $\hat\delta$, regret decomposes naturally into an exploration cost and an exploitation risk:
\begin{align}
\mathrm{Reg}(m,\hat\delta,\mu_1,\mu_0)
=
C(m,\mu_1,\mu_0)+R(m,\mu_1,\mu_0), \label{eq:decomoposition_regret}
\end{align}
where the exploration cost is
$$
C(m,\mu_1,\mu_0):=\frac{m}{2}|\tau|
$$
and the exploitation risk is
$$
R(m,\mu_1,\mu_0):=(N-m)|\tau|\,e(m,\mu_1,\mu_0).
$$
Here
\begin{align}
e(m,\mu_1,\mu_0)
:=
\begin{cases}
\mathbb{P}\left[\hat\mu_1<\hat\mu_0\right]+\frac12\mathbb{P}\left[\hat\mu_1=\hat\mu_0\right], & \mu_1>\mu_0,\\[4pt]
\frac12, & \mu_1=\mu_0,\\[4pt]
\mathbb{P}\left[\hat\mu_1>\hat\mu_0\right]+\frac12\mathbb{P}\left[\hat\mu_1=\hat\mu_0\right], & \mu_1<\mu_0
\end{cases} \label{eq:e_m}
\end{align}
is the probability that the empirical success rule fails to select the better arm.
The exploration cost $C(m,\mu_1,\mu_0)$ represents the welfare loss from assigning, ex post, half of the experimental sample to the inferior arm. The exploitation risk $R(m,\mu_1,\mu_0)$ represents the welfare loss from making an erroneous rollout decision and assigning the inferior arm to the remaining $N-m$ individuals.
This decomposition implies that the choice of the experimental sample size $m$ involves a tradeoff between the exploration cost and the exploitation risk. A larger $m$ increases the exploration cost but reduces the exploitation risk, while a smaller $m$ does the reverse. A sensible choice of $m$ should therefore balance these two components.
We first consider selecting a policy $(m,\delta)$ by minimizing worst-case regret.
\medskip
\begin{definition}[Minimax regret criterion]
The absolute minimax regret criterion selects
$$
(\hat m^{\mathrm{MR}}, \hat \delta^{\mathrm{MR}})
=
\operatorname*{arg\,min}_{(m,\delta)\in\{0,2,\dots,N\} \times \mathcal{D}}
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\mathrm{Reg}(m,\delta,\mu_1,\mu_0).
$$
\end{definition}
\medskip
Previous work, such as \citet{feit2019test_and_roll}, typically chooses sample sizes by placing priors on $(\mu_1,\mu_0)$, taking the treatment rule as given. Such priors can be difficult to specify, and the resulting design can be sensitive to misspecification. By contrast, the minimax regret criterion selects policies based on worst-case performance, thereby avoiding the arbitrariness associated with prior selection.
We next ask whether the empirical success rule $\hat{\delta}$ remains minimax-regret optimal in our test-and-roll setting. The following proposition shows that, for any fixed experimental sample size $m$, it remains globally minimax-regret optimal, extending the result of \citet{stoye2009minimax_regret_finite_samples} to the finite-population setting.
\medskip
\begin{proposition}[Finite-population minimax optimality for fixed experimental size]\label{prop:fixed-delta-justification}
For every even $m$, the empirical success rule $\hat\delta$ is minimax-regret optimal:
\[
\hat\delta
\in
\operatorname*{arg\,min}_{\delta\in\mathcal{D}}
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\mathrm{Reg}(m,\delta,\mu_1,\mu_0).
\]
Consequently,
\[
\hat m^{\mathrm{MR}}
\in
\operatorname*{arg\,min}_{m\in\{0,2,\dots,N\}}
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\mathrm{Reg}(m,\hat\delta,\mu_1,\mu_0).
\]
\end{proposition}
\medskip
\begin{proof}[Proof of Proposition~\ref{prop:fixed-delta-justification}]
Fix an even $m$. For any rule $\delta$,
$$
\mathrm{Reg}(m,\delta,\mu_1,\mu_0)
=
\frac{m}{2}|\mu_1-\mu_0|+(N-m)\mathrm{SReg}(\delta,\mu_1,\mu_0).
$$
The first term does not depend on $\delta$. Therefore,
$$
\operatorname*{arg\,min}_{\delta\in\mathcal{D}}
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\mathrm{Reg}(m,\delta,\mu_1,\mu_0)
=
\operatorname*{arg\,min}_{\delta\in\mathcal{D}}
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\mathrm{SReg}(\delta,\mu_1,\mu_0).
$$
By Proposition~\ref{prop:stoye-optimal-treatment}, the right-hand side is the empirical success rule $\hat\delta$. This proves the first claim. The expression for $\hat m^{\mathrm{MR}}$ follows immediately by optimizing the resulting worst-case regret over $m$.
\end{proof}
\medskip
This result justifies our restriction to the empirical success rule $\hat{\delta}$ and reduces the design problem to the choice of the experimental sample size $m$.
\subsection{Limitations of Absolute Minimax Regret}
This section illustrates sample-size selection under the standard absolute minimax regret criterion through a numerical study. We show that, in the finite-population setting, this criterion can recommend experimental sample sizes that are implausibly small for practical applications.
We first examine the behavior of $\hat m^{\mathrm{MR}}$ across different population sizes. Table~\ref{tab:minimax_m} reports the minimax-regret sample size $\hat m^{\mathrm{MR}}$ and the corresponding experimental fraction $\hat m^{\mathrm{MR}}/N$ for several values of $N$. The criterion recommends very small experiments: the optimal experimental fraction is about $9\%$ when $N=200$ and falls to only $2.3\%$ when $N=10{,}000$.
\bigskip
\begin{table}[H]
\centering
\caption{Optimal sample sizes $\hat m^{\mathrm{MR}}$ under the absolute minimax regret criterion for several population sizes $N$}
\label{tab:minimax_m}
\begin{tabular}{@{}lrrrrr@{}}
\toprule
$N$ & 200 & 500 & 1000 & 5000 & 10000 \\
\midrule
$\hat m^{\mathrm{MR}}$ & 18 & 32 & 50 & 146 & 230 \\
$\hat m^{\mathrm{MR}}/N$ & 0.090 & 0.064 & 0.050 & 0.029 & 0.023 \\
\bottomrule
\end{tabular}
\end{table}
\bigskip
Figure~\ref{fig:absolute_regret_worstcase} illustrates the adversary's strategy and the resulting minimax regret for $N=500$. Panel~(a) shows the worst-case parameter values as a function of $m$. When $m$ is small, the adversary seeks to maximize exploitation risk: because the rollout population, $N-m$, carries much greater weight than the experimental sample, the adversary favors interior parameter values with $\mu_1\approx\mu_0$, where the better arm is difficult to identify. When $m$ is large, the adversary instead maximizes exploration cost by choosing extreme parameter values, such as $(\mu_1,\mu_0)=(1,0)$, that maximize $|\tau|$. Panel~(b) shows the resulting worst-case regret for each value of $m$.
\bigskip
\begin{figure}[H]
\centering
\caption{Worst-case behavior under absolute minimax regret for $N=500$.}
\label{fig:absolute_regret_worstcase}
\begin{minipage}{0.48\textwidth}
\centering
\subcaption{Worst-case parameter values $(\mu_1,\mu_0)$ as a function of $m$.}
\label{fig:absolute_regret_worstcase_panel}
\IfFileExists{plot/paper_worstcase_mu_absolute_regret.pdf}{\includegraphics[width=\linewidth]{plot/paper_worstcase_mu_absolute_regret.pdf}}{
\fbox{\parbox[c][0.22\textheight][c]{0.88\linewidth}{\centering Missing figure file\\[3pt]\texttt{\detokenize{plot/paper_worstcase_mu_absolute_regret.pdf}}}}
}
\end{minipage}\hfill
\begin{minipage}{0.48\textwidth}
\centering
\subcaption{Worst-case absolute regret as a function of the experimental sample size $m$.}
\label{fig:absolute_regret_m_panel}
\IfFileExists{plot/paper_minimax_absolute_regret.pdf}{\includegraphics[width=\linewidth]{plot/paper_minimax_absolute_regret.pdf}}{
\fbox{\parbox[c][0.22\textheight][c]{0.88\linewidth}{\centering Missing figure file\\[3pt]\texttt{\detokenize{plot/paper_minimax_absolute_regret.pdf}}}}
}
\end{minipage}
\begin{minipage}{0.92\textwidth}
\medskip
\footnotesize
\emph{Note:} Panel~(a) shows the worst-case parameter values $(\mu_1,\mu_0)$ as a function of $m$.
Panel~(b) shows the resulting worst-case absolute regret as a function of $m$, which yields $\hat m^{\mathrm{MR}}=32$.
\end{minipage}
\end{figure}
\bigskip
Although these two adversarial regimes reflect the exploration--exploitation tradeoff, the standard absolute minimax regret criterion suppresses exploration to an excessive degree. To see this, consider the regret decomposition
\[
\mathrm{Reg}(m,\hat\delta,\mu_1,\mu_0)
=
\frac{m}{2}|\tau|
+
(N-m)|\tau|\,e(m,\mu_1,\mu_0),
\]
where the first term is the exploration cost and the second term is the exploitation risk. The exploitation risk is scaled by the error probability $e(m,\mu_1,\mu_0)$, which is at most one. Hence, for large $m$, the adversary can generate large regret simply by choosing parameter values with a large treatment-effect magnitude $|\tau|$, such as $(\mu_1,\mu_0)=(1,0)$, even if the probability of a mistaken rollout decision is small. As a result, the maximum regret is often driven by the exploration-cost component, $(m/2)|\tau|$. The criterion therefore penalizes experimentation too severely and pushes the optimal sample size toward very small values.
A similar issue arises under other common criteria, including relative regret,
$$
\mathrm{RReg}(m,\hat\delta,\mu_1,\mu_0)
:=
\frac{\mathrm{Reg}(m,\hat\delta,\mu_1,\mu_0)}{g^{\mathrm{oracle}}},
$$
which normalizes regret by the oracle gain. Appendix~\ref{sec:relative-regret} shows that this criterion also fails to yield a meaningful interior optimum.
These limitations suggest that total-regret criteria may not provide an appropriate balance between exploration and exploitation. We therefore turn to a criterion that evaluates this tradeoff at the margin, as developed in the next section.
\section{Proposed Method: Worst-case Marginal Benefit Rule}\label{sec:wmb}
The undesirable behavior of absolute minimax regret stems from an imbalance in the exploration--exploitation tradeoff: the exploration cost receives too much weight relative to the exploitation risk. We therefore propose a criterion that evaluates this tradeoff directly at the margin.
As the experimental sample size increases, the exploration cost rises while the exploitation risk falls. For small $m$, additional experimentation is typically worthwhile because the resulting reduction in exploitation risk exceeds the incremental exploration cost. Since information exhibits diminishing marginal returns, however, there should be a stopping point at which this inequality reverses.
We formalize this idea by comparing the marginal reduction in exploitation risk from adding one matched pair to the experiment with the corresponding increase in exploration cost.
\medskip
\begin{definition}[Marginal cost--benefit ratio]
\label{def:marginal_cost_benefit_ratio}
For even $m\le N-2$, we define the marginal cost--benefit ratio as
$$
\eta(m,\mu_1,\mu_0)
:=
-
\frac{R(m+2,\mu_1,\mu_0)-R(m,\mu_1,\mu_0)}
{C(m+2,\mu_1,\mu_0)-C(m,\mu_1,\mu_0)}.
$$
\end{definition}
\medskip
The ratio $\eta(m,\mu_1,\mu_0)$ measures the marginal reduction in exploitation risk from adding one matched pair to the experiment, normalized by the corresponding marginal increase in exploration cost. When $\eta(m,\mu_1,\mu_0)>1$, this additional experimentation is worthwhile in expectation; when $\eta(m,\mu_1,\mu_0)\le 1$, it is not justified because the expected marginal benefit no longer exceeds the expected marginal cost.
Because
$
C(m+2,\mu_1,\mu_0)-C(m,\mu_1,\mu_0)=|\tau|,
$
this criterion can be written equivalently as
\begin{equation}
\eta(m,\mu_1,\mu_0)
=
(N-m)e(m,\mu_1,\mu_0)-(N-m-2)e(m+2,\mu_1,\mu_0),
\label{eq:bernoulli_discrete_eta}
\end{equation}
where $(N-m)e(m,\mu_1,\mu_0)$ represents the expected number of individuals who receive the inferior arm during the rollout phase if the experiment stops at size $m$. Thus the expression~(\ref{eq:bernoulli_discrete_eta}) measures how many individuals are saved from the inferior treatment in the rollout by adding one more matched pair to the experiment.
We now define the optimal experimental sample size by applying this marginal cost--benefit comparison in a worst-case manner over all possible states of nature.
\medskip
\begin{definition}[Worst-case marginal benefit (WMB) rule]
\label{def:wmb-rule}
The WMB sample size is the smallest even sample size at which the marginal benefit of continued exploration is no greater than its marginal cost uniformly over all possible states of nature:
$$
\hat m^{\mathrm{WMB}}
=
\min\left\{
m\in\{0,2,\dots,N-2\}:
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\eta(m,\mu_1,\mu_0)\le 1
\right\}.
$$
\end{definition}
\medskip
Under this criterion, the adversary no longer maximizes total regret. Instead, it chooses the parameter values that maximize the marginal benefit of continued experimentation. Conceptually, the WMB rule stops as soon as the decision-maker can guarantee that switching from exploration to exploitation would not entail a marginal welfare loss from stopping too early. In this sense, the rule guards against under-experimentation: there is no possible scenario in which one more matched pair would have reduced exploitation losses by more than its marginal exploration cost.
\section{Numerical Study for the Bernoulli WMB Rule}\label{sec:simulation}
This section illustrates the behavior of the Bernoulli WMB rule through a numerical study, which motivates the theoretical analysis in Section~\ref{sec:theoretical-results}. We highlight three key findings. First, when $N$ is small, the least favorable states lie on the boundary, leading to an optimal sample size $\hat m^{\mathrm{WMB}}$ close to $N/2$. Second, as $N$ grows, the least favorable parameters move away from the boundary and localize near the diagonal. Third, within this interior regime, the WMB sample size approaches the rule-of-thirds benchmark, $\hat m^{\mathrm{WMB}}\approx N/3$.
For the numerical computations, we calculate the WMB sample size over discrete grids with $\epsilon=0.01$ and $\epsilon=0.005$:
$$
\Theta_{\epsilon}^{\mathrm{grid}}
=
\left\{
(\mu_1,\mu_0)\in[0,1]^2:
\mu_1,\mu_0\in\{0,\epsilon,2\epsilon,\ldots,1\},
\ |\mu_1-\mu_0|\ge \epsilon
\right\}.
$$
Specifically, we compute
$$
\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N)
=
\min\left\{
m\in\{0,2,\ldots,N-2\}:
\max_{(\mu_1,\mu_0)\in\Theta_{\epsilon}^{\mathrm{grid}}}
\eta(m,\mu_1,\mu_0)\le 1
\right\}.
$$
Figure~\ref{fig:eta_stopping_N500} shows the maximized marginal cost--benefit ratio and illustrates where it crosses one for $N=500$ and $\epsilon=0.01$. The shape of the curve indicates that the marginal benefit of exploration declines as the experimental sample size increases.
\bigskip
\begin{figure}[htbp]
\centering
\caption{Maximized marginal cost-benefit ratio at $N=500$ for $\epsilon=0.01$.}
\label{fig:eta_stopping_N500}
\IfFileExists{plot/integrated_eta_stopping_rule_N500.pdf}{\includegraphics[width=0.7\textwidth]{plot/integrated_eta_stopping_rule_N500.pdf}}{
\fbox{\parbox[c][0.22\textheight][c]{0.88\linewidth}{\centering Missing figure file\\[3pt]\texttt{\detokenize{plot/integrated_eta_stopping_rule_N500.pdf}}}}
}
\begin{minipage}{0.92\textwidth}
\medskip
\footnotesize
\emph{Note:}
This figure plots the maximized marginal cost-benefit ratio at $N=500$ for $\epsilon=0.01$.
The vertical line shows the optimal sample size $\hat m_{0.01,\mathrm{grid}}^{\mathrm{WMB}}(500) = 182$.
\end{minipage}
\end{figure}
\bigskip
Table~\ref{tab:worstcase_mu_summary} reports the calculated WMB sample size $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N)$ for each population size $N$, together with the corresponding least favorable parameters $(\mu_0,\mu_1)$. When $N$ is small relative to $1/\epsilon$, the required experimental share is close to $1/2$ because the least favorable states lie on the boundary, for example with one parameter equal to zero. As $N$ increases, the calculated experimental share decreases and stabilizes near the $N/3$ benchmark, while the least favorable parameters move to the interior and localize near the diagonal.
\begin{table}[htbp]
\centering
\caption{$\epsilon$-WMB sample sizes and least favorable parameters by $N$}
\label{tab:worstcase_mu_summary}
\begin{tabular}{@{}rrrrrr@{}}
\toprule
$\epsilon$ & $N$ & $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}$ & $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}/N$ & Least favorable $\mu_0$ & Least favorable $\mu_1$ \\
\midrule
$0.01$ & 200 & 90 & 0.450 & 0.010 & 0.000 \\
$0.01$ & 500 & 188 & 0.376 & 0.010 & 0.000 \\
$0.01$ & 1000 & 332 & 0.332 & 0.500 & 0.490 \\
$0.01$ & 5000 & 1608 & 0.322 & 0.500 & 0.490 \\
$0.01$ & 10000 & 3106 & 0.311 & 0.500 & 0.490 \\
\midrule
$0.005$ & 200 & 96 & 0.480 & 0.005 & 0.000 \\
$0.005$ & 500 & 216 & 0.432 & 0.005 & 0.000 \\
$0.005$ & 1000 & 376 & 0.376 & 0.005 & 0.000 \\
$0.005$ & 5000 & 1652 & 0.330 & 0.495 & 0.500 \\
$0.005$ & 10000 & 3274 & 0.327 & 0.500 & 0.505 \\
\bottomrule
\end{tabular}
\bigskip
\end{table}
These findings suggest that the pathological result $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N) \approx N/2$ is driven by least favorable states on the boundary. Such boundary cases can arise for any population size $N$ unless the state space $\Theta$ is restricted by the grid parameter $\epsilon$. After excluding these cases, the optimal sample size $\hat m^{\mathrm{WMB}}$ approaches the $N/3$ benchmark as $N$ becomes sufficiently large.
\section{Theoretical Results}\label{sec:theoretical-results}
This section formalizes the numerical findings from Section~\ref{sec:simulation}. We first show that the pathological result $\hat m^{\mathrm{WMB}}\approx N/2$ arises from a boundary rare-event detection problem, which is conceptually distinct from the comparison of two nondegenerate treatments. We then prove that, under a regular growth regime, the least favorable Bernoulli states localize near the diagonal.
This localization allows the exact Bernoulli WMB ratio to be uniformly approximated by an asymptotic Gaussian limit, which leads to our main result: the rule-of-thirds benchmark, $\hat m^{\mathrm{WMB}}\approx N/3$.
\subsection{The Boundary Rare-Event Pathology}\label{sec:boundary_pathology}
Before turning to our main analysis, it is useful to clarify why the exact Bernoulli WMB problem over the unrestricted parameter space $[0,1]^2$ can demand $\hat m_{\epsilon,\mathrm{grid}}^{\mathrm{WMB}}(N) \approx N/2$. This extreme sample size requirement arises not from comparing two well-behaved distributions, but from the pathological nature of the boundary, where one arm has zero variance and the other's success probability vanishes at the rate of $O(1/N)$.
\medskip
\begin{example}[Boundary rare-event states can force $m\approx N/2$]
\label{ex:boundary_pathology_half}
Consider the boundary sequence
$$
(\mu_1,\mu_0)=\left(\frac{c}{N},0\right),
\qquad c>0,
$$
where $\tau = \frac{c}{N} > 0$ since $c > 0$. Because the control arm never succeeds ($\mu_0=0$), its sample mean is $\hat\mu_0 = 0$ almost surely. The experiment therefore no longer compares two noisy averages; instead, it degenerates into rare-event detection in the treatment arm, where observing even a single success perfectly reveals the treatment's superiority. Let $S_1 := \frac{m}{2}\hat\mu_1$ denote the number of successes in the treatment arm, which is distributed as
$$
S_1 \sim \mathrm{Binomial}\left(\frac{m}{2},\tau\right),
\qquad
\tau=\frac{c}{N}.
$$
Under the empirical success rule, the experiment ends in a tie ($\hat\mu_1=\hat\mu_0$) a.s. if and only if $S_1=0$. In this event, the remaining $N-m$ individuals are split equally across the two arms. Since the treatment arm is better by $\tau$, the resulting exploitation risk is
$$
R(m)
= (1-\tau)^{m/2} \times \frac{N-m}{2} \tau.
$$
Since the marginal exploration cost is $\tau/2$ per additional individual, the continuous relaxation of the corresponding marginal ratio is
$$
\eta(m)
\approx
\frac{-\frac{\partial}{\partial m}R(m)}{\tau/2}
=
(1-\tau)^{m/2}
\left(
1-\frac{N-m}{2}\log(1-\tau)
\right).
$$
Now write
$$
m=\alpha N,
\qquad
\tau=\frac{c}{N}.
$$
As $N\to\infty$,
$$
\eta(\alpha N)
\to
e^{-\alpha c/2}
\left(
1+\frac{1-\alpha}{2}c
\right)
=
1+\frac{1-2\alpha}{2}c+O(c^2),
$$
where the equality follows from a Taylor expansion around $c=0$. If $\alpha<1/2$, then $\eta(\alpha N)>1$ for all sufficiently small $c>0$ and all sufficiently large $N$. Therefore, along this boundary sequence, the exact full-space WMB feasibility condition therefore fails below approximately $m=N/2$.
\end{example}
\medskip
Example~\ref{ex:boundary_pathology_half} shows that the near-$N/2$ behavior is generated by a pathological rare-event detection mechanism.
We threfore proceed by excluding this pathological case. To rule out the boundary pathology, the remainder of the analysis uses the centered parametrization
$$
\mu:=\frac{\mu_1+\mu_0}{2},
\qquad
\delta:=\frac{\mu_1-\mu_0}{2},
$$
and imposes the interior midpoint restriction $\mu\in[\kappa,1-\kappa]$ for some fixed $\kappa\in(0,1/2)$. In this notation, $\delta$ denotes the half-gap between the two arms.
\subsection{Localization near the Diagonal}
As a foundation for the subsequent analysis, we first examine the least favorable state of nature that maximizes the marginal benefit ratio $\eta(m,\mu_1,\mu_0)$. The next result formalizes the simulation finding that, as the experimental sample size grows with the population size, the least favorable Bernoulli configuration moves toward the diagonal.
\begin{proposition}[The exact Bernoulli maximizer localizes near the diagonal]
\label{prop:bernoulli_localization_body}
Let $\{m_N\}$ be a sequence with
$
m_N\in\{0,2,\dots,N-2\}.
$
For each $N$, let $(\mu_{1,N}^\star,\mu_{0,N}^\star)$ satisfy
$$
\eta(m_N,\mu_{1,N}^\star,\mu_{0,N}^\star)
=
\max_{(\mu_1,\mu_0)\in[0,1]^2}
\eta(m_N,\mu_1,\mu_0).
$$
Suppose that, for some $\alpha>0$, $m_N/N\to\alpha$ as $N\to\infty$.
Then
$$
|\mu_{1,N}^\star-\mu_{0,N}^\star|
\le
2\sqrt{\frac{\log N}{m_N}}
=
O\left(\sqrt{\frac{\log N}{N}}\right)
\to 0.
$$
\end{proposition}
Proposition~\ref{prop:bernoulli_localization_body} identifies the relevant asymptotic behavior of the least favorable states: the least favorable Bernoulli parameters concentrate near the diagonal. As shown in the next subsection, this localization places the problem on a scale on which a Gaussian approximation becomes uniformly valid.
\subsection{Uniform Gaussian Approximation and the Rule of Thirds}
Building on Proposition~\ref{prop:bernoulli_localization_body}, this subsection establishes a uniform Gaussian approximation on the relevant interior region and solves the resulting asymptotic Gaussian objective. This analysis yields the analytical rule-of-thirds benchmark.
For the remainder of this subsection, we parameterize the treatment and control means by their midpoint $\mu$ and half-gap $\delta$:
$$
\mu_1 = \mu+\delta,
\qquad
\mu_0 = \mu-\delta,
$$
and write $P_{\mu,\delta}$ for probabilities evaluated under this state.
The exclusion of the pathological boundary case discussed in Section~\ref{sec:boundary_pathology}, together with the near-diagonal localization result in Proposition~\ref{prop:bernoulli_localization_body}, motivates us to restrict attention to the following local parameter space:
$$
\mathcal A_N(\kappa,K)
=
\left\{
(\mu,\delta):
\mu\in[\kappa,1-\kappa],\
0<\delta\le K\sqrt{\frac{\log N}{N}},\
\mu\pm\delta\in[0,1]
\right\}.
$$
This class rules out the pathological boundary case by imposing $\mu\in[\kappa,1-\kappa]$ and captures the near-diagonal behavior by allowing the half-gap $\delta$ to shrink with $N$.
Define the Gaussian limit function as
$$
f_k(t):=2\Phi(-t)+kt\phi(t),
$$
where $\Phi$ and $\phi$ denote the standard normal distribution and density functions. For any even sequence of sample sizes $\{m_N\}$, set
$$
k_N:=\frac{N-m_N}{m_N},
\qquad
q:=\mu(1-\mu),
\qquad
t_N(\mu,\delta):=\frac{\sqrt{m_N}\,\delta}{\sqrt{q-\delta^2}}.
$$
To analyze the tie-breaking event, let $S_{m_N}$ denote the difference in the total number of successes between the treatment and control arms:
$$
S_{m_N} := \sum_{i=1}^{m_N/2} Y_i(1) - \sum_{i=m_N/2+1}^{m_N} Y_i(0).
$$
The next theorem shows that, uniformly over the full interior region, the exact Bernoulli marginal ratio is asymptotically governed by the Gaussian limit function $f_{k_N}$.
\medskip
\begin{theorem}[Uniform Gaussian approximation on $\mathcal A_N(\kappa,K)$]
\label{thm:full_region_uniform_gaussian_approx}
Fix $\kappa\in(0,1/2)$, $K>0$, and $\alpha_0\in(0,1)$. Let $\{m_N\}$ be any even sequence such that
$$
\alpha_0 N\le m_N\le N-2
\qquad (N\ge 1).
$$
Then, uniformly over $(\mu,\delta)\in\mathcal A_N(\kappa,K)$,
\begin{equation}
\eta(m_N,\mu+\delta,\mu-\delta)
=
f_{k_N}\bigl(t_N(\mu,\delta)\bigr)
+
o(1).
\label{eq:full_region_eta_profile}
\end{equation}
\end{theorem}
\medskip
Theorem~\ref{thm:full_region_uniform_gaussian_approx} provides a bridge between the exact Bernoulli problem and a continuous asymptotic analysis. Once the least favorable sequence is interior in the midpoint and local in the gap, the exact Bernoulli WMB problem reduces asymptotically to a Gaussian approximation problem, with the marginal ratio governed by the Gaussian limit function $f_{k_N}\bigl(t_N(\mu,\delta)\bigr)$.
This reduction allows us to characterize the asymptotic behavior of the Bernoulli WMB sample size $\hat m^{\mathrm{WMB}}$. The resulting normal-approximation objective admits a closed-form solution, as shown in the following lemma.
\medskip
\begin{lemma}[Rule of thirds under the Bernoulli normal approximation] \label{thm:bernoulli_normal_rule_of_thirds}
Let
\[
\tau := \mu_1-\mu_0,
\qquad
v:=\mu_1(1-\mu_1)+\mu_0(1-\mu_0),
\qquad
t:=\frac{\sqrt{m}\,|\tau|}{\sqrt{2v}}.
\]
Define the Bernoulli normal approximation to the marginal ratio by
\[
\eta^{\mathrm{NA}}(m,\mu_1,\mu_0)
=
2\Phi(-t)
+
\frac{N-m}{m}\,t\phi(t),
\]
and define the normal-approximation WMB sample size by
\[
\hat m^{\mathrm{NA}}
=
\min\left\{
m\in\{2,4,\dots,N-2\}:
\sup_{(\mu_1,\mu_0)\in(0,1)^2}
\eta^{\mathrm{NA}}(m,\mu_1,\mu_0)\le 1
\right\}.
\]
Then,
\[
\sup_{(\mu_1,\mu_0)\in(0,1)^2}
\eta^{\mathrm{NA}}(m,\mu_1,\mu_0)
\le 1
\quad\Longleftrightarrow\quad
m\ge \frac{N}{3}.
\]
Consequently,
\[
\hat m^{\mathrm{NA}}
=
\min\left\{m\in\{2,4,\dots,N-2\}: m\ge \frac{N}{3}\right\}.
\]
\end{lemma}
\medskip
\begin{comment}
\begin{remark}
We note that if $(\mu_1, \mu_0) = (\mu + \delta, \mu - \delta)$, then the treatment effect is $\tau = 2\delta$ and the total variance parameter becomes $v = (\mu+\delta)(1-\mu-\delta) + (\mu-\delta)(1-\mu+\delta) = 2(\mu(1-\mu) - \delta^2).$ Letting $q = \mu(1-\mu)$, we have $v = 2(q - \delta^2)$. Consequently, the standardized parameter $t$ simplifies to $t = \frac{\sqrt{m}\,|2\delta|}{\sqrt{4(q-\delta^2)}} = \frac{\sqrt{m}\,|\delta|}{\sqrt{q-\delta^2}},$ which explicitly coincides with the argument $t_N(\mu,\delta)$ driving the uniform Gaussian approximation in Theorem~\ref{thm:full_region_uniform_gaussian_approx}.
\end{remark}
\end{comment}
Together with Proposition~\ref{prop:bernoulli_localization_body} and Theorem~\ref{thm:full_region_uniform_gaussian_approx}, this theorem establishes the rule-of-thirds benchmark, $\hat m^{\mathrm{NA}}\approx N/3$, as the asymptotically optimal experimental sample size under the WMB criterion for Bernoulli outcomes. This conclusion applies once the boundary rare-event pathology described in Example~\ref{ex:boundary_pathology_half} has been ruled out.
The following theorem formalizes the rule-of-thirds benchmark as an asymptotic stopping threshold.
\medskip
\begin{theorem}[Rule of thirds as an asymptotic stopping threshold]
\label{cor:profile_rule_of_thirds_from_uniform_gaussian}
Fix $\kappa\in(0,1/2)$ and $K>0$. Let $\{m_N\}$ be an even sequence such that
$$
\frac{m_N}{N}\to \alpha\in(0,1)
\qquad (N\to\infty).
$$
Then the following statements hold:
\begin{enumerate}
\item If $\alpha<1/3$, then
$$
\liminf_{N\to\infty}
\sup_{(\mu,\delta)\in\mathcal A_N(\kappa,K)}
\eta(m_N,\mu+\delta,\mu-\delta)
>1.
$$
\item If $\alpha>1/3$, then
$$
\limsup_{N\to\infty}
\sup_{(\mu,\delta)\in\mathcal A_N(\kappa,K)}
\eta(m_N,\mu+\delta,\mu-\delta)
\le 1.
$$
\end{enumerate}
\end{theorem}
\medskip
\section{Gaussian Outcomes with a Common Variance}\label{sec:gaussian-known-variance}
This section examines the experimental design problem under Gaussian outcomes with a common variance. We show that the two main conclusions from the Bernoulli analysis continue to hold in a non-asymptotic Gaussian setting. Unlike in the Bernoulli case, the WMB criterion yields the rule of thirds without excluding the boundary pathology discussed in Section~\ref{sec:boundary_pathology}.
\subsection{Gaussian Setup and the limitations of absolute minimax regret}
We suppose that the potential outcomes follow Gaussian distributions:
\[
Y_i(1) \sim N(\mu_1,\sigma^2), \qquad
Y_i(0) \sim N(\mu_0,\sigma^2),
\]
where $(\mu_1,\mu_0)\in\mathbb{R}^2$ and the variance $\sigma^2>0$ is common across arms.
As in the Bernoulli setting, the superpopulation regret is
$$
\mathrm{SReg}(\delta,\mu_1,\mu_0) = \max\{\mu_1,\mu_0\} - \Bigl[ \mu_0\{1-\mathbb{E}\left[\delta(\omega)\right]\} + \mu_1\mathbb{E}\left[\delta(\omega)\right] \Bigr].
$$
The following proposition shows that the empirical success rule remains minimax optimal for the Gaussian superpopulation problem.
\medskip
\begin{proposition}[Gaussian superpopulation minimax rule]
\label{prop:gaussian-minimax-regret-treatment-rule}
Under the Gaussian model with a known common variance, the empirical success rule remains minimax optimal in the superpopulation problem:
$$
\hat\delta(\omega):=\mathbf{1}\{\hat\mu_1>\hat\mu_0\} \in \operatorname*{arg\,min}_{\delta\in\mathcal{D}}
\max_{(\mu_1,\mu_0)\in\mathbb R^2}
\mathrm{SReg}(\delta,\mu_1,\mu_0).
$$
\end{proposition}
\medskip
We next consider sample-size selection under the absolute minimax regret criterion. For a fixed policy $(m,\delta)$, the expected finite-population gain is
$$
g(m,\delta,\mu_1,\mu_0)
=
\frac{m}{2}(\mu_1+\mu_0)
+
(N-m)\Bigl[
\mu_0\{1-\mathbb{E}\left[\delta(\omega)\right]\}
+
\mu_1\mathbb{E}\left[\delta(\omega)\right]
\Bigr].
$$
The oracle benchmark is
$
g^{\mathrm{oracle}}=N\max\{\mu_1,\mu_0\},
$
and the corresponding regret is
$$
\mathrm{Reg}(m,\delta,\mu_1,\mu_0)
=
\frac{m}{2}|\tau|+(N-m)\mathrm{SReg}(\delta,\mu_1,\mu_0).
$$
Unlike in the Bernoulli model, the Gaussian parameter space is unbounded. Consequently, for every fixed $m$ and every rule $\delta$,
$$
\sup_{(\mu_1,\mu_0)\in\mathbb{R}^2}
\mathrm{Reg}(m,\delta,\mu_1,\mu_0)=\infty.
$$
Thus, without an arbitrary compact restriction on the parameter space, the absolute minimax regret criterion provides no meaningful guidance in the Gaussian model. This highlights the usefulness of the WMB criterion in the present setting.
\subsection{Exact Gaussian Analysis of the WMB Rule}
We now turn to sample-size selection under the WMB criterion. In the Gaussian model, the rollout error probability can be derived exactly in closed form.
\medskip
\begin{lemma}[Gaussian rollout error probability]
\label{lem:gaussian_error_probability}
Under the Gaussian model with a common variance,
$$
\hat\mu_1-\hat\mu_0
\sim
N\left(\tau,\frac{4\sigma^2}{m}\right).
$$
Consequently, under the empirical success rule,
$$
e(m,\tau)=\Phi\left(-\frac{\sqrt{m}\,|\tau|}{2\sigma}\right),
$$
where $e(m,\tau)$ denotes $e(m,\mu_1,\mu_0)$.
\end{lemma}
\medskip
To facilitate an analysis, we define continuous relaxations of the cost, risk, and marginal ratio introduced in the Bernoulli setting.
\medskip
\begin{definition}[Continuous-relaxation marginal ratio]
\label{def:continuous_marginal_ratio}
For $m \in (0, N]$, the continuous relaxations of the experimental cost and the rollout risk are defined respectively as
$$
C(m,\tau):=\frac{m}{2}|\tau|, \qquad R(m,\tau):=(N-m)|\tau|\,e(m,\tau).
$$
The continuous-relaxation marginal ratio is then defined as
$$
\eta(m,\tau)
:=
\frac{-\frac{\partial}{\partial m}R(m,\tau)}
{\frac{\partial}{\partial m}C(m,\tau)}.
$$
\end{definition}
\medskip
Using this continuous relaxation, the marginal ratio can be evaluated explicitly. The following theorem shows that, in the Gaussian model, the rule of thirds emerges as the exact WMB threshold without requiring any asymptotic approximation or restriction of the parameter space.
\medskip
\begin{theorem}[Exact Gaussian rule of thirds]
\label{thm:gaussian_safe_exact}
Let
$$
t:=\frac{\sqrt{m}\,|\tau|}{2\sigma}, \qquad k:=\frac{N-m}{m},
$$
and define
$$
f_k(t):=2\Phi(-t)+kt\phi(t), \qquad t\ge 0.
$$
Then,
$$
\eta(m,\tau)=f_k(t).
$$
Moreover:
\begin{enumerate}
\item If $m\ge N/3$, equivalently $k\le 2$, then $f_k$ is nonincreasing on $[0,\infty)$ and
$$
\sup_{t\ge 0}f_k(t)=f_k(0)=1.
$$
\item If $m<N/3$, equivalently $k>2$, then $f_k$ has a unique interior maximizer
$$
t_k^\star:=\sqrt{1-\frac{2}{k}}
$$
and satisfies
$$
\sup_{t\ge 0}f_k(t)>1.
$$
\end{enumerate}
Therefore,
$$
\sup_{\tau\in\mathbb{R}}\eta(m,\tau)\le 1
\quad\Longleftrightarrow\quad
m\ge \frac{N}{3}.
$$
Hence, the continuous-relaxation Gaussian WMB threshold is exactly
$$
m^{\mathrm{WMB}}=\frac{N}{3}.
$$
\end{theorem}
\medskip
Theorem~\ref{thm:gaussian_safe_exact} provides the exact Gaussian analogue of the Bernoulli rule-of-thirds benchmark. In contrast to the Bernoulli model, no local approximation is required: the function $f_k$ characterizes the marginal ratio exactly, and the WMB threshold is obtained directly from its worst-case behavior.
\section{Conclusion}\label{sec:conclusion}
This paper studies welfare-aware sample-size selection for test-and-roll experiments. In this setting, increasing the experimental sample size improves the accuracy of the rollout decision, but also exposes more experimental units to the inferior treatment ex post. The design problem therefore involves an exploration--exploitation tradeoff that is not captured by conventional power-based sample-size calculations.
We show that the standard absolute minimax regret criterion can recommend implausibly small experiments because its worst-case objective over-penalizes exploration at extreme parameter values. To address this limitation, we propose the Worst-case Marginal Benefit (WMB) rule, which compares the worst-case marginal benefit of adding one more matched pair with the corresponding marginal exploration cost.
Our main result is a simple rule-of-thirds benchmark. For Bernoulli outcomes, after excluding a boundary rare-event pathology, the WMB criterion yields $\hat m^{\mathrm{NA}} \approx N/3$ through a Gaussian approximation. For Gaussian outcomes with a common variance, the same benchmark arises exactly in the continuous-relaxation problem. These results provide a prior-free and practically implementable guide for welfare-based sample-size design.
\bibliographystyle{apalike}
\bibliography{refs}
\clearpage