The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
142,120 characters
When is the statistical evidence strong enough? Using hypothesis tests to value data collection
\maketitle
\vspace{-24pt}
\begin{abstract}
We recast statistical significance as a choice between making an immediate policy recommendation and deferring it until further evidence is collected. We show that the welfare-optimal decision corresponds, under minimax regret, to a statistical test whose level depends on the cost and precision of additional evidence. Inverting this rule, we introduce and recommend reporting the abstention-value (\(A\)-value) alongside traditional \(p\)-values to determine where additional data collection is most needed. The $A$-value defines the break-even welfare cost of abstaining and recommending further experimentation given the initial evidence. When experimentation capacity is limited, prioritizing additional data collection where \(A\)-values are the largest yields finite-sample welfare guarantees. We illustrate its implications for economic program evaluation.
\end{abstract}
\newpage
\onehalfspacing
\section{Introduction}
Statistical significance plays a prominent role in evidence-based policy. From the regulatory approval of new drugs \citep{guidance2017multiple} to the adoption of economic programs \citep{gertleretal2016}, researchers routinely compare \(p\)-values with conventional thresholds to determine whether evidence is ``strong enough'' to implement a new policy. Yet the economic rationale for this practice remains unclear and actively debated \citep[e.g.][]{wassersteinlazar2016,benjaminetal2018,amrheinetal2019,abadie_nonsignificance,imbens2021,frankel2022findings}. For instance, when choosing between two policies, a five-percent test can generally be rationalized only by assigning highly asymmetric preferences to Type I and Type II errors \citep[][]{tetenov2012,tetenov2016testing}, which is difficult to justify for policies with different welfare implications. At the same time, this binary decision-theoretic foundation overlooks the possibility that a ``failure to reject'' reflects insufficient evidence to support an immediate policy recommendation, rather than evidence in favor of the status quo \citep{fisher1935design}.
This paper re-interprets statistical significance through the lens of the cost incurred by declaring ``we do not know yet'', defined as the cost of deferring a policy recommendation and collecting further evidence.
Our main goal is to characterize the trade-off between making an immediate policy recommendation and deferring it, and to use this characterization to construct a welfare-based measure that complements traditional \(p\)-values to guide where additional evidence is most valuable and where an immediate policy action is most justified.
We formalize this trade-off by embedding the classical binary testing problem \citep{neymanpearson1933} in a two-stage sequential decision framework in the spirit of \citet{wald1947sequential}.
Specifically, we study a two-stage experiment (and multi-stage in Section \ref{sec:multiple_periods}) in which researchers collect a Gaussian, noisy and unbiased estimate of welfare effects. After observing the initial estimate, the researcher chooses among three actions: recommend implementing the policy, recommend retaining the status quo, or defer any policy choice. We assume that deferral implies that a decision-maker will make a terminal policy recommendation based on both the first unbiased signal and, in addition, on a second unbiased estimate collected in the second period, but unobserved to researchers in the first period. We remain agnostic over the values of treatment effects and evaluate each strategy by its worst-case regret over possible policy effects \citep{savage1951}, corresponding to the welfare difference with an oracle that knows the true effect and always makes the correct recommendation. Equivalently, regret corresponds to negative welfare effects when the policy recommendation differs from the correct policy recommendation, net of costs of additional data collection if recommended.
We characterize the optimal decision as a three-region structure familiar from the classical testing literature \citep[e.g.][]{wald1947sequential} where sufficiently positive (negative) initial evidence leads to implementation (rejection), and intermediate evidence leads to further data collection. Here the boundaries solely depend on two economic fundamentals: the cost per standard deviation measure and the ratio of standard errors between the two periods.
We use this characterization to measure the economic value of further experimentation. Given the initial evidence, we calculate the break-even cost at which the researcher switches between making an immediate policy decision to recommend collecting additional evidence (with equal precision). We call this threshold the abstention-value (\(A\)-value for short). A smaller \(A\)-value indicates more decisive current evidence; as we show, further experimentation is optimal if and only if it can be conducted at a lower cost than the $A$-value.
We establish two main properties of the \(A\)-value. First, holding experimental precision fixed, it increases with the \(p\)-value, i.e., weaker statistical evidence makes additional information more valuable. Second, holding the \(p\)-value fixed, the \(A\)-value is smaller when the experimental design is more precise. The intuition is that with higher precision, the policy effects that remain difficult to distinguish from zero are smaller in welfare units. The worst-case welfare of an incorrect immediate decision, and hence the value of collecting further evidence, are therefore smaller. Thus, unlike the \(p\)-value, the \(A\)-value reflects both the strength of the statistical evidence and the welfare implication of the policy decision.
Our next main result shows how \(A\)-values can guide experimentation when researchers face several policy questions but can suggest future experimentation for a limited number of studies. We propose assigning the available capacity to the policies with the largest \(A\)-values, corresponding to those for which further information is most valuable. Our ranking guarantees that the worst-case regret relative to the best feasible strategy is bounded by the expected \(A\)-value of the policy at the experimentation margin. Thus, when the marginal \(A\)-value is small, the ranking rule is approximately minimax-regret optimal.
With equally precise experiments, the rule prioritizes the largest \(p\)-values. When experimental precision differs across policies, \(A\)-values may produce a different ranking because they account for both the potential welfare loss from an incorrect immediate decision and the informativeness of the proposed follow-up experiment, two features that \(p\)-values do not capture.
We illustrate our findings by reanalyzing 42 evaluations of unconditional cash transfer programs in low- and middle-income countries \citep{crosta2024unconditional}, measuring welfare by their effects on household consumption. We find that the distribution of \(A\)-values is strongly right-skewed, with many evaluations leaving little value in deferring a policy recommendation, while a few indicate substantially greater value of further experimentation. When allocating ten follow-up experiments, the \(A\)-value rule shares seven or eight selections with a ranking by the largest \(p\)-values and five or six with a ranking by standard errors, depending on the cost specification. These disagreements illustrate the importance of accounting jointly for statistical inconclusiveness and the scale of uncertainty. For instance, selecting studies solely by the largest \(p\)-values excludes some program with smaller \(p\)-values but substantially larger standard errors. The uncertainty in these evaluations concerns effects on a larger welfare scale, so an incorrect immediate recommendation can entail greater losses.
One more comparison is with an empirical Bayes (EB) benchmark which leverages studies distribution to estimate a Gaussian prior and then posterior welfare effects. While useful in the presence of many studies drawn from the same distribution \citep[e.g.][]{abadie2023estimating}, the approach is not applicable in settings where studies across different countries are not exchangeable, and can be sensitive to the prior when a small number of studies make non-parametric approaches infeasible. In our empirical findings, the two rules share only three to five of the ten studies selected for additional experimentation. These differences arise because, for programs with imprecise estimates, empirical Bayes places most of the weight on the fitted prior. Countries with noisy but negative estimated effects exhibit positive precise posterior effects and confident policy recommendations due to more aggressive shrinkage; a second similarly-precise experiment is unlikely to change the EB policy decision.
The \(A\)-value instead reflects the welfare losses that remain possible when assessing each evaluation without imposing a common prior across programs. It therefore values experimentation especially in settings where both $p$-values and standard errors are the largest, since policy decisions face the largest uncertainty in terms of welfare effects. It is robust to settings where previous evidence is not comparable or available when making decisions.
\subsection{Related literature}
This paper builds on the classical literature on statistical decision theory and sequential analysis \citep{arrowblackwellgirshick1949,waldwolfowitz1950bayes,wald1950statistical}, which develop the foundations for our problem. While these papers mostly focus on trade-off between Type I and
II error control, here our focus is on welfare maximization under a continuous parameter space and its implications for the opportunity costs of immediate policy recommendations.
Specifically, we connect the practice of hypothesis testing to the classical literature on sequential experimentation \citep[e.g.][]{fisher1952sequential,robbins1952aspects,sobel1953,maurice1957,degroot1960,chernoff1961normal,johnsonmaurice1963}, the broader recent literature on best-arm identification \citep[e.g.][]{trapeznikov2013supervised, kasy2021adaptive, manskitetenov2016, garivier2016optimal,qin2017improving}, and to the generalized Wald problem recently studied by \citet{adusumilli2026wald}. While \citet{adusumilli2026wald} derives minimax-regret stopping rules in a continuous-time experiment, here we focus on a discrete two-stage problem with a single prespecified follow-up experiment; because continuation is a single batch decision, our
continuation value differs from
continuous-time solutions.\footnote{As \citet[Section 6.1]{adusumilli2026wald} note, simply discretizing continuous-time rules is not minimax optimal.} Also, different from sequential binary prediction problems \citep{trapeznikov2013supervised}, here we study a welfare-optimization problem.
Our main contribution relative to all these references is to characterize the exact threshold in our terminal two-batches problem, and, most importantly, to invert it and introduce and establish properties of the $A$-value.
Because the least-favorable distribution can change with the cost
of data collection, monotonicity of the minimax threshold in the
cost does not follow directly from its posterior characterization under a least-favorable prior.
We establish this monotonicity and continuity, to justify
the $A$-value as the break-even cost.
The \(A\)-value is conceptually distinct from other useful statistics in the testing literature. For instance, the \(q\)-value introduced by \citet{storey2002} measures the minimum positive false discovery rate associated with treating an observed statistic as a discovery. Unlike the \(A\)-value, it does not account for the cost and precision of future evidence.
An \(e\)-value is instead a nonnegative statistic whose expectation under the null is at most one. Sequences of \(e\)-values are often used to preserve Type I error control when the decision to collect additional observations depends on previously observed evidence \citep{vovkwang2021,ramdas2025hypothesis}. However these values do not determine whether the value of additional information justifies its cost. The $e$-value therefore differs in its scope and construction from the \(A\)-value whose goal is not to adaptively control Type I error, but rather to measure the welfare-implications of future data collection through its break-even cost.
The $A$-value is instead related to notions of economic value of experimentation often studied under risk criteria that average with respect to a chosen (prior) distribution. Classical contributions include \citet{grundyreeshealy1954,grundyhealyrees1956,raiffaschlaifer1961,batherwalker1962,scott1968,schleifer1969}, while recent Bayesian or empirical Bayes analyses include \citet{abadie2023estimating, hendren2020unified, henry2019research, fudenberg2018speed,liang2022dynamically,morris2019wald,azevedoetal2023,azevedo2020b}. Here, we evaluate experimentation with minimax regret, connecting the value of information to classical frequentist hypotheses testing. The minimax perspective is particularly useful when researchers want to characterize the evidence strength without relying on (or having access to) a prior distribution over studies. The \(A\)-value serves this purpose and can be interpreted as a frequentist analog to Bayesian values of information.
The study of statistical testing from a decision-theoretic perspective has a long tradition in statistical decision theory and econometrics.
The difficulty of motivating hypothesis testing in binary treatment-choice problems \citep{manski2004, stoye2009} has motivated the literature to (i) either take Type I error control as an exogenous part of researcher's preferences, and study approximately optimal decisions up-to the probability of making Type I error mistake \citep{manski2021econometrics,andrews2025certified}; (ii) or study hypothesis testing in communication models involving multiple (often misaligned) stakeholders \citep[e.g.][]{abadie_nonsignificance,tetenov2016testing,frankel2022findings,vivianoetal2026, jagadeesan2025publication, bates2026principal}; (iii) or focus on single-agent binary-decisions with highly asymmetric losses \citep[e.g.][]{das1994decision, kimchoi2021, tetenov2012}. All such references focus on a terminal (policy) decision. We complement this literature
by studying statistical testing as a choice between immediate decision and future data collection. This allows us to characterize the significance threshold endogenously as a function of the economic cost of future data. With prohibitively costly data collection, we recover \cite{manski2004}'s empirical success rule.
The three-action structure of our problem connects to the literature on classification with a rejection option, where ``rejection'' means declining to classify \citep{chow1970optimum,cortes2016learning,bartlettwegkamp2008,elyanivwiener2010,mozannarsontag2020}. A related strand of work gives statistical testing a three-decision interpretation \citep[e.g.][]{kaiser1960,ricekrakauer2023}.
In these contributions, the middle region represents a terminal conclusion that the available evidence does not support making a decision. In our framework, by contrast, deferring the policy decision implies conducting a follow-up experiment. This distinction requires a separate analysis. Instead of minimizing a loss function for a three point decision that only depends on immediate information, here our optimization explicitly accounts for the follow up decision to be contingent on the evidence accumulated in each period, making the cost of abstention a function of future decisions and of the precision of future data collection. These differences yield a distinct characterization of the optimal abstention threshold and motivate the \(A\)-value introduced in this paper.
Finally, recent work in econometrics uses abstention to address other decision problems. \citet{horowitzlee2025} study partial identification in a maximum-score model and develop a computationally tractable linear-programming procedure. \citet{brezaetal2025} use an option to admit ignorance when aggregating heterogenous causal evidence across environments. We share \citet{brezaetal2025}'s view that abstention should be interpreted under the lens of future data collection; we complement this literature with an explicit characterization of optimal abstention threshold with sequential decisions. This characterization is key to study its connection to hypothesis testing and the notion and properties of $A$-value introduced here.
\section{From statistical testing to economic decisions}
\subsection{Setup}
For a given policy question, a planner (or researcher) may reject or implement the given policy. If implemented, the policy generates a welfare effect $\Delta\in\left[-\Bar{\Delta},\Bar{\Delta}\right]$, ex-ante unknown to the planner. We standardize welfare at status quo to be equal to zero.
We assume that $\Bar{\Delta}$, the maximum possible magnitude of the welfare difference, is known (as in the case of bounded outcomes). If unknown, $\Bar{\Delta}$ can be arbitrary large without affecting our results.
\begin{ass}[Unbiased experiment] \label{ass:experiment} Researchers collect an (asymptotically) Gaussian unbiased estimate of the welfare effect with variance $s_1^2$
\begin{equation*}
X_1 \sim \mathcal{N}\left(\Delta,s_1^2\right),
\end{equation*}
where the variance $s_1^2$ is assumed to be known.
\end{ass}
Assumption \ref{ass:experiment} states that researcher conducts an experiment with an unbiased signal of welfare. Known $s_1^2$ can be replaced by a consistent estimate for $s_1^2$, where $s_1^2$ is implicitly a function of the sample in the first-stage experiment. In this case our analysis should be interpreted as asymptotic. While the standard Neyman-Pearson framework assumes a two-point decisions (either reject or accept a null hypothesis), here we study a sequential testing procedure in the spirit of \cite{wald1945sequential}, where researchers may recommend to accept, reject or \textit{collect more evidence}. The third option connects the idea of rejection option in \cite{chow1970optimum} to classical literature on sequential experimentation \citep[e.g.][]{maurice1957}, which we revisit here in the context of a three point decision with two sequential batch experiments.
\begin{defn}[Classification with rejection option, \cite{chow1970optimum}] Denote $\delta_1$ a decision as a function of evidence $X_1$ with
$$
\begin{aligned}
\delta_1(X_1) & = \begin{cases} 1 & \text{ if accept policy} \\
0 & \text{ if reject policy} \\
\texttt{NA} & \text{ if abstain}
\end{cases}
\end{aligned}
$$
\end{defn}
If abstention carried no cost, no immediate decision would be made. The core question is therefore how to model its consequences. Here, we give abstention an explicit sequential interpretation and find optimal decisions under such interpretation. Abstention implies that a decision-maker will make policy decision after conducting a follow-up experiment.
\begin{ass}[Abstention as a sequential sampling rule] \label{ass:2} Upon admitting ignorance $(\delta_1 = \texttt{NA})$, researchers recommend collecting more evidence. Additional evidence will be collected in the form of $X_2 \sim \mathcal{N}(\Delta, s_2^2), X_2 \perp X_1$ \text{ at cost } $c$.\footnote{Independence is without loss because with jointly Gaussian and correlated $X_1,X_2$, we can re-define the second experiment signal as $X_2'$ after residualizing $X_1$. Specifically, for $X_1,X_2$ with correlation $\rho < 1$ and for $\rho s_2/s_1 \neq 1$, we can define $a = \rho \frac{s_2}{s_1}$ and $X_2' = (X_2 - a X_1)/(1 - a)$, independent of $X_1$ and centered around $\Delta$. The same analysis then continues where we replace $X_2$ with $X_2'$.}
If $\delta_1 = \texttt{NA}$, the terminal decision takes the form:
$$
\delta_2(X_1, X_2) = \begin{cases} 1 & \text{ if accept policy} \\
0 & \text{ if reject policy}
\end{cases}
$$
where no further experimentation is possible.
\qed
\end{ass}
Assumption \ref{ass:2} gives abstention a welfare interpretation where abstention leads to additional evidence but entails the cost of collecting such evidence and delaying the policy decision. This structure will allow us to connect the optimal decision rule to standard practices of hypothesis testing. In the baseline analysis, we treat the design of the follow-up experiment, and hence its variance $s_2^2$, as given. This matches common practices of fixed budget constraints for replication studies. Section \ref{rem:endogenous_precision} discusses how the precision of the follow-up experiment can instead be chosen optimally ex-ante.
The cost $c$ can be an implicit function of $s_2^2$, omitted for notational convenience.
\begin{figure}[t]
\centering
\begin{tikzpicture}[
>=latex,
timeline/.style={thick},
event/.style={
rectangle, draw, rounded corners,
align=left, inner sep=4pt, font=\footnotesize,
text width=3.35cm
},
note/.style={
align=center, font=\scriptsize,
text width=3.35cm
}
]
\draw[timeline,->] (0,0) -- (10.7,0);
\foreach \x/\lab in {0/$t=0$,5/$t=1$,10/$t=2$} {
\draw[timeline] (\x,0.1) -- (\x,-0.1);
\node[below=0.22cm] at (\x,0) {\lab};
}
\node[event, above=0.55cm] (design) at (0,0) {
\textbf{Data collection}\\
Researcher observe $X_1 \sim \mathcal{N}(\Delta,s_1^2)$
};
\node[event, above=0.55cm] (report) at (5,0) {
\textbf{Decision}\\
Researcher recommends a policy or defers and asks for more evidence
};
\node[event, above=0.55cm] (audience) at (10,0) {
\textbf{Loss}\\
The cost of deferral is $c$ plus the expected loss from making decisions using additional evidence
};
\draw[->,timeline] (design.east) -- (report.west);
\draw[->,timeline] (report.east) -- (audience.west);
\end{tikzpicture}
\caption{\footnotesize Timeline of the decision process.}
\label{fig:communication_model}
\end{figure}
\subsection{Objective function}
Given the decisions $\delta_1, \delta_2$, the corresponding loss function depends on the welfare effect of adopting the policy $\Delta$ net of cost of future experimentation. This is defined as follows:
$$
\ell(\delta_1, \delta_2; c, \Delta) = - \Big(\underbrace{\Delta 1\{\delta_1 = 1\}}_{\text{Welfare effect today}} + \underbrace{\Delta1\{\delta_2 = 1\} 1\{\delta_1 = \texttt{NA}\}}_{\text{Welfare effect from future decision}} - \underbrace{c 1\{\delta_1 = \texttt{NA}\}}_{\text{Cost from future experimentation}} \Big).
$$
Because $\Delta$ is unknown, and
following \cite{savage1951}, we study optimal decisions from a minimax regret perspective. Specifically, we compare the loss function to the highest achievable welfare corresponding to $\Delta 1\{\Delta \ge 0\}$. Expected regret therefore takes the following form:
$$
\small
\begin{aligned}
\mathrm{Reg}_1(\delta_1, \delta_2; \Delta,c) & = \underbrace{\mathbb{E}_{\Delta}\Big[\ell(\delta_1(X_1), \delta_2(X_1, X_2);\Delta,c) \Big]}_{\text{Expected researcher's loss}} + \underbrace{\Delta 1\{\Delta \ge 0\}}_{\text{highest achievable welfare}} \\
&= \underbrace{\Delta_+ \mathbb{E}_\Delta\Big[1\{\delta_1(X_1) = 0\} + 1\{\delta_2(X_1,X_2) = 0\} 1\{\delta_1(X_1) = \texttt{NA}\}\Big]}_{\text{loss from wrongly recommending status quo}} \\ & \qquad +\underbrace{\Delta_- \mathbb{E}_\Delta \Big[ 1\{\delta_1(X_1) = 1\} + 1\{\delta_2(X_1,X_2) = 1\} 1\{\delta_1(X_1) = \texttt{NA}\}\Big]}_{\text{loss from wrongly recommend new policy}} + \underbrace{c \mathbb{E}_\Delta[1\{\delta_1(X_1) = \texttt{NA}\}]}_{\text{abstention cost}}
\end{aligned}
$$
where $\Delta_+ = \max\{0,\Delta\}, \Delta_- = \max\{-\Delta,0\}$.
That is, under the regret objective function, a correct policy decision incurs no loss while an incorrect decision incurs a loss equal to the magnitude of the treatment effect; $c$ measures the cost (in welfare units) of additional data.
We define the minimax regret optimal decisions over all measurable functions $\mathcal{D}$ as
$$
(\delta_1^\star,\delta_2^\star) \in \mathrm{arg} \min_{(\delta_1,\delta_2) \in \mathcal{D}} \sup_{\Delta: |\Delta| \le \bar{\Delta}} \mathrm{Reg}_1(\delta_1,\delta_2;\Delta,c).
$$
\begin{rem}[Comparison with minimax solution]
Minimax regret is a common criterion for evaluating statistical decision rules;
see, for example, \cite{manski2021econometrics} for a review. Other objectives
are possible. A seemingly natural alternative is to minimize the worst-case
expected loss $\ell$, instead of the loss corresponding to regret. In our setting, however, this criterion admits a trivial
solution. In particular, nature can make the welfare for rules that approve the policy arbitrary small, making the rule that never approves minimax optimal. Minimax
regret avoids this asymmetry between new policy and status quo and assigns a cost both to approving a harmful policy and to rejecting a
beneficial one. \qed
\end{rem}
\begin{rem}[Comparison with Bayes and Empirical Bayes]
Another alternative is a Bayesian objective that averages $\ell$ with respect
to a prior $\Delta\sim\pi$. This criterion is natural when the researcher can
elicit a credible distribution of policy effects. It may be less attractive when such
information is unavailable or when the decision is sensitive to the chosen
prior.\footnote{Even when multiple studies evaluate similar interventions across different contexts, using their distribution of effects to construct a prior for a new site requires an exchangeability assumption. This assumption may fail when sites selected for future implementation are not drawn from the same distribution of observable sites (e.g., because of selection).}
An empirical Bayes analysis would use a collection of comparable studies to estimate a distribution of policy effects under similar exchangeability assumptions (potentially controlling for observable characteristics), requiring that this evidence be available at the time of the policy or deferral decision. Our approach complements empirical Bayes by accommodating decisions made before a sufficiently large body of comparable evidence has accumulated. \qed
\end{rem}
\subsection{Optimal decisions}
We can now state our first theorem.
\begin{theorem}[Confidence interval representation] \label{thm:dead_zone}
Let Assumptions \ref{ass:experiment}, \ref{ass:2} hold with $s_2,s_1 > 0$.
There exists minimax-regret optimal decisions $(\delta_1^\star,\delta_2^\star)$, and a threshold $\tau^\star \ge 0$ so that
$$
\begin{aligned}
\delta_{1}^\star(X_1) = \begin{cases}
1 & \text{ if } \frac{X_1}{s_1} \ge \tau^\star \\
0 & \text{ if } \frac{X_1}{s_1} \le -\tau^\star \\
\texttt{NA} & \text{ if } \frac{X_1}{s_1} \in (-\tau^\star,\tau^\star)
\end{cases}
\end{aligned}, \qquad \delta_2^\star(X_1,X_2) = 1\Big\{ \frac{s_2^2}{s_1^2} X_1 + X_2 \ge 0\Big\}
$$
where for $\Phi$ and $\phi$ denoting the Gaussian CDF and PDF respectively, and $w = \frac{s_2}{s_1}$
$$
\small
\begin{aligned}
\tau^\star \in \mathrm{arg} \min_{\tau: \tau \ge 0} \max_{d: d \in [0, \bar{\Delta}/s_1]}&
\Big\{d\underbrace{\Phi\!\left(-\tau-d\right)}_{
\substack{\text{probability of an incorrect}\\
\text{first-period decision}}}
+d\underbrace{\int_{-\tau}^{\tau}
\Phi\!\left(-wx-d/w\right)
\phi\!\left(x-d\right)\,dx}_{
\substack{\text{probability of continuation and}\\
\text{an incorrect second-period decision}}}
\\ & \quad
+\frac{c}{s_1} \underbrace{\left[
\Phi\!\left(\tau-d\right)
-\Phi\!\left(-\tau-d\right)
\right]}_{\text{probability of continuation}}\Big\}.
\end{aligned}
$$
\end{theorem}
\begin{proof}
See Appendix \ref{proof:dead_zone}.
\end{proof}
Theorem \ref{thm:dead_zone} shows that in the original pilot study, the optimal decision takes the form of a standard interval-based decision. For sufficiently large positive (resp. negative) estimated effects, the policy is accepted (resp. rejected), while sufficiently small estimated effects lead to deferral. The confidence intervals for which a policy recommendation is deferred depends on how costly is a follow-up experiment. A particularly costly experiment, as for large-scale experimentations discussed in \cite{muralidharan2017experimentation}, policy recommendation may demand more lenient (i.e., smaller) thresholds $\tau^\star$ because of the cost of additional data collection. Easily replicable experiments may require more stringent (i.e., larger) thresholds.
The following proposition identifies key properties of $\tau^\star$.
\begin{prop}[Properties of $\tau^\star$] \label{cor:1} Let the conditions in Theorem \ref{thm:dead_zone} hold and let $\bar{\Delta}/s_1 \ge 0.76$. Then the following hold:
\begin{itemize}
\item[(A)]
$\tau^\star$ is constant in $\bar{\Delta}$ and is only a function of $(s_2/s_1,c/s_1)$. Denote this as $\tau^\star(w,\gamma)$, with $w = s_2/s_1, \gamma = c/s_1$.
\item[(B)] For every fixed $w>0$, $\tau^\star(w,\gamma)$
is unique, continuous and nonincreasing in $\gamma\geq0$.
Moreover, $\tau^\star(w,0)=\infty, \lim_{\gamma\downarrow0}\tau^\star(w,\gamma)=\infty$, and
$\tau^\star(w,\gamma)<\infty$ for every $\gamma>0$.
\end{itemize}
\end{prop}
\begin{proof} See Appendix \ref{proof:cor:1}.
\end{proof}
The critical threshold is independent of the bound $\bar{\Delta}$ for sufficiently precise initial experiment (small $s_1$).\footnote{Intuitively, regret is small when $|\Delta|$ is close to zero because the welfare consequences of an incorrect decision are small. Regret is also small when $|\Delta|$ is large relative to $s_1$ because the sign of the effect is easier to determine. The least-favorable value of $|\Delta|$ therefore has an intermediate magnitude. Once the parameter space contains this value (which occurs when $\bar{\Delta}/s_1\geq 0.76$) further increases in $\bar{\Delta}$ do not affect the worst-case regret or the optimal threshold $\tau^\star$.
} In particular, $\tau^\star$ is only a function of the variance of the follow up study relative to the next experiment $s_2^2/s_1^2$, and of the cost of deferral per standard deviation unit $c/s_1$. We will therefore write throughout $\tau^\star(w,\gamma)$ where $w = \frac{s_2}{s_1},\gamma = \frac{c}{s_1}$ as a function of these two quantities, assuming that the conditions in Corollary \ref{cor:1} hold.
The threshold also exhibits a desiderable behavior: it is monotonic in the welfare cost of future data collection, for given precision of the first and follow up study.
\begin{table}[!ht]
\centering
\caption{Optimal thresholds and normalized costs for unconstrained $\Delta$.}
\label{tab:optimal_thresholds}
\begin{tabular}{ccc}
\toprule
\multicolumn{3}{l}{
\textit{Panel A: Optimal threshold $\tau^\star$ for selected normalized costs}
} \\
\addlinespace[2pt]
$\displaystyle \frac{c}{s_1}$
&
\begin{tabular}{c}
Perfect follow-up \\[-2pt]
$s_2=0$
\end{tabular}
&
\begin{tabular}{c}
Equal precision \\[-2pt]
$s_2=s_1$
\end{tabular}
\\
\midrule
$0$ & $\infty$ & $\infty$ \\
$0.025$ & $5.564$ & $0.880$ \\
$0.050$ & $2.757$ & $0.564$ \\
$0.100$ & $1.295$ & $0.265$ \\
$0.150$ & $0.761$ & $0.109$ \\
$0.200$ & $0.469$ & $0.010$ \\
$0.250$ & $0.280$ & $0$ \\
$0.300$ & $0.145$ & $0$ \\
$0.400$ & $0$ & $0$ \\
\midrule
\multicolumn{3}{l}{
\textit{Panel B: Normalized cost corresponding to $\tau^\star=1.96$}
} \\
\addlinespace[2pt]
Follow-up design
&
Target threshold
&
$\displaystyle \frac{c}{s_1}$
\\
\midrule
Perfect follow-up, $s_2=0$
&
$1.96$
&
$0.06918$
\\
Equal precision, $s_2=s_1$
&
$1.96$
&
$0.0015$
\\
\bottomrule
\end{tabular}
\begin{minipage}{0.92\textwidth}
\vspace{0.15cm}
\footnotesize
\textit{Notes:} Panel A reports the minimax-optimal
threshold $\tau^\star$ for as a function of $c/s_1$. Panel B reports the cost that generates
$\tau^\star=1.96$, corresponding to a two-sided nominal significance
level of $5\%$.
\end{minipage}
\end{table}
\subsection{$A$-value: an economic notion of significance value}
To gain insights on the behavior of the threshold $\tau^\star$, Table \ref{tab:optimal_thresholds} reports
the value of $\tau^\star$ as a function of the cost per standard deviation units $\gamma = c/s_1$.
We consider two regimes. The first regime sets $w = s_2/s_1$ equal to one, so that the follow up experiment has the same precision as the first study. The second regime sets $s_2/s_1 = 0$, so that the follow up experiment is perfectly precise. In practice, we may expect to be within these two scenarios.
As $c/s_1 \rightarrow 0$, the threshold $\tau^\star \rightarrow \infty$, so that whenever the cost of abstention is small, critical values become more stringent. Similar behavior for $\tau^\star$ occurs as $w$ decreases, so that follow-up data collection is more precise. To motivate the standard critical value $1.96$ instead, we need either a perfect follow up study and $c= 7\% s_1$ or with an equally precise follow up study (arguably the most credible case), we require a much smaller cost $c = 0.15\% s_1$.
While the exact cost $c$ may be difficult to elicit in practice, our recommendation is that researchers report the $A$-value alongside conventional $p$-values. The $A$-value measures the break-even welfare cost of additional data collection at which the planner is indifferent between collecting further evidence and making an immediate
policy decision.
\begin{defn}[$A$-value] Under the conditions in Corollary \ref{cor:1}, denote the $A$-value the smallest welfare cost of follow up data collection so that results are ``deemed significant''. Specifically, denote
$$
A_{\mathrm{value}}(X_1;s_1, w) = s_1 \inf\{\gamma: |X_1|/s_1 \ge \tau^\star(w,\gamma)\}, \qquad w = \frac{s_2}{s_1}
$$
with $\tau^\star$ in Theorem \ref{thm:dead_zone}.
Denote the benchmark as $A_{\mathrm{value}}^\star(X_1;s_1) = A_{\mathrm{value}}(X_1; s_1, w = 1)$.
\end{defn}
The $A$-value relates statistical significance to the immediate accept/reject policy decision based on welfare considerations. In words:
\begin{center}
\textit{A smaller $A$-value indicates more decisive current evidence, as it is optimal to defer the policy decision only when additional evidence can be collected at a sufficiently low cost.}
\end{center}
We establish its properties below.
\begin{prop}[Properties of the $A$-value]
\label{prop:c_value_properties}
Denote $p(|X_1|/s_1) = 2\left\{1 - \Phi\Big(\frac{|X_1|}{s_1}\Big)\right\}$ the two-sided $p$-value and $ \Gamma_w(t) = \inf\{\gamma \ge 0: t\ge \tau^\star(w,\gamma)\}$.
The following properties hold.
\begin{enumerate}
\item
The $A$-value depends on the initial estimate only through its
two-sided $p$-value:
\[
A_{\mathrm{value}}(X_1;s_1, w)
=
s_1\Gamma_w\left(
\Phi^{-1}\left(1-\frac{p(|X_1|/s_1)}{2}\right)
\right).
\]
For fixed $(s_1,w)$, the $A$-value is weakly decreasing in
$|X_1/s_1|$ and weakly increasing in $p(|X_1|/s_1)$.
\item For fixed $p$-value $p(|X_1|/s_1)$ and $w$, the $A$-value is increasing in the variance $s_1^2$.
\item
The $A$-value is scale equivariant. For every $a>0$,
$
A_{\mathrm{value}}(aX_1;as_1, w)
=
aA_{\mathrm{value}}(X_1;s_1, w).
$
\item $\delta_1^\star(X_1) = \texttt{NA}$ as defined in Theorem \ref{thm:dead_zone} if and only if $c < A_{\mathrm{value}}(X_1;s_1, w)$.
\end{enumerate}
\end{prop}
\begin{proof} See Appendix \ref{proof:prop:c_value_properties}.
\end{proof}
Proposition \ref{prop:c_value_properties} establishes four main properties of the $A$-value. First, after normalization by $s_1$, the $A$-value depends on the initial estimate $X_1$ only through its $p$-value and on the design of the follow-up experiment through $w=s_2/s_1$. As a benchmark, we recommend setting $w=1$, which corresponds to a follow-up estimate with the same precision as the initial estimate. For fixed $(s_1,w)$, the $A$-value is increasing in the $p$-value. Thus, as with the $p$-value, larger values indicate weaker initial evidence and a greater value of collecting additional information.
Second, unlike the $p$-value, the $A$-value also depends on the uncertainty through $s_1$ separately. Holding the $p$-value and the relative precision $s_2/s_1$ of the follow-up experiment fixed, the $A$-value is proportional to $s_1$. More precise initial experiments therefore lead to smaller $A$-values (and therefore more conclusive evidence). Intuitively, at a fixed test statistic, a smaller $s_1$ implies both a smaller estimated effect and less posterior uncertainty. The welfare consequences of an incorrect immediate decision, and hence the value of collecting additional evidence, are therefore smaller.
Third, the $A$-value changes proportionally with the units in which $X_1$ and $s_1$ are measured. Comparisons across studies therefore require estimated effects to be expressed in common welfare units, as we discuss in our application.
Fourth, any maximin optimal rule in Theorem \ref{thm:dead_zone} chooses to abstain if and only if the welfare cost of abstention is below the $A$-value. This implies that more stringent $A$-values require smaller cost of deferral to justify abstention.
Table \ref{tab:c_value_by_s1} tabulates the $A$-value as a function of the $p$-value and the standard error of the initial estimate. We set $w=1$ and report the resulting $A$-values relative to the $A$-value evaluated at the $5\%$ $p$-value with standard error $s_1 = 1$. The table illustrates the role of precision. Holding the $p$-value fixed, the $A$-value increases proportionally with $s_1$. An imprecise estimate leaves greater uncertainty and therefore makes additional evidence more valuable. Similarly, holding $s_1$ fixed, the $A$-value increases with the $p$-value.
\begin{table}[t]
\centering
\caption{Normalized $A$-values for different initial standard errors when $s_2/s_1=1$.}
\label{tab:c_value_by_s1}
\begin{tabular}{lccccc}
\toprule
& \multicolumn{5}{c}{$A_{\mathrm{value}}/A_{\mathrm{ref}}$} \\
\cmidrule(lr){2-6}
$p$-value
& $s_1=0.05$
& $s_1=0.1$
& $s_1=0.2$
& $s_1=0.5$
& $s_1=1$ \\
\midrule
0.05 & 0.050 & 0.100 & 0.200 & 0.500 & $\mathbf{1.000}$ \\
0.10 & 0.126 & 0.252 & 0.504 & 1.260 & 2.520 \\
0.15 & 0.219 & 0.438 & 0.876 & 2.190 & 4.380 \\
0.20 & 0.327 & 0.654 & 1.308 & 3.270 & 6.540 \\
0.25 & 0.449 & 0.898 & 1.796 & 4.490 & 8.980 \\
\bottomrule
\end{tabular}
\begin{minipage}{0.92\textwidth}
\vspace{0.18cm}
\centering
\small
\textbf{Reference value:}\quad
$
A_{\mathrm{ref}}$ indicates the $A$-value at $p = 0.05$ and $s_1 = 1$ and $C_\mathrm{ref} = 0.0015$
\vspace{0.3cm}
\raggedright
\footnotesize
\textit{Notes:} The table reports the $A$-value relative to
$A_{\mathrm{ref}}$ for an equal-precision follow-up design,
$s_2/s_1=1$.
\end{minipage}
\end{table}
\subsection{Using $A$-values for policy rankings} \label{sec:multiple_comparisons}
In practice, the \(A\)-value is particularly useful when researchers face several policy questions and must decide which follow-up experiments to conduct subject to a common resource constraint. This problem is related in spirit to multiple testing, but differs in its objective since rather than controlling erroneous policy recommendations, it allocates scarce capacity for future data collection across policies.
Specifically, researchers face \(J\) independent policy questions but can collect follow-up evidence for exactly \(q\) of them.
\begin{ass}[Multiple policy comparisons] \label{ass:multiple} Consider $J$ policies. For each policy \(j\), suppose that
$$
X_{1j} \sim\mathcal N(\Delta_j,s_{1j}^2),
\qquad
X_{2j} \sim\mathcal N(\Delta_j,s_{2j}^2),
$$
where the $X_{tj}$ are mutually independent across policies and periods. Let
$
\boldsymbol\Delta
=
(\Delta_1,\ldots,\Delta_J)
\in
\prod_{j=1}^J[-\bar\Delta_j,\bar\Delta_j],
$
and write
$
A_j
=
A_{\mathrm{value}}(X_{1j};s_{1j},\frac{s_{2j}}{s_{1j}}).
$
\end{ass}
Independence is natural when the policy-specific signals are constructed from separate samples; this is common when studying interventions in different experimental sites or populations. We next define the class of feasible strategies under the capacity constraint.
\begin{defn}[Strategy with $q$-experimentation constraints] A strategy
\(\boldsymbol\delta=(\boldsymbol\delta_1,\boldsymbol\delta_2)\)
consists of a measurable first-period rule
$
\boldsymbol\delta_1:
\mathbb R^J
\rightarrow
\{0,1,\texttt{NA}\}^J,
$ where \(\delta_{1j}(\mathbf X_1)=1\) denotes immediate implementation,
\(\delta_{1j}(\mathbf X_1)=0\) denotes immediate rejection, and
\(\delta_{1j}(\mathbf X_1)=\texttt{NA}\) denotes collecting follow-up evidence for policy \(j\). Feasibility requires
$
\sum_{j=1}^J
1\{\delta_{1j}(\mathbf X_1)=\texttt{NA}\}
=
q
$
for a given capacity constraint $q\leq J$, almost surely.
After the first-period decisions, researchers observe \(X_{2j}\) only for policies satisfying
\(\delta_{1j}(\mathbf X_1)=\texttt{NA}\). For each such policy,
\(\delta_{2j}\in\{0,1\}\) denotes the terminal implementation decision and may depend on \(\mathbf X_1\) and all the follow-up evidence collected under the first-period rule. Let \(\mathcal D_q\) denote the class of all measurable strategies satisfying these constraints. \qed
\end{defn}
We evaluate a strategy \(\boldsymbol\delta\in\mathcal D_q\) using regret. To simplify notation, write
\(\delta_{1j}=\delta_{1j}(\mathbf X_1)\), with \(\delta_{2j}\) denoting the corresponding terminal decision whenever \(\delta_{1j}=\texttt{NA}\). Define
$$
\small
\begin{aligned}
\mathrm{Reg}(\boldsymbol\delta;\boldsymbol\Delta)
=
\mathbb E_{\boldsymbol\Delta}
\Bigg[
\frac{1}{J} \sum_{j=1}^J
\Big\{
&
1\{\delta_{1j}=0\}\Delta_{j,+}
+
1\{\delta_{1j}=1\}\Delta_{j,-}
+
1\{\delta_{1j}=\texttt{NA}\}
\big[
(1-\delta_{2j})\Delta_{j,+}
+
\delta_{2j}\Delta_{j,-}
\big]
\Big\}
\Bigg]
\end{aligned}
$$
where
$
\Delta_{j,+}=\max\{\Delta_j,0\},
\Delta_{j,-}=\max\{-\Delta_j,0\},
$
and $\boldsymbol{\delta} \in \mathcal{D}_q$ so that we re-interpret the cost of experimentation as a constraint on the number of policies for which researchers may recommend collecting more evidence. See Remark \ref{rem:cost} for further discussion.
An intuitive approach is to rank policy decisions based on the $A$-values.
\begin{defn}[Top-\(q\) \(A\)-values procedure]
\label{def:top_q_c_value}
After observing \(\mathbf X_1\), let
\(\mathcal J_q(\mathbf X_1)\) contain the indices of the \(q\) largest \(A\)-values (with arbitrary tie-breaking rule). The top-\(q\) \(A\)-values procedure
\(\boldsymbol\delta^{A,q}\in\mathcal D_q\) is defined by
$$
\delta_{1j}^{A,q}(\mathbf X_1)
=
\begin{cases}
\texttt{NA},
& j\in\mathcal J_q(\mathbf X_1),\\
1,
& j\notin\mathcal J_q(\mathbf X_1)
\text{ and }X_{1j}\geq0,\\
0,
& j\notin\mathcal J_q(\mathbf X_1)
\text{ and }X_{1j}<0,
\end{cases}
$$
and, for every \(j\in\mathcal J_q(\mathbf X_1)\),
$
\delta_{2j}^{A,q}(X_{1j},X_{2j})
=
1\left\{
\frac{s_{2j}^2}{s_{1j}^2}X_{1j}
+
X_{2j}
\geq0
\right\}.
$
Thus, the procedure collects follow-up evidence for the \(q\) policies with the largest \(A\)-values, makes an immediate decision based on the sign of \(X_{1j}\) for every other policy, and uses the combined first- and second-period evidence for the selected policies. \qed
\end{defn}
Our goal is to study how the top-$q$ $A$-value solution compared to the regret of the uncostrained solution defined as
$$
R_J^\star = \min_{\boldsymbol{\delta} \in \mathcal{D}_J} \sup_{\boldsymbol{\Delta} \in \prod_{j=1}^J[-\bar\Delta_j,\bar\Delta_j]} \mathrm{Reg}(\boldsymbol\delta;\boldsymbol\Delta).
$$
Here the minimization is over all measurable functions $\boldsymbol{\delta}$ where the capacity constraint is not binding ($q = J$), therefore allowing for each policy to elicit additional data at \textit{no cost}.
\begin{theorem}[Regret bound for the top-\(q\) \(A\)-values procedure]
\label{prop:top_q_c_value}
Suppose that Assumption \ref{ass:multiple} holds and $s_{1j},s_{2j} > 0, \bar{\Delta}_j/s_{1j} > 0.76$. For given realization of \(\mathbf X_1\), let
$
A_{(1)}
\geq
A_{(2)}
\geq
\cdots
\geq
A_{(J)}$
denote the ordered \(A\)-values. Then, for every
\(q\in\{1,\ldots,J-1\}\), $\boldsymbol\Delta\in
\prod_{j=1}^J[-\bar\Delta,\bar\Delta_j]$
$$
\begin{aligned}
\mathrm{Reg}
\big(
\boldsymbol\delta^{A,q};
\boldsymbol\Delta
\big)
-
R_J^\star
\leq \frac{J - q}{J}
\mathbb E_{\boldsymbol{\Delta}}
\left[
A_{(q)}
\right].
\end{aligned}
$$
In addition $\sup_{\boldsymbol\Delta\in
\prod_{j=1}^J[-\bar\Delta,\bar\Delta_j]} E_{\boldsymbol{\Delta}}
\left[
A_{(q)}
\right] \le E_{\boldsymbol{\Delta} = \mathbf{0}}
\left[
A_{(q)}
\right]$.
\end{theorem}
\begin{proof}
See Appendix \ref{proof:prop:top_q_c_value}.
\end{proof}
Theorem \ref{prop:top_q_c_value} provides an approximation guarantee for the top-\(q\) \(A\)-values procedure. The theorem characterizes the difference between its worst-case aggregate regret and the lowest worst-case regret attainable by any strategy that can elicit more evidence with no constraint/additional cost. We can therefore interpret $R_J^\star$ as the worst-case irreducible regret.
The bound depends only on the expected \(A\)-values at the experimentation margin. In particular, \(A_{(q)}\) is the smallest \(A\)-value among the policies receiving follow-up evidence. Thus, when these marginal \(A\)-values are small, ranking policies by their \(A\)-values achieves approximately optimal worst-case regret relative to all feasible strategies. This result further highlights the interpretation of the \(A\)-value as a measure of the value of additional evidence: policies with larger \(A\)-values are those for which an immediate decision is potentially most costly and therefore receive priority when experimentation capacity is scarce.
While the bound holds for every treatment-effect vector \(\boldsymbol\Delta\), because the worst-case bound $\mathbb E_{\boldsymbol{\Delta} = \boldsymbol 0}
\left[
A_{(q)}
\right]$ can be computed as a function of the precision of each study in analytic form, researchers may ex-ante commit to a given $q$ to guarantee control of the excess risk.
\begin{rem}[\(A\)-values and \(p\)-value rankings]
Suppose that all policies have the same initial and follow-up precisions, so that \(s_{1j}=s_1\) and \(s_{2j}=s_2\) for every \(j\). The \(A\)-value is then increasing in the two-sided \(p\)-value. Consequently, the top-\(q\) \(A\)-values procedure collects additional evidence for the \(q\) policies with the largest \(p\)-values. Experimentation therefore follows the reverse of the conventional multiple-testing ranking where policies with the least decisive initial evidence receive priority for additional data collection. Whenever $s_j$ differs across policies, the ranking under the $A$-value may differ from the ranking under the $p$-value as the $A$-value accounts in addition for the precision of the estimates. \qed
\end{rem}
\begin{rem}[Budget constraint and common cost of experimentation] \label{rem:cost} We can interpret $\mathrm{Reg}(\boldsymbol\delta;\boldsymbol\Delta)$ under the $q$ budget constraints under a dual representation of worst-case regret with experimentation costs, where we assume that the \textit{worst-case} cost of each policy decision is the same. Because the cost of deferring policy decision $j$ depends on the precision $s_{2j}$ of the follow-up experiment, we can interpret this dual representation as restricting attention to follow-up designs where the follow-up precision $s_{2j}$ may differ across policies and is calibrated so that the corresponding maximum welfare cost is common across studies.\footnote{If the analyst wants to include heterogeneous costs directly in the objective function, it is possible to do so by ranking $A$-values minus $c_j$, with $c_j$ denoting the per-study follow up welfare cost. We leave this extension to future research.}
\qed
\end{rem}
\section{Extensions}
\subsection{Multiple time periods}
\label{sec:multiple_periods}
We now allow researchers to collect evidence over multiple periods. The setup extends Assumption \ref{ass:2}, with researchers choosing in each period whether to recommend or reject the policy or to defer the decision and collect additional evidence.
\begin{ass}[Multiple rounds of data collection]
\label{ass:multiple_periods}
In each period $h=1,\ldots,H$, researchers observe
$
X_h\mid\Delta
\sim
\mathcal{N}(\Delta,s_h^2),
X_1,\ldots,X_H
\text{ mutually independent}.
$
After observing $(X_1,\ldots,X_h)$, researchers make a decision
$$
\delta_h(X_1,\ldots,X_h)
\in
\{0,1,\texttt{NA}\},
\qquad h<H,
$$
where $\delta_h=1$ corresponds to recommending the policy,
$\delta_h=0$ corresponds to rejecting the policy, and
$\delta_h=\texttt{NA}$ corresponds to collecting additional evidence.
If $\delta_h=\texttt{NA}$, researchers collect $X_{h+1}$ at welfare cost
$c_h\geq0$. In the terminal period, no additional evidence can be collected and $
\delta_H(X_1,\ldots,X_H)\in\{0,1\}.
$
\qed
\end{ass}
Assumption \ref{ass:multiple_periods} allows researchers to defer the policy recommendation several times. The standard errors $s_h$ and the costs $c_h$ may vary across periods. For example, researchers may first conduct an inexpensive pilot study, subsequently collect evidence in a larger experiment, and finally conduct an expensive confirmatory experiment. Denote by
$$
\mathsf{h}(\boldsymbol{\delta})
=
\inf\left\{
h\leq H:
\delta_h(X_1,\ldots,X_h)\in\{0,1\}
\right\}
$$
the period in which the terminal policy decision is made, where
$\boldsymbol{\delta}=(\delta_1,\ldots,\delta_H)$. The realized loss equals
$
\ell_H(\boldsymbol{\delta};\boldsymbol{c},\Delta)
=
-\left(
\Delta
1\left\{
\delta_{\mathsf{h}(\boldsymbol{\delta})}=1
\right\}
-
\sum_{r<\mathsf{h}(\boldsymbol{\delta})}c_r
\right)
$
where $\boldsymbol{c}=(c_1,\ldots,c_{H-1})$. Thus, if researchers stop in period $h$, they implement or reject the policy using all the evidence collected up to that period and incur the cumulative experimentation cost
$
\sum_{r<h}c_r.
$
As in the two-period problem, we compare the researcher's loss to the highest achievable welfare under knowledge of $\Delta$. Expected regret is
$$
\mathrm{Reg}_H(
\boldsymbol{\delta};
\Delta,\boldsymbol{c}
)
=
\mathbb{E}_\Delta\left[
\ell_H(
\boldsymbol{\delta};
\boldsymbol{c},\Delta
)
\right]
+
\Delta 1\{\Delta\geq0\}.
$$
Researchers choose a complete sequence of contingent decisions to solve
$$
\boldsymbol{\delta}^\star
\in
\operatorname*{arg\,min}_{\boldsymbol{\delta}}
\sup_{\Delta:|\Delta|\leq\bar{\Delta}}
\mathrm{Reg}_H(
\boldsymbol{\delta};
\Delta,\boldsymbol{c}
),
$$
over the set of measurable decision sequences.
The multiple-period problem introduces two features relative to the two-period problem. First, the value of collecting evidence in period $h+1$ depends on all the subsequent experimentation opportunities. Second, reaching a later period is endogenous because it depends on the evidence collected and the continuation decisions made in the previous periods. Minimax-regret therefore evaluates the costs and policy consequences over the entire path.
Despite this path dependence, the Gaussian structure reduces the relevant history to a one-dimensional statistic. Define cumulative precision and precision-weighted evidence as
$$
S_h^2
=
\sum_{r=1}^h\frac{1}{s_r^2},
\qquad
T_h
=
\frac{\sum_{r=1}^h\frac{X_r}{s_r^2}}{S_h}.
$$
Note that $\mathbb{V}(T_h) = 1$ so that the statistic $T_h$ is standardized.
We can now state the multiple-period extension of Theorem \ref{thm:dead_zone}.
\begin{theorem}[Multiple-period dead-zone rule]
\label{thm:multi_period_dead_zone}
Under Assumption \ref{ass:multiple_periods}, with $s_h^2 > 0$ for all $h$, there exist minimax-regret optimal decisions
$
\boldsymbol{\delta}^\star
=
(\delta_1^\star,\ldots,\delta_H^\star)
$
and thresholds
$
\tau_h^\star\in[0,\infty],
h=1,\ldots,H,
$
with $\tau_H^\star=0$, such that, for every $h<H$,
$$
\delta_h^\star(X_1,\ldots,X_h)
=
\begin{cases}
1
& \text{if } T_h \geq\tau_h^\star,\\
0
& \text{if } T_h \leq-\tau_h^\star,\\
\texttt{NA}
& \text{if } |T_h|<\tau_h^\star,
\end{cases}, \qquad \delta_H^\star(X_1,\ldots,X_H)
=
1\{T_H\geq0\}.
$$
To define the thresholds jointly, let
$
\boldsymbol{\tau}
=
(\tau_1,\ldots,\tau_{H-1},0)
\in
\mathcal{T}_H
=
[0,\infty]^{H-1}\times\{0\},
$
and denote
$$
\sigma(\boldsymbol{\tau})
=
\inf\left\{
h\leq H:
|T_h|\geq\tau_h
\right\}, \qquad R_H(
\boldsymbol{\tau};
\Delta,\boldsymbol{c}
)
=
\Delta
\underbrace{
\mathbb{P}_\Delta\left(
T_{\sigma(\boldsymbol{\tau})}<0
\right)
}_{\substack{\text{probability of an incorrect}\\
\text{terminal policy decision}}}
+
\sum_{h=1}^{H-1}
c_h
\underbrace{
\mathbb{P}_\Delta\left(
\sigma(\boldsymbol{\tau})>h
\right)
}_{\text{probability of collecting }X_{h+1}}
$$
The optimal thresholds can be selected so that
$
\boldsymbol{\tau}^\star
\in
\operatorname*{arg\,min}_{\boldsymbol{\tau}\in\mathcal{T}_H}
\max_{\Delta\in[0,\bar{\Delta}]}
R_H(
\boldsymbol{\tau};
\Delta,\boldsymbol{c}
).
$
\end{theorem}
\begin{proof}
See Appendix \ref{proof:multi_period_dead_zone}.
\end{proof}
Theorem \ref{thm:multi_period_dead_zone} generalizes the confidence-interval representation in Theorem \ref{thm:dead_zone}. In every non-terminal period, sufficiently positive estimates lead researchers to recommend the policy, sufficiently negative estimates lead researchers to reject it, and intermediate estimates lead to additional data collection. The sign of $T_h$ determines the direction of the policy recommendation, while $|T_h|$ measures the strength of the accumulated evidence.
The definition of $\tau_h^\star$ provides a direct economic interpretation as it depends on the cost of deferral in each period and the precision of each subsequent estimate. As for the definition of the $A$-value we can use $\tau_h^\star$ to define each interim $A$-value representation, omitted for brevity.
To gain intuition on the behavior of $\tau_h$, Figure \ref{fig:threshold} reports the threshold for a five-period horizon as a function of the costs of future data collections, assuming for simplicity $s_1 = s_2 = \cdots = s_H = 1$ and costs are constant over each period ($c_1 = c_2 = \cdots c_{H -1}$), with $\Delta \in \mathbb{R}$. The figure shows two expected patterns. First, as the number of subsequent periods decreases (i.e., we move from the first period $h = 1$, to $h = 2$ and so on), the threshold becomes less stringent (smaller). This is natural as more possible experimentation periods ahead implies higher chance of detecting the welfare-improving policy and more possibility of learning. Second, as the continuation cost increases, the threshold also becomes less stringent due to higher costs of future data collection.
\begin{figure}[H]
\centering
\includegraphics[scale = 0.7]{figures/plot_threshold.pdf}
\caption{\footnotesize Optimal threshold with five experimentation periods.}
\label{fig:threshold}
\end{figure}
\subsection{Endogenous precision of the follow-up experiment}
\label{rem:endogenous_precision}
The precision of the follow-up experiment can be endogenized by allowing researchers to choose its sample size. Let $n\in\mathcal{N}$ denote a feasible follow-up sample size and suppose that
$$
X_2(n)
\sim
\mathcal{N}\left(\Delta,\frac{\sigma_2^2}{n}\right)
$$
at welfare cost $c(n)$. The cost $c(n)$ may include both the direct cost of collecting additional observations and the opportunity cost of delaying the policy decision. Each sample size therefore generates a feasible pair
$
\left(s_2^2(n),c(n)\right)
=
\left(\frac{\sigma_2^2}{n},c(n)\right),
$
capturing the trade-off between the informativeness and the cost of the follow-up experiment. If the follow-up design must be chosen before observing $X_1$, its optimal sample size solves
$$
n_2^\star
\in
\argmin_{n\in\mathcal{N}}
\;
\min_{(\delta_1,\delta_2)\in\mathcal{D}}
\sup_{|\Delta|\leq\bar{\Delta}}
\mathrm{Reg}_1
\left(
\delta_1,\delta_2;
\Delta,c(n)
\right),
\qquad
s_2^2=\frac{\sigma_2^2}{n}.
$$
Conditional on $n_2^\star$, Theorem \ref{thm:dead_zone} continues to apply directly with
$$
\frac{s_2}{s_1} =\frac{\sigma_2}{s_1\sqrt{n_2^\star}},
\qquad
\frac{c}{s_1} =\frac{c(n_2^\star)}{s_1}.
$$
We can think of this problem as in contexts where funding agencies must commit to a given budget before seeing the evidence reported in pilot studies.
When the sample size $n_2^\star$ is chosen as a function of $X_1$ the problem becomes however more complex, and left to future research.
\section{Empirical application} \label{sec:application}
In this section, we use data from \cite{crosta2024unconditional} to analyze unconditional cash transfer (UCT) programs in low- and middle-income countries. Our goal is to show how researchers may report the $A$-value alongside $p$-values as a useful measure of evidence strength.
\subsection{Data description}
\cite{crosta2024unconditional} cover 115 studies of 72 randomized evaluations in 34 countries. We focus on monthly household consumption, which is available and has been harmonized across 45 studies. These are publicly available through the online data supplement of \cite{crosta2024unconditional}. Treatment effects and standard errors are expressed as monthly household consumption per \$100 of total transfer in 2010 USD PPP.
For each study, we use a single point estimate selected as follows. First, we keep estimates which use the overall sample and omit those based on smaller subgroups. Second, within each program, we retain estimates from the longest available follow-up horizon. Third, we select the arm with the largest total transfer delivered.\footnote{For lump-sum transfers, we use the reported total transfer. For stream transfers, we calculate the reported monthly tranche amount multiplied by the number of months since the first transfer.} The longest-follow-up horizon focuses on the most persistent effects, while the largest-transfer restriction focuses on the highest-intensity arm. This procedure yields 42 program-level estimates out of the original 45 studies for which household consumption is available.\footnote{Three programs are excluded because they report consumption effects only for particular subgroups.}
For each of the 42 candidate programs, we construct an estimate of average monthly consumption, expressed per \$100 of total transfers. Let \(m_j\) denote the number of months between the first transfer and the selected follow-up.
Define \(\widehat b_j\) and \(s_{1j}\) the resulting monthly consumption estimate and standard error, both expressed per \$100 of total transfers.\footnote{Lump-sum programs measure monthly household consumption effects per \$100 of total transfers, whereas those for stream programs measure monthly effects per \$100 of the monthly tranche. Because the monthly tranche equals cumulative transfers divided by \(m_j\), we divide both the reported estimate and its standard error by \(m_j\) for stream programs and leave them unchanged for lump-sum programs. Dividing the reported stream estimate and its standard error by \(m_j\) therefore converts their denominator to \$100 of cumulative transfers without requiring continuous monthly payments through follow-up.}
We consider two scenarios. In the first scenario we set
$
X_{1j}
=\widehat b_j,
$
so that in the baseline specification we assume that treatment effects measure directly the researcher's welfare criterion, abstracting from the opportunity and administrative costs of transfer expenditure (i.e., assuming that a dollar has the same value if transferred to the households in the program or invested in a different program). As a second specification, we consider
administrative costs of the program which we assume equals 24 percent of the transfer amount, following the cost assumption in \cite{crosta2024unconditional}, and set
$
X_{1j}^{\mathrm{admin}}
=\widehat b_j - \frac{24}{m_j},
$ assuming welfare denotes the average consumption over the $m_j$ horizon.
The standard error remains \(s_{1j}\), treating the administrative-cost assumption and normalization factors as fixed. Clearly other specification are possible depending on the target welfare criterion. (Also, note that our framework does not require that the interpretation of treatment effects $\Delta_j$ is the same across studies.)
\subsection{Problem description}
Our goal is to assess when the existing evidence is strong enough to recommend implementing or rejecting a UCT program, and when further experimentation is warranted. We take the perspective of a researcher reporting the results of an individual policy evaluation.
After observing the evidence from a given program, the research team can recommend scaling-up the program, reject it in favor of the status quo, or defer the decision and commission a follow-up experiment. The follow-up experiment provides an independent estimate of the same policy effect with the same precision as the initial experiment, \(s_{2j}=s_{1j}\).\footnote{This is our preferred specification, although the framework accommodates other choices of \(s_{2j}\).}
We take a perspective where the individual research team may lack a collection of comparable studies from which to estimate a distribution of policy effects, and readers may differ in their prior beliefs about each given study effect. For instance, studies in different countries may be drawn from different distributions or
the evidence assembled in a subsequent meta-analysis need not have been available when each study reported its results and make a policy recommendation. This exercise therefore differs from standard Bayes approach that require observing the same distribution across studies before making policy decisions.
We use the \(A\)-value to assess the value of further experimentation for each program. The availability of multiple studies also allows us to examine, in aggregate, how alternative cost thresholds affect recommendations for further data collection and how programs should be prioritized when experimentation capacity is limited.
\subsection{Analysis}
Figure \ref{fig:distribution} and Table \ref{tab:a-value-distribution} summarize the distribution of \(A\)-values across the 42 programs. The distribution is strongly right-skewed, with many values close to zero and a few substantially larger values. Without administrative costs, the median \(A\)-value is 0.003, compared with a mean of 0.258; with administrative costs equal to 24 percent of transfers, the corresponding values are 0.019 and 0.251. Small \(A\)-values indicate that even low welfare costs of further experimentation favor an immediate policy recommendation.
Selecting the ten programs with the largest \(A\)-values ($\sim$ one fourth of the sample) yields cutoffs of 0.07 without administrative costs and 0.09 with administrative costs, measured in monthly consumption units per \$100 of total transfers. Accounting for administrative costs therefore raises both the median \(A\)-value and the cutoff for allocating ten follow-up experiments. These changes reflect two opposing effects: administrative costs can bring estimated welfare effects closer to zero, increasing the value of further experimentation, or make rejection more decisive, reducing that value.
\begin{table}[tbp]
\centering
\setstretch{1}
\small
\caption{Distribution of A-values}
\label{tab:a-value-distribution}
\begin{tabular}{lr@{\hspace{1cm}}lr}
\toprule
\multicolumn{2}{c}{No administrative costs} & \multicolumn{2}{c}{Administrative costs (24\%)} \\
\cmidrule(lr){1-2}\cmidrule(lr){3-4}
Statistic & A-value & Statistic & A-value \\
\midrule
Studies & 42 & Studies & 42 \\
Mean & 0.258 & Mean & 0.251 \\
Standard deviation & 0.977 & Standard deviation & 0.706 \\
Minimum & $1.91\times 10^{-40}$ & Minimum & $6.64\times 10^{-26}$ \\
10th percentile & $2.61\times 10^{-8}$ & 10th percentile & $6.42\times 10^{-6}$ \\
25th percentile & $1.25\times 10^{-5}$ & 25th percentile & $8.94\times 10^{-4}$ \\
Median & 0.003 & Median & 0.019 \\
75th percentile & 0.059 & 75th percentile & 0.059 \\
90th percentile & 0.163 & 90th percentile & 0.547 \\
95th percentile & 1.160 & 95th percentile & 1.585 \\
Maximum & 5.680 & Maximum & 3.732 \\
\bottomrule
\end{tabular}
\par\vspace{0.6em}
\begin{minipage}{\linewidth}\footnotesize\raggedright Each panel summarizes all 42 program-level A-values in monthly consumption units per \$100 of total transfers (2010 USD PPP). The administrative-cost specification subtracts 24/$m$ (where $m$ is the length of the study in months) before computing A-values.\end{minipage}
\end{table}
\begin{figure}[!ht]
\centering
\includegraphics[scale = 0.55]{figures/distribution_a.pdf}
\caption{\footnotesize Histogram in logarithmic scale of the distribution of $A$-values across experiments.}
\label{fig:distribution}
\end{figure}
Table \ref{tab:country-a-values-selected-panels} reports the countries with at least one program selected for further experimentation under the global top-ten \(A\)-value rule. The countries represented are the same across specifications, except for Liberia, which appears only when administrative costs are excluded. Among countries with multiple evaluations, the baseline rule selects one of seven programs in Kenya, one of six in Uganda, one of four in Malawi, one of two in Liberia, and both programs in Mexico. Administrative costs increase the number selected in Kenya to three and reduces the number in Mexico to one, while Uganda and Malawi remain unchanged.
These selections illustrate how both sampling uncertainty and proximity to the implementation threshold determine the value of further experimentation. Without administrative costs, the selected Kenyan study has an estimate-to-standard-error ratio of 0.02 and a sampling variance of 216, compared with a country average variance of 37. In Uganda, the selected study has a smaller variance than the country average, one versus ten, but its estimate-to-standard-error ratio is only 0.4. Thus, a relatively precise study can still merit further experimentation when its estimated welfare effect remains sufficiently close to zero.
Within-country comparisons can also help researchers contextualize how decisive their evidence is for a policy recommendation. A practical reporting exercise would place a study's \(A\)-value within the distribution of previous comparable evaluations, even when only a few are available. For example, without administrative costs, Kenya's mean \(A\)-value is \(0.4\), and six of its seven studies have values below \(0.07\). Naturally, one may refine further these comparisons to similar transfer amounts, study periods or specific regions if available.
\begin{table}[H]
\centering
\setstretch{1}
\footnotesize
\setlength{\tabcolsep}{4pt}
\caption{Countries with top-ten A-value selections}
\label{tab:country-a-values-selected-panels}
\begin{tabular}{lrrrrrr}
\toprule
\multicolumn{7}{l}{\textbf{Panel A: No administrative costs}} \\
\addlinespace[0.4em]
Country & \shortstack{Mean\\A-value} & \shortstack{Mean\\variance} & \shortstack{Share of selected\\top-10 studies} & \shortstack{Mean A-value\\(selected)} & \shortstack{Mean variance\\(selected)} & \shortstack{Mean $X_{1j}/s_{1j}$\\(selected)} \\
\midrule
Nepal & 5.680 & 1393.156 & 100\% (1) & 5.680 & 1393.156 & -0.104 \\
Ghana & 1.208 & 65.610 & 100\% (1) & 1.208 & 65.610 & 0.111 \\
Kenya & 0.413 & 37.524 & 14.3\% (7) & 2.845 & 216.090 & 0.020 \\
Bangladesh & 0.165 & 111.831 & 100\% (1) & 0.165 & 111.831 & 1.087 \\
Mexico & 0.106 & 34.817 & 100\% (2) & 0.106 & 34.817 & 0.556 \\
Tanzania & 0.070 & 8.410 & 100\% (1) & 0.070 & 8.410 & 0.897 \\
Liberia & 0.070 & 7.650 & 50\% (2) & 0.139 & 15.210 & -0.718 \\
Malawi & 0.064 & 2.378 & 25\% (4) & 0.241 & 5.621 & -0.258 \\
Uganda & 0.024 & 9.825 & 16.7\% (6) & 0.072 & 1.000 & 0.400 \\
\midrule
\addlinespace[0.8em]
\multicolumn{7}{l}{\textbf{Panel B: Administrative costs (24\%)}} \\
\addlinespace[0.4em]
Country & \shortstack{Mean\\A-value} & \shortstack{Mean\\variance} & \shortstack{Share of selected\\top-10 studies} & \shortstack{Mean A-value\\(selected)} & \shortstack{Mean variance\\(selected)} & \shortstack{Mean $X_{1j}/s_{1j}$\\(selected)} \\
\midrule
Nepal & 3.732 & 1393.156 & 100\% (1) & 3.732 & 1393.156 & -0.265 \\
Ghana & 1.606 & 65.610 & 100\% (1) & 1.606 & 65.610 & -0.012 \\
Bangladesh & 0.583 & 111.831 & 100\% (1) & 0.583 & 111.831 & 0.520 \\
Kenya & 0.514 & 37.524 & 42.9\% (7) & 1.152 & 85.605 & 0.048 \\
Tanzania & 0.220 & 8.410 & 100\% (1) & 0.220 & 8.410 & 0.379 \\
Mexico & 0.108 & 34.817 & 50\% (2) & 0.216 & 69.444 & 0.864 \\
Malawi & 0.037 & 2.378 & 25\% (4) & 0.092 & 5.621 & -0.680 \\
Uganda & 0.027 & 9.825 & 16.7\% (6) & 0.125 & 1.000 & 0.178 \\
\bottomrule
\end{tabular}
\par\vspace{0.6em}
\begin{minipage}{\linewidth}\footnotesize\raggedright Only countries with at least one study selected under the respective global top-ten rule are shown. Countries are ordered by decreasing overall mean A-value within each specification. Each study is one of the 42 program-level estimates. Overall means use all studies in the country. The selection share is the number selected in the country divided by all studies in that country, expressed as a percentage. The three selected-study means use only studies in that country selected under the global top 10 A-value rule for the specification. Mean variance is the average $s_{1j}^2$ in the respective group. Mean $X_{1j}/s_{1j}$ is the average signed estimate-to-standard-error ratio among selected studies, using the specification-specific X. A-values are in monthly household consumption per \$100 of total transfers (2010 USD PPP). Numbers in parentheses after percentages are the total numbers of studies in the respective countries.\end{minipage}
\end{table}
\subsection{Comparisons with $p$-values, variance and empirical Bayes}
To understand what drives the \(A\)-value ranking, we compare it with two simple benchmarks. The first selects the ten programs with the largest two-sided \(p\)-values, prioritizing estimates closest to zero relative to their standard errors. The second selects the ten programs with the largest standard errors \(s_{1j}\), prioritizing the least precise estimates. The \(p\)-value rule does not account for the scale of uncertainty in welfare units, while the standard-error rule ignores whether the estimated effect already supports an immediate policy recommendation.
Figures \ref{fig:rank-no-admin} compares these rankings with and without administrative costs. The \(A\)-value ranking generally tracks the \(p\)-value ranking more closely than the standard-error ranking. Without administrative costs, the \(A\)-value and \(p\)-value rules share eight of their ten selections; with administrative costs, they share seven. The corresponding overlaps with the standard-error rule are five and six programs.
Table \ref{tab:selection-disagreements-p_value} reports the programs on which these rules disagree. The disagreements with the \(p\)-value rule illustrate the importance of measuring uncertainty in welfare units. Without administrative costs, the \(A\)-value rule selects a study in Bangladesh with \(p=0.277\) and \(s_{1j}=10.575\), whereas the \(p\)-value rule selects a study in Kenya with \(p=0.377\) and \(s_{1j}=1.017\). Although the Kenyan estimate is less decisive relative to its standard error, the uncertainty of the Bangladeshi estimate is substantially larger. Because a mistaken policy recommendation incurs a loss equal to the magnitude of the true welfare effect, this greater uncertainty allows for substantially larger losses from an immediate decision, increasing the value of further experimentation. Their \(A\)-values are consequently 0.025 and 0.165, respectively. The \(A\)-value rule also selects a Mexican study with a relatively large standard error of 8.333 despite its smaller \(p\)-value of 0.270.
This distinction becomes even more stark when administrative costs are included. The \(p\)-value rule selects two Kenyan studies with \(p\)-values of 0.9 and 0.8, both with standard errors of only 0.3. Their estimates provide little evidence about the sign of net welfare, but the uncertainty concerns effects on a relatively small welfare scale as their \(A\)-values are 0.043 and 0.034. The \(A\)-value rule instead prioritizes, among others, again the Bangladeshi study with significantly larger standard error.
The disagreements with the standard-error rule reveal the complementary limitation of prioritizing imprecision alone. Without administrative costs, this rule selects studies in Rwanda and Uganda with standard errors of 11 and 5.8, respectively, even though their \(p\)-values are close to zero. By contrast, the \(A\)-value rule selects another Mexican study with a much smaller standard error of 0.435 but a \(p\)-value of 0.993. Moreover, the standard-error rule selects the same programs in both specifications, since subtracting administrative costs leaves standard errors unchanged, without incorporating its welfare effects.
\begin{figure}[!ht]
\centering
\includegraphics[scale = 0.55]{figures/rank_1.pdf}
\includegraphics[scale = 0.55]{figures/rank_p1.pdf}
\caption{\footnotesize Comparison of rankings for further experimentation. Each point represents one of the 42 programs. The vertical axis reports the \(A\)-value rank; the horizontal axis reports the \(p\)-value rank (left) or standard-error rank (right). Rank 1 corresponds to the largest value. The upper panels include administrative costs equal to 24 percent of transfers; the lower panels exclude administrative costs. Dashed lines indicate identical rankings. Red points are selected only by the \(A\)-value rule, blue points only by the comparison rule, red points with blue outlines by both rules, and gray points by neither. Each rule selects ten programs, with cutoff ties broken by program ID. \(A\)-values assume \(s_{2j}=s_{1j}\), and two-sided \(p\)-values test a zero welfare effect under the respective specification.}
\label{fig:rank-no-admin}
\end{figure}
\begin{table}[!ht]
\centering
\setstretch{1}
\footnotesize
\setlength{\tabcolsep}{3pt}
\caption{Top-ten selection disagreements: $A$-value versus $p$-value and standard error}
\label{tab:selection-disagreements}
\label{tab:selection-disagreements-p_value}
\label{tab:selection-disagreements-standard_error}
\begin{tabular*}{\linewidth}{
@{\extracolsep{\fill}}
lrrr
@{\hspace{0.5cm}}
lrrr
@{}
}
\toprule
\multicolumn{8}{l}{
\textbf{Panel A: $A$-value versus $p$-value}
} \\
\addlinespace[0.5em]
\multicolumn{4}{c}{No administrative costs}
& \multicolumn{4}{c}{Administrative costs (24\%)} \\
\multicolumn{4}{c}{\textit{8 of 10 selections shared}}
& \multicolumn{4}{c}{\textit{7 of 10 selections shared}} \\
\cmidrule(lr){1-4}\cmidrule(lr){5-8}
Country & $A$-value & $p$ & $s_{1j}$
& Country & $A$-value & $p$ & $s_{1j}$ \\
\midrule
\multicolumn{4}{l}{\textit{Selected only by $A$-value}}
& \multicolumn{4}{l}{\textit{Selected only by $A$-value}} \\
\addlinespace[0.2em]
Bangladesh & 0.165 & 0.277 & 10.575
& Bangladesh & 0.583 & 0.603 & 10.575 \\
Mexico & 0.125 & 0.270 & 8.333
& Mexico & 0.216 & 0.388 & 8.333 \\
& & &
& Malawi & 0.092 & 0.496 & 2.371 \\
\addlinespace[0.6em]
\multicolumn{4}{l}{\textit{Selected only by $p$-value}}
& \multicolumn{4}{l}{\textit{Selected only by $p$-value}} \\
\addlinespace[0.2em]
Kenya & 0.025 & 0.377 & 1.017
& Kenya & 0.043 & 0.902 & 0.300 \\
Uganda & 0.070 & 0.370 & 2.900
& Kenya & 0.034 & 0.833 & 0.300 \\
& & &
& Malawi & 0.054 & 0.668 & 0.800 \\
\addlinespace[0.6em]
\midrule
\addlinespace[0.4em]
\multicolumn{8}{l}{
\textbf{Panel B: $A$-value versus standard error}
} \\
\addlinespace[0.5em]
\multicolumn{4}{c}{No administrative costs}
& \multicolumn{4}{c}{Administrative costs (24\%)} \\
\multicolumn{4}{c}{\textit{5 of 10 selections shared}}
& \multicolumn{4}{c}{\textit{6 of 10 selections shared}} \\
\cmidrule(lr){1-4}\cmidrule(lr){5-8}
Country & $A$-value & $p$ & $s_{1j}$
& Country & $A$-value & $p$ & $s_{1j}$ \\
\midrule
\multicolumn{4}{l}{\textit{Selected only by $A$-value}}
& \multicolumn{4}{l}{\textit{Selected only by $A$-value}} \\
\addlinespace[0.2em]
Malawi & 0.241 & 0.796 & 2.371
& Tanzania & 0.220 & 0.704 & 2.900 \\
Liberia & 0.139 & 0.473 & 3.900
& Uganda & 0.125 & 0.859 & 1.000 \\
Mexico & 0.087 & 0.993 & 0.435
& Kenya & 0.112 & 0.821 & 1.017 \\
Uganda & 0.072 & 0.689 & 1.000
& Malawi & 0.092 & 0.496 & 2.371 \\
Tanzania & 0.070 & 0.370 & 2.900
& & & & \\
\addlinespace[0.6em]
\multicolumn{4}{l}{\textit{Selected only by standard error}}
& \multicolumn{4}{l}{\textit{Selected only by standard error}} \\
\addlinespace[0.2em]
Rwanda & 0.001 & 0.006 & 11.133
& Rwanda & 0.002 & 0.010 & 11.133 \\
South Sudan & 0.004 & 0.021 & 7.700
& South Sudan & 0.009 & 0.040 & 7.700 \\
Mali & 0.026 & 0.102 & 6.625
& Mali & 0.039 & 0.138 & 6.625 \\
Kenya & 0.010 & 0.053 & 6.300
& Uganda & $3.44\times10^{-5}$ & $5.89\times10^{-4}$ & 5.858 \\
Uganda & $7.18\times10^{-6}$ & $1.58\times10^{-4}$ & 5.858
& & & & \\
\bottomrule
\end{tabular*}
\par\vspace{0.6em}
\begin{minipage}{\linewidth}
\footnotesize\raggedright
\textit{Notes:}
Within each comparison and cost specification, only programs selected
by exactly one rule are shown. Each rule selects the ten largest values
of its measure. Repeated country
names within a specification refer to different programs.
$A$-values assume $s_{2j}=s_{1j}$.
$p$-values are from two-sided normal tests of a zero welfare effect
under the respective cost specification.
$A$-values and harmonized standard errors $s_{1j}$ are expressed in
monthly household consumption per \$100 of total transfers
(2010 USD PPP).
\end{minipage}
\end{table}
\paragraph{Empirical Bayes}
As a second comparison, we consider a Gaussian empirical Bayes benchmark based on \cite{abadie2023estimating} (Section 2.3). Empirical Bayes uses evidence across programs to estimate the distribution of policy effects, allowing each evaluation to benefit from information contained in other studies. This approach is particularly useful when a collection of comparable evaluations is available at the time of the decision, but not feasible when the study effect may not have a reference distribution when making a policy recommendation.
For the empirical Bayes benchmark, we estimate a common Gaussian prior by maximum likelihood, allowing sampling variances to differ across the 42 programs. This specification treats program effects as exchangeable and independent of their standard errors. The estimated prior has mean \(2.985\) and standard deviation \(2.088\). We then compute the expected welfare gain from an independent follow-up experiment with \(s_{2j}=s_{1j}\).
Figure \ref{fig:comparison-eb} shows substantial differences between the rankings. The two rules share five of their ten selected programs without administrative costs and three with administrative costs. Pooling also affects immediate recommendations: without administrative costs, all 42 posterior means are positive, including those for the four programs with negative initial estimates. With administrative costs, 32 posterior means remain positive.
Table \ref{tab:selection-disagreements-V_EB} reports the programs where the two ranking disagree.
The program in Nepal illustrates the mechanism behind these differences. Its estimated consumption effect is \(-3.875\), with a standard error of \(37.325\), but its posterior mean is \(2.963\): the empirical Bayes calculation assigns approximately \(99.7\%\) of the weight to the fitted prior mean. This program ranks first by \(A\)-value and last by empirical Bayes value in both specifications. Under the fitted model, a second estimate with the same large standard error would provide little information relative to the existing posterior and would be very unlikely to change the policy recommendation. The large \(A\)-value reflects the welfare losses that remain possible when assessing this imprecise evaluation without relying on the chosen prior distribution. The empirical Bayes procedure on the other hand strongly relies on the underlying exchangeability assumption in the policy ranking for this setting.
A similar pattern appears in Kenya and Bangladesh. Without administrative costs, programs with standard errors of \(14.7\) and \(10.6\), respectively, rank second and fifth by \(A\)-value but 38th and 37th by empirical Bayes value. The fitted prior mean receives approximately \(98\%\) and \(96\%\) of the weight in their respective posterior means, making an equally imprecise follow-up experiment unlikely to change the policy recommendation. Both programs remain selected by the top-ten \(A\)-value rule and excluded by the empirical Bayes rule when administrative costs are included.
\begin{figure}[!ht]
\centering
\includegraphics[scale = 0.55]{figures/comparison_EB.pdf}
\caption{ \footnotesize Comparison of $A$-value and empirical Bayes rankings.
Each point represents one of the 42 programs, with its $A$-value rank
on the vertical axis and empirical Bayes sample-information value
rank on the horizontal axis. Rank 1 denotes the largest value.
The left panel excludes administrative costs; the right panel includes
administrative costs equal to 24\% of transfers.
The dashed diagonal indicates identical ranks.
Red points denote programs selected only by the $A$-value rule,
blue points those selected only by empirical Bayes, red points with
blue outlines those selected by both, and gray points those selected
by neither. Each rule selects ten programs, with cutoff ties broken
by program ID.
Both measures assume $s_{2j}=s_{1j}$.
Empirical Bayes values condition on the initial evidence and hold
the common Gaussian prior.}
\label{fig:comparison-eb}
\end{figure}
\begin{table}[p]
\centering
\setstretch{1}
\small
\setlength{\tabcolsep}{4pt}
\caption{Top-ten selection disagreements: $A$-value versus empirical Bayes}
\label{tab:selection-disagreements-V_EB}
\begin{tabular*}{\linewidth}{
@{\extracolsep{\fill}}lrrrrrr@{}
}
\toprule
\multicolumn{7}{l}{
\textbf{Panel A: No administrative costs}
} \\
\addlinespace[0.4em]
Country
& $A$-value
& $V_j^{\mathrm{EB}}$
& \shortstack{Estimate\\$X_{1j}$}
& \shortstack{Standard error\\$s_{1j}$}
& \shortstack{EB Prior weight\\(\%)}
& \shortstack{EB Posterior mean} \\
\midrule
\multicolumn{7}{l}{\textit{Selected only by the $A$-value rule}} \\
\addlinespace[0.2em]
Nepal & 5.680 & $<0.001$ & -3.875 & 37.325 & 99.7 & 2.963 \\
Kenya & 2.845 & $<0.001$ & 0.300 & 14.700 & 98.0 & 2.932 \\
Ghana & 1.208 & $<0.001$ & 0.900 & 8.100 & 93.8 & 2.855 \\
Bangladesh & 0.165 & $<0.001$ & 11.500 & 10.575 & 96.2 & 3.304 \\
Mexico & 0.125 & $<0.001$ & 9.200 & 8.333 & 94.1 & 3.352 \\
\addlinespace[0.5em]
\multicolumn{7}{l}{\textit{Selected only by the empirical Bayes rule}} \\
\addlinespace[0.2em]
Uganda & 0.070 & 0.042 & -2.600 & 2.900 & 65.9 & 1.078 \\
Kenya & 0.025 & 0.004 & 0.897 & 1.017 & 19.2 & 1.297 \\
Malawi & 0.015 & 0.003 & 0.800 & 0.800 & 12.8 & 1.080 \\
Malawi & 0.002 & 0.002 & 0.100 & 0.100 & 0.2 & 0.107 \\
Lesotho & 0.004 & $<0.001$ & 1.404 & 0.896 & 15.5 & 1.650 \\
\addlinespace[0.5em]
\midrule
\addlinespace[0.4em]
\multicolumn{7}{l}{
\textbf{Panel B: Administrative costs (24\%)}
} \\
\addlinespace[0.4em]
Country
& $A$-value
& $V_j^{\mathrm{EB}}$
& \shortstack{Estimate\\$X_{1j}$}
& \shortstack{Standard error\\$s_{1j}$}
& \shortstack{EB Prior weight\\(\%)}
& \shortstack{EB Posterior mean} \\
\midrule
\multicolumn{7}{l}{\textit{Selected only by the $A$-value rule}} \\
\addlinespace[0.2em]
Nepal & 3.732 & $<0.001$ & -9.875 & 37.325 & 99.7 & -3.037 \\
Kenya & 2.165 & $<0.001$ & -1.700 & 14.700 & 98.0 & 0.932 \\
Ghana & 1.606 & $<0.001$ & -0.100 & 8.100 & 93.8 & 1.855 \\
Kenya & 1.179 & $<0.001$ & 0.200 & 6.300 & 90.1 & -8.103 \\
Bangladesh & 0.583 & $<0.001$ & 5.500 & 10.575 & 96.2 & -2.696 \\
Tanzania & 0.220 & 0.021 & 1.100 & 2.900 & 65.9 & 1.353 \\
Mexico & 0.216 & $<0.001$ & 7.200 & 8.333 & 94.1 & 1.352 \\
\addlinespace[0.5em]
\multicolumn{7}{l}{\textit{Selected only by the empirical Bayes rule}} \\
\addlinespace[0.2em]
Uganda & 0.028 & 0.310 & -3.743 & 2.900 & 65.9 & -0.065 \\
Liberia & 0.048 & 0.244 & -4.646 & 3.900 & 77.7 & -0.150 \\
Malawi & 0.054 & 0.173 & -0.343 & 0.800 & 12.8 & -0.063 \\
Kenya & 0.034 & 0.071 & -0.063 & 0.300 & 2.0 & -0.027 \\
Kenya & 0.043 & 0.053 & 0.037 & 0.300 & 2.0 & 0.071 \\
Lesotho & 0.058 & 0.033 & 0.404 & 0.896 & 15.5 & 0.650 \\
Rwanda & 0.025 & 0.032 & 0.186 & 0.400 & 3.5 & 0.224 \\
\bottomrule
\end{tabular*}
\par\vspace{0.6em}
\begin{minipage}{\linewidth}
\footnotesize\raggedright
\textit{Notes:}
Only programs selected by exactly one rule are shown.
Both rules select their ten largest values within each specification.
Both information measures assume $s_{2j}=s_{1j}$.
$V_j^{\mathrm{EB}}$ is the expected empirical Bayes welfare gain from a follow-up
experiment conditional on the initial evidence, excluding the cost
of conducting the experiment.
The common Gaussian prior is fitted to consumption effects,
with $\hat\mu_0=2.985$ and $\hat\tau^2=4.360$.
Prior weight is reported as $100(1-\hat\tau^2/(\hat\tau^2+s_{1j}^2))$.
\end{minipage}
\end{table}
Finally, it is interesting to study similar comparisons where follow up data collection are significantly more precise instead of equally precise. Here we find that changing the precision of the follow-up study does not change the choice of the experiment for the top-10 $A$-value ranking choice with a single exception.
\begin{figure}[!ht]
\centering
\includegraphics[scale = 0.55]{figures/rank_by_w.pdf}
\caption{\footnotesize The figure illustrates that $A$-value ranking are strongly correlated as we vary the precision of the follow up study from $s_2/s_1 = 1$ to $s_2/s_1 = 0.1$. }
\label{fig:different_precision}
\end{figure}
\section{Conclusions}
This paper reinterprets statistical significance through the choice between making an immediate policy recommendation and collecting further evidence. We propose reporting the \(A\)-value alongside \(p\)-values as a measure of the break-even welfare cost of further experimentation under minimax regret. Ranking studies by their \(A\)-values yields finite-sample welfare guarantees when experimentation capacity is limited. Our application to cash transfer programs illustrates how accounting for uncertainty in welfare units changes experimentation priorities, without requiring a common distribution of policy effects across studies.
This paper opens several questions for future research. The first is to combine Bayesian procedures with minimax regret through ambiguity-averse objectives that incorporate prior information while allowing uncertainty about its specification \citep[e.g.][]{banerjeeetal2020, gilboa2008probability, andrews2025certified}, here taking into account opportunity costs of future data collection. The second is to choose the size of the follow-up sample as a function of the initial evidence, jointly determining whether and how to collect additional data. A third direction is to extend the analysis to settings where unbiased estimates of welfare effects are unavailable \citep{cesa2025adaptive}.
\bibliographystyle{plainnat}
\bibliography{bibliography}
\section{Proofs of main results}
\subsection{Proof of Theorem \ref{thm:dead_zone}}
\label{proof:dead_zone}
Theorem \ref{thm:dead_zone} is a corollary of
the more general Theorem \ref{thm:multi_period_dead_zone} with $H=2$ and
$c_1=c$. Under this specialization, write
$
S_1^2=\frac{1}{s_1^2},
T_1
=
\frac{X_1/s_1^2}{S_1}
=
\frac{X_1}{s_1},
S_2^2
=
\frac{1}{s_1^2}+\frac{1}{s_2^2},
T_2
=
\frac{X_1/s_1^2+X_2/s_2^2}{S_2}.
$
Because $S_2>0$,
\[
1\{T_2\geq0\}
=
1\left\{
\frac{X_1}{s_1^2}
+
\frac{X_2}{s_2^2}
\geq0
\right\}
=
1\left\{
\frac{s_2^2}{s_1^2}X_1+X_2\geq0
\right\}.
\]
Moreover, the loss and regret in the $H=2$ problem coincide with
$\mathrm{Reg}_1(\delta_1,\delta_2;\Delta,c)$. Theorem
\ref{thm:multi_period_dead_zone} therefore implies that there exists a
minimax-regret rule of the form
\[
\delta_1^\star(X_1)
=
\begin{cases}
1,
& X_1/s_1\geq\tau^\star,\\
0,
& X_1/s_1\leq-\tau^\star,\\
\texttt{NA},
& |X_1|/s_1<\tau^\star,
\end{cases}, \qquad \delta_2^\star(X_1,X_2)
=
1\left\{
\frac{s_2^2}{s_1^2}X_1+X_2\geq0
\right\}
\]
where $\tau^\star=\tau_1^\star$. It remains only to specialize the threshold characterization in
Theorem \ref{thm:multi_period_dead_zone}. Let
\[
d=\frac{\Delta}{s_1},
\qquad
w=\frac{s_2}{s_1}.
\]
By symmetry (see the proof of Theorem \ref{thm:multi_period_dead_zone}), it is sufficient to consider $\Delta\geq0$. Under a
general threshold $\tau$, an incorrect first-period decision occurs
when $X_1/s_1\leq-\tau$, with probability
$
\Phi(-\tau-d).
$
Conditional on $X_1/s_1=x$, continuation followed by an incorrect
terminal decision occurs when
$
\frac{s_2^2}{s_1^2}X_1+X_2<0.
$
Since $X_2\sim\mathcal N(s_1d,s_2^2)$ independently of $X_1$, its
conditional probability is
$
\Phi\left(-wx-\frac{d}{w}\right).
$
Finally, the probability of continuation is
$
\mathbb P_\Delta
\left(
-\tau<\frac{X_1}{s_1}<\tau
\right)
=
\Phi(\tau-d)-\Phi(-\tau-d).
$
Since $X_1/s_1\sim\mathcal N(d,1)$, the $H=2$ regret function in
Theorem \ref{thm:multi_period_dead_zone} satisfies
\[
\begin{aligned}
\frac{1}{s_1}
R_2(\tau;\Delta,c)
={}&
d\Phi(-\tau-d)
+
d\int_{-\tau}^{\tau}
\Phi\left(-wx-\frac{d}{w}\right)
\phi(x-d)\,dx +
\frac{c}{s_1}
\left[
\Phi(\tau-d)-\Phi(-\tau-d)
\right].
\end{aligned}
\]
For $\Delta<0$, symmetry gives the same expression with
$d=|\Delta|/s_1$. Therefore, the threshold characterization in
Theorem \ref{thm:multi_period_dead_zone} becomes
\[
\begin{aligned}
\tau^\star
\in
\operatorname*{arg\,min}_{\tau\in[0,\infty]}
\max_{0\leq d\leq\bar\Delta/s_1}
\Bigg\{
&
d\Phi(-\tau-d)
+
d\int_{-\tau}^{\tau}
\Phi\left(-wx-\frac{d}{w}\right)
\phi(x-d)\,dx +
\frac{c}{s_1}
\left[
\Phi(\tau-d)-\Phi(-\tau-d)
\right]
\Bigg\}.
\end{aligned}
\]
\subsection{Proof of Proposition \ref{cor:1}} \label{proof:cor:1}
Throughout the proof we define
$
d=\frac{|\Delta|}{s_1},
w=\frac{s_2}{s_1},
\gamma=\frac{c}{s_1}.
$
We prove each part separately.
\paragraph{Proof of Part (A)}
Fix a threshold \(\tau\geq0\). By symmetry, it is sufficient to consider
\(\Delta\geq0\).
For \(\Delta=s_1d\geq0\), let \(p(d,\tau)\) denote the probability that the policy is ultimately rejected under the first-period rule with threshold \(\tau\) and the optimal terminal rule \(\delta_2^\star\). Formally,
$$
\begin{aligned}
p(d,\tau)
=&
\mathbb P_{\Delta=s_1d}
\left(
\frac{X_1}{s_1}\leq-\tau
\right) +
\mathbb P_{\Delta=s_1d}
\left(
-\tau<\frac{X_1}{s_1}<\tau,
\ \frac{s_2^2}{s_1^2}X_1+X_2<0
\right).
\end{aligned}
$$
We suppress the dependence of \(p(d,\tau)\) on \(w=s_2/s_1\) to simplify notation. Let
$$
q(d,\tau)
=
\Phi(\tau-d)-\Phi(-\tau-d)
$$
denote the probability of collecting the follow-up signal. The normalized
regret can then be written as
$$
\frac{1}{s_1}
\mathrm{Reg}_1
(\delta_1^\tau,\delta_2^\star;\Delta,c)
=
d\,p(d,\tau)+\gamma q(d,\tau).
$$
We show that both terms on the right-hand side are nonincreasing for
\(d\geq u_0\), where \(u_0\) is the unique positive solution to
$
\Phi(-u_0)=u_0\phi(u_0).
$
Numerically, \(u_0\simeq0.7518\).
First,
$$
\frac{\partial q(d,\tau)}{\partial d}
=
\phi(\tau+d)-\phi(\tau-d)
\leq0,
\qquad d,\tau\geq0.
$$
Next, define
$$
Z_1=\frac{X_1}{s_1},
\qquad
Z_2=\frac{X_2}{s_2} \Rightarrow
Z_1\sim\mathcal N(d,1), \qquad V=\frac{wZ_1+Z_2}{\sqrt{1+w^2}},
\qquad
a=\frac{\sqrt{1+w^2}}{w}, \qquad \Gamma =\tau\sqrt{1+w^2}.
$$
The terminal decision rejects the policy if
\(wZ_1+Z_2<0\).
Also, \(V\sim\mathcal N(ad,1)\), and
$
Z_1\mid V=v
\sim
\mathcal N\left(
\frac{w}{\sqrt{1+w^2}}v,
\frac{1}{1+w^2}
\right).
$
Importantly, this conditional distribution does not depend on \(d\).
Conditional on $V=v$, the probability that the policy is ultimately rejected is
\[
\mathbb P_{\Delta=s_1d}\left(
\left.
\left\{\frac{X_1}{s_1}\leq-\tau\right\}
\cup
\left\{
-\tau<\frac{X_1}{s_1}<\tau,\,
\frac{s_2^2}{s_1^2}X_1+X_2<0
\right\}
\,\right|\,V=v
\right)
=
\begin{cases}
\Phi(\Gamma-wv), & v<0,\\[3pt]
\Phi(-\Gamma-wv), & v>0.
\end{cases}
\]
Indeed, when \(v<0\), the second-period decision rejects the policy, so the
policy is ultimately rejected unless \(Z_1\geq\tau\). When \(v>0\), the
second-period decision implements the policy, so rejection occurs only when
\(Z_1\leq-\tau\).
Consider a symmetric random variable \(U\), independent of \(V\), with
$
P(U = 0) = 2 \Phi(\Gamma) - 1
$
and density
$
f_U(u)=w\phi(\Gamma+w|u|),
u\neq0.
$
It follows that
$$
p(d,\tau)
=
\mathbb P(U>V)
=
\mathbb P(U-\varepsilon>ad), \qquad \varepsilon\sim\mathcal N(0,1).
$$
For \(\Gamma>0\), the continuous part of the density of \(U\) satisfies
$
f_U(u)
=
w\phi(\Gamma)
\exp\left\{
-\Gamma w|u|-\frac{w^2u^2}{2}
\right\}.
$
Using the identity
$$
e^{-b|u|}
=
\frac{b}{2\sqrt{\pi}}
\int_0^\infty
t^{-3/2}
\exp\left\{-\frac{b^2}{4t}\right\}
\exp\{-tu^2\}\,dt,
\qquad b>0,
$$
we can represent \(U\) as a mixture of centered Gaussian random variables
whose variances \(\sigma^2\) satisfy
$
0\leq\sigma^2\leq\frac{1}{w^2},
$
where the atom at zero corresponds to \(\sigma^2=0\). When \(\Gamma=0\), the same
representation holds directly because
\(U\sim\mathcal N(0,1/w^2)\).
For $w>0$ and $\Gamma>0$, let $H(\cdot;\tau,w)$ be the probability measure on
$[0,1/w^2]$ defined, for every Borel set $B\subseteq[0,1/w^2]$, by
\[
\begin{aligned}
H(B;\tau,w)
={}&
\bigl(2\Phi(\Gamma)-1\bigr)1\{0\in B\} +
\int_{B\cap(0,\,1/w^2)}
\frac{\Gamma w^2\phi(\Gamma)}
{(1-w^2s)^{3/2}}
\exp\left\{
-\frac{\Gamma^2w^2s}{2(1-w^2s)}
\right\}ds.
\end{aligned}
\]
We can write
\[
p(d,\tau)
=
\int_{[0,\,1/w^2]}
\Phi\left(
-\frac{ad}{\sqrt{1+s}}
\right)
\,dH(s;\tau,w).
\]
When $\Gamma=0$, define $H(\cdot;0,w)$ to place probability one on $s=1/w^2$.
Moreover, for every $\sigma^2\in[0,1/w^2]$,
\[
1
\leq
\lambda(\sigma^2)
\equiv
\frac{a}{\sqrt{1+\sigma^2}}
\leq a,
\]
where the first inequality follows from
$a^2=1+1/w^2$. Hence,
\[
p(d,\tau)
=
\int_{[0,\,1/w^2]}
\Phi\bigl(-\lambda(\sigma^2)d\bigr)
\,dH(\sigma^2;\tau,w).
\]
For every \(\lambda\geq1\),
$
\frac{\partial}{\partial d}
\left\{
d\Phi(-\lambda d)
\right\}
=
\Phi(-\lambda d)-\lambda d\phi(\lambda d).
$
By the definition of \(u_0\),
$
\Phi(-x)-x\phi(x)\leq0$ for every $x\geq u_0.
$
Hence, if \(d\geq u_0\), then \(\lambda d\geq u_0\), and
$
\frac{\partial}{\partial d}
\left\{
d\Phi(-\lambda d)
\right\}
\leq0.
$
Because \(d\,p(d,\tau)\) is a mixture of these terms, it is also
nonincreasing for \(d\geq u_0\). The same conclusion holds for
\(\tau=\infty\), for which
$
p(d,\tau)=\Phi(-ad),
q(d,\tau)=1.
$
Therefore, for every threshold \(\tau\), normalized regret is nonincreasing
in \(d\) for \(d\geq u_0\). If
\(\bar\Delta/s_1\geq0.76>u_0\), it follows that
$$
\max_{0\leq d\leq\bar\Delta/s_1}
\left\{
d\,p(d,\tau)+\gamma q(d,\tau)
\right\}
=
\max_{0\leq d\leq u_0}
\left\{
d\,p(d,\tau)+\gamma q(d,\tau)
\right\}.
$$
This equality holds for every \(\tau\). Consequently, the minimax objective
as a function of \(\tau\), and hence its set of minimizers, does not depend
on \(\bar\Delta\) once \(\bar\Delta/s_1\geq0.76\). Finally, the normalized
objective depends on the remaining primitives only through
$
w=\frac{s_2}{s_1},
\gamma=\frac{c}{s_1}.
$
\paragraph{Proof of Part (B)}
Next, retain the notation from Part (A) and write
\begin{equation}
\label{eq:tau_cost_risk}
\begin{aligned}
L(\tau,d)&=d\,p(d,\tau),\qquad
R(\tau,d,\gamma)=L(\tau,d)+\gamma q(d,\tau), \qquad
F(\tau,\gamma)=\max_{0\leq d\leq u_0}R(\tau,d,\gamma).
\end{aligned}
\end{equation}
Part (A) shows that minimizing $F$ over $\tau\in[0,\infty]$
is equivalent to the threshold problem in Theorem~\ref{thm:dead_zone}.
The values at $\tau=\infty$ are those specified in Part (A).
Subscripts on $L,R,F,p,q$, and the auxiliary functions below denote
partial derivatives.
We first prove uniqueness, continuity, and the endpoint statements,
and then establish cost monotonicity through four additional lemmas.
For finite $\tau>0$ and $d>0$, introduce the following auxiliary quantities:
\begin{align*}
A&=\phi(\tau+d),& B&=\phi(\tau-d),\\
k&=A+B,& z&=\frac{B-A}{A+B}=\tanh(\tau d),\\
P_+&=\Phi(d/w-w\tau),& P_-&=\Phi(-d/w-w\tau),\\
J&=AP_+-BP_-,& E&=AP_++BP_-,\\
M&=2\Phi(\Gamma)-1,& K&=a\phi(ad)M,\\
S&=E+zJ,& C&=\phi(\Gamma)\phi(ad)=A\phi(d/w-w\tau).
\end{align*}
Define
\begin{align}
b(\tau,d)&=\frac{dJ}{k}
=d\frac{\Phi(d/w-w\tau)-e^{2\tau d}\Phi(-d/w-w\tau)}{1+e^{2\tau d}},
\label{eq:tau_cost_b}\\
G(\tau,d)&=p(d,\tau)-d(K+S).
\label{eq:tau_cost_G}
\end{align}
Dependence on $w$ is suppressed unless $w$ is varied explicitly.
Differentiating the Gaussian expressions gives
\begin{equation}
p_\tau(d,\tau)=-J,\qquad
p_d(d, \tau)=-(E+K),\qquad
J_\tau=-\tau J-dE,
\label{eq:tau_cost_pderiv}
\end{equation}
and
\begin{equation}
\begin{aligned}
q_\tau(d,\tau)&=k,& q_d(d, \tau)&=-kz,\\
q_{dd}(d, \tau)&=-k(\tau-dz),&
q_{d\tau}(d, \tau)&=k(\tau z-d).
\end{aligned}
\label{eq:tau_cost_qderiv}
\end{equation}
Consequently,
\begin{align}
R_\tau&=k(\gamma-b),\label{eq:tau_cost_Rt}\\
G&=R_d+zR_\tau=L_d+zL_\tau,\label{eq:tau_cost_Gidentity}\\
R_{\tau\tau}+\tau R_\tau&=d^2E+\gamma d k z>0.\label{eq:tau_cost_convexityidentity}
\end{align}
The cancellation in \eqref{eq:tau_cost_Gidentity} makes $G$ independent of
$\gamma$. Also,
\begin{equation}
b_\tau=-\frac{d^2}{2}(1-z^2)(P_++P_-)<0.
\label{eq:tau_cost_bt}
\end{equation}
\begin{lem}[Curvature and the location of worst-case effects]
\label{lem:tau_cost_curvature}
For every finite $\tau\geq0$ and $d>0$:
\begin{enumerate}
\item If $a^2d^2\leq2$, then $L_{dd}(\tau,d)\leq0$ and
$L_{\tau d}(\tau,d)\leq0$.
\item If $ad\geq1$ and $d(\tau+d)\geq1$, then $L_d(\tau,d)<0$.
\end{enumerate}
For $\tau,\gamma>0$, every maximizer in \eqref{eq:tau_cost_risk} belongs to
$(0,u_0)$, satisfies $R_d=0$ and $R_{dd}\leq0$, and has $L_d>0$.
\end{lem}
\begin{proof}
Differentiating the Gaussian-mixture representation from Part (A) gives
\begin{equation}
L_{dd}(\tau,d)=
\int_{[0,\,1/w^2]}\lambda(s)\phi(\lambda(s)d)
\bigl((\lambda(s)d)^2-2\bigr)\,dH(s;\tau,w)\leq0
\quad\text{if }a^2d^2\leq2,
\label{eq:tau_cost_Ldd}
\end{equation}
because $1\leq\lambda(s)\leq a$. The mixing measure does not depend
on $d$, and its bounded support justifies differentiation.
For the mixed derivative, we can write
\begin{equation}
L_\tau=-2w\phi(\tau)\int_0^\infty
\phi\bigl(w(\tau+y)\bigr)d e^{-a^2d^2/2}\sinh(dy)\,dy.
\label{eq:tau_cost_Ltintegral}
\end{equation}
The derivative in $d$ of the last three factors in this
integrand is
\begin{align*}
\frac{\partial}{\partial d}
\left[d e^{-a^2d^2/2}\sinh(dy)\right]
&=e^{-a^2d^2/2}\bigl[(1-a^2d^2)\sinh(dy)+dy\cosh(dy)\bigr]\\
&\geq e^{-a^2d^2/2}(2-a^2d^2)\sinh(dy)\geq0,
\end{align*}
using $x\cosh x\geq\sinh x$ for $x\geq0$.
Differentiation under the integral is justified by Gaussian domination
on compact intervals of $d$. This proves the first assertion.
For the second assertion, condition on $V=-y<0$ using the conditional
distribution of $Z_1$ given $V$ obtained in Part (A).
The probability of continuation followed by terminal rejection is
\begin{equation}
\begin{aligned}
p_{\mathrm{cont}}(\tau,d)
&:=\mathbb P_{\Delta=s_1d}
\left(-\tau<Z_1<\tau,\ V<0\right) =\int_0^\infty h(y,\tau)\phi(y+ad)\,dy,
\end{aligned}
\label{eq:tau_cost_pcont}
\end{equation}
where, by Gaussian conditioning and symmetry,
\[
\begin{aligned}
h(y,\tau)
&:=\mathbb P_{\Delta=s_1d}
\left(-\tau<Z_1<\tau\mid V=-y\right) =\Phi(\Gamma-wy)-\Phi(-\Gamma-wy)\geq0.
\end{aligned}
\]
The integral in \eqref{eq:tau_cost_pcont} follows because
$V\sim\mathcal N(ad,1)$. Thus the total rejection probability satisfies
\[
p(d, \tau)=\Phi(-\tau-d)+p_{\mathrm{cont}}(\tau,d),
\]
where the two terms correspond to immediate rejection and rejection
Its contribution to $L_d$ is
\[
\frac{\partial}{\partial d}\{d p_{\mathrm{cont}}(\tau,d)\}
=\int_0^\infty h(y,\tau)\phi(y+ad)
\bigl(1-a^2d^2-ady\bigr)\,dy\leq0
\]
when $ad\geq1$. The derivative of the stopping component is strictly
negative when $d(\tau+d)\geq1$, since
\[
\Phi(-\tau-d)-d\phi(\tau+d)
<\phi(\tau+d)\left(\frac{1}{\tau+d}-d\right)\leq0.
\]
Here we used the Gaussian Mills inequality
$\Phi(-x)<\phi(x)/x$ for $x>0$, which follows by comparing
$\int_x^\infty\phi(y)\,dy$ with
$x^{-1}\int_x^\infty y\phi(y)\,dy$.
Finally, symmetry gives $R_d(\tau,0,\gamma)=1/2>0$.
Part (A) gives $L_d(\tau,u_0)\leq0$, whereas
$q_d(d =u_0, \tau)<0$ for $\tau>0$. Thus
$R_d(\tau,u_0,\gamma)<0$ for $\gamma>0$.
Every maximizer is therefore interior, and at such a maximizer
$L_d=-\gamma q_d=\gamma kz>0$.
\end{proof}
\begin{lem}[Existence, uniqueness, continuity, and endpoint behavior]
\label{lem:tau_cost_unique}
For every $\gamma\geq0$, $F(\cdot,\gamma)$ has a unique minimizer
in $[0,\infty]$. This minimizer equals infinity at $\gamma=0$ and is
finite at every $\gamma>0$. It is continuous as a function of
$\gamma\geq0$ with values in $[0,\infty]$, and therefore tends to
infinity as $\gamma\downarrow0$.
In addition, $\tau^\star(w,\gamma)=0$ whenever $\gamma>u_0$.
\end{lem}
\begin{proof}
Make the increasing change of variable
$x=2\Phi(\tau)-1\in[0,1]$, with $x=1$ corresponding to
$\tau=\infty$. The threshold rule and the rule that always continues
can differ in their terminal decisions only on the stopping event.
Hence $|p(d,\tau)-\Phi(-ad)|\leq1-q(d,\tau)$, and
\begin{equation}
\sup_{0\leq d\leq u_0}
|R(\tau,d,\gamma)-R(\infty,d,\gamma)|
\leq2(u_0+\gamma)\Phi(u_0-\tau)\longrightarrow0.
\label{eq:tau_cost_endpoint_continuity}
\end{equation}
Together with continuity at finite thresholds, this makes the
transformed objective continuous on $[0,1]$, so it attains a minimum.
Moreover, because $0\leq q(d,\tau)\leq1$,
\begin{equation}
|F(\tau,\gamma)-F(\tau,\eta)|\leq|\gamma-\eta|
\qquad(\tau\in[0,\infty],\ \gamma,\eta\geq0).
\label{eq:tau_cost_cost_lipschitz}
\end{equation}
The transformed objective is thus jointly continuous in $(x,\gamma)$,
including at $x=1$.
For $d>0$, \eqref{eq:tau_cost_convexityidentity} implies
\[
\frac{\partial^2 R}{\partial x^2}
=\frac{R_{\tau\tau}+\tau R_\tau}{4\phi(\tau)^2}>0
\qquad(0<x<1).
\]
For every fixed $d>0$ and $\gamma\geq0$, the normalized regret
$R(\tau,d,\gamma)$ is strictly convex as a function of
$x=2\Phi(\tau)-1$, including its continuous extensions at
$x=0$ and $x=1$. For $d=0$, it equals the affine function
$\gamma x$.
For any $0\leq x_1<x_2\leq1$ and $0<\theta<1$, choose a maximizing
effect at $x_\theta=(1-\theta)x_1+\theta x_2$. This effect is positive
because $R_d(\tau,0,\gamma)=1/2$. Strict convexity of its regret
function gives strict convexity of $F$ in $x$. Consequently the
minimizer is unique.
Equation \eqref{eq:tau_cost_Ltintegral} gives $J>0$ for $d>0$.
Extend $b(\tau,d)$ to $d=0$ by setting $b(\tau,0)=0$. Uniformly over
$d\in[0,u_0]$,
\[
0\leq b(\tau,d)\leq u_0\Phi(u_0/w-w\tau)\longrightarrow0.
\]
For $\gamma>0$, \eqref{eq:tau_cost_Rt} therefore implies that
$R_\tau(\tau,d,\gamma)>0$ for every $d\in[0,u_0]$ once $\tau$ is
sufficiently large. Thus $F$ is strictly increasing beyond a finite
threshold, and infinity cannot minimize it.
At zero cost, \eqref{eq:tau_cost_Ltintegral} gives $L_\tau<0$ for every $d>0$.
Every finite-threshold maximizer is positive. Evaluating at a maximizer
for the larger of any two finite thresholds shows that $F(\tau,0)$
is strictly decreasing in $\tau$. Its unique minimum is at infinity.
To prove continuity of the minimizer, let $\gamma_n\to\gamma_0\geq0$
and put
$x_n=2\Phi(\tau^\star(w,\gamma_n))-1$, with $\Phi(\infty)=1$.
Every subsequence of $(x_n)$ has a further subsequence converging
to some $\bar x\in[0,1]$.
For every fixed $x\in[0,1]$, optimality of $x_{n_k}$ gives
\[
F\left(\Phi^{-1}\left(\frac{1+x_{n_k}}{2}\right),\gamma_{n_k}\right)
\leq
F\left(\Phi^{-1}\left(\frac{1+x}{2}\right),\gamma_{n_k}\right),
\]
where $\Phi^{-1}(1)=\infty$.
Since $x_{n_k}\to\bar x$ and $\gamma_{n_k}\to\gamma_0$,
joint continuity of the transformed objective implies
\[
F\left(\Phi^{-1}\left(\frac{1+\bar x}{2}\right),\gamma_0\right)
\leq
F\left(\Phi^{-1}\left(\frac{1+x}{2}\right),\gamma_0\right).
\]
This inequality holds for every $x\in[0,1]$. Thus $\bar x$
minimizes the transformed objective at cost $\gamma_0$.
By uniqueness,
\[
\bar x=2\Phi\bigl(\tau^\star(w,\gamma_0)\bigr)-1.
\]
If $\gamma_0>0$, the limit is smaller than one, and continuity of
$x\mapsto\Phi^{-1}((1+x)/2)$ gives
$\tau^\star(w,\gamma_n)\to\tau^\star(w,\gamma_0)$.
If $\gamma_0=0$, then $x_n\to1$, which gives
$\tau^\star(w,\gamma_n)\to\infty$.
This proves continuity.
Finally, the terminal decisions under thresholds $0$ and $\tau$
can differ only when the latter rule continues. Hence
$p(d, \tau = 0)-p(d,\tau)\leq q(d,\tau)$ and
\[
R(\tau,d,\gamma)\geq R(0,d,\gamma)+(\gamma-d)q(d,\tau).
\]
By the definition of $u_0$, $F(0,\gamma)=u_0\Phi(-u_0)$.
For $\gamma>u_0$ and any $\tau\in(0,\infty]$, evaluating the last
inequality at $d=u_0$ gives
$F(\tau,\gamma)>F(0,\gamma)$ because $q(u_0, \tau)>0$.
This proves the additional assertion.
\end{proof}
\begin{lem}[Shape of the continuation benefit]\label{lem:tau_cost_bshape}
For fixed $\tau,w>0$, the positive function $d\mapsto b(\tau,d)$ is strictly
log-concave on $(0,\infty)$. In addition:
\begin{enumerate}
\item If $b_d(\tau,d)\leq0$, then $\tau d\{1+\tanh(\tau d)\}>1$.
\item If $d/w=\sqrt2$ and $\tau d\leq1$, then $b_d(\tau,d)>0$.
\end{enumerate}
\end{lem}
\begin{proof}
Put $u=w\tau$, $v=d/w$, $r=\tau d=uv$, and
\[
h(v)=\Phi(v-u)-e^{2uv}\Phi(-v-u).
\]
Changing variables in the two Gaussian integrals gives
\begin{equation}
h(v)=\int_0^\infty\phi(v-u-y)(1-e^{-2vy})\,dy>0.
\label{eq:tau_cost_h}
\end{equation}
The integrand is jointly log-concave in $(v,y)\in(0,\infty)^2$,
and remains log-concave when extended by zero outside this quadrant.
To verify the only non-Gaussian factor, let $f(s)=\log(1-e^{-s})$.
The diagonal entries of the Hessian of $f(2vy)$ are negative, and its
determinant is
\[
4\{f'(s)\}^2\left(\frac{2s}{1-e^{-s}}-1\right)>0,
\qquad s=2vy>0.
\]
For completeness, log-concavity is preserved by the integration in
\eqref{eq:tau_cost_h} for the following reason. Denote the integrand by
$g(v,y)$, fix $v_0,v_1>0$ and $0<\theta<1$, and set
$f_i(y)=g(v_i,y)/h(v_i)$ for $i=0,1$. These are positive smooth
probability densities on $(0,\infty)$. Let $T$ be their increasing
quantile transport, defined by
\[
\int_0^{T(y)} f_1(s)\,ds=\int_0^y f_0(s)\,ds.
\]
Then $T'(y)=f_0(y)/f_1(T(y))>0$, and
$Y_\theta(y)=(1-\theta)y+\theta T(y)$ is an increasing bijection of
$(0,\infty)$ onto itself. Joint log-concavity of $g$ and the change
of variables $s=Y_\theta(y)$ give
\begin{align*}
h((1-\theta)v_0+\theta v_1)
&\geq h(v_0)^{1-\theta}h(v_1)^\theta
\int_0^\infty f_0(y)
\frac{(1-\theta)+\theta T'(y)}{T'(y)^\theta}\,dy\\
&\geq h(v_0)^{1-\theta}h(v_1)^\theta.
\end{align*}
The last inequality uses
$(1-\theta)+\theta s\geq s^\theta$ for $s>0$.
Thus $h$ is log-concave.
Since
\[
\log b(\tau,d)=\log d+\log h(d/w)-\log(1+e^{2\tau d}),
\]
the strictly concave term $\log d$ makes $b$ strictly log-concave.
Moreover,
\[
h'(v)=2\phi(v-u)-2u e^{2uv}\Phi(-v-u)>0,
\]
where strict positivity follows from
$e^{2uv}\phi(v+u)=\phi(v-u)$ and
$\Phi(-v-u)<\phi(v+u)/(v+u)$.
The function $d/(1+e^{2\tau d})$ has a nonnegative derivative whenever
$r(1+\tanh r)\leq1$. Multiplying by the strictly increasing positive
factor $h(d/w)$ proves the first additional assertion.
For the second, differentiating \eqref{eq:tau_cost_bt} in $d$ gives, for $r\leq1$,
\begin{equation}
b_{\tau d}
=-d(1-\tanh^2 r)\left[
(1-r\tanh r)(P_++P_-)
+\frac{v}{2}\{\phi(v-u)-\phi(v+u)\}
\right]<0
.
\label{eq:tau_cost_btd}
\end{equation}
Hold $d,w$ fixed and increase $\tau$, so $v$ is fixed and $r=\tau d$ increases.
Equation \eqref{eq:tau_cost_btd} implies that $b_d$ decreases in $r$ on
$0<r\leq1$. Differentiating \eqref{eq:tau_cost_b} shows that $b_d$ depends only
on $(r,v)$, so it suffices to check $v=\sqrt2$ and $r=1$.
At this point put $z_1=\tanh1$ and $\delta=1/\sqrt2$. Direct
differentiation of \eqref{eq:tau_cost_b} gives
\[
(1+e^2)b_d
=-z_1\Phi(\delta)-e^2(2-z_1)\Phi(-3\delta)
+2\sqrt2\phi(\delta).
\]
By the Mills inequality and $e^2\phi(3\delta)=\phi(\delta)$, the
right-hand side is strictly larger than
\[
\frac{\sqrt2(4+z_1)}{3}\phi(\delta)-z_1\Phi(\delta).
\]
This expression decreases with $z_1$. Using $z_1<4/5$ and
$\Phi(\delta)<1/2+1/(2\sqrt\pi)$, it is strictly larger than
\[
\frac{8e^{-1/4}-2\sqrt\pi-2}{5\sqrt\pi}>0.
\]
The final inequality follows from $e^{-1/4}>3/4$ and $\pi<4$.
This proves the second assertion.
\end{proof}
\begin{lem}[Key sign restriction]\label{lem:tau_cost_key}
Fix $\tau,w>0$. If $d>0$ satisfies
\[
L_d(\tau,d)\geq0,\qquad a^2d^2>2,\qquad b_d(\tau,d)\leq0,
\]
then $G(\tau,d)<0$.
\end{lem}
\begin{proof}
Write $r=\tau d$ and $v=d/w$; thus $z=\tanh r$.
Since $ad>\sqrt2>1$, Lemma~\ref{lem:tau_cost_curvature} implies
\begin{equation}
r+d^2<1.\label{eq:tau_cost_rbound}
\end{equation}
We first show that $v^2>2$. If instead $v^2\leq2$, then
\[
v^2>2-d^2>1+r>2r^2,
\]
where the last inequality uses $0<r<1$.
Let $\bar d=\sqrt2w\geq d$. It follows that
\[
\tau\bar d=\frac{\sqrt2r}{v}<1.
\]
Lemma~\ref{lem:tau_cost_bshape} gives $b_d(\tau,\bar d)>0$.
Strict log-concavity of $b$ then gives $b_d(\tau,d)>0$ as well,
contradicting the hypothesis. Thus
\begin{equation}
v^2=(a^2-1)d^2>2.\label{eq:tau_cost_vbound}
\end{equation}
Now hold $(\tau,d)$ fixed and regard $G$ as a function of the precision
parameter $a>1$, so $w=(a^2-1)^{-1/2}$ and
$\Gamma=\tau a/\sqrt{a^2-1}$.
Differentiating the Gaussian expressions gives
\begin{align*}
\partial_a p(d,\tau)&=-dK/a,\\
K_a&=(1-a^2d^2)K/a-\frac{2a\tau C}{(a^2-1)^{3/2}},\\
S_a&=2aC\left(\frac{dz}{\sqrt{a^2-1}}
+\frac{\tau}{(a^2-1)^{3/2}}\right).
\end{align*}
For the first identity, differentiating the Gaussian integral for $p(d,\tau)$ in $w$ cancels
the odd part of the integrand and gives
\[
\partial_w p(d,\tau)=\frac{d}{w^2}\int_{-\tau}^\tau
\phi(wx+d/w)\phi(x-d)\,dx
=\frac{d\,\phi(ad)M}{w^2\sqrt{1+w^2}}.
\]
Combining this with $da/dw=-1/(aw^3)$ yields $\partial_a p(d,\tau)=-dK/a$.
Consequently,
\begin{equation}
G_a=\frac{dK}{a}\left[
a^2d^2-2-\frac{d}{\tau}z\,
\frac{2\Gamma\phi(\Gamma)}{2\Phi(\Gamma)-1}
\right].\label{eq:tau_cost_Ga}
\end{equation}
Since
\[
0<\frac{2\Gamma\phi(\Gamma)}{2\Phi(\Gamma)-1}<1,
\qquad z<\tau d,
\]
the bracket in \eqref{eq:tau_cost_Ga} is strictly larger than
$(a^2-1)d^2-2$. By \eqref{eq:tau_cost_vbound}, $G$ is strictly increasing as
$a$ increases from its current value to infinity.
Its limit is
\[
\lim_{a\to\infty}G(\tau,d)
=\Phi(-\tau-d)-d\phi(\tau+d)(1+z).
\]
The first assertion of Lemma~\ref{lem:tau_cost_bshape} gives $\tau d(1+z)>1$.
Thus the Mills inequality implies
\[
\Phi(-\tau-d)-d\phi(\tau+d)(1+z)
<\phi(\tau+d)\left(\frac1{\tau+d}-d(1+z)\right)<0.
\]
The current value of $G$ is smaller still, proving the claim.
\end{proof}
For $\tau,\gamma>0$, let
\[
\mathcal M(\tau,\gamma)=\argmax_{0\leq d\leq u_0}R(\tau,d,\gamma).
\]
This is a nonempty compact subset of $(0,u_0)$ by
Lemma~\ref{lem:tau_cost_curvature}. In particular, \eqref{eq:tau_cost_Gidentity} implies
\begin{equation}
G(\tau,d)=zR_\tau(\tau,d,\gamma)=kz\{\gamma-b(\tau,d)\},
\qquad d\in\mathcal M(\tau,\gamma).
\label{eq:tau_cost_Gactive}
\end{equation}
\begin{lem}[Zeros and ties]\label{lem:tau_cost_zeros}
Fix $\tau,\gamma>0$.
\begin{enumerate}
\item If $d\in\mathcal M(\tau,\gamma)$ and $G(\tau,d)=0$, then $G_d(\tau,d)<0$.
\item There do not exist $d_1,d_2\in\mathcal M(\tau,\gamma)$ with
$d_1<d_2$ and $G(\tau,d_2)\geq0\geq G(\tau,d_1)$.
\end{enumerate}
\end{lem}
\begin{proof}
For the first assertion, \eqref{eq:tau_cost_Gactive} gives $R_\tau=0$ and
$\gamma=b(\tau,d)$. Differentiating \eqref{eq:tau_cost_Gidentity} in $d$ at this
point yields
\begin{equation}
G_d=R_{dd}+zR_{\tau d}=R_{dd}-kzb_d.
\label{eq:tau_cost_Gdzero}
\end{equation}
If $b_d>0$, this is strictly negative since $R_{dd}\leq0$.
If $b_d\leq0$, Lemma~\ref{lem:tau_cost_key} and $G=0$ imply $a^2d^2\leq2$.
Using Lemma~\ref{lem:tau_cost_curvature} and \eqref{eq:tau_cost_qderiv},
\begin{align*}
G_d
&=L_{dd}+zL_{\tau d}+\gamma(q_{dd}(d, \tau)+z q_{d\tau}(d,\tau))\\
&=L_{dd}+zL_{\tau d}-\gamma \tau k(1-z^2)<0.
\end{align*}
This argument does not require $R_{dd}<0$.
For the second assertion, suppose such $d_1<d_2$ existed.
Equation \eqref{eq:tau_cost_Gactive} would give
\[
b(\tau,d_1)\geq\gamma\geq b(\tau,d_2).
\]
Strict log-concavity of $b$ implies
\[
\frac{b_d(\tau,d_2)}{b(\tau,d_2)}
<\frac{\log b(\tau,d_2)-\log b(\tau,d_1)}{d_2-d_1}\leq0.
\]
Thus $b_d(\tau,d_2)<0$. If $a^2d_2^2>2$, Lemma~\ref{lem:tau_cost_key}
gives $G(\tau,d_2)<0$, a contradiction.
If $a^2d_2^2\leq2$, Lemma~\ref{lem:tau_cost_curvature} gives
$L_{dd}\leq0$ throughout $[0,d_2]$. Also $q_\tau''(d)<0$ there, because
\[
\tau-d\tanh(\tau d)\geq \tau(1-d^2)>0
\qquad(0\leq d\leq d_2<u_0<1).
\]
Hence $R$ is strictly concave on $[0,d_2]$, which precludes two
distinct global maximizers $d_1,d_2$.
\end{proof}
\begin{lem}[Cost single crossing at a fixed threshold]
\label{lem:tau_cost_crossing}
Fix $\tau>0$, and select
\[
m(\gamma)=\min\mathcal M(\tau,\gamma),\qquad \gamma>0.
\]
There do not exist $0<\gamma_1<\gamma_2$ such that
\[
G(\tau,m(\gamma_1))>0>G(\tau,m(\gamma_2)).
\]
\end{lem}
\begin{proof}
Every maximizer at a higher cost is weakly smaller than every maximizer
at a lower cost. Indeed, the optimality inequalities for
$d_i\in\mathcal M(\tau,\gamma_i)$, where $\gamma_2>\gamma_1$, imply
\[
(\gamma_2-\gamma_1)\{q(d_2, \tau)-q(d_1, \tau)\}\geq0.
\]
Since $q(d,\tau)$ is strictly decreasing in $d>0$, $d_2\leq d_1$.
Continuity of $R$ and compactness of the parameter interval then imply
\begin{equation}
m(\gamma+)=\min\mathcal M(\tau,\gamma)=m(\gamma),
\qquad
m(\gamma-)=\max\mathcal M(\tau,\gamma).
\label{eq:tau_cost_limits}
\end{equation}
The left-limit statement follows in the same way.
Suppose the claimed single-crossing property failed, and set
$g(\gamma)=G(\tau,m(\gamma))$.
This function is right-continuous and has left limits by
\eqref{eq:tau_cost_limits}. Let
\[
\bar\gamma=\inf\{\gamma\in[\gamma_1,\gamma_2]:g(\gamma)\leq0\}.
\]
Then $\bar\gamma>\gamma_1$, $g(\bar\gamma)\leq0$, and
$g(\gamma)>0$ for $\gamma_1\leq\gamma<\bar\gamma$.
Write
\[
d_-=m(\bar\gamma-)=\max\mathcal M(\tau,\bar\gamma),
\qquad
d_+=m(\bar\gamma)=\min\mathcal M(\tau,\bar\gamma).
\]
We have
\[
d_-\geq d_+,\qquad G(\tau,d_-)\geq0\geq G(\tau,d_+).
\]
If $d_->d_+$, this contradicts the second assertion of
Lemma~\ref{lem:tau_cost_zeros}.
If $d_-=d_+=\bar d$, then $G(\tau,\bar d)=0$, and the first assertion
gives $G_d(\tau,\bar d)<0$.
For $\gamma<\bar\gamma$ sufficiently close to $\bar\gamma$,
$m(\gamma)\geq\bar d$ and $m(\gamma)\to\bar d$.
Local strict decrease of $G$ in $d$ therefore gives
$g(\gamma)\leq G(\tau,\bar d)=0$, again a contradiction.
\end{proof}
We now complete the proof of Part (B). Existence, uniqueness,
continuity, and all endpoint statements follow from
Lemma~\ref{lem:tau_cost_unique}.
We now prove cost monotonicity. Suppose that
$0<\gamma_1<\gamma_2$ but
$\tau_1:=\tau^\star(w,\gamma_1)
<\tau_2:=\tau^\star(w,\gamma_2)$.
Choose $\tau\in(\tau_1,\tau_2)$ and let
$d_i=\min\mathcal M(\tau,\gamma_i)$.
For each $i=1,2$, hold $d_i$ and $\gamma_i$ fixed.
By Lemma~\ref{lem:tau_cost_unique}, the function
\[
x\longmapsto
R\left(\Phi^{-1}\left(\frac{1+x}{2}\right),d_i,\gamma_i\right)
\]
is convex. Its derivative at $x=2\Phi(\tau)-1$ equals
$R_\tau(\tau,d_i,\gamma_i)/(2\phi(\tau))$.
The first-order inequality for a differentiable convex function
therefore gives
\[
R(\tau_i,d_i,\gamma_i)
\geq R(\tau,d_i,\gamma_i)
+\frac{R_\tau(\tau,d_i,\gamma_i)}{2\phi(\tau)}(x_i-x).
\]
The preceding strict comparison and $x_1<x<x_2$ therefore imply
$R_\tau(\tau,d_1,\gamma_1)>0$ and
$R_\tau(\tau,d_2,\gamma_2)<0$.
Equation \eqref{eq:tau_cost_Gactive} gives
$G(\tau,d_1)>0>G(\tau,d_2)$, contradicting
Lemma~\ref{lem:tau_cost_crossing}.
Thus $\tau^\star(w,\gamma)$ is nonincreasing for $\gamma>0$.
The comparison with zero cost follows from
$\tau^\star(w,0)=\infty$.
Together with Lemma~\ref{lem:tau_cost_unique},
this proves Part (B).
\subsection{Proof of Proposition \ref{prop:c_value_properties}}
\label{proof:prop:c_value_properties}
Let
$
t=\frac{|X_1|}{s_1}.
$
By the definitions of the $A$-value and $\Gamma_w$,
$
A_{\mathrm{value}}(X_1;s_1,w)
=
s_1\Gamma_w(t).
$
Moreover, since
$
p(t)=2\{1-\Phi(t)\},
$
and $t\geq0$, we have
$
t
=
\Phi^{-1}\left(1-\frac{p(t)}{2}\right).
$
Substitution gives
\begin{equation} \label{eqn:characterization}
A_{\mathrm{value}}(X_1;s_1,w)
=
s_1\Gamma_w\left(
\Phi^{-1}\left(
1-\frac{p(|X_1|/s_1)}{2}
\right)
\right).
\end{equation}
To establish monotonicity, consider $0\leq t_1\leq t_2$. Then
\[
\left\{
\gamma\geq0:
t_1\geq\tau^\star(w,\gamma)
\right\}
\subseteq
\left\{
\gamma\geq0:
t_2\geq\tau^\star(w,\gamma)
\right\}.
\]
Taking infima over the two sets gives
$
\Gamma_w(t_2)\leq\Gamma_w(t_1).
$
Hence, for fixed $(s_1,w)$, the $A$-value is weakly decreasing in
$|X_1|/s_1$. Because
$
p'(t)=-2\phi(t)<0,
$
it is equivalently weakly increasing in the two-sided $p$-value.
Next, holding the $p$-value and $w$ fixed, the $A$-value is proportional
to $s_1$ and is therefore weakly increasing in $s_1^2$. The increase is
strict whenever the $A$-value is positive.
The third property follows directly from Equation \eqref{eqn:characterization}.
We conclude by proving the fourth property.
Fix $w>0$, a finite $t\geq0$, and $\gamma\geq0$.
Recall that
\[
\Gamma_w(t)
=
\inf\{\eta\geq0:t\geq\tau^\star(w,\eta)\},
\]
with the convention $\inf\varnothing=\infty$.
If $t\geq\tau^\star(w,\gamma)$, then $\gamma$ belongs to the
set defining the infimum, and therefore
$\Gamma_w(t)\leq\gamma$.
Conversely, suppose that $t<\tau^\star(w,\gamma)$.
By Proposition \ref{cor:1}(B), there exists $\varepsilon>0$
such that
$
t<\tau^\star(w,\gamma+\varepsilon).
$
For $\gamma>0$, this follows from continuity at $\gamma$.
For $\gamma=0$, it follows from
$\tau^\star(w,\eta)\to\infty$ as $\eta\downarrow0$.
Monotonicity then implies that, for every
$0\leq\eta\leq\gamma+\varepsilon$,
$
\tau^\star(w,\eta)
\geq\tau^\star(w,\gamma+\varepsilon)>t.
$
Thus no such $\eta$ belongs to the set defining $\Gamma_w(t)$,
and
$
\Gamma_w(t)\geq\gamma+\varepsilon>\gamma.
$
Combining the two implications gives
$
t<\tau^\star(w,\gamma)$ if and only if $\gamma<\Gamma_w(t).
$
Taking $t=|X_1|/s_1$ and $\gamma=c/s_1$, and using
$A_{\mathrm{value}}(X_1;s_1,w)=s_1\Gamma_w(t)$,
proves Part (4).
\subsection{Proof of Theorem \ref{prop:top_q_c_value}}
\label{proof:prop:top_q_c_value}
We break the proof into several steps.
\paragraph{A one-policy bound}
For each policy $j$, let
\[
r_j^\star(c)
=
\inf_{(\delta_{1j},\delta_{2j})}
\sup_{|\Delta_j|\leq\bar\Delta_j}
\mathrm{Reg}_1
(\delta_{1j},\delta_{2j};\Delta_j,c)
\]
denote the one-policy minimax-regret value when the continuation cost is
$c$. When $c=0$, Theorem \ref{thm:dead_zone} implies that it is optimal
to collect $X_{2j}$ after every realization of $X_{1j}$. Using this rule
when the continuation cost is $c$ gives
\begin{equation}
\label{eqn:a1}
r_j^\star(c)
\leq
r_j^\star(0)+c.
\end{equation}
We next establish the implication of Equation \eqref{eqn:a1} for the
$A$-value. Fix $\lambda\geq0$ and consider the one-policy rule that
collects $X_{2j}$ when $A_j\geq\lambda$, makes an immediate decision
based on the sign of $X_{1j}$ otherwise, and uses the terminal rule in
Theorem \ref{thm:dead_zone} after continuation. Denote this rule by
\[
\delta_{1j}(X_{1j};\lambda)
=
\begin{cases}
\texttt{NA},
& \text{if } A_j\geq\lambda,\\
1,
& \text{if } A_j<\lambda \text{ and } X_{1j}\geq0,\\
0,
& \text{otherwise}.
\end{cases}, \qquad
\delta_{2j}^\star(X_{1j},X_{2j})
=
1\left\{
\frac{s_{2j}^2}{s_{1j}^2}X_{1j}+X_{2j}\geq0
\right\}.
\]
Write $w_j=s_{2j}/s_{1j}$. By the definition of the $A$-value, recall
from Proposition \ref{prop:c_value_properties} that
$
A_j
=
s_{1j}\Gamma_{w_j}
\left(
\frac{|X_{1j}|}{s_{1j}}
\right).
$
Fix $\lambda>0$ and write
\[
t_j=\frac{|X_{1j}|}{s_{1j}},
\qquad
\bar\tau_j(\lambda)
=
\inf_{0\leq\eta<\lambda/s_{1j}}
\tau^\star(w_j,\eta).
\]
By the definition of $\Gamma_{w_j}$,
$
\left\{
\Gamma_{w_j}(t_j)\geq\frac{\lambda}{s_{1j}}
\right\}
=
\bigcap_{0\leq\eta<\lambda/s_{1j}}
\left\{
t_j<\tau^\star(w_j,\eta)
\right\}.
$
It follows that
\[
\left\{t_j<\bar\tau_j(\lambda)\right\}
\subseteq
\left\{A_j\geq\lambda\right\}
\subseteq
\left\{t_j\leq\bar\tau_j(\lambda)\right\}.
\]
The two outer events differ only when
$t_j=\bar\tau_j(\lambda)$. Because $t_j$ has a continuous distribution
under every $\Delta_j$,
\[
\mathbb P_{\Delta_j}\left(
\{A_j\geq\lambda\}
\setminus
\left\{
t_j<\bar\tau_j(\lambda)
\right\}
\right)
=0.
\]
Consequently, for every $\Delta_j$, the rule that continues when
$A_j\geq\lambda$ has the same regret as the threshold rule in
Theorem \ref{thm:dead_zone} with threshold $\bar\tau_j(\lambda)$.
Although $\bar\tau_j(\lambda)$ need not itself be an optimal threshold
at any particular continuation cost, it is a limit of optimal
thresholds associated with costs strictly below $\lambda$.
Taking limits
yields\footnote{
If $\bar\tau_j(\lambda)<\infty$, the definition of the infimum implies
that there exist $0\leq\eta_n<\lambda/s_{1j}$ such that
$\tau_n\equiv\tau^\star(w_j,\eta_n)\rightarrow
\bar\tau_j(\lambda)$. Let $c_n=s_{1j}\eta_n<\lambda$. The rule with
threshold $\tau_n$ is minimax at cost $c_n$. Therefore, for every
$|\Delta_j|\leq\bar\Delta_j$,
\[
\begin{aligned}
\mathrm{Reg}_1
(\delta_{1j}^{\tau_n},\delta_{2j}^\star;\Delta_j,0)
=
\mathrm{Reg}_1
(\delta_{1j}^{\tau_n},\delta_{2j}^\star;\Delta_j,c_n)
-
c_n\mathbb P_{\Delta_j}(t_j<\tau_n)
& \leq
r_j^\star(c_n)
-
c_n\mathbb P_{\Delta_j}(t_j<\tau_n)
\leq
r_j^\star(0)
+
c_n\mathbb P_{\Delta_j}(t_j\geq\tau_n)
\\ & \leq
r_j^\star(0)
+
\lambda\mathbb P_{\Delta_j}(t_j\geq\tau_n).
\end{aligned}
\]
For fixed $\Delta_j$, the zero-cost regret and stopping probability
are continuous in $\tau$. Taking $n\to\infty$ and using the
almost-sure equivalence between the $A$-value rule and the rule with
threshold $\bar\tau_j(\lambda)$ proves the desired inequality.
If $\bar\tau_j(\lambda)=\infty$, the $A$-value rule always continues,
and its regret is bounded by $r_j^\star(0)$.
}
\begin{equation}
\label{eqn:a2}
\mathrm{Reg}_1
(\delta_{1j}(\,\cdot\,;\lambda),
\delta_{2j}^{\star};
\Delta_j,0)
\leq
r_j^\star(0)
+
\lambda\mathbb P_{\Delta_j}(A_j<\lambda).
\end{equation}
When $\lambda=0$, the $A$-value rule always continues, so the same
inequality holds directly.
We will also use the version of Equation \eqref{eqn:a2} for the rule
that continues when $A_j>\lambda$. Applying
Equation \eqref{eqn:a2} with cutoff $\lambda+1/n$ and taking
$n\to\infty$, dominated convergence gives the same bound with
$\mathbb P_{\Delta_j}(A_j\leq\lambda)$ in place of
$\mathbb P_{\Delta_j}(A_j<\lambda)$. Thus, under either convention,
the additional term is $\lambda$ times the probability of making
an immediate decision.
\paragraph{Relaxed problem}
Consider the relaxed problem in which all follow-up signals
$\mathbf X_2=(X_{21},\ldots,X_{2J})$ are observed at zero cost and the
terminal action for each policy may depend on
$(\mathbf X_1,\mathbf X_2)$. By definition, the minimax value of this
problem is $R_J^\star$. We claim that
\begin{equation}
\label{eqn:a3}
R_J^\star
=
\frac{1}{J}\sum_{j=1}^J r_j^\star(0).
\end{equation}
For the upper bound, apply the one-policy minimax rule in
Theorem \ref{thm:dead_zone} with zero continuation cost separately
to each policy. Rectangularity of the parameter space then gives
\[
R_J^\star
\leq
\sup_{\boldsymbol\Delta}
\frac{1}{J}\sum_{j=1}^J
\mathrm{Reg}_1
(\delta_{1j}^\star,\delta_{2j}^\star;\Delta_j,0)
=
\frac{1}{J}\sum_{j=1}^J r_j^\star(0).
\]
For the reverse inequality, let $\pi_j^\star$ be a least-favorable
prior for the zero-cost one-policy problem for policy $j$, and
consider the product prior
$
\pi^\star
=
\bigotimes_{j=1}^J\pi_j^\star.
$
By independence of $(X_{1j},X_{2j})$ across policies $j$,
\[
\pi^\star
\bigl(
d\boldsymbol\Delta
\mid
\mathbf X_1,\mathbf X_2
\bigr)
=
\bigotimes_{j=1}^J
\pi_j^\star
\bigl(
d\Delta_j
\mid
X_{1j},X_{2j}
\bigr).
\]
Consequently, conditional on $(\mathbf X_1,\mathbf X_2)$, the posterior
expected regret from action $a_j\in\{0,1\}$ is
\[
\bigl(1-a_j\bigr)
\mathbb E_{\pi_j^\star}
\left[
\Delta_{j,+}\mid X_{1j},X_{2j}
\right]
+
a_j
\mathbb E_{\pi_j^\star}
\left[
\Delta_{j,-}\mid X_{1j},X_{2j}
\right],
\]
which depends on the data only through $(X_{1j},X_{2j})$. The Bayes
decision problem therefore separates across policies, and its Bayes
risk under $\pi^\star$ is
$
\frac{1}{J}\sum_{j=1}^J r_j^\star(0).
$
Since the supremum risk of any rule is at least its Bayes risk under
$\pi^\star$,
$
R_J^\star
\geq
\frac{1}{J}\sum_{j=1}^J r_j^\star(0),
$
which proves the claim.
\paragraph{Capacity constrained problem}
We next bound the regret of the top-$q$ $A$-value rule. Fix
$\boldsymbol\Delta\in
\prod_{j=1}^J[-\bar\Delta_j,\bar\Delta_j]$.
For each policy $j$, define its realized regret by
\[
\begin{aligned}
L_j^{A,q}
={}&
\Delta_{j,+}
1\{\delta_{1j}^{A,q}=0\}
+
\Delta_{j,-}
1\{\delta_{1j}^{A,q}=1\}
+
1\{\delta_{1j}^{A,q}=\texttt{NA}\}
\left[
\Delta_{j,+}1\{\delta_{2j}^\star=0\}
+
\Delta_{j,-}1\{\delta_{2j}^\star=1\}
\right].
\end{aligned}
\]
Thus,
\[
\mathrm{Reg}
(\boldsymbol\delta^{A,q};\boldsymbol\Delta)
=
\frac{1}{J}\sum_{j=1}^J
\mathbb E_{\boldsymbol\Delta}[L_j^{A,q}].
\]
Fix policy $j$, and let $\lambda_j$ be the $q$-th largest $A$-value
among the remaining $J-1$ policies, using the same fixed ordering to
resolve ties as in the top-$q$ procedure. Then $\lambda_j$ is measurable
with respect to $\mathbf X_{1,-j}$. Moreover,
\[
\{A_j>\lambda_j\}
\subseteq
\{j\in\mathcal J_q(\mathbf X_1)\}
\subseteq
\{A_j\geq\lambda_j\}.
\]
Conditional on $\mathbf X_{1,-j}$, the value of $\lambda_j$ is fixed
and $(X_{1j},X_{2j})$ has its marginal distribution under $\Delta_j$,
independently of $\mathbf X_{1,-j}$. The conditional rule for policy
$j$ therefore coincides with the one-policy $A$-value rule with cutoff
$\lambda_j$. Applying
Equation \eqref{eqn:a2}
\[
\begin{aligned}
\mathbb E_{\boldsymbol\Delta}
\left[
L_j^{A,q}
\mid
\mathbf X_{1,-j}
\right]
\leq{}&
r_j^\star(0) +
\lambda_j
\mathbb P_{\boldsymbol\Delta}
\left(
j\notin\mathcal J_q(\mathbf X_1)
\mid
\mathbf X_{1,-j}
\right)
\end{aligned}
\]
$\mathbb P_{\boldsymbol\Delta}$-almost surely.
Taking expectations and summing over $j$ gives
\[
\begin{aligned}
\mathrm{Reg}
(\boldsymbol\delta^{A,q};\boldsymbol\Delta)
\leq{}&
\frac{1}{J}\sum_{j=1}^J r_j^\star(0) +
\mathbb E_{\boldsymbol\Delta}
\left[
\frac{1}{J}\sum_{j=1}^J
\lambda_j
1\{j\notin\mathcal J_q(\mathbf X_1)\}
\right].
\end{aligned}
\]
For every realization of $\mathbf X_1$, the definition of $\lambda_j$
implies that $\lambda_j=A_{(q)}$ whenever
$j\notin\mathcal J_q(\mathbf X_1)$. Because
$|\mathcal J_q(\mathbf X_1)|=q$,
\[
\frac{1}{J}\sum_{j=1}^J
\lambda_j
1\{j\notin\mathcal J_q(\mathbf X_1)\}
=
\frac{J-q}{J}A_{(q)}.
\]
\paragraph{Maximization of expectation}
It remains to show that the expectation of $A_{(q)}$ is maximized
under $\boldsymbol\Delta=\mathbf0$. For
$
d_j=\frac{|\Delta_j|}{s_{1j}},
$
and every $t\geq0$,
\[
\mathbb P_{\Delta_j}
\left(
\frac{|X_{1j}|}{s_{1j}}\leq t
\right)
=
\Phi(t-d_j)-\Phi(-t-d_j).
\]
The right-hand side is nonincreasing in $d_j$, since its derivative is
$
-\phi(t-d_j)+\phi(t+d_j)\leq0.
$
Therefore, $|X_{1j}|/s_{1j}$ is stochastically smallest when
$\Delta_j=0$. Proposition \ref{prop:c_value_properties} shows that
$A_j$ is weakly decreasing in $|X_{1j}|/s_{1j}$. Thus, $A_j$ is
stochastically largest under $\Delta_j=0$. Independence across policies
and the fact that the $q$-th largest order statistic is increasing in
each coordinate imply
$
\sup_{\boldsymbol\Delta\in
\prod_{j=1}^J[-\bar\Delta_j,\bar\Delta_j]}
\mathbb E_{\boldsymbol\Delta}[A_{(q)}]
=
\mathbb E_{\boldsymbol\Delta=\mathbf0}[A_{(q)}].
$
\subsection{Proof of Theorem \ref{thm:multi_period_dead_zone}}
\label{proof:multi_period_dead_zone}
We break the proof into several steps.
\paragraph{Existence and worst-case prior}
The case $\bar\Delta=0$ is immediate, so suppose that
$\bar\Delta>0$. Temporarily enlarge the class of admissible decisions $\boldsymbol{\delta} \in \mathcal D_H$ to include
randomized behavioral strategies.
Our setting satisfies Assumptions 3.1--3.7 of
\citet{wald1950statistical}. Indeed, the Gaussian experiment is
absolutely continuous and dominated; the parameter space
$[-\bar\Delta,\bar\Delta]$ is compact; the distribution of
$(X_1,\ldots,X_H)$ is continuous in $\Delta$ in total variation; and
regret is bounded and continuous in $\Delta$, uniformly over decision
rules. Moreover, the horizon and action sets are finite, and
the class of randomized strategies is convex and closed.
Therefore, \citet[Theorem 3.4]{wald1950statistical} implies that
\[
\begin{aligned}
V
&:=
\inf_{\boldsymbol\delta\in\mathcal D_H}
\sup_{|\Delta|\leq\bar\Delta}
\mathrm{Reg}_H
(\boldsymbol\delta;\Delta,\boldsymbol c) =
\sup_{\pi\in\mathcal P([-\bar\Delta,\bar\Delta])}
\inf_{\boldsymbol\delta\in\mathcal D_H}
\int
\mathrm{Reg}_H
(\boldsymbol\delta;\Delta,\boldsymbol c)
\,\pi(d\Delta).
\end{aligned}
\]
By \citet[Theorems 3.7 and 3.14]{wald1950statistical}, respectively,
there exist a minimax strategy and a
least-favorable prior $\pi^\star$. In particular, defining
\[
b(\pi)
=
\inf_{\boldsymbol\delta\in\mathcal D_H}
\int
\mathrm{Reg}_H
(\boldsymbol\delta;\Delta,\boldsymbol c)
\,\pi(d\Delta),
\]
we have
\[
b(\pi^\star)=V, \qquad \pi^\star
\in
\operatorname*{arg\,max}_{\pi\in
\mathcal P([-\bar\Delta,\bar\Delta])}
b(\pi)
\]
\paragraph{Symmetric worst-case prior}
We may take the least-favorable prior to be symmetric. For any prior
$\pi$, define its reflection around zero by
\[
\pi^-(A)=\pi(-A),
\qquad
-A=\{-\Delta:\Delta\in A\},
\]
for every Borel set
$A\subseteq[-\bar\Delta,\bar\Delta]$. For any strategy
$\boldsymbol\delta$, define its reflected strategy by reversing the
signs of all observed signals, interchanging terminal actions $0$ and
$1$, and leaving continuation unchanged. Formally, for $h\leq H$,
\[
\delta_h^-(x_1,\ldots,x_h)
=
\begin{cases}
1-\delta_h(-x_1,\ldots,-x_h),
&
\delta_h(-x_1,\ldots,-x_h)\in\{0,1\},\\
\texttt{NA},
&
\delta_h(-x_1,\ldots,-x_h)=\texttt{NA}.
\end{cases}
\]
Because the distribution of
$(-X_1,\ldots,-X_H)$ under $\Delta$ is the same as the distribution
of $(X_1,\ldots,X_H)$ under $-\Delta$, while the continuation costs
are unchanged,
\[
\mathrm{Reg}_H
(\boldsymbol\delta^-;\Delta,\boldsymbol c)
=
\mathrm{Reg}_H
(\boldsymbol\delta;-\Delta,\boldsymbol c).
\]
It follows that
\[
b(\pi^-)=b(\pi).
\]
The function $b(\pi)$ is concave in $\pi$, because it is the infimum
over strategies of functions that are linear in $\pi$. Hence, if
$\pi^\star$ is least favorable, then
\[
b\left(
\frac{\pi^\star+(\pi^\star)^-}{2}
\right)
\geq
\frac{1}{2}b(\pi^\star)
+
\frac{1}{2}b((\pi^\star)^-)
=
V.
\]
Since $V$ is the largest attainable Bayes risk, equality must hold.
Thus, the symmetrized prior
\[
\frac{\pi^\star+(\pi^\star)^-}{2}
\]
is also least favorable, and we may take $\pi^\star$ to be symmetric.
\paragraph{Nondegeneracy}
The symmetric least-favorable prior is nondegenerate. To establish
this, fix $a\in(0,\bar\Delta]$ and consider the symmetric two-point
prior $\pi(a) = \pi(-a) = 1/2$.
Consider the more informative experiment in which all $H$ signals are
observed at zero cost. Under $\Delta=\pm a$,
\[
T_H\sim\mathcal N(\pm aS_H,1).
\]
The likelihood ratio between $\Delta=a$ and $\Delta=-a$ is increasing
in $T_H$, so the Bayes terminal decision under $\pi_a$ implements the
policy if and only if $T_H\geq0$. Its probability of error under
either parameter value is $\Phi(-aS_H)$. Since an incorrect terminal
decision has regret $a$, the Bayes regret in this more informative,
costless experiment is
$
a\Phi(-aS_H)>0.
$
The original sequential problem cannot have smaller Bayes regret.
Therefore,
\[
b(\pi_a)
\geq
a\Phi(-aS_H)
>0,
\]
and consequently
\[
V
=
\sup_{\pi}b(\pi)
\geq
b(\pi_a)
>0.
\]
By contrast, if $\delta_0$ denotes the point mass at $\Delta=0$, then
$b(\delta_0)=0$: either immediate terminal action has zero regret, and
the planner can avoid all continuation costs by stopping in the first
period. Hence a least-favorable prior cannot equal $\delta_0$. Because
$\pi^\star$ is symmetric, it follows that $\pi^\star$ is
nondegenerate.
\paragraph{Bayes characterization of the minimax strategy}
Taking $\boldsymbol\delta^M$ to be a minimax strategy and $\pi^\star$ is
least favorable,
\[
\begin{aligned}
V
=
b(\pi^\star)
&\leq
\int
\mathrm{Reg}_H
(\boldsymbol\delta^M;\Delta,\boldsymbol c)
\,\pi^\star(d\Delta) \leq
\sup_{|\Delta|\leq\bar\Delta}
\mathrm{Reg}_H
(\boldsymbol\delta^M;\Delta,\boldsymbol c)
=
V.
\end{aligned}
\]
Both inequalities must therefore hold with equality. In particular,
$\boldsymbol\delta^M$ attains the minimum Bayes risk under
$\pi^\star$. Hence the minimax strategy is Bayes under a symmetric,
nondegenerate least-favorable prior. It remains to characterize its
Bayes form.
\paragraph{Posterior policy decision}
Define
\[
Y_h
=
\sum_{r=1}^h\frac{X_r}{s_r^2}
=
S_hT_h.
\]
The likelihood of $(X_1,\ldots,X_h)$ satisfies
\[
\begin{aligned}
\prod_{r=1}^h
\exp\left\{
-\frac{(X_r-\Delta)^2}{2s_r^2}
\right\}
&\propto
\exp\left\{
\Delta\sum_{r=1}^h\frac{X_r}{s_r^2}
-
\frac{\Delta^2}{2}
\sum_{r=1}^h\frac{1}{s_r^2}
\right\} =
\exp\left\{
Y_h\Delta-\frac{S_h^2}{2}\Delta^2
\right\}.
\end{aligned}
\]
Hence, $Y_h$ is sufficient for $\Delta$. Define
\begin{equation} \label{eqn:M_h}
M_h(y)
=
\int_{-\bar\Delta}^{\bar\Delta}
\exp\left\{
y\Delta-\frac{S_h^2}{2}\Delta^2
\right\}
\pi^\star(d\Delta).
\end{equation}
The posterior distribution after period $h$ is
\[
\pi^\star(d\Delta\mid Y_h=y)
=
\frac{
\exp\left\{
y\Delta-\frac{S_h^2}{2}\Delta^2
\right\}
}{
M_h(y)
}
\pi^\star(d\Delta).
\]
Its posterior mean is
\[
\mu_h(y)
=
\mathbb E_{\pi^\star}[\Delta\mid Y_h=y]
=
\frac{M_h'(y)}{M_h(y)}.
\]
Symmetry of $\pi^\star$ implies that $M_h$ is even. Therefore,
$M_h'$ and $\mu_h$ are odd, and in particular $\mu_h(0)=0$.
Moreover, because $\Delta$ has bounded support, differentiation under
the integral is justified and
\[
\begin{aligned}
\mu_h'(y)
&=
\frac{M_h''(y)}{M_h(y)}
-
\left(
\frac{M_h'(y)}{M_h(y)}
\right)^2 =
\mathbb E_{\pi^\star}
[\Delta^2\mid Y_h=y]
-
\left(
\mathbb E_{\pi^\star}
[\Delta\mid Y_h=y]
\right)^2 =
\operatorname{Var}_{\pi^\star}
(\Delta\mid Y_h=y)
>0.
\end{aligned}
\]
The final inequality follows because the posterior is obtained from
the nondegenerate prior $\pi^\star$ through a strictly positive
exponential tilt and therefore has the same support. Thus, $\mu_h$ is
strictly increasing. Together with oddness, this gives
\[
\operatorname{sign}\{\mu_h(y)\}
=
\operatorname{sign}(y)
=
\operatorname{sign}(T_h).
\]
If the planner stops in period $h$, the posterior regret from rejecting
the policy is
$
\mathbb E_{\pi^\star}[\Delta_+\mid Y_h=y],
$
whereas the posterior regret from implementing it is
$
\mathbb E_{\pi^\star}[\Delta_-\mid Y_h=y].
$
Their difference is
\[
\mathbb E_{\pi^\star}
[\Delta_+-\Delta_-\mid Y_h=y]
=
\mathbb E_{\pi^\star}[\Delta\mid Y_h=y]
=
\mu_h(y).
\]
Consequently, the Bayes terminal action is
\[
1\{\mu_h(Y_h)\geq0\}
=
1\{Y_h\geq0\}
=
1\{T_h\geq0\}.
\]
At $Y_h=0$, the two actions have the same posterior regret, and the
display adopts implementation as the tie-breaking convention.(Since
$Y_h$ has a continuous distribution under every $\Delta$, this
convention does not affect frequentist regret.)
\paragraph{Continuation decision}
Let
\[
r_h(y)
=
\min\left\{
\mathbb E_{\pi^\star}[\Delta_+\mid Y_h=y],
\mathbb E_{\pi^\star}[\Delta_-\mid Y_h=y]
\right\}
\]
denote the posterior regret from stopping in period $h$. Excluding
costs already incurred, define the Bayes value functions recursively
by
\[
V_H(y)=r_H(y), \qquad V_h(y)
=
\min\left\{
r_h(y),\,
c_h+
\mathbb E_{\pi^\star}
\left[
V_{h+1}(Y_{h+1})
\mid Y_h=y
\right]
\right\}, \quad h < H.
\]
Continuation is optimal in period $h$ if and only if
$A_h(y)>0$, where
\[
A_h(y)
=
r_h(y)-c_h-
\mathbb E_{\pi^\star}
\left[
V_{h+1}(Y_{h+1})
\mid Y_h=y
\right].
\]
We show by backward induction that $A_h$ is even and strictly
decreasing in $|y|$.
\paragraph{Posterior-predictive ordering}
Recall he expression of $M_h(y)$ in Equation \eqref{eqn:M_h}.
Conditional on $\Delta$,
\[
Y_{h+1}-Y_h
=
\frac{X_{h+1}}{s_{h+1}^2}
\sim
\mathcal N\left(
\frac{\Delta}{s_{h+1}^2},
\frac{1}{s_{h+1}^2}
\right).
\]
Let $K_h(y,y')$ denote the posterior-predictive density of
$Y_{h+1}=y'$ conditional on $Y_h=y$. Integrating the preceding
Gaussian density with respect to the posterior distribution of
$\Delta$ gives
\[
K_h(y,y')
=
\phi_{h+1}(y'-y)
\frac{M_{h+1}(y')}{M_h(y)},
\]
where $\phi_{h+1}$ is the density of
$\mathcal N(0,s_{h+1}^{-2})$. For $y_2>y_1$,
\[
\begin{aligned}
\frac{K_h(y_2,y')}{K_h(y_1,y')}
=
\frac{M_h(y_1)}{M_h(y_2)}
\exp\left\{
-\frac{s_{h+1}^2}{2}(y_2^2-y_1^2)
+
s_{h+1}^2(y_2-y_1)y'
\right\}.
\end{aligned}
\]
This likelihood ratio is strictly increasing in $y'$. Therefore,
$Y_{h+1}\mid Y_h=y$ is strictly stochastically increasing in $y$. In addition, symmetry of $\pi^\star$ implies that $M_h$ is even and
$
K_h(-y,-y')=K_h(y,y').
$
For $x\geq0$ and $y\geq0$, the conditional density of
$|Y_{h+1}|=x$ is
\[
\widetilde K_h(y,x)
=
K_h(y,x)+K_h(y,-x) \propto \frac{M_{h+1}(x)}{M_h(y)}
\exp\left\{
-\frac{s_{h+1}^2}{2}(x^2+y^2)
\right\}
\cosh(s_{h+1}^2xy),
\]
where $\cosh(z) = e^z/2 + e^{-z}/2$.
Hence, if $0\leq y_1<y_2$,
\[
\frac{\widetilde K_h(y_2,x)}
{\widetilde K_h(y_1,x)}
=
\frac{M_h(y_1)}{M_h(y_2)}
\exp\left\{
-\frac{s_{h+1}^2}{2}(y_2^2-y_1^2)
\right\}
\frac{
\cosh(s_{h+1}^2xy_2)
}{
\cosh(s_{h+1}^2xy_1)
}.
\]
The right-hand side is increasing in $x\geq0$. Thus,
$|Y_{h+1}|\mid Y_h=y$ is stochastically increasing in
$y\geq0$.
\paragraph{Backward induction}
Set $A_H\equiv0$. Suppose that $A_{h+1}$ is even and nonincreasing in
$|y|$, which holds for $h+1=H$. From the definitions of $V_{h+1}$ and
$A_{h+1}$,
\[
V_{h+1}(y)
=
r_{h+1}(y)-(A_{h+1}(y))_+.
\]
Therefore,
\begin{equation}
\label{eqn:a7}
\begin{aligned}
A_h(y)
={}&
r_h(y)-c_h-
\mathbb E_{\pi^\star}
\left[
r_{h+1}(Y_{h+1})
\mid Y_h=y
\right] +
\mathbb E_{\pi^\star}
\left[
(A_{h+1}(Y_{h+1}))_+
\mid Y_h=y
\right].
\end{aligned}
\end{equation}
Posterior symmetry implies that $r_h$ is even. Together with
$K_h(-y,-y')=K_h(y,y')$ and the induction hypothesis, this shows that
$A_h$ is even.
It remains to establish monotonicity for $y\geq0$. For every $y$,
\[
r_h(y)
=
\frac{1}{2}
\left\{
\mathbb E_{\pi^\star}
[|\Delta|\mid Y_h=y]
-
|\mu_h(y)|
\right\}.
\]
When $y\geq0$, $\mu_h(y)\geq0$. By the law of iterated expectation applied to the posterior mean under $\pi^\star$, which implies
$
\mathbb E_{\pi^\star}
\left[
\mu_{h+1}(Y_{h+1})
\mid Y_h=y
\right]
=
\mu_h(y)
$,
we have
\begin{equation}
\label{eqn:a8}
\begin{aligned}
&r_h(y)-
\mathbb E_{\pi^\star}
\left[
r_{h+1}(Y_{h+1})
\mid Y_h=y
\right] =
\mathbb E_{\pi^\star}
\left[
\bigl(-\mu_{h+1}(Y_{h+1})\bigr)_+
\mid Y_h=y
\right].
\end{aligned}
\end{equation}
Because $\mu_{h+1}$ is strictly increasing,
$y'\mapsto(-\mu_{h+1}(y'))_+$ is nonconstant and nonincreasing.
The strict stochastic ordering of
$Y_{h+1}\mid Y_h=y$ therefore implies that the sum of the first three
terms in \eqref{eqn:a7} is strictly decreasing in $y\geq0$.
By the induction hypothesis,
$y'\mapsto(A_{h+1}(y'))_+$ is even and nonincreasing in $|y'|$.
The stochastic ordering of
$|Y_{h+1}|\mid Y_h=y$ therefore implies that the final term in
\eqref{eqn:a7} is nonincreasing in $y\geq0$. Consequently, $A_h$ is
even and strictly decreasing in $|y|$, completing the induction.
\paragraph{Threshold representation}
The functions $r_h$, $V_h$, and $A_h$ are continuous by backward
induction. Because $A_h$ is even and strictly decreasing in $|y|$,
there exists $b_h^\star\in[0,\infty]$ such that
$
A_h(y)>0$ if and only if $
|y|<b_h^\star.$
Set $b_H^\star=0$ and define
\[
\tau_h^\star
=
\frac{b_h^\star}{S_h},
\qquad h=1,\ldots,H.
\]
Since $Y_h=S_hT_h$, the Bayes rule continues in period $h<H$ if and
only if
$
|T_h|<\tau_h^\star.
$
Upon stopping, the posterior policy decision derived above implements
the policy if and only if $T_h\geq0$. The Bayes action is unique except when $Y_h=0$ or
$|Y_h|=b_h^\star$. Each of these events has probability zero under
every $\Delta$. Since the minimax strategy
$\boldsymbol\delta^M$ is Bayes under $\pi^\star$, it agrees almost
surely, under every $\Delta$, with $\boldsymbol{\delta}^\star$.
\paragraph{Characterization of the optimal thresholds}
For a symmetric threshold vector $\boldsymbol\tau$ and $\Delta\geq0$,
regret is incurred if the terminal statistic is negative and whenever
an additional signal is collected. Therefore,
\[
\mathrm{Reg}_H
(\boldsymbol\delta^{\boldsymbol\tau};
\Delta,\boldsymbol c)
=
\Delta\,
\mathbb P_\Delta
\left(
T_{\sigma(\boldsymbol\tau)}<0
\right)
+
\sum_{h=1}^{H-1}
c_h\,
\mathbb P_\Delta
\left(
\sigma(\boldsymbol\tau)>h
\right)
=
R_H(\boldsymbol\tau;\Delta,\boldsymbol c).
\]
Reflection symmetry implies that the worst-case regret over
$[-\bar\Delta,\bar\Delta]$ equals the maximum of this expression over
$[0,\bar\Delta]$. Since
$\boldsymbol\delta^{\star}$ is minimax,
$
\boldsymbol\tau^\star
\in
\operatorname*{arg\,min}_{\boldsymbol\tau\in\mathcal T_H}
\max_{\Delta\in[0,\bar\Delta]}
R_H(\boldsymbol\tau;\Delta,\boldsymbol c).
$