Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
85,226 characters · 21 sections · 71 citation commands
Valuing Winners: When and How to Correct for Selection Bias in Randomized Experiments
\thispagestyle{empty}
Keywords: A/B Testing, Selection Bias, Field Experiments, Empirical Likelihood, Randomized Control Trials
\setcounter{page}{1}
A common task for decision-makers after an experiment (e.g., randomized-controlled trial or A/B test) is to decide whether to implement the best-performing policy going forward. This decision usually appears in one of two variants. In “champion-challenger” experiments, an existing treatment (the champion) is already implemented and is used as a default treatment or control condition to determine if switching to the winner (the challenger) will provide enough gain simester2020efficiently. In “pick-the-best” experiments, there is no pre-existing default treatment to compare to, and the goal is to implement the best treatment among those tested Feit2019.
In both scenarios, a common approach is to compute the average outcome for each treatment, and then pick the one with the highest outcome for implementation. This approach, however, involves post-selection inference and leads to what is called the winner's curse andrews2024inference, xu2025 or the optimizer's curse SmithWinkler2006: Making a prediction about future performance based on the observed average outcome is upwards biased because of selection. Intuitively, one does not know if the best performing treatment has a high average outcome in the data because it was lucky in the chosen sample (i.e., due to noise), or because it is truly better on average than other treatments in the population.
The literature proposes multiple solutions to this problem. These include sample splitting, conditional inference andrews2024inference and resampling methods such as bootstrap-based methods xu2025. Some of this research focuses on finite-sample bias reduction, while some focuses on inference, i.e., computing confidence intervals. Most of these papers, however, do not consider the decision maker's objective and whether their winner's curse correction approach supports the goal of the decision maker.
In this paper, we study how the winner's curse affects decision-makers who rely on experimental estimates to choose among treatments. We distinguish three goals a decision-maker may have: (1) Precisely predicting how a specific winning policy will perform if deployed, (2) Estimating how a specific winning policy will perform compared to the truly best treatment, and (3) Minimizing regret with respect to the performance of the truly best arm. We call the bias arising from the first goal the selective winner's curse, which is the difference between the na\"{i}ve estimate and the selected treatment's true mean. We call the bias from the second goal the global winner's curse, which is the difference between the na\"{i}ve estimate and the true mean of the best treatment. We then make the connection between these two concepts to the concept of regret in decision-theoretic analysis.
Our paper makes four contributions. First, we derive an identity linking all three quantities in (ref) and show that this connection has practical consequences: methods that perform well on one objective can perform poorly on another. Second, for each objective, we analyze how proposed corrections perform and what their implications are for decision-makers. We also show that in some cases, too much of a good thing can hurt the decision-maker: In the champion-challenger scenario, for example, it is sometimes important to know if the winner outperforms the control to make a decision. It is well-known that commonly used bias correction techniques increase the variance of the estimate due to the bias-variance tradeoff (e.g., bootstrap bias correction efron1993bootstrap). As a result, for champion-challenger experiments, bias correction might increase the variance of the estimate so much that one can no longer confidently say if it is worthwhile to deploy the winner.
Third, after analyzing the different solutions and their performance for different decision-making goals, we focus on three common scenarios that decision makers encounter: (1) null effects (Cohen's $d = 0$), (2) small treatment effects or small sample sizes (Cohen's $d = 0.2-0.5$) and (3) large treatment effects (Cohen's $d = 0.8$). Interestingly, across simulations, some of the proposed solutions in the literature perform inconsistently across these scenarios; many of the solutions hinge on a tuning parameter and its optimal value is context dependent. A method optimized for small effect regimes underperforms with null effects and vice-versa. These results persist when we extend the study to experiments with many treatments and in a simulation study calibrated to experiments from an online A/B testing platform berman2022false.
Consequently, our fourth contribution is a new inference method for experiments subject to the winner's curse. Building on empirical likelihood techniques owen2001empirical, we develop an adaptive procedure ((ref)) that constructs valid confidence intervals across all settings we consider. We establish that this procedure achieves pointwise asymptotic coverage ((ref)). Compared to resampling methods, our approach is both less sensitive to tuning parameters and more computationally efficient. In simulations, it consistently achieves nominal coverage across all scenarios.
We distinguish among three different targets that a decision-maker might have with respect to the winner's curse and note that the literature has not identified an optimal method for many of these scenarios. This framework allows us to identify optimal correction procedures for realistic scenarios decision-makers might encounter.
We contribute to several interconnected literature streams on the winner's curse, post-selection inference, and optimal decision-making in experiments. SmithWinkler2006 introduced the optimizer's curse, showing that point estimates of selected alternatives are systematically upward biased. thaler2025 review similar reversion-to-the-mean phenomena in behavioral economics. In statistics, the selective inference literature tackles a related problem of conducting valid inference after model selection, where the choice of model parallels our setting of choosing the winner Kuchibhotla2022. We extend this literature by distinguishing three objectives (global winner's curse, selected winner's curse, and regret) and showing that the optimal correction depends on the decision-maker's objective.
A related literature studies inference for rankings Bechhofer1954. Gu2023 show that ranking and best-arm identification are directly connected and propose an empirical Bayes procedure. Mogstad2024 develop a resampling-based method for constructing statistical tests on ranks. While related, this literature focuses on identifying the best arm rather than estimating its value. Identifying the best arm addresses regret, but practitioners also need accurate estimates of the selected treatment's effect size to justify deployment and set expectations.
Estimating the maximum of means presents a parameter boundary problem: standard inference methods fail at the kink where $\mu_1 = \mu_2$. andrews2000inconsistency shows that standard resampling methods become inconsistent when parameters lie at boundaries, and proposes pre-test, shrinkage, and subsampling solutions. fang2019inference and hong2018numerical develop derivative-based bootstrap procedures for directionally differentiable statistics. The post-selection inference literature distinguishes simultaneous from conditional (selective) inference Kuchibhotla2022. andrews2024inference develop conditional inference specifically for winners, proposing a hybrid procedure that bounds interval length near ties. For bias correction, efron1993bootstrap establishes the classical non-parametric bootstrap approach, and xu2025 apply it to the selected winner's curse. Our simulations reveal that these resampling methods, while asymptotically valid, are sensitive to tuning parameters in finite samples. The hybrid procedure proposed by andrews2024inference is more stable but produces overly conservative confidence intervals in our simulation study.
On the decision-making side, simester2020efficiently and Feit2019 study experimental design for treatment selection, and bastani2025beating propose inference-aware policy optimization. The minimax-regret literature (e.g., song2014local, JooChiong2025, and joo2025) shows that the plug-in estimator is locally asymptotically minimax optimal; our regret analysis corroborates this finding.
On the inference side, we contribute to the empirical likelihood (EL) literature owen2001empirical by using chi-bar-squared critical values Dykstra1991 to achieve valid coverage of the confidence interval at the kink. Our adaptive procedure then builds on the pre-tests suggested by andrews2000inconsistency, and we show that it provides pointwise asymptotic coverage of the confidence interval ((ref) in (ref)) both away from and at the kink. The EL approach is both less sensitive to tuning parameters and less computationally burdensome than resampling methods and thus offers both an efficient and robust alternative for constructing confidence intervals.
Although A/B testing uses relatively standardized methods and analysis, it is applied to make decisions in distinct decision making scenarios, which imply that the same numerical results might yield different decisions under different scenarios and objectives of the experimenter. We provide two vignettes which we use to illustrate different solutions for the winner's curse below:
\paragraph{Vignette 1: Champion-Challenger with Optimized Default}
An experimenter in this scenario already has a previously well-tested and optimized version which is the current default. They are testing new variations, nudges or interventions, to see if they improve on the existing default. Because the existing default is already optimized, it is likely that most new proposed challenger interventions will not improve on the champion and will have null effects. The goal of the experimenter is to select a new intervention that improves over the default with high confidence, and in addition properly estimate the amount of improvement over the default to assess whether it justifies the additional cost of switching to the challenger.
\paragraph{Vignette 2: One-shot pick the best} An experimenter in this scenario runs a mega-study comparing many interventions, with a goal of selecting the best one going forward milkman2021megastudies. The results of this study will be published and used to drive future public policy. It is anticipated that the best interventions will have a non-null effect compared to the other interventions Milkman2024Nature,Milkman2025PNAS. The goal of the experimenter is to have a precise estimate of the expected outcome of the intervention to be deployed, in order to assess whether the outcomes outweigh the costs. This setting also appears in the behavioral economics literature thaler2025, which emphasizes reversion to the mean for "winners" where the chosen winner may perform worse after being selected.
Although these two scenarios appear similar a priori, we present below how they impact decision-making when accounting for the winner's curse.
\definecolor{ECBlue}{RGB}{224,232,242} \definecolor{ECSand}{RGB}{241,235,225} \definecolor{ECSage}{RGB}{228,238,231} \definecolor{ECHeader}{RGB}{245,245,245} \definecolor{ECNA}{RGB}{250,250,250}
\definecolor{ECLavender}{HTML}{D9D2E9} \definecolor{ECPeach}{HTML}{FCE5CD}
\definecolor{ECRose}{HTML}{F4CCCC}
We present three alternative definitions for the magnitude of the winner's curse that have been used in the literature. We illustrate how they affect decision making in our three illustrative scenarios. We focus our analysis on experiments with two treatments (arms), and note that they naturally extend to more than two.
An experimenter runs an A/B test with two treatments/variations $k \in \{1,2\}$. The observed outcome of person $i$ who is exposed to treatment $k$, $y_{ik}$, is drawn from a distribution with mean $\mu_{k}$ and variance $\sigma^2$ where $\sigma$ is assumed known for simplicity. When treatment effects are null, we assume $\mu_1 = \mu_2$. When one treatment is better than the other, we assume it is treatment one, i.e., $\mu_1 = \mu_2 + \Delta \mu$ with $\Delta \mu >0$.
For each treatment group, the experimenter computes an estimate $\hat{\mu}_k$ of the true population mean $\mu_k$. A common way to compute the estimate is by taking the sample mean $\hat{\mu}_k = \frac{1}{n_k} \sum_i y_{ik}$, where $n_k$ is the number of individuals exposed to treatment $k$. However other methods (e.g., OLS regression) are also often used.
After the experiment, the experimenter picks one treatment to deploy in the future. If they pick the treatment with the highest $\hat{\mu}_k$ (denoted with $\hat{k}$), and use the value of $\hat{\mu}_{\hat{k}}$ (the plug-in estimator) to predict the future outcomes, then this value is expected to be upwards biased. Because the maximum function is convex, Jensen's inequality tells us that $E[\max(\hat{\mu}_1,\hat{\mu}_2) ] \ge \max(E[\hat{\mu}_1],E[\hat{\mu}_2)] ] = \max(\mu_1, \mu_2)$. There is also some chance that the chosen treatment is not the truly optimal one, i.e., that $\hat{k} \neq \operatorname*{arg\,max}(\mu_1, \mu_2)$.
This upward bias leads to a definition of the size of the winner's curse as the difference between the best estimated mean outcome and the truly best mean outcome in the population:
In other words, this estimate answers ”If one implements the treatment with the highest estimated mean, how far is the estimate from the truly best outcome”.
Although estimating (and minimizing) $WC_{global}$ is desirable, the true values of $\mu_k$ are unobserved and may not be precisely estimated. However, one can pick a benchmark to compare to (for example, a default control group). Let that benchmark be denoted as treatment $0$, we can rewrite
Minimizing $\max(\hat{\mu}_1, \hat{\mu}_2) - \hat{\mu}_0$ is equivalent to minimizing $WC_{global}$ because the last term in parenthesis is fixed and does not depend on the identity of the estimated winner $\hat{k}$ or the value $\max(\hat{\mu}_1, \hat{\mu}_2)$. This approach to measuring or correcting the winner's curse, which focuses on comparing to a control, addresses the objective of the experimenter in Vignette 1, by comparing to a champion treatment. An empirical application which focuses on this type of objective is the inference-aware policy optimization of bastani2025beating. The focus in this analysis is often on having an improvement with a statistical guarantee of a positive effect, which implies that performing precise inference on $WC_{global}$ is desirable.
The literature on the winner's curse often focuses on a different objective, answering ”For the treatment selected for implementation, how far is the estimate from its true unobserved mean”. The winner's curse of the selected treatment is defined as:
This metric is useful in Vignette 2, where a policy maker is interested in estimating (or minimizing) $WC_{select}$ in order to predict their returns when implementing the winning treatment xu2025.
These two definitions lead to the third, which is regret, commonly used in decision theory bell1982regret. A decision maker minimizing regret is not interested in estimates of outcomes, but only true outcomes. Hence, they might ask ”if I implement treatment $\hat{k}$, how much do I lose because I didn't implement the true optimal treatment?”. This question yields the definition of ex-post regret:
We observe that regret only depends on the selected arm $\hat k$ and not its estimate $\hat\mu$. Thus, the selecting the best arm is necessary and sufficient for zero ex-post regret. Further, since only arm selection matters, we can focus our efforts in estimating only $\hat k$ instead of both $\hat\mu_k$ and $\hat k$. This is why we expect the plug-in estimates to perform well in minimizing regret (as they are using the whole data set to just figure out $\hat k$).
(ref) illustrates the difference between these definitions. Noting that $\hat{\mu}_{\hat{k}} = \max(\hat{\mu}_1, \hat{\mu}_2)$, we can combine these definitions to write the identity:
This identity implies a tradeoff between minimizing $WC_{global}$ and $WC_{select}$ if one keeps regret constant.
Given these three definitions and their applicability in different experimentation scenarios and objectives, we investigate if the correction procedures offered in the literature properly help the experimenter minimize the winner's curse. Because these definitions do not coincide, we would not expect one correction procedure to be the best for all scenarios and all objectives.\footnote{There is no tradeoff when all of the values are zero, which means both the correct treatment has been selected and the value of the treatment has been correctly estimated.} Also, we note that the correction procedures sometimes target different objectives such as correcting bias or conducting inference, and these are sometimes conflicting. For example, in the case of $WC_{select}$, a debiasing procedure will aim to minimize $E[WC_{select}]$, which will allow an experimenter in Scenario 2 to have the most precise prediction of the future performance of the selected intervention. For scenario 1, however, debiasing estimates is expected to increase their variance; a larger variance makes statistical inference about the probability of improving over the optimized default harder. Some methods might also be too conservative, over estimate the variance, and lead to overcovering confidence intervals, which may lead managers to decide against deployment despite it being beneficial.
In our analysis, we will compare the different proposed methods on both their ability to debias, and on providing tight confidence intervals. We will also consider a mean square error (MSE) metric that trades off bias and variance, and allows a manager to consider how much might be lost by debiasing or choosing a method that is overly conservative. Table (ref) summarizes the different definitions and provides the notation for each one of the metrics we will analyze.
In the remainder of the paper we first describe and then analyze the different solutions offered to correct the winner's curse by comparing their performance across the different objectives. Our analysis shows that the appropriate correction procedure should be matched to the target objective of the decision maker and the expected effect sizes measured by Cohen's $d$. Table (ref) previews the recommended procedure for each scenario for reducing bias and Table (ref) contains our full recommendation for all seven objectives.
This section introduces the families of solutions that address the winner's curse. We first formally set up the problem and outline the mathematical properties of the maximum of means, which is central to understanding the winner's curse. The discussion highlights the challenges with standard inferential techniques when dealing with the non-differentiability of the maximum function. Then, we describe the proposed solutions, and categorize them by their underlying machinery. One downside of these prior methods is that they require tuning parameters, which we show can lead to substantial degradation in performance if picked incorrectly. Hence, we introduce a novel techniques for constructing confidence intervals using empirical likelihood, which is less sensitive to underlying tuning parameters.
The winner's curse can be recast as a problem studying the maximum of means. We focus on the two-arm setting and the parameter $\theta = \max(\mu_1, \mu_2) = \phi(\mu_1, \mu_2)$, where $\mu_1$ and $\mu_2$ are the means of two independent populations. We let $\{X_{1i}\}_{i=1}^{n_1}$ and $\{X_{2i}\}_{i=1}^{n_2}$ be independent and identically distributed (i.i.d.) samples from distributions with means $\mu_1, \mu_2$ and variances $\sigma_1^2, \sigma_2^2$ respectively. The sample estimators for the means are $\hat\mu_1 = \bar{X}_1$ and $\hat\mu_2 = \bar{X}_2$.
A natural estimator for this statistic is the plug-in estimator $\hat{\theta}_n = \max(\bar{X}_1, \bar{X}_2) = \phi(\hat\mu_1,\hat\mu_2)$ that plugs the sample mean estimates into the max function. It is straightforward to show this function is continuous and Lipschitz in its arguments. Further, it is differentiable except at the kink where $\hat\mu_1=\hat{\mu_2}$. The maximum function is convex, and by Jensen's inequality, we have $E[\hat{\theta}_n] \ge \max(\mu_1, \mu_2) = \theta$. This upward bias is precisely the "winner's curse".
In (ref), we establish the consistency of the estimator and derive its asymptotic distribution. We show that the asymptotic distribution is a maximum of two normal distributions, which behaves irregularly when at the kink ($\mu_1 = \mu_2$) due to the non-differentiability of $\phi$ (the maximum function) at this point when the means of the two groups are equal. As a result, standard approaches that rely on the Delta method will fail at the kink hirano2012impossibility, and the vanilla bootstrap methods for bias correction and uncertainty quantification are inconsistent for this statistic mammen1992bootstrap.
We categorize the several families of proposed solutions into five categories: sample-splitting, resampling, conditional inference, empirical Bayes, and empirical likelihood. In our simulations, we will evaluate the performance of these methods in terms of the different objectives outlined in (ref).
Sample splitting decouples the selection of the winner from the estimation of its value. The data is partitioned into two independent folds; one fold is used to select the arm with the highest sample mean, and the other fold is used to estimate the mean of that selected arm. While this eliminates winner's curse, it reduces the effective sample size and can be inefficient. Sample splitting methods are reviewed in Kuchibhotla2022 and are considered as a general, model-agnostic method for de-biasing. However, the splitting of the data will widen the confidence interval and accurate selection of the winner is not guaranteed.
Cross-fitting can recover some efficiency by averaging estimates over exchanged folds. The procedure recycles the sample-splitting folds for selection and estimation and then averages the results across the folds to provide a more data-efficient estimator.
Resampling methods, such as the bootstrap, are widely used for bias correction and inference efron1993bootstrap. fang2019inference show that a statistic needs to be fully Hadamard differentiable for the bootstrap to be consistent. In (ref), we show that the maximum operator $\max(\cdot)$ would not yield consistent bootstrap estimates because it is only directionally (and not fully) Hadamard differentiable.
andrews2000inconsistency provides four classes of adjustments for resampling methods to be consistent for estimating a parameter on the boundary of its domain. We adapt these methods for our setting where we have two arms and the statistic of interest is the maximum of the two arms. The boundary is at the kink when the two arms yield equal means.
First, andrews2000inconsistency proposes using a pre-test that initially tests whether the two arms' means are equal or not for each bootstrap iteration. If the pre-test fails to reject the null hypothesis that the two means are equal, then the asymptotic distribution of the max of two normals is used. Otherwise, the limiting distribution of the higher arm is used. The critical value of the pre-test is chosen to scale with the number of observations which ensures asymptotic consistency.
Second, andrews2000inconsistency proposes a shrinkage parametric bootstrap. The parametric approaches uses the limiting distribution of the problem. Like in the first approach, for each bootstrap iteration, a pre-test is run to see if the means are equal. If the test fails to reject, the means are pooled and draws from a normal distribution with the pooled means are used. Otherwise, the maximum of the draws of two normals centered around the sample means of each arm are used. The pre-test critical value is also chosen to ensure asymptotic consistency.
Intuitively, the pre-test in both approaches will ”snap” the estimates to the kink if the sample means are close enough and smaller than the pre-test critical value. In practice, the choice of the pre-test value is a tuning parameter, and the adjustment leads to pointwise asymptotically consistent performance when at or away from the kink, but possibly worse performance in finite samples when close to the kink.
As the third and fourth solutions, andrews2000inconsistency proposes two subsampling procedures: the $m$-out-of-$n$ subsampling estimator Politis1994 and the $m$-out-of-$n$ or rescaled bootstrap. Both use a smaller subsample size to smooth out the limit distribution and can be generally applied to i.i.d. settings. The subsampling size, $m$, acts as the tuning parameter and as long as $m\to\infty$ and $m/n\to 0$, the estimator is asymptotically consistent. As $m$ gets close to $n$, for the $m$-out-of-$n$ bootstrap, we recover the standard nonparametric bootstrap.
fang2019inference and hong2018numerical offer two additional resampling procedures based on the Hadamard derivative of $\phi$ and are similar to the pre-test procedures outlined by andrews2000inconsistency. fang2019inference uses a pre-test for each bootstrap iteration to determine if the means are equal and then uses the analytical Hadamard derivative at the kink if it fails to reject. Just like the pre-test approaches, the effective critical value of the test is a tuning parameter that needs to be chosen during implementation. hong2018numerical expand on this approach and numerically compute the Hadamard derivative for each bootstrap iteration, and the step size of the numerical derivative becomes the tuning parameter. The numerical derivative approach is readily generalizable to many arms because it does not require a pre-test across the different arms.
Separately, these resampling methods can be used for bias correction efron1993bootstrap, which includes the winner's curse or bias of the plug-in estimator xu2025. Any of these resampling methods that explicitly adjust for the kink will provide consistent bias correction.
These approaches have two drawbacks: sensitivity to tuning parameters and heavy computational cost, with the latter especially acute for bootstrap-within-bootstrap confidence intervals around bootstrap bias corrections. While theory guarantees asymptotic consistency, in finite samples the choice of tuning parameter will substantially shift the sampling distribution. Each method has its own tuning parameter: $m$ in $m$-in-$n$ resampling, the critical value for pre-test and shrinkage estimators, and the step size for the numerical Hadamard derivative. Our simulations confirm that results depend meaningfully on these choices.
A related literature examines constructing confidence intervals that are valid conditional on the selection event Kuchibhotla2022. By explicitly conditioning on the fact that a specific arm was chosen (e.g., $\hat{\mu}_1 > \hat{\mu}_2$), these methods adjust the inference to account for the truncation of the sampling distribution induced by selection and are known in the literature as selective inference. In contrast, the sample-splitting and resampling approaches provide simultaneous inference, which covers both the arm selection and the mean estimation in our example.
andrews2024inference studies the problem of conducting inference conditional on choosing the best performing arm. In their framework, a manager faces a set of normally distributed arms and chooses the "winner" with the highest estimate. To remedy the winner's curse, they propose a conditional inference procedure based on the distribution that is conditional on the selection. By conditioning on the identity of the winning arm, they show that the residual distribution of the winner's estimator follows a truncated normal distribution rather than a standard normal. Inference proceeds by inverting this truncated distribution, which yields estimators that are median-unbiased and confidence intervals with valid coverage conditional on the specific arm selected. However, a limitation of this approach is that when the "best" arms are close in performance (close to the kink), the conditioning event becomes highly restrictive, leading to potentially infinite-length confidence intervals.
To resolve this concern, andrews2024inference propose a hybrid inference procedure that combines the conditional inference procedure with a simultaneous "projection" interval, which is an overly conservative interval valid for all arms simultaneously. Specifically, the projection interval is constructed to cover the true parameters of all potential winners with a high probability, ensuring unconditional validity regardless of which arm is ultimately selected. The hybrid procedure conditions on the selection event but restricts the parameter space to the projection interval, effectively using the simultaneous interval to bound the length of the conditional interval. The authors show that this approach prevents the confidence intervals from becoming excessively wide and ensures valid unconditional coverage. Our simulation study evaluates the performance of these two approaches for our outcomes of interest.
SmithWinkler2006 propose a Bayesian correction in which the posterior mean of each arm is a convex combination of its sample mean and a prior mean. We implement an empirical Bayes version that shrinks each arm toward the grand mean, with the prior variance estimated from the data by method of moments:
where $\bar{\mu} = K^{-1}\sum_k \bar{Y}_k$ is the grand mean and $\hat{\sigma}_\mu^2$ is a method-of-moments estimator of the between-arm variance. The winner is selected as $\hat{k} = \arg\max_k \hat{v}_k$ and its shrunk mean $\hat{v}_{\hat{k}}$ is reported as the point estimate.
The intuition is that the naive estimator $\bar{Y}_{\hat{k}}$ is unbiased conditional on the true means but becomes upward biased once the selection step is introduced, because extreme sample means are disproportionately likely to come from arms whose noise realizations happened to be large. Shrinking toward the grand mean reduces these extremes: an arm whose observed mean is far above the others is partially attributed to noise rather than to a genuinely superior true mean, and its estimate is pulled back accordingly. The amount of shrinkage is governed by the signal-to-noise ratio $\alpha_k$. When sampling noise is large relative to the spread of true means, $\alpha_k$ is small and estimates are shrunk heavily; when the spread of true means dominates, $\alpha_k$ approaches one and the shrinkage vanishes. This approach is the Bayesian analogue of the James--Stein estimator JamesStein1961, EfronMorris1973.
The Empirical Bayes approach is most advantageous when the true means are plausibly drawn from a common distribution and when $K$ is large enough to estimate $\sigma_\mu^2$ reliably. It is straightforward to implement, requires no resampling, and has no tuning parameter beyond the variance components, which are estimated from the data. Its main limitation in our setting is that the estimate of $\sigma_\mu^2$ is noisy when $K$ is small, and shrinkage toward the grand mean can introduce downward bias when one arm is genuinely far better than the rest.
Empirical likelihood (EL) offers a computationally efficient, non-parametric approach to constructing confidence intervals. We incorporate EL for our problem and introduce an adaptive procedure based on pre-tests that accounts for the kink and ensures proper pointwise asymptotic coverage of its confidence interval.
EL constructs confidence intervals by inverting a profile likelihood ratio test: For a candidate value $\theta$, one computes the maximum of the empirical likelihood subject to the constraint that the parameter equals $\theta$, and rejects when the resulting deviance exceeds a $\chi^2$ critical value owen2001empirical. The likelihood ratio test holds under smooth, interior constraints.
However, the constraint $\max(\mu_1, \mu_2) = \theta$ introduces a constraint at the boundary. When $\mu_1 = \mu_2 = \theta$, the constraint set has a kink, and the standard $\chi^2$ calibration for the critical value is incorrect. In (ref), we show the profile deviance at the kink instead converges to a chi-bar-squared distribution, which is a mixture of $\chi^2$ variates, and accounts for the boundary constraint Dykstra1991. We also show that applying the standard $\chi^2$ critical value at the kink leads to under-coverage, but using the chi-bar-squared critical value everywhere leads to over-coverage away from the kink.
In practice, we do not know if we are at the kink or not. We therefore propose an adaptive procedure based on pre-tests to adaptively use the $\chi^2$ or the chi-bar-squared critical value. In (ref), we show that this approach is asymptotically consistent and provides pointwise asymptotic coverage (ref) and (ref) in (ref).
(ref) establishes that the pre-test consistently classifies whether the data generating process is at the kink ($\mu_1 = \mu_2$) or away from it.\footnote{$\hat K_n$ is a binary variable that represents the sample classification (See (ref)).}
The intuition behind the proof relies on the behavior of the scaled difference between the sample means. At the kink (when the true means are equal), this difference is driven purely by random noise and remains bounded. Because our pre-test threshold $\kappa_n$ grows with the sample size, it eventually exceeds this noise, correctly identifying the tie. Away from the kink, the true difference between the means causes the scaled statistic to grow very quickly (at a rate proportional to $\sqrt{n}$). Because our threshold is chosen to grow slower than this rate, the test statistic easily exceeds it, ensuring the pre-test correctly detects that the means are different.
Building on (ref), (ref) shows that the adaptive EL confidence interval achieves correct pointwise asymptotic coverage.
The proof evaluates the coverage probability by splitting it into two scenarios based on the pre-test's decision. If the true means are equal (at the kink), the pre-test consistently identifies the tie. This triggers the use of the chi-bar-squared critical value, which matches the nonstandard distribution of the test statistic at the boundary, ensuring correct coverage. Conversely, if the true means differ (away from the kink), the pre-test consistently detects the difference. The procedure then applies the standard $\chi^2$ critical value, which provides correct coverage because the test statistic follows a standard, smooth distribution in this regime.
We then extend our analysis $J$ arms. We derive the associated chi-bar-squared critical value ((ref)) and propose an active-set approach ((ref)), in which we only consider comparisons of the arms to the empirically best performing arm. (ref) and (ref) respectively show that the procedure is asymptotically consistent for identifying the active-set of best performing arms and provides pointwise asymptotic coverage ((ref)).
The advantages of our EL method are twofold: compared to resampling techniques, its tuning parameter is more rigorously justified by theory and, in practice, demonstrates lower sensitivity to its tuning parameter. In finite samples, the performance of $m$-out-of-$n$ resampling methods depends heavily on the chosen subsample size, $m$. As our simulations demonstrate, varying this tuning parameter causes the method to systematically over- or under-correct. Further, there is no established theoretical guidance for selecting an optimal $m$. The EL approach requires a pre-test cutoff $\kappa_n$ that is guided by asymptotic theory, requiring $\kappa_n \to \infty$ and $\kappa_n = o(\sqrt{n})$. With theoretically guided cutoff $\kappa_n = \sqrt{\log\log(\min\{n\})}$, the EL method avoids the finite-sample volatility observed in resampling procedures in our simulation study; the tuning parameter for EL also only impacts the critical value and not the sampling distribution, which provides stability in its estimate.
While our EL method uses a pre-test like the adaptive resampling techniques of andrews2000inconsistency and fang2019inference, the impact of the pre-test is fundamentally different. In adaptive bootstrap methods, the pre-test determines which distribution to simulate. A misclassification forces the estimator to draw from the wrong distribution, causing significant volatility in finite samples. In contrast, our EL procedure uses the pre-test only to select the critical value (toggling between $\chi_1^2$ and $\bar{\chi}^2$), while the underlying test statistic remains unchanged. Therefore, a pre-test misclassification in the EL framework merely applies a slightly different threshold to the correct test statistic and the EL approach is structurally more stable.
Like all pre-test procedures, our EL method achieves pointwise, rather than uniform, asymptotic coverage andrews2000inconsistency. Because methods relying on a tuning parameter to adjust for the kink inherently suffer from this limitation, we advocate the EL approach primarily through its finite sample stability and its significant computational advantages. While resampling methods require computationally expensive bootstrap-with-bootstraps procedures to construct valid confidence intervals, our EL method completely sidesteps this burden. Instead, it relies on a highly efficient one-dimensional root-finding procedure, making it vastly more scalable in practice.
Our Monte Carlo simulations are designed to compare the four categories of solutions at representative treatment effect values that match the three scenarios from (ref). We focus on the seven different objectives: bias correction, MSE, and coverage for WC select and WC global as well as regret.
\paragraph{Data generating process} The simulation generates data from a two-arm randomized experiment with balanced assignment. In each Monte Carlo replication, we draw an i.i.d. sample of size $N$ with sample sizes $n_1=\lfloor N/2\rfloor$ and $n_2=N-n_1$, where $N \in \{20, 40, 60, 80, 100\}$. Individual outcomes for unit $i$ are from $Y_i \overset{iid}{\sim} \mathcal{N}(\mu_k,\sigma^2)$, with $\mu_1 = 1$ and $\mu_2 = 1+\Delta\mu$. Throughout, we report Cohen's $d \equiv \Delta\mu / \sigma$ as a standardized effect-size summary, so that the grid spans $d \in \{0, 0.1, 0.2, 0.5, 0.8, 1.0\}$ when varying $\sigma$. For each $(N, d ,\sigma)$ tuple, we run $R=20,000$ replications. Resampling-based procedures use $B=200$ resamples per replication.
\paragraph{Objectives for comparison} We evaluate all methods on the following criteria:
\paragraph{List of methods}
(ref) summarizes the procedures we analyze in our simulations. Here, we report results of analyses for the following methods: (1) standard nonparametric bootstrap; (2) $m$-out-of-$n$ bootstrap with $m=\lfloor N^{\gamma}\rfloor$ and $\gamma\in\{0.45,0.95\}$; (3) parametric shrinkage bootstrap; (4) FS19 derivative bootstrap; (5) sample splitting; (6) cross fitting; (7) AKM24 hybrid; (8) empirical likelihood (EL); and (9) Bayesian shrinkage. (ref) reports the full set of simulation results for all methods we analyze.
For the bootstrap-based bias-correction methods, we use two correction amounts that mirror our two evaluation targets (select vs.\ global). Let $D=\{(T_i,Y_i)\}_{i=1}^N$ denote the realized dataset, and define the arm sample means $\bar{Y}_k(D)$ and the plug-in winner estimator $\hat{\phi}(D)=\max\{\bar{Y}_1(D),\bar{Y}_2(D)\}$. Each bootstrap procedure generates bootstrap resamples $D^{*(b)}$ for $b=1,\ldots,B$ and computes the corresponding bootstrap winner $\hat{\phi}^{*(b)}=\hat{\phi}(D^{*(b)})$, from which we form two bias estimates.
The global correction estimates the upward bias of the plug-in winner by centering each bootstrap winner at the original plug-in value, \[ \widehat{WC}_{\mathrm{glo}} =\frac{1}{B}\sum_{b=1}^B\big(\hat{\phi}^{*(b)}-\hat{\phi}(D)\big). \] The select correction re-centers each bootstrap winner around the mean of the arm selected within that bootstrap sample, evaluated on the original data, \[ \widehat{WC}_{\mathrm{sel}} =\frac{1}{B}\sum_{b=1}^B\big(\hat{\phi}^{*(b)}-\bar{Y}_{\hat{k}^{*(b)}}(D)\big), \qquad \hat{k}^{*(b)}=\arg\max_{k\in\{1,2\}}\bar{Y}_k(D^{*(b)}). \] For either choice $\diamond\in\{\mathrm{sel},\mathrm{glo}\}$, the corrected estimator has the common form $\hat{\phi}_{\diamond}=\hat{\phi}(D)-\widehat{WC}_{\diamond}$, where $\widehat{WC}_{\diamond}$ is computed using $B=200$ bootstrap resamples.\\
\newcolumntype{Y}{>{\arraybackslash}X}
\newlength{\methodcolw} {0.22\textwidth}
\begingroup \linespread{1.0} {3pt} {0pt} {0pt}
\endgroup \linespread{1.5}
This section presents the results of our simulations and provides an analysis for our seven objectives of interest.
\paragraph{Winner’s-curse bias.} Figures (ref) and (ref) report the average estimation error for the global and select targets. The magnitude of winner’s-curse bias is largest in regimes with low signal-to-noise ratios: small sample sizes, large noise $\sigma$, and near-ties (small $\Delta\mu$), and it diminishes as $N$ increases or the gap $\Delta\mu$ grows. The direction of bias reflects how aggressively a method corrects for selection: The plug-in estimator exhibits a large upward bias in tie/near-tie settings, while some resampling-based corrections can over-shoot and become negatively biased in these same regimes (most visibly for the more aggressive $m$-out-of-$n$ choice with smaller $m$).
Methods that explicitly decouple selection from estimation such as sample splitting and cross-fitting produce biases close to zero for the select target, especially under ties or at the kink, at the cost of reduced effective sample size.
\paragraph{Mean squared error (MSE)} Figures (ref) and (ref) summarize the MSE trade-off. The ranking of methods depends on whether the problem is a tie/near-tie or a clear-winner regime. In ties and near-ties, bias reduction is valuable, yet overly aggressive correction can inflate MSE through increased variance and/or negative bias. This is most apparent when using a small $m$ in the $m$-out-of-$n$ bootstrap, which can lead to systematic negative bias and larger MSE. In contrast, when $\Delta\mu$ is large (clear separation), the winner’s-curse bias is already small and additional correction is typically unnecessary; in these regimes, the plug-in estimator often dominates in the MSE because it avoids the extra variability introduced by correction or sample splitting.
\paragraph{Regret} Regret is driven by selection mistakes choosing the inferior arm, rather than by post-selection bias correction. From the simulation, the expected regret has the closed form \[ \mathbb{E}[\mathrm{Regret}] = |\mu_1-\mu_2|\, \Phi\!\left( -\frac{|\mu_1-\mu_2|}{\sigma\sqrt{1/n_1+1/n_2}} \right), \] where $\Phi(\cdot)$ is the standard normal CDF. Procedures that devote only part of the sample to selection (e.g., sample splitting or cross-fitting) increase the variance of the comparison $\hat\mu_1-\hat\mu_2$, thereby raising the probability of selecting the inferior arm and increasing regret relative to selecting with the full sample. For a sample split with equal size folds, selection uses $n_k/2$ observations per arm, implying \[ \mathbb{E}[\mathrm{Regret}_{\mathrm{SS}}] = |\mu_1-\mu_2|\, \Phi\!\left( -\frac{|\mu_1-\mu_2|}{\sigma\sqrt{1/(n_1/2)+1/(n_2/2)}} \right) = |\mu_1-\mu_2|\, \Phi\!\left( -\frac{|\mu_1-\mu_2|}{\sigma\sqrt{2(1/n_1+1/n_2)}} \right). \]
Bootstrap-based bias corrections have the same regret as the plug-in rule because the selection decision still uses the full-sample comparison $\hat\mu_1-\hat\mu_2$.
In clear-winner regimes, the plug-in rule is therefore preferred when regret is the primary objective, whereas under ties ($\mu_1=\mu_2$) regret is uninformative since $\mathrm{Regret} = 0$.
\paragraph{Coverage} (ref) and (ref) report the empirical coverage of $95\%$ confidence intervals for the global and select targets. In exact ties, sample splitting yields close to nominal coverage for both targets, whereas cross-fitting can under-cover at small $N$.\footnote{(ref) provides a detailed discussion of why cross-fitting can under-cover in this setting.} When Cohen's $d$ is moderate, the empirical likelihood and AKM24 are the preferred methods. When Cohen's $d$ is large, the plug-in approach yields intervals whose coverage converges to the target level as $N$ grows.
The bootstrap-of-bootstrap is the most computationally expensive method, yet it underperforms relative to empirical likelihood. Empirical likelihood requires root-finding, which can be slower than closed-form alternatives, but it remains relatively fast compared to resampling methods.
Table (ref) summarizes a practical rule-of-thumb for experimenters who might need to choose a winner's curse correction method.
We provide additional simulation, scenario-based recommendations for (1) when arms are binomial and (2) for many (normals) arms. The details are in Appendix (ref).
To verify that the results do not depend on the Gaussian noise assumption, we re-run the two-arm Cohen's $d$ grid with Bernoulli outcomes $Y_{ik}\sim\text{Bernoulli}(p_k)$. We sweep base rates $p_1 \in \{0.1, 0.3, 0.5\}$ and target effect sizes $d \in \{0, 0.1, 0.2, 0.5, 0.8, 1.0\}$, with $p_2$ chosen so that the Cohen's $d$ matches the column label: $ d \;=\; \frac{p_2 - p_1}{\sqrt{\tfrac{1}{2}\big(p_1(1-p_1) + p_2(1-p_2)\big)}}.$ Figures (ref)--(ref) report the same simulation with Bernoulli outcomes. Bias and MSE rankings are essentially unchanged. Coverage is the dimension where the noise distribution matters: Sample Splitting and andrews2024inference under-cover at small $N$, while the empirical-likelihood interval remains near nominal under both DGPs and is the safer default for binary outcomes.
We extend the simulation to $K \in \{2, 3, 4\}$ arms with true means drawn from a Gaussian prior, $\tau_k \overset{iid}{\sim} \mathcal{N}(1, \sigma_0^2)$, and observations $Y_i \mid T_i = k \sim \mathcal{N}(\tau_k, \sigma^2)$. The prior standard deviation $\sigma_0 \in \{0, 0.05, 0.1, 0.2, 0.5, 1\}$ controls treatment-effect heterogeneity, $\sigma \in \{0.1, 0.2, 0.5, 1\}$, and $N = 100$ is fixed and balanced across arms. Method-by-method generalizations are summarized in Appendix Table (ref). Figures (ref)--(ref) report the full results faceted by $\sigma_0$ (columns) and $\sigma$ (rows), with $K$ on the $x$-axis. The qualitative patterns extend consistently to $K > 2$.
To study the effectiveness of the methods in real-world settings, we calibrate the simulation to data from the Optimizely A/B experimentation platform from berman2022false. The simulation mimics how the methods would perform if deployed on the platform. There are two main differences: (1) we consider many arms and (2) the underlying data generating process is more complicated and contains a mix of small effects and null effects.
In contrast to our prior simulation's data generating process, outcomes here are binary: each visitor $i$ in arm $k$ produces a Bernoulli outcome $Y_{ik} \sim \text{Bernoulli}(p_k)$, representing a conversion event. The conversion probability $p_k$ is constructed from a null/non-null mixture. Further, each arm is independently designated as null with probability $\pi_0$ and non-null arms receive a treatment effect $\tau_k \sim \mathcal{N}(\mu_e, \sigma_e^2)$, so that $p_k = p_0 + D_k \cdot \tau_k$ where $D_k \sim \text{Bernoulli}(1 - \pi_0)$.
We set $p_0 =0.633$, $\pi_0 = 0.82$, $\mu_e = -0.0032$, and $\sigma_e = 0.0815$ to match the empirical distribution of treatment effects observed on the Optimizely platform, and fix $N = 5,700$ with balanced allocation across arms.\footnote{$\pi_0 = 0.82$ implies 82% of the arms are null effects.} We vary the number of arms from $K = 2$ to $K = 10$. As $K$ grows, selection bias and the Winner's Curse increase. We report standardized effect size as Cohen's $d$. Across Monte Carlo replications $r = 1, \ldots, R$, let $\mu_{(1)}^{(r)} \geq \mu_{(2)}^{(r)}$ denote the top-two true arm rates and $\hat{s}_{(1)}^{(r)}, \hat{s}_{(2)}^{(r)}$ their sample standard deviations. Cohen's $d$ increases with the number of arms and ranges from $0.04-0.08$, which implies small effects.\footnote{We define $d = \frac{\tfrac{1}{R}\sum_{r=1}^{R} \left(\mu_{(1)}^{(r)} - \mu_{(2)}^{(r)}\right)} {\tfrac{1}{R}\sum_{r=1}^{R} \sqrt{\tfrac{1}{2}\!\left(\hat{s}_{(1)}^{(r)\,2} + \hat{s}_{(2)}^{(r)\,2}\right)}}$ the MC-averaged top-two gap divided by the MC-averaged pooled empirical SD of those two arms.} Figures (ref)--(ref) report the results.
The qualitative patterns broadly mirror those of the prior simulation. For bias (Figure (ref)), cross-fitting performs best for both the selective and global targets at small $K$ and small $d$; as the number of arms and Cohen's $d$ increase, the Bayes shrinkage estimator and bootstrap corrections become more competitive. For MSE (Figure (ref)), the Bayes estimator performs best at small $K$, while bootstrap corrections overtake it as $K$ grows. For coverage (Figure (ref)), empirical likelihood and andrews2024inference methods provide almost nominal coverage across the board. For regret (Figure (ref)), the plug-in estimator achieves the lowest regret across all $K$.
Addressing the winner's curse is important when the results of experiments are used to decide which treatment to deploy and to inform policy making. Our paper identifies that by itself, the term "winner's curse" does not imply a well-defined objective, and there are three different goals that an experimenter might care about: the global winner's curse, the selected winner's curse, and regret (to the optimal decision). While each of these three objectives have been explored in past literature, we provide a framework to evaluate all three and discuss when each case is relevant. Our Winner's Curse identity in (ref) links these three objectives together.
Our discussion of the the problem from a statistical perspective showed that the standard plug-in estimator (the max of two mean estimator) exhibits a Winner's Curse, causing an upward bias over the true value. The estimator contains a kink, or non-differentiable point, when the group means are equal, which poses a challenge for estimation and inference.
By reviewing and categorizing the different solutions in the literature, we establish the statistical properties for each estimator: bias, mean-squared error (MSE), and coverage of the confidence interval around the true value. We then provide a simulation study in (ref) to contrast these solutions, where we focus on the most common scenarios an experimenter might encounter: null effects, small effects, moderate effects, and large effects.
Methodologically, we provide a novel adaptation of empirical likelihood (EL) techniques for conducting inference and constructing confidence intervals. EL offers a non-parametric approach for likelihood-based inference. We first account for the kink in the problem by adapting critical value to be a chi-bar-squared distribution instead of the typical $\chi^2$ distribution used for a likelihood ratio test. We then propose an adaptive procedure based on pre-tests and theoretically ensure this adaptive procedure provides pointwise asymptotic coverage of the confidence interval ((ref) and (ref)). In practice, since we cannot know whether we are at the kink, the adaptive procedure ensures that our constructed confidence intervals attains proper coverage both at or away from the kink.
We then provide a calibrated, simulation to a real-world A/B testing platform, with many arms and non-normally distributed arms. Increasing the number of arms does have implications for computational complexity and sample size. If more arms imply smaller samples per arms, many of the methods that underperform in the near-tie regime suffer more. This is especially true for bootstrap methods. The computational burden of bootstrap methods with multiple arms also makes them less feasible, making the applicability of our EL method even more relevant.
(ref) summarizes our main simulation results. The optimal correction method depends critically on the anticipated treatment effect regime. When group means are equal or close (at or near the kink), cross-fitting performs best for bias reduction, while bootstrap bias correction achieves the lowest MSE. The efficiency loss inherent in sample-splitting explains why cross-fitting underperforms on MSE despite its strong bias properties. As group means diverge, the plug-in estimator increasingly dominates: It performs best for large treatment effects across all objectives and consistently minimizes regret regardless of effect size. For inference, our empirical likelihood method achieves correct coverage across the simulation designs and proves less sensitive to tuning parameters than resampling alternatives.
There are direct implications from our results for applied work. First, the choice of the global winner's curse ($WC_{global}$), selected winner's curse ($WC_{select}$), and regret will depend on the setting the decision-maker is facing. If they aim to choose the best option and not re-optimize in the future, then the selected winner's curse will apply as they care about bias and uncertainty of their chosen option. If they aim to repeatedly experiment and choose the best arm over time, then our global winner's curse analysis will provide them guidance. However, if the decision-maker is concerned about the loss from not implementing the optimal treatment, then our analysis with regret will apply. While the three scenarios are linked by our identity, they are not identical and thus the optimal solutions for each will be different. As a result, decision-makers need to determine which solution best suits their setting.