Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
51,660 characters · 14 sections · 20 citation commands
Evaluating A/B Testing Methodologies via Sample Splitting: Theory and Practice
Measuring the impact of proposed improvements to the way in which A/B tests are analyzed and used to inform decision making is critical for large-scale A/B testing products. Over the last few decades, there have been several proposed improvements to A/B testing methodologies, including new estimators of impacts (for example, deng_etal_2013, deng_etal_2023, and deng_etal_2024) and new rules for deciding when to launch (for example, goldberg_johndrow_2017, azevedo_etal_2019 and azevedo_etal_2020). The scope of proposed improvements continues to expand, with more recent work exploring AI-driven approaches (for example, delriochanona_etal_2023 and delriochanona_etal_2025). A/B testing products must determine which of these methodologies deliver meaningful value relative to their implementation costs.
The key challenge is that the true impacts of A/B tests are unobservable. Each test yields only a noisy estimate of the true impact, making it difficult to tell whether a proposed methodology actually leads to an overall improvement in estimates or decision making across tests. While simulation methods can be used to evaluate performance, they require strong assumptions about the true underlying data generating process.
Sample splitting offers a potential solution to this problem (tripuraneni_etal_2023), but its theoretical foundation remains undeveloped. Under sample splitting, each A/B test is divided into two, non-overlapping splits: one to implement the proposed methodology, the other to construct an independent, unbiased proxy for the true impact. Comparing these two quantities across A/B tests allows us to measure the average performance of the proposed methodology, without making strong distributional assumptions (tripuraneni_etal_2023). However, the statistical properties of these sample-split estimators have yet to be characterized formally: their bias, variance, and asymptotic distributions remain unknown, and there are no established methods for conducting statistical inference. Without such foundations, products lack guidance on exactly how to use sample splitting, limiting its adoption in practice.
In this paper, we introduce a theoretical framework for sample splitting in A/B testing environments, characterizing both what these methods can measure and how to conduct valid inference. We show that while sample splitting enables performance evaluation when true impacts are unobserved, it does so at a cost. Because methodologies are implemented on only a split of the data, sample-split estimators are generally biased for what we ultimately care about --- how the methodologies perform on the full sample. We show that sample-split estimators consistently estimate sample-split analogues of full-sample performance, derive their asymptotic distributions, and show how to construct valid asymptotic confidence intervals. We demonstrate that while a careful choice of how the sample is split can reduce bias for full-sample performance, this improvement generally comes at the cost of increased variance. We validate our main theoretical results through simulations.
Our theoretical framework has important practical implications for A/B testing products. First, it shows that, in some cases, products can dramatically reduce the computational costs of sample splitting by simulating sample-split estimates rather than repeatedly partitioning unit-level data. Second, it provides a principled approach to statistical inference, enabling products to conduct hypothesis testing (for example, to determine whether one methodology significantly outperforms another) and statistical power calculations (for example, to determine how many A/B tests are needed to reliably detect a given performance difference between methodologies). Third, it demonstrates that common sample splitting practices --- such as splitting data equally or using hundreds of random partitions --- are often suboptimal. These insights, together with the step-by-step algorithms we provide, enable products to implement principled sample splitting efficiently at scale, making it a practical and powerful tool for evaluating proposed methodological improvements.
More generally, our work contributes to the literature on cross-validation and resampling methods (stone_1971). Like cross-validation, sample splitting separates the application of a methodology from its evaluation by partitioning data. The key difference is that in cross-validation the prediction target is observed directly, whereas in our setting the prediction target is itself an estimate subject to sampling variation. To our knowledge, tripuraneni_etal_2023 were the first to apply the idea of sample splitting in the A/B testing context, building upon earlier work in the estimation of heterogeneous treatment effects by athey_imbens_2016 and chernozhukov_2025. tripuraneni_etal_2023 demonstrate the potential of sample splitting in evaluating the performance of A/B testing methodologies but leave its statistical properties largely unformalized.\footnote{The authors acknowledge that their work is guided primarily by intuition and heuristics rather than principled theory. For example, they note the use “$t$-test heuristics” (page 7) and suggest that the confidence intervals they report “should be interpreted cautiously” (page 11).} We extend their work by introducing a theoretical framework for sample splitting in A/B testing environments, characterizing both what these methods can measure and how to conduct valid inference.
We consider a set of $i \in \{1,...,I\}$ feature changes, each evaluated through an A/B test where the change is exposed to a random subset of units. Each feature change has a true impact $\Delta_i$, which is a random variable with realized values $\delta_i$. Each test is further characterized by latent variables $Z_i$, with realized values $z_i$, capturing features of the testing environment such as sample size and outcome variability. The pairs $(\Delta_i, Z_i)$ are independently and identically distributed according to an unknown distribution $G$.
Conditional on the true impact $\Delta_i = \delta_i$ and test environment $Z_i = z_i$, the A/B test generates random data $\hat{X}_i$ according to a sampling distribution $F_i$, with realized values $\hat{x}_i$. To simplify notation, we suppress the dependence on $i$ and write $F$ in place of $F_i$. This sampling distribution reflects the sampling of units and random assignment of treatment in each test and may vary across tests through the true impact and test environment $(\Delta_i, Z_i)$.
We define an A/B testing methodology $\kappa(\hat{x}_i)$ to be a mapping of the data $\hat{x}_i$ to a real-valued output. The methodology could be an estimator of the true impact $\delta_i$ or a decision rule that determines whether to launch the given treatment ($\kappa(\hat{x}_i) = 1$) or not ($\kappa(\hat{x}_i) = 0$).
Given the randomization of the feature change and the central limit theorem, we assume that each test yields an unbiased estimator $\hat{\Delta}_i$ of the impact $\delta_i$ that is normally distributed, with realized values $\hat{\delta}_i$ and sampling variance $\tau_i^2 = \tau_i^2(\delta_i, z_i)$:
Following standard practice in the A/B testing literature (see, for example, azevedo_etal_2020), we treat $\tau_i^2$ as known for each test, while allowing it to vary across tests.
We measure the performance of the methodology $\kappa(\hat{x}_i)$ given a true impact of $\Delta_i = \delta_i$ via a real-valued performance function $p(\kappa(\hat{x}_i), \delta_i)$. As illustrated in the examples below, the specification of the performance function depends on the type of methodology being evaluated.
Example 1 (Measuring performance of an estimator): We may want to measure the performance of a shrinkage estimator $\kappa(\hat{x}_i) = w_i \cdot \hat{\delta}_i$ for $w_i \in (0, 1)$ under squared error in which case we could specify the performance function $p(\kappa(\hat{x}_i), \delta_i) = (\kappa(\hat{x}_i) - \delta_i)^2$.
Example 2 (Measuring performance of a decision rule): We may want to measure the performance of the decision rule $\kappa(\hat{x}_i) = 1(\hat{\delta}_i > \tau_i \cdot \Phi^{-1}(0.975))$ to launch treatments when the estimated impact is positive and statistically significant at the 5 percent level. In this case, we could specify the performance function $p(\kappa(\hat{x}_i), \delta_i) = \kappa(\hat{x}_i) \cdot \delta_i - (1 - \kappa(\hat{x}_i)) \cdot \delta_i$ to capture the idea that we get $\delta_i$ when we launch and $-\delta_i$ when we do not. We hereafter refer to this measure of performance and its counterpart $p(\kappa(\hat{x}_i), \delta_i) = \kappa(\hat{x}_i) \cdot \delta_i$ as decision value and launch-only decision value, respectively.
In what follows, for any random variable $K_i$, we write $F_K$ to denote its distribution induced by the sampling distribution $F$ and $G_K$ to denote its distribution induced by the distribution $G$. We write $E_F(\cdot)$ and $\text{Var}_F(\cdot)$ for the expectation and variance with respect to randomness induced by $F$, and $E_G(\cdot)$ and $\text{Var}_G(\cdot)$ for the expectation and variance with respect to randomness induced by $G$. Finally, we write $E(\cdot)$ and $\text{Var}(\cdot)$ for the expectation and variance with respect to both sources of randomness, integrating over both $F$ and $G$. We denote by $\phi(\cdot)$ and $\Phi(\cdot)$ the probability density function and cumulative distribution function of the standard normal distribution, respectively.
Our goal is to quantify how well different methodologies perform across A/B tests. To this end, we begin by defining a set of ideal estimands that capture how we would measure performance if the true impacts $\delta_i$ were observable. These estimands serve as our conceptual targets and help clarify the sense in which our estimators (introduced in section (ref)) may be biased for what we ultimately want to measure.
For each A/B test $i$, the expected performance of a methodology $\kappa$ under performance measure $p$ given a true impact of $\Delta_i = \delta_i$ and test environment $Z_i = z_i$ is given by:
Here $\kappa(\hat{X}_i)$ is the methodology's output (estimate or launch decision) given data $\hat{X}_i$. The function $p(\kappa(\hat{X}_i), \delta_i)$ measures how well this output performs given a true impact $\delta_i$. Taking the expectation over the sampling variability induced by $F$ averages out the randomness in the data $\hat{X}_i$, yielding the expected performance $Y_i(\kappa)$ for a given true impact and test environment.
While $Y_i(\kappa)$ captures the expected performance of $\kappa$ for a given test, it varies across tests due to variation in $(\Delta_i, Z_i)$:
One ideal estimand is therefore the average performance of the methodology across tests, $\theta(\kappa)$. The quantity $\theta(\kappa)$ accounts for both the sampling variation in each A/B test and the variation in true impacts across tests, making it a natural target when evaluating methodological performance at scale.
While average performance can be helpful in benchmarking a given methodology, we are often more interested in differences in average performance across methodologies. Another ideal estimand is therefore:
where $\kappa^1$ and $\kappa^2$ are different methodologies, and we normalize by $\theta(\kappa^1)$ so that $\theta(\kappa^1, \kappa^2)$ can be interpreted as the percent change in performance of methodology $\kappa^2$ relative to $\kappa^1$.
Example 3 (Illustrating performance estimands for estimators): Assume that $G$ is such that $\tau_i = \tau$ for all $i=1,..., I$ and $\Delta_i \sim N(0, \sigma^2)$. Consider evaluating the relative performance of two estimators under squared error $p(\kappa(\hat{x}_i), \delta_i) = (\kappa(\hat{x}_i) - \delta_i)^2$: the unbiased estimator $\kappa^1(\hat{x}_i)=\hat{\delta}_i$ and the Bayes estimator $\kappa^2(\hat{x}_i) = \sigma^2 (\sigma^2 + \tau^2)^{-1} \hat{\delta}_i$. In this case, we have that:
consistent with the fact that the Bayes estimator minimizes mean squared error (casella_berger_2002).
Example 4 (Illustrating performance estimands for decision rules): Assume that $G$ is such that $\tau_i = \tau$ for all $i=1,..., I$ and $\Delta_i \sim N(0, \sigma^2)$. Consider evaluating the relative performance of two decision rules under launch-only decision value $p(\kappa(\hat{x}_i), \delta_i) = \kappa(\hat{x}_i) \cdot \delta_i$: a frequentist decision rule $\kappa^1(\hat{x}_i)= 1(\hat{\delta}_i > \tau \cdot \Phi^{-1}(0.975))$ which launches treatments when the estimated impact is positive and statistically significant at the 5 percent level and a Bayes decision rule $\kappa^2(\hat{x}_i) = 1(\sigma^2 (\sigma^2 + \tau^2)^{-1} \hat{\delta}_i > 0)$ which launches treatments when the posterior mean is positive. In this case, we have that:
consistent with the fact that the Bayes decision rule maximizes mean launch-only decision value (azevedo_etal_2020).
Note that these examples make restrictive assumptions about the distribution $G$ only to help make the estimands concrete. Our sample splitting framework, introduced below, is designed to evaluate performance of methodologies without requiring any assumptions about $G$.
We evaluate the performance of methodologies via sample splitting, where we partition each A/B test into two splits: one to implement the proposed methodology, the other to construct an independent, unbiased proxy for the true impact. This approach enables performance measurement when true impacts are unobserved (tripuraneni_etal_2023). However, because the proposed methodology is implemented on only a split of the sample, we cannot directly estimate the ideal estimands in section (ref). In what follows, we outline a general approach to generating sample-split data, which applies to any methodology. We then show that this approach can be simplified for “plug-in methodologies” that depend on the data only through a set of sufficient statistics. With this data foundation, we then define sample-split analogues of the ideal estimands, introduce corresponding unbiased estimators, and outline an approach to statistical inference.
We implement sample splitting as follows. For each A/B test, we randomly partition the units assigned to each experience (control and treatment) into two splits, labeled $a$ and $b$, with a fraction $\alpha \in (0, 1)$ allocated to split $a$ and the remaining $(1-\alpha)$ to split $b$. For a given random partition $s$, let $\hat{X}_{ias}$ and $\hat{X}_{ibs}$ be the data associated with units in splits $a$ and $b$, with $(\hat{x}_{ias} \cup \hat{x}_{ibs}) = \hat{x}_{i}$ and $(\hat{x}_{ias} \cap \hat{x}_{ibs}) = \emptyset$. We use split $a$ for training, applying both the methodology of interest and the unbiased estimator $\hat{\Delta}_i$ of $\delta_i$. This yields an estimate or launch decision $\kappa(\hat{x}_{ias})$ and an estimate $\hat{\delta}_{ias}$. We use split $b$ for evaluation, applying the unbiased estimator $\hat{\Delta}_i$ of $\delta_i$ to obtain a proxy for the true impact $\hat{\delta}_{ibs}$. Because we treat $\tau_i^2$ as known, we do not estimate sampling variances of $\hat{\Delta}_{ias}$ and $\hat{\Delta}_{ibs}$ based on data but instead compute them deterministically based on $\tau_i^2$ and the split fraction $\alpha$. Specifically, $\tau_{ias}^2 = \tau_i^2/\alpha$ and $\tau_{ibs}^2 = \tau_i^2 / (1 - \alpha)$, reflecting the standard inverse relationship between sampling variance and sample size.
Because the methodology $\kappa(\hat{X}_{ias})$ and the proxy for the true impact $\hat{\Delta}_{ibs}$ are based on disjoint samples of units, they are statistically independent conditional on the true impact $\Delta_i = \delta_i$ and testing environment $Z_i = z_i$:
Additionally, because the partitions are determined at random, the proxy for the true impact $\hat{\Delta}_{ibs}$ is (under equation ((ref))) unbiased and normally distributed conditional on the true impact $\Delta_i = \delta_i$:
Together, equations ((ref)) and ((ref)) provide the statistical foundation for our sample splitting methodology. The conditional independence property in equation ((ref)) ensures that noise affecting the methodology's output does not also influence its validation, guarding against spurious estimates of performance. The normality and unbiasedness properties in equation ((ref)) ensure that split $b$ provides a valid proxy for the true impact $\delta_i$.
A single partition suffices to estimate performance, but results can vary substantially across different partitions. To reduce this variability, we repeat our sample splitting procedure $S$ times, generating new random partitions that each divide units into splits $a$ and $b$. This yields estimates $\{\kappa(\hat{x}_{ias}), \hat{\delta}_{ias}, \tau_{ias}^2, \hat{\delta}_{ibs}, \tau_{ibs}^2\}_{s=1}^S$ which we then average for each test in hopes of obtaining more stable results. Importantly, while splits $a$ and $b$ remain independent within each partition $s$ (maintaining the property in equation ((ref))), they are not independent across partitions, as they are based on overlapping data. Our approach to estimation and inference, outlined in section (ref), takes this dependence into account.
While the previous section illustrated how sample-split data can be generated by repeatedly partitioning the unit-level data of a test, many methodologies depend on the unit-level data only through the sufficient statistics $(\hat{\delta}_{ias}, \tau_{ias}^2)$. We refer to these as “plug-in methodologies”, as they simply plug the estimates $(\hat{\delta}_{ias}, \tau_{ias}^2)$ into deterministic functions, without changing the way in which the estimates themselves are constructed. Plug-in methodologies include shrinkage estimators $\kappa(\hat{x}_{ias}) = w_i \cdot \hat{\delta}_{ias}$ for $w_i \in (0, 1)$ and threshold decision rules $\kappa(\hat{x}_{ias}) = 1(\hat{\delta}_{ias} > \tau_{ias} \cdot c)$ for launch threshold $c \geq 0$.
For these methodologies, we can simplify sample-split data generation. Instead of repeatedly partitioning unit-level data and re-computing estimates, we can directly sample $(\hat{\Delta}_{ias}, \hat{\Delta}_{ibs})$ from their joint distribution given the full-sample estimates $(\hat{\delta}_i, \tau_i^2)$ and true impact $\delta_i$. This shortcut, formalized in the proposition below, eliminates the need for unit-level data in partitioning while preserving the statistical guarantees that sample splitting provides in performance evaluation.
Proposition 1 (Evaluating plug-in methodologies without unit-level partitioning): Let splits $a$ and $b$ be formed by randomly partitioning units in an A/B test with proportions $\alpha$ and $(1-\alpha)$, respectively. Under equation ((ref)), the joint distribution of $(\hat{\Delta}_{ias}, \hat{\Delta}_{ibs})$ conditional on the full-sample estimates $(\hat{\delta}_i, \tau_i^2)$ and true impact $\delta_i$ is given by:
For a given test, the process of repeatedly partitioning units and re-estimating $(\hat{\delta}_{ias}, \hat{\delta}_{ibs})$ can therefore be approximated by resampling from the conditional distribution in equation ((ref)) given the full-sample estimates $(\hat{\delta}_i, \tau_i^2)$. Given draws from this distribution, plug-in methodologies (being deterministic functions of $(\hat{\delta}_{ias}, \tau_{ias}^2)$) can then be applied, yielding estimates $\{\kappa(\hat{x}_{ias}), \hat{\delta}_{ias}, \tau_{ias}^2, \hat{\delta}_{ibs}, \tau_{ibs}^2\}_{s=1}^S$.\footnote{Equation ((ref)) implies that, conditional on full-sample estimates $(\hat{\delta}_i, \tau_i^2)$, $\hat{\Delta}_{ias}$ and $\hat{\Delta}_{ibs}$ are perfectly negatively correlated across partitions. This follows from the fact that $\hat{\Delta}_i = \alpha \hat{\Delta}_{ias} + (1-\alpha) \hat{\Delta}_{ibs}$, so increases in one require decreases in the other.}
Proof: See appendix (ref) $\hfill \square$
Proposition 1 is similar in spirit to lewis_2025, who generate “doppelganger” observations without access to unit-level data by deconvolving the overall estimates for a test.
By eliminating the need for unit-level data in partitioning, proposition 1 lowers the computational cost of evaluating plug-in methodologies. Without the shortcut, evaluating plug-in methodologies requires processing $N$ unit observations for each of the $I$ tests $S$ times. With it, we simply draw from the conditional distribution $S$ times for each of the $I$ tests. In contexts with millions of units and thousands of tests, the savings can be substantial, enabling us to increase the scale of our analyses by orders of magnitude and obtain significantly more precise estimates of performance.
However, not all methodologies can be evaluated via this shortcut. The shortcut relies on the methodology treating the estimates $(\hat{\delta}_{ias}, \tau_{ias}^2)$ as sufficient statistics for the underlying data $\hat{x}_{ias}$. Some methodologies --- such as covariate adjustment, stratification, and outlier handling --- affect how these estimates are constructed and therefore still require repeated unit-level partitioning.
To evaluate performance of methodologies under sample splitting, we introduce generalized, sample-splitting analogues of the ideal estimands introduced in section (ref). These estimands reflect that performance depends not only on the methodology $\kappa$ and performance function $p$, but also on the share of the sample $\alpha$ used in implementing the methodology. Given $\alpha$, the expected performance of a methodology $\kappa$ under performance measure $p$ given a true impact of $\Delta_i = \delta_i$ and testing environment $Z_i = z_i$ is given by:
This generalization nests the ideal case: as $\alpha \rightarrow 1$ the methodology is implemented on the full sample and we recover the ideal performance measure in equation ((ref)).
Just like its ideal counterpart $Y_i(\kappa)$, $Y_i(\kappa, \alpha)$ varies across tests due to variation in $(\Delta_i, Z_i)$:
We denote realizations of $Y_i(\kappa, \alpha)$ by $y_i(\kappa, \alpha)$.
As in section (ref), our sample-splitting estimands are then the average performance of the methodology across tests, $ \theta\left(\kappa, \alpha\right)$, and percent differences in average performance across methodologies:
Example 5 (Illustrating performance estimands for estimators, continued): Returning to example 3, we now consider the sample-split analogue of the difference in mean squared error across the estimators. For any $\alpha < 1$, sample splitting overestimates the performance benefits of the Bayes estimator. Replacing $\tau^2$ with $\tau^2 / \alpha$, we have that:
For example, when $\sigma^2 = \tau^2 = 1$, the Bayes estimator reduces mean squared error by 50 percent. Sample splitting with $\alpha = 0.50$ would, however, suggest a reduction of 66 percent.
Example 6 (Illustrating performance estimands for decision rules, continued): Returning to example 4, we now consider the sample-split analogue of the difference in launch-only decision value across the decision rules. For any $\alpha < 1$, sample splitting overestimates the performance benefits of the Bayes decision rule. Replacing $\tau^2$ with $\tau^2 / \alpha$, we have that:
For example, when $\sigma^2 = \tau^2 = 1$, the Bayes decision rule increases the launch-only decision value by 160 percent. Sample splitting with $\alpha = 0.50$ would, however, suggest a benefit of 260 percent.
As these examples illustrate, the sample-split estimands can differ substantially from their ideal counterparts, with the magnitude of this difference governed by the split fraction $\alpha$. In subsequent sections, we discuss how to estimate these sample-split estimands and the tradeoffs underlying different choices of $\alpha$.
Sample splitting enables us to estimate the performance of methodologies by replacing the unobserved true impacts $\delta_i$ with the independent, unbiased proxies $\hat{\Delta}_{ibs}$ from split $b$. This leads to natural estimators of the sample-split estimands $Y_i(\kappa, \alpha)$ introduced in the previous section.
For each A/B test $i$ and partition $s$, we consider an estimator $\hat{Y}_{is}(\kappa, \alpha)$ whose average across partitions $\hat{Y}_{i}(\kappa, \alpha, S)=\frac{1}{S}\sum_{s=1}^S \hat{Y}_{is}(\kappa, \alpha)$ satisfies:
The variance $\Omega_i^2\left(\kappa, \alpha, S\right)$ reflects both sampling variability in the full-sample data $\hat{X}_i$ given the true impact $\Delta_i = \delta_i$ and testing environment $Z_i = z_i$ and variability in the sample-split data $(\hat{X}_{ias}, \hat{X}_{ibs})$ given the full-sample data.
Given $\hat{Y}_{i}(\kappa, \alpha, S)$ satisfying equation ((ref)), we estimate the average performance of methodologies across tests and differences in average performance across methodologies. We estimate the average performance across tests $\theta(\kappa, \alpha)$ via:
We estimate the difference in average performance across methodologies $\theta(\kappa^1, \kappa^2, \alpha)$ via:
Because each $\hat{Y}_{i}(\kappa, \alpha, S)$ is unbiased for $y_i(\kappa, \alpha)$, the estimator $\hat{\theta}(\kappa, \alpha, S)$ is unbiased for $\theta(\kappa, \alpha)$. We formalize this via the following proposition.
Proposition 2 (Unbiasedness): Under equations ((ref)) and ((ref)), the estimator of average performance $\hat{\theta}(\kappa, \alpha, S)$ is unbiased for $\theta(\kappa, \alpha)$.
Proof: See appendix (ref) $\hfill \square$
The challenge is therefore to construct estimators $\hat{Y}_{is}(\kappa, \alpha)$ that satisfy equation ((ref)) for different performance functions $p$. Table (ref) presents such estimators for common choices of $p$, including bias, mean squared error, and decision value. We provide proofs of the unbiasedness condition in ((ref)) for each estimator in appendix (ref), relying on the sample splitting conditions in equations ((ref)) and ((ref)).
Although $\hat{\theta}(\kappa, \alpha, S)$ is unbiased for $\theta(\kappa, \alpha)$, the estimator of the difference in average performance across methodologies $\hat{\theta}(\kappa^1, \kappa^2, \alpha, S)$ is generally biased for $\theta(\kappa^1, \kappa^2, \alpha)$. The bias arises from the nonlinearity of the percent change: even when $\hat{\theta}(\kappa^1, \alpha, S)$ and $\hat{\theta}(\kappa^2, \alpha, S)$ are unbiased for $\theta(\kappa^1, \alpha)$ and $\theta(\kappa^2, \alpha)$, respectively, their ratio $\hat{\theta}(\kappa^1, \kappa^2, \alpha, S)$ need not be unbiased for $\theta(\kappa^1, \kappa^2, \alpha)$.
Having established the bias properties of the estimators $\hat{\theta}(\kappa, \alpha, S)$ and $\hat{\theta}(\kappa^1, \kappa^2, \alpha, S)$, we now turn to their large-sample behavior. In practice, we care not only about whether the estimators are close to their true values on average, but also about their sampling variability and the rate at which they converge to their true values as the number of tests $I$ grows large.
We summarize our results via the two following propositions. For brevity, we focus the propositions on $\hat{\theta}(\kappa, \alpha, S)$; analogous results for $\hat{\theta}(\kappa^1, \kappa^2, \alpha, S)$ follow immediately from standard asymptotic theory.
Proposition 3 (Asymptotic normality): Under equations ((ref)) and ((ref)) and the regularity condition that $\sup_i E(\hat{Y}_i(\kappa, \alpha)^2) < \infty$, the estimator $\hat{\theta}(\kappa, \alpha)$ is asymptotically normally distributed:
where $\zeta^2(\kappa, \alpha, S) = \lambda^2(\kappa, \alpha) + E_G\left(\Omega_i^2\left(\kappa, \alpha, S\right)\right)$.
Proof: See appendix (ref) $\hfill \square$
Proposition 3 shows that the estimator $\hat{\theta}(\kappa, \alpha, S)$ is asymptotically normally distributed. Its asymptotic variance $\zeta^2(\kappa, \alpha, S)$ captures heterogeneity in true performance across tests via $\lambda^2(\kappa, \alpha)$ and the average sampling variance of estimated performance across tests via $E_G\left(\Omega_i^2\left(\kappa, \alpha, S\right)\right)$.
Proposition 4 shows that $\zeta^2(\kappa, \alpha, S)$ can be consistently estimated, enabling valid asymptotic inference.
Proposition 4 (Valid asymptotic inference): Under equations ((ref)) and ((ref)) and the regularity condition that $\sup_i E(\hat{Y}_i(\kappa, \alpha, S)^2) < \infty$, the estimator:
is consistent for the asymptotic variance $\zeta^2(\kappa, \alpha, S) = \lambda^2(\kappa, \alpha) + E_G\left(\Omega_i^2\left(\kappa, \alpha, S\right)\right)$. Therefore, the $(1-\upsilon)$ confidence intervals:
will cover $\theta(\kappa, \alpha)$ with probability $(1-\upsilon)$ as $I \longrightarrow \infty$.
Proof: See appendix (ref) $\hfill \square$
Together, these results enable principled estimation of the performance of methodologies via sample splitting. The unbiasedness of the estimators allows us to reliably estimate average performance under sample splitting, while the asymptotic normality enables us to quantify uncertainty in our estimates. The asymptotic confidence intervals can be used both for hypothesis testing (for example, to determine whether one methodology significantly outperforms another) and statistical power calculations (for example, to determine how many A/B tests are needed to reliably detect a given performance difference between methodologies). Algorithm 1 provides a step-by-step guide towards implementation.
Note, however, that care must be taken in applying these asymptotic results in smaller samples of A/B tests. The normal approximation in proposition 3 requires a sufficient number of A/B tests to be reliable. Similarly, the variance estimator in proposition 4 may be unstable in small samples, potentially leading to poor coverage of the resulting confidence intervals. Our simulation results in the next section suggest that these approximations work well with thousands of tests, but practitioners should exercise caution when working with significantly smaller samples.
In this section, we simulate a stylized A/B testing environment in which the difference in average performance between two methodologies is known. These simulations serve three purposes. First, they validate theoretical predictions outlined in previous sections. Second, they assess the adequacy of the asymptotic approximations underlying our approach to estimation and inference in finite samples. Third, they allow us to assess how performance of our estimators varies as a function of key sample-split design parameters.
We simulate from the following data generating process (DGP):
We vary five key parameters: the variance of true impacts $\sigma^2$, baseline sampling variance $\tau^2$, split fraction $\alpha$, number of A/B tests $I$, and number of random partitions $S$.
This DGP captures three key features of standard A/B testing environments. First, we capture heteroskedasticity across tests by modeling the sampling variance of the estimated impacts via $\tau_i^2 = \tau^2 + \epsilon_i$, where $\tau$ is a constant and $\epsilon_i$ is a chi-squared random variable with 1 degree of freedom. Second, we capture different signal-to-noise ratios by considering different pairwise combinations of the variance of true impacts $\sigma^2$ and the baseline sampling variance $\tau^2$. Third, we capture the real-world correlations across partitions $s$ for a given test by drawing $(\hat{\Delta}_{ias}, \hat{\Delta}_{ibs})$ from their joint distribution given the full-sample estimates $(\hat{\delta}_i, \tau_i^2)$.
As a benchmark specification, we consider $I = 5K$ A/B tests, $S=30$ partitions, a split fraction $\alpha = 0.5$, a variance of true impacts $\sigma^2 = 1$, and a baseline sampling variance $\tau^2 = 2$.
We use the simulations to evaluate the relative performance of different point estimators and decision rules. For point estimators, we compare the performance of the unbiased estimator $\kappa^1(\hat{x}_i)=\hat{\delta}_i$ to that of the Bayes estimator $\kappa^2(\hat{x}_i) = \sigma^2 (\sigma^2 + \tau_i^2)^{-1} \hat{\delta}_i$ under squared error, building upon example 3. For decision rules, we compare the performance of the frequentist decision rule $\kappa^1(\hat{x}_i)= 1(\hat{\delta}_i > \tau_i \cdot \Phi^{-1}(0.975))$ to that of a Bayes decision rule $\kappa^2(\hat{x}_i) = 1(\sigma^2 (\sigma^2 + \tau_i^2)^{-1} \hat{\delta}_i > 0)$ under launch-only decision value, building upon example 4.
For each methodology type, we measure the bias of the estimator $\hat{\theta}(\kappa^1, \kappa^2, \alpha, S)$, both relative to the ideal estimand $\theta(\kappa^1, \kappa^2)$ and the sample-split estimand $\theta(\kappa^1, \kappa^2, \alpha)$. We also measure the variance of $\hat{\theta}(\kappa^1, \kappa^2, \alpha, S)$ and the coverage of its 95 percent confidence interval. Finally, we assess how the bias and variance vary as a function of the number of tests $I$, number of partitions $S$, and split fraction $\alpha$.
Figure (ref) confirms the theoretical predictions outlined in previous sections. Sample splitting estimators are biased for the ideal estimands but are consistent for the sample-split estimands, resulting in negligible bias in the sample of $I = 5K$ A/B tests. The confidence intervals achieve nominal coverage, covering the sample-split estimand approximately 95 percent of the time.
Figure (ref) highlights a bias-variance tradeoff underlying the choice of the split fraction $\alpha$. As more data is allocated to the training split (that is, as $\alpha \rightarrow 1$), bias for the ideal estimand falls to zero but the variance of the estimated difference in performance rises. Under the baseline DGP, this produces a $U$-shaped curve in terms of mean-squared error, with the minimizing $\alpha$ greater than 0.5. We present results for alternative specifications of the DGP in appendix (ref).
Together, the results in figures (ref) and (ref) underscore both the value and limitations of sample splitting. Sample splitting makes estimation possible by using a split of the data to construct an independent, unbiased proxy for the true impact, thereby overcoming the fundamental challenge that true impacts are unobserved. However, because we implement the methodology on only a split of the data, we cannot construct unbiased estimators of the quantity we ultimately care about: the performance of the methodology if it had been implemented on the full sample. We can reduce bias for this ideal estimand by increasing $\alpha$ but this comes at the cost of increasing the variance of our estimates of performance.
Figure (ref) illustrates how the variance of sample-split estimators depends on both the number of A/B tests $I$ and partitions $S$. While increasing either dimension reduces variance, the marginal benefits of increasing $I$ and $S$ differ substantially. Increasing the number of tests $I$ consistently reduces variance. By contrast, increasing the number of partitions $S$ yields diminishing returns, with minimal improvements beyond $S = 30$ under the baseline DGP. This pattern reflects fundamental differences between additional tests and additional partitions: while increasing the number of tests results in new, independent information about performance, increasing the number of partitions eventually just provides different realizations of the same underlying information. We present results for alternative specifications of the DGP in appendix (ref).
We conclude by distilling our theoretical and simulation findings into five recommendations for sample-splitting practitioners: