Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
83,031 characters · 21 sections · 57 citation commands
On the Performance of the Neyman Allocation with Small Pilots
{\it JEL Classification:} C21, C90
{\it Keywords:} Experiment design, treatment effect estimation, alternative asymptotics
A growing literature on experiment design provides researchers with tools for reducing the asymptotic variance of their average treatment effect (ATE) estimates. Many do so in the context of two-wave experiments, where the researcher has access to a pilot study that can be used to improve the main study. Pilots are typically assumed to be large, allowing population parameters to be well-estimated. However, large pilots may not be realistic in practice. In this paper, we study the implications of small pilots for experiment design through the lens of the Neyman Allocation.
The Neyman Allocation (neyman1934two) is a simple method for minimizing the variance of the difference-in-means estimator of ATE. In a setting without covariates, suppose that the standard deviations of the treated and control outcomes are known. The Neyman Allocation assigns more units to either treatment or control in proportion to the ratio of their standard deviations. Intuitively, the optimal experiment entails more measurements of the noisier quantity. Since the variances are not known in practice, the feasible Neyman Allocation (FNA) estimates the variances using the pilot study and then plugs the estimates into the assignment rule.
The FNA is an important part of many experiment design procedures. In the econometrics literature alone there are several notable works. hahn2011adaptive propose to estimate the variance of outcome and control groups conditional on covariate value, implementing the FNA conditional on covariates. In a similar vein, tabord2021stratification employs tree-based techniques to stratify units based on their covariates. Units are then assigned to treatment and control based on the FNA conditional on strata. cytrynbaum2021designing proposes local randomization to select representative units for participation and treatment in experiments. In what cytrynbaum2021designing terms the “fully efficient" case, treatment proportion conditional on the randomization group is chosen by the FNA. Despite their differences, the above papers study their proposals in asymptotic frameworks that take the pilot size to infinity. Their analyses, appropriate for large pilots, essentially assume that population parameters are arbitrarily well-estimated from the pilots alone. In practice, pilots are often conducted for logistical reasons and may be small. In such settings, accurately estimating the relevant variances may be difficult. Indeed, all of the above authors caution against the use of their methods when pilot sizes are small.
To understand the implications of small pilots, we study the properties of the Neyman Allocation in an asymptotic framework that takes pilot size to be fixed even as the size of the main wave tends to infinity. In this setting, uncertainty in parameter estimation is non-negligible in the limit, and the FNA may not be the optimal allocation. In fact, we show that the FNA can do worse than the naive, balanced allocation that assigns half the units to treatment and half to control. This occurs when outcomes have similar variances across treatment and control -- that is, when outcomes are relatively homoskedastic with respect to treatment status. To assess how much homoskedasticity exists in practice, we examine the first 10 completed experiments in the AER RCT Registry. We ask the hypothetical question: if researchers conducted these studies as a two-wave experiment with a random sample from the same population, would they do better with the FNA or with balanced randomization? We find that the treatment and control groups are often highly homoskedastic across a range of outcomes and across experiments. This suggests that if faced with a small pilot, the authors of those studies would likely not have benefited from implementing the FNA. Finally, we show that as the pilot size increases, the amount of heteroskedasticity needed for the FNA to be preferable to the balanced allocation decreases, but at a rate that depends on the kurtosis of the outcome variables. Hence, even when researchers believe they are in a setting with high heteroskedasticity, they may want to avoid the FNA if they also believe that the outcomes are fat-tailed.
The above findings suggest that researchers should be cautious when designing experiments using the FNA with small pilots. However, even when pilots are large, methods which condition on many covariates may end up estimating the FNA using a small conditional sample. Furthermore, if researchers believe that units exhibit cluster-dependence -- a common assumption in empirical work -- the number of “effective" observations may be smaller still, impeding the estimation of the FNA.
However, our results do not imply that the FNA should never be used. Instead, researchers may consider alternative procedures with better performance under homoskedasticity. We explore three candidate solutions via simulations. One is based on testing for homoskedasticity, and the other two continuously regularize the FNA towards the balanced allocation. Our simulations suggest that the procedure we term “exponential regularization" offers the best trade-offs in that performance under heteroskedasticity is the least compromised in exchange for better performance under homoskedasticity. Alternatively, researchers might consider optimizing other aspects of the design, such as how strata are formed, or in the selection of covariates for stratification. bai2022optimality and cytrynbaum2023optimal discuss methods to robustify these approaches for small pilots.
The challenges that small pilots pose for the FNA were noted as early as sukhatme1935contribution, which is concerned with both small pilots and small main wave experiments. Given the relative intractability of the set-up, the author concludes using simulations that the FNA performs well. Authors such as hahn2011adaptive, tabord2021stratification, bai2022optimality have noted the limitations of their method in a small pilot setting. In particular, cytrynbaum2021designing,cytrynbaum2023optimal highlights that the FNA leads to estimators with residual variation from variance estimation when pilots are small. cytrynbaum2023optimal further shows that estimating stratification variables using small pilots may be counterproductive. In a network setting, viviano2022experimental characterizes the regret of the designed experiment as a function of pilot size, providing a rule for choosing pilot size. Our paper builds on this body of work by setting up balanced randomization as the natural alternative to the FNA when pilots are small, focusing on the relative efficiency of the two methods. Our analysis specifically addresses the use of the FNA in reducing the asymptotic variance of the difference-in-means estimator in two-wave experiments. It does not speak to papers which take the treatment assignment probability as given, such as bai2022optimality.
Our work stands in contrast to blackwell2022batch, who argue that the FNA can perform well even when pilots are small. However, their theoretical results rely on pilot sizes going to infinity, which we consider less suitable for addressing the small pilot case. Their simulations, which consider only normally distributed outcomes, may understate the challenges arising from high kurtosis.
Our paper is closely related to papers pointing out a similar issue in the design of sequential experiments. melfi1998variablility argue by simulation that treatment assignment rules based on estimated outcomes can do worse than non-adaptive rules due to estimation noise. Theoretical analysis is provided in hu2003optimality, in an asymptotic framework that does not nest ours. Our paper is also related to those studying small sample problems in experiments. deChaisemartin2019level are concerned with the problems small strata pose for inference. bruhn2009pursuit consider the effectiveness of various randomization strategies in achieving balance when a single-wave experiment is small. Finally, we note that there is a large literature discussing the large-pilot properties of the Neyman Allocation for alternative criteria such as power (e.g. brittain1982optimal, azriel2012optimal), minimax optimality (e.g. bai2021randomize) or ethical considerations (e.g. Chapter 8 of hu2006theory). These criteria fall outside the scope of this present paper.
The remainder of this paper is organized as follows. Section (ref) presents the theoretical framework. Section (ref) contains analytical and simulation results using a stylized toy example. Our main theoretical results can be found in Section (ref). We assess the level of homoskedasticity in selected empirical applications in Section (ref). Section (ref) concludes the paper. All proofs as well as additional empirical examples are contained in the online appendix.
We use a standard binary treatment potential outcomes framework assuming an infinite superpopulation. The potential outcomes are \((Y (0), Y (1))\), where \(Y (0)\) denotes the potential outcome under control or status quo and \(Y (1)\) denotes the potential outcome under treatment or the innovation.
The estimand of interest is the Average Treatment Effect (ATE), \(\theta = \mathbb{E} [Y (1) - Y (0)]\). To estimate the ATE, the experimenter conducts a two-wave experiment. The smaller first wave, also known as the pilot, is used to inform the experimenter about aspects of the design of the larger main wave (i.e. the second wave). The following assumptions about the two experimental waves will be maintained throughout the paper. For notational clarity, the tilde symbol (e.g. \(\widetilde{X}\)) refers to quantities associated with the pilot.
There are various ways in which the experimenter could implement treatment assignment in the main wave. We describe a few common ones below. Note that in all of these allocation schemes, treatment assignment is independent of potential outcomes.
In this section, we illustrate the main problems with the FNA in small samples with a toy model. In the context of this simple example, we ask the question: when does the FNA do worse than the balanced allocation? It turns out that this happens for a range of plausible values of population parameters. Section (ref) describes the extension of our findings into more general settings.
We assume in this section that the potential outcomes are bivariate normal:
Suppose we have a pilot of size $m$ where $m$ is even. Suppose treatment is assigned deterministically as follows:
Using standard arguments, it follows that the variance estimates are distributed as independent \(\chi^{2}\) random variables:
Conditional on the pilot sample, the FNA assigns
proportion of the units in the main wave to treatment. Suppose for simplicity again these are the first $n\widetilde{p}$ units (rounding $n\widetilde{p}$ if necessary). Then,
Terms in the above expression have conditional distributions:
Furthermore, they are independent. Hence,
Taking expectation over $\widetilde{p}$, we have that:
Letting $\widetilde{p}$ be constant at $p$, we obtain the variance of the difference-in-means estimator under simple random assignment as a special case:
Our goal is then to compare the two expressions above when we set $p = \frac{1}{2}$. First, note that we can rewrite (ref) as
where $Z_m \sim \sqrt{F\left(\frac{m}{2}-1, \frac{m}{2}-1\right)}$.
The Neyman Allocation does worse than balanced randomization whenever the following obtains:
The final implication uses the fact that $Z_m \overset{d}{=} 1/Z_m$ under bivariate normality and balanced randomization. By the quadratic formula, the above inequality is satisfied if and only if
Note that reciprocal symmetry together with Jensen's inequality guarantees that the discriminant is strictly positive:
Furthermore, we have that
In other words, the interval has the form
where $c_m > 1$ is the upper bound in (ref). Hence, there is a range of parameter values under which the FNA does strictly worse than balanced randomization. We first note that $1 \in C_m$ for all $m$. This is intuitive since $p = \frac{1}{2}$ is the infeasible Neyman Allocation when $\sigma(1)/\sigma(0) = 1$. Secondly, $x \in C_m \Leftrightarrow 1/x \in C_m$. That is, the relative performance of the FNA to the balanced allocation does not change when we relabel treatment and control.
Given an underlying distribution, it is simple to compute $C_m$ by Monte Carlo integration. In this subsection, we present the values of $C_m$ for some simple models and argue that for plausible values of $\sigma(1)/\sigma(0)$, the FNA performs worse than the balanced allocation.
We start with the toy model, where $Y(1) \sim {N}(\mu(1), \sigma^2(1))$ and $Y(0) \sim {N}(\mu(0), \sigma^2(0))$. Results are shown in the left panel of Figure (ref). The set of parameter values over which the FNA does worse is colored in blue. This area is larger when $m$ is smaller. In fact, we show in Section (ref) that if the data generating process (DGP) is sub-Gaussian, the length of $C_m$ is $O\left(m^{-1/2}\right)$. In the toy model, $C_{20} = [0.70, 1.43]$, while $C_{50}$ is $[0.81, 1.23]$. Suppose instead that $Y(1)\sim \mu(1) + \sigma(1) \chi^2_1$, $Y(0)\sim \mu(0) + \sigma(0) \chi^2_1$. The right panel of Figure (ref) shows that \(C_{m}\) is wider across the range of \(m\). In particular, $C_{20} = [0.49, 2.22]$, while $C_{50} = [0.64, 1.61]$. While the intervals may appear rather narrow at first glance, we provide numerous examples in Section (ref) in which the amount of heteroskedasticity falls within this range.
As our toy model shows, estimation of $\widetilde{\sigma}(1)$ and $\widetilde{\sigma}(0)$ causes problems when $m$ is small. It is therefore intuitive that $C_m$ will be wide when the distributions are fat tailed, that is, when kurtosis is high. As such, we consider the following parametrization: $Y(1)\sim \mu(1) + \sigma(1) \cdot \text{Pareto}(l, s)$ and $Y(0)\sim \mu(0) + \sigma(0) \cdot \text{Pareto}(l, s)$. Here, $l$ and $s$ are the location and scale parameters respectively. Figure (ref) plots $C_m$ for $l = 1$ and $s \in \{3,4,5\}$. Indeed, we see that the bands are much wider than in the previous models. For $s = 3$, even when $m = 50$, $C_m = [0.50, 2.15]$. Given the examples in Section (ref), this appears to be fairly extreme amounts of heteroskedasticity. Finally, we note that, $C_m$ decreases in width as we move from $s = 3$ to $s = 5$.
In sum, we see that the FNA does worse than the balanced allocation when the treatment and control groups are relatively homoskedastic. Furthermore, \(C_{m}\) can be quite large when \(m\) is small. The small pilot problem is exacerbated when observations exhibit cluster dependence, so that the “effective observations” are fewer in number. Small pilot issues can also arise when researchers perform stratified randomization with many strata, so that each stratum ends up with few observations. In Section (ref), we argue that many empirical applications in fact have fairly homoskedastic outcomes. As such, unless a researcher has reason to believe that their outcomes are highly heteroskedastic, they should exercise caution in using the FNA with small pilots.
In this section, we study the theoretical properties of the FNA. We first review known results on properties of the FNA under large and small pilot asymptotics. We then present novel results in which we compare the efficiency of FNA to balanced randomization.
Consider an asymptotic regime where both \(m, n \to \infty\), corresponding to situations where both the pilot and the main wave are large. It is straightforward to see that $\widetilde{p} \overset{p}{\to} p_*$. Furthermore, we have that:
The above result obtains by applying Theorem 1 in hahn2011adaptive to the case with only one stratum. In words, \( \widehat{\theta}_{\widetilde{p}} \) and \( \widehat{\theta}_{p_{\ast}} \) have the same limiting distribution after suitable centering and scaling. This occurs because in the limit, noise coming from estimating $p_*$ with \(\widetilde{p}\) is negligible in comparison to the sampling error of the difference-in-means estimator. Researchers employing the large $m$ framework essentially assume that \(\widetilde{p}\) is an arbitrarily good estimator for $p_*$ so that its sampling error can be ignored. In practice, error in \(\widetilde{p}\) can be large, particularly when $m$ is small. Asymptotic approximations that do not take this into account will likely perform poorly in finite sample.
To better understand the behavior of the FNA under small pilots, we follow cytrynbaum2021designing in studying its properties in an asymptotic framework that takes $m$ to be fixed, even as $n \to \infty$.
A growing literature in econometrics uses fixed-sample asymptotics to study settings in which the “effective sample size" is small. For example, when data exhibit cluster-dependence, settings with few clusters pose unique challenges for estimation and inference. To tailor their analyses to these problems, papers such as ibragimov2010t, canay2017randomization and canay2021wild employ asymptotic frameworks in which the number of clusters is fixed in the limit. Similarly, to model difference-in-differences studies involving few treated units, conley2011inference keep the number of treated units fixed even as the number of untreated units tend to infinity. In the same vein, inference and specification testing for regression discontinuity designs typically involves few observations around the discontinuity. To capture this, 2017canayApproximatePermutationTests analyze a permutation test for continuity in the distribution of baseline covariates under an asymptotic regime with a fixed number of observations on either side of the discontinuity.
The small pilot approach is similar in spirit to these papers. In keeping $m$ fixed, $\widetilde{p}$ is a noisy estimate of $p_*$ even in the limit. Preserving this important feature of the statistical problem makes our chosen framework more appropriate for analyzing experiments with small pilots. In this setting, $\widehat{\theta}_{\widetilde{p}}$ converges in distribution to a mixture of Gaussians instead of $\mathcal{N}(0, \Sigma_*)$. Specifically, the form of the limiting mixture distribution depends on the distribution of $\widetilde{p}$. This is the content of the following result due to cytrynbaum2021designing (also see the discussion following equation (3.5) of cytrynbaum2023optimal):
To see the intuition for this result, recall that for each \(p \in (0, 1)\), \[ \sqrt{n} \left( \widehat{\theta}_{p} - \theta \right) \overset{d}{\to} {N} \left( 0, \frac{\sigma^{2} (1)}{p} + \frac{\sigma^{2} (0)}{1 - p} \right) = {N} \left( 0, s^{2} (p) \right). \] When \(m\) is held fixed, in the limit as \(n \to \infty\), \(\widetilde{p}\) remains a non-degenerate random variable. In particular, the limiting distribution of $\widetilde{p}$ is its finite-sample distribution. Thus, the distribution of \(\sqrt{n} \left( \widehat{\theta}_{\widetilde{p}} - \theta \right)\) becomes a mixture of the marginal distributions of the process \[ \left\{ \sqrt{n} \left( \widehat{\theta}_{p} - \theta \right) : p \in (0, 1) \right\}~, \] where the mixing distribution is the distribution of \(\widetilde{p}\), denoted here by \(G_{m}\).
Proposition (ref) shows that when pilots are small, $\hat{\theta}_{\tilde{p}}$ has asymptotic distribution that is different from $\hat{\theta}_{p_\ast}$. It stands to reason that the FNA may not be optimal in this regime. In this sub-section, we compare the asymptotic variances of $\widehat{\theta}_{\widetilde{p}}$, \(\widehat{\theta}_{p_{\ast}}\) as well as \(\widehat{\theta}_{p}\) with \(p = \frac{1}{2}\). We first show that \(\widehat{\theta}_{\widetilde{p}}\) has larger asymptotic variance in the fixed-$m$ regime than in the large-$m$ regime. We next show that under reasonable ranges of parameter values, the asymptotic variance of \(\widehat{\theta}_{\widetilde{p}}\) can exceed that of \(\widehat{\theta}_{p}\) with \(p = \frac{1}{2}\).
We begin with the following observation:
In words, the asymptotic variance of $\widehat{\theta}_{\widetilde{p}}$ is larger under the fixed-$m$ regime than under the large-$m$ regime. When pilots are small, uncertainty in $\widetilde{p}$ may be large and could affect the asymptotic variance of $\widehat{\theta}_{\widetilde{p}}$. In particular, $\widehat{\theta}_{\widetilde{p}}$ will not be able to attain the optimal asymptotic variance of the infeasible allocation $p_*$. Conventional large-$m$ asymptotics may be too optimistic about the effectiveness of the Neyman Allocation with small pilots.
In addition to not attaining $\Sigma_*$, the $\widehat{\theta}_{\widetilde{p}}$ can do worse than $\widehat{\theta}_p$ for certain values of $\sigma^2(1)$ and $\sigma^2(0)$, as our next two results asserts. For convenience, define the following:
Here, $Z_m$ is our estimator of the ratio of $\sigma(1)/\sigma(0)$, appropriately normalized. $B_m-1$ is therefore the average bias of the estimators for $\sigma(1)/\sigma(0)$ and $\sigma(0)/\sigma(1)$, up to normalization. Our next proposition characterizes the region $C_m$ in terms of $B_m$:
The properties of $C_m$ are intuitive. Firstly, $x \in C_m$ implies that $1/x \in C_m$. That is, the relative performance of the FNA to the balanced allocation does not change when we relabel treatment and control. Secondly, $1 \in C_m$. This is because when $ {\sigma(1)}/{\sigma(0)} = 1$, the balanced allocation is optimal. Finally, note that $|C_m|$ depends on the bias of $\widetilde{\sigma}_m(1)/\widetilde{\sigma}_m(0)$ and $\widetilde{\sigma}_m(0)/\widetilde{\sigma}_m(1)$. In particular, if both terms are unbiased, $B_m = 1$ and $|C_m| = 0$. However, $\left\lvert C_m \right\rvert$ is strictly positive as long as $\widetilde{p}$ is not degenerate.
The exact properties of $C_m$ depend on the underlying distributions of potential outcomes. To understand its behavior in a more general setting, we study its first-order approximation under a sub-Gaussian assumption. Recall that \(F\) is the distribution of \((Y (0), Y (1))\) in Assumption (ref).
That is, a random variable is sub-Gaussian if its tails decay at least as quickly as a normal/Gaussian distribution. This yields the following:
Provided that the potential outcomes are sub-Gaussian, the relative efficiency of $\widehat{\theta}_{\widetilde{p}}$ and $\widehat{\theta}_p$ under $p = 1/2$ is, to a first order, determined by the kurtosis of $Y_i(1)$ and $Y_i(0)$. Intuitively, if the potential outcomes have fatter tails, $\widetilde{p}$ will be poorly estimated, leading to larger variance in $\widehat{\theta}_{\widetilde{p}}$.
Furthermore, $|C_m|$ is shrinking to $0$ at the rate $1/\sqrt{m}$. Letting $m \to \infty$, $\widehat{\theta}_{\widetilde{p}}$ has weakly lower asymptotic variance across all parameter values. Hence, we recover the classic result concerning the optimality of the Neyman Allocation. When $m$ is small, however, $C_m$ can be wide, as Proposition (ref) suggests. As we will argue in Section (ref), many empirical applications have $\sigma(1)/\sigma(0)$ close to $1$, so that for small $m$, these ratios fall within the range in which balanced randomization is preferred.
While the sub-Gaussian assumption limits the applicability of Proposition (ref), it covers binary and bounded outcomes, which are relevant in empirical work. Furthermore, we consider it to be a negative result: even when potential outcomes are well-behaved, the FNA is sensitive to the kurtosis of the potential outcomes. It can still perform poorly relative to the balanced allocation as a result.
The previous section showed that the FNA can perform worse than the balanced allocation when the pilots are small. This raises the question: what is the efficiency loss from using the FNA in such scenarios? To answer this question, consider the following criteria:
In words, we measure efficiency loss using either the difference or the ratio of the asymptotic variances arising from the FNA and the balanced allocation. Larger values of $\mathcal{L}^d$ and $\mathcal{L}^r$ indicate that the FNA is doing worse relative to balanced randomization.
Clearly, the efficiency losses depend on $F$. For a given $F$, the two criteria are equivalent in the sense that the difference criterion is positive if and only if the ratio criterion exceeds one. Depending on the set-up, one will be more convenient to work with than the other. From Definition (ref), it is straightforward to see that:
The above proposition shows us how the efficiency loss from using the FNA depends on $B_m$ as well as the amount of heteroskedasticity in $F$. To better understand the trade-off from the two terms, suppose we fix $B_m$ and reparametrize $\sigma(0) = 1/\sqrt{1+\rho^2}$, $\sigma(1) = \rho/\sqrt{1+\rho^2}$. Now let us consider the effect of changing $\rho$. This is the thought experiment in which we change the relative scaling of $\varepsilon(1) = Y(1) - \mu(1)$ and $\varepsilon(0) = Y(0) - \mu(0)$ while keeping their functional forms fixed. Additionally, this entails keeping $\text{AVar}(\hat{\theta}_{\frac{1}{2}}) = \text{Var}(Y(1)) + \text{Var}(Y(0))$ fixed at $2$. This scaling is useful for discussing the properties of $\mathcal{L}^d$ since it scales linearly with $\sigma(0)$ and $\sigma(1)$. $\mathcal{L}^r$ is invariant to the scale of $\sigma(0)$ and $\sigma(1)$. With this parametrization, we have that:
The above expressions are intuitive. When $\rho$ exceeds $1$, increasing $\rho$ amounts to increasing heteroskedasticity. FNA is more preferable to balanced randomization and the ratio loss decreases.
Furthermore, it straightforward to see that both $\mathcal{L}^d$ and $\mathcal{L}^r$ are maximized when $\rho = 1$, with the respective values of $B_m$ and $\frac{1}{2}(1+B_m)$. On the other hand, $\mathcal{L}^d$ and $\mathcal{L}^r$ have minimum values $-1$ and $\frac{1}{2}$ respectively, which are obtained when either $\rho = 0$ or $\rho \to \infty$. In summary, for a given functional form of $\varepsilon(0)$ and $\varepsilon(1)$, the loss from using FNA depends on $B_m$ but the maximum possible gain does not. This asymmetry foreshadows the next result.
Since $F$ is unknown, we proceed by comparing the supremum and infimum of $\mathcal{L}^d(F)$ and $\mathcal{L}^r(F)$ (i.e. maximum losses and gains) over appropriate classes of DGPs. To that end, consider the following.
Then we have that:
In words, the worst case loss from using the FNA -- whether measured as a difference or as a ratio relative to the variance of the balanced allocation -- is unbounded. On the other hand, the gains are bounded. Most strikingly, the fact that $\inf_{F \in \mathcal{F}(\infty)} \mathcal{L}^r(F) = \frac{1}{2}$ suggests that the best a researcher can do with the FNA is to obtain main wave confidence intervals for $\hat{\theta}$ that are $1/\sqrt{2} \approx 70\%$ the length of that from balanced randomization. This comes at the cost of potentially doing much worse as a result of noise from the pilot estimation.
We can also interpret the result in terms of regret. Suppose a researcher is constrained to either using the FNA or the balanced allocation. If their utility over the second stage estimator is given by $U(\hat{\theta}) = -\left({\hat{\theta}-\theta}\right)^2$, then $\mathcal{L}^d(F)$ is the asymptotic regret of choosing the FNA, while $-\mathcal{L}^d(F)$ is the asymptotic regret of choosing the balanced allocation. A similar argument can be made for $\mathcal{L}^r(F)$ with $U(\hat{\theta}) = -\log\left({\hat{\theta}-\theta}\right)$. Proposition (ref) then says that the regret of using the FNA over the balanced allocation is unbounded, while the regret of using the balanced allocation over the FNA is not. The balanced allocation is therefore preferred in terms of maximum regret. This is admittedly a very conservative way of choosing an estimation procedure. However, it is coherent in the following sense. Researchers may prefer the FNA if they want to guard against large amounts of heteroskedasticity in the population. However, if researchers are being agnostic about $\sigma(1)/\sigma(0)$, they should also be agnostic about $F$, in which case the balanced allocation may ultimately be preferable.
The efficiency losses recorded in Proposition (ref) are pessimistic and may not be relevant in practice. To better appreciate the losses that researchers might experience, we next present mean-squared errors for $\hat{\theta}_{\tilde{p}}$ under various data-generating processes, computed via Monte Carlo simulations. Our choice of parameter values is inspired by blackwell2022batch.
Let $Y(1) = 0.075 + \sigma(1)\varepsilon(1)$ and $Y(0) = \sigma(0)\varepsilon(0)$. We are interested in the MSE of $\hat{\theta}_{\tilde{p}}$ as $\sigma(1)/\sigma(0)$ varies over $[0,1]$. We do not need to consider values greater than $1$ given the reciprocal symmetry. We set $\sigma^2(1) + \sigma^2(0) = 2$ so that the variance of $\hat{\theta}_{\frac{1}{2}}$ is fixed. As in Section (ref), consider the following models:
where $\varepsilon(1)$ and $\varepsilon(0)$ are standardized to have mean $0$ and variance $1$. We set the pilot to have $m = 20$, while main wave has $n = 1000$. We run the simulations with 100,000 draws for each set of parameter values.
Results for Models 1 and 2 are presented in Figure (ref), while those for Models 3, 4 and 5 are presented in Figure (ref). First note that the MSE curves follow the theoretical description provided in Section (ref). The MSE of the FNA is monotonically increasing in $\sigma(1)/\sigma(0)$. As $\sigma(1)/\sigma(0) \to 0$, the MSE of $\hat{\theta}_{\tilde{p}}$ approaches 1/2 that of $\hat{\theta}_{\frac{1}{2}}$. As $\sigma(1)/\sigma(0) \to 1$, the MSE of $\hat{\theta}_{\tilde{p}}$ exceeds that of $\hat{\theta}_{\frac{1}{2}}$ by $B_m$.
Next, consider Model 1. With a normal distribution, the MSE of the FNA is 2.9% larger than the balanced allocation when $\sigma(1)/\sigma(0) = 1$. Weighed against a potential reduction of 50%, the FNA is definitely preferred in this setting. However, the picture becomes less clear once we move to Model 2. Here, the MSE of the FNA is around 17% larger than that of the balanced allocation. This is true up to $\sigma(1)/\sigma(0) = 0.75$, which is more heteroskedasticity in many of the outcomes considered in our empirical examples. Similar results obtain with Models 3, 4 and 5, in which the FNA has MSEs that are 26%, 20% and 17% more that of the balanced allocation when $\sigma(1)/\sigma(0) = 1$. The above results suggest that the FNA may not be advisable when $m = 20$. On the other hand, the worst-case efficiency loss shrinks quickly as $m$ increases, so that the FNA performs favorably. Results for $m = 50$ can be found in Appendix (ref). At $m = 50$, the loss in efficiency over Models 1 to 5 are, respectively, 1.5%, 6.4%, 13%, 11% and 8.6%.
To assess the amount of heteroskedasticity that empirical researchers face, we revisit the first 10 completed experiments in the AER RCT Registry. In each experiment, we ask the following question: suppose the authors had access to a small pilot prior to the main study, would they have done better using the FNA instead of balanced randomization? In each experiment presented, we use the full experimental sample to estimate the standard deviations of each treatment arm and compute the corresponding ratios to see if these are close to one. In practice, researchers cannot do this given a small pilot since they do not have access to consistent variance estimators. Our findings suggest that these authors would likely not have done better with the FNA. We present two examples in this section: avvisati2014getting conduct an experiment in which the outcomes are relatively homoskedastic. ashraf2006tying study outcome variables which are heteroskedastic. In this latter example, we also provide estimates of the interval \(C_{m}\) and show that it will be wide even when \(m\) is large. The remaining eight experiments are qualitatively similar to avvisati2014getting and can be found in Section (ref) of the online appendix.
There is a significant body of research examining the impact of school-level factors such as class size or teacher quality on educational performance of students. These are typically seen as the primary instruments for educational policy intervention. A large body of work also examines the impact of parental inputs on educational outcomes. avvisati2014getting study whether or not parental inputs can be effectively manipulated through simple participation programs at schools. They do so via a large-scale randomized control trial in middle schools in the Cr\'eteil educational district of Paris. The experiment targeted families of 6th graders and the program consisted of a sequence of three meetings with parents every 2--3 weeks. The sessions focused on how parents can help their children by participating at school and at home in their education and additionally included advice on how to adapt to results in end-of-term report cards. Participation in the program was randomized at the class level -- half of the classes at a given school were assigned to the participation program. Classes were groups of 20--30 students. The overall sample comprised 183 classes and a total of 4,308 students. The study tracked three types of outcomes: (1) parental involvement attitudes and behavior; (2) children's behavior, namely truancy, disciplinary record and work effort; and (3) children's academic results. Since randomization was done at the class-level, we examine heteroskedasticity with respect to treatment status at both the individual level and at the class level.
Table (ref) reports student-level standard deviations in treatment and control groups, as well as their ratios, for the main outcomes of interest. These are the outcomes considered in Tables 2, 3 and 5 in avvisati2014getting. The ratios are all close to one, so that by and large the treatment and control groups are relatively homoskedastic at the student level. This indicates that if the experimenters were to run a randomized control trial in the same population with treatment assigned at the individual-level, the FNA would likely yield no improvement relative to the balanced allocation.
Table (ref) reports standard deviations and their ratios in class-level means of outcomes. This corresponds to a scenario in which classes are the units of interest, with class-level means as the relevant outcomes. We first calculate class-level means and then compute their standard deviations across classes for the treatment and control groups respectively. The standard deviation ratios for class-level means are by and large also close to one. Hence, if the hypothetical experiment were to be conducted at the class-level, the treatment and control groups would still be relatively homoskedastic. In this case, the FNA would again not improve upon the balanced allocation.
A large body of economic models posit that individuals have time inconsistent preferences, exhibiting more impatience for near-term trade-offs than for future trade-offs. The implication of these models is that those who engage in commitment devices ex ante may improve their welfare. To test this hypothesis, ashraf2006tying conducted an RCT in the Philippines, in which individuals were offered randomly offered the chance to open a SEED (Save, Earn, Enjoy Deposits) account. Money deposited into the account cannot be withdrawn until the owner reached a goal, such as reaching a savings amount or until a pre-specified month in which they expected large expenditures.
Partnering with a rural bank in Mindanao, the authors surveyed 1,777 of their existing or former clients, of which 842 were placed into the treatment group, while 469 were placed in the control group. As treatment involved receiving a briefing on the importance of savings, the remaining 466 individuals were placed in the marketing group, receiving the briefing but not access to SEED. We focus on the Table VI of ashraf2006tying, containing results on saving behavior. The parameter of interest is the Intent-to-Treat effect, with approximately 25% of the treated taking up treatment. Here, the authors find that relative to the control group, the treatment group had a higher change in savings 6 months (6m) and 12 months (12m) after treatment. Comparing the treatment to the marketing group led to weaker but still positive results.
We present standard deviations of the outcomes as well as their ratios in Table (ref). The outcomes of interest are
We first note that \(\Delta\) Tot. Bal. $> 0\%$ (12m) and \(\Delta\) Tot. Bal. $> 20\%$ (12m) are binary outcomes which are relatively homoskedastic. \(\Delta\) Tot. Bal. (6m) and \(\Delta\) Tot. Bal. (12m), measured in Philippine pesos, exhibit more heteroskedasticity. In particular, comparing the treatment group to the marketing group at the 12 month period, we observe a standard deviation ratio of $3.13$. At first glance, this suggests that the FNA might outperform balanced randomization, at least with respect to this specific outcome. This turns out to be false once we investigate other features of \(\Delta\) Tot. Bal. (6m) and \(\Delta\) Tot. Bal. (12m). Quantiles of these variables are displayed in Table (ref). Clearly, they have extremely fat right tails, which we confirm by computing the kurtosis, contained in Table (ref). Fat tails worsen the performance of a variety of statistical techniques, including the FNA, as our analysis in Section (ref) shows.
Researchers cannot estimate $C_m$ if they only have access to a small pilot. However, this is feasible with data from the full experiment. To do so, we draw samples of size $m$ with replacement and compute $Z_m$ for each draw. We then take the average of $Z_m + 1/Z_m$ over 10,000 draws to evaluate $B_m$, essentially evaluating the expectation with respect to the empirical CDF. Proposition (ref) then allows us to compute $C_m$. We evaluate $C_m$ for $m \leq 2000$, using grids of size $20$. Results are displayed in Figure (ref). Given the high kurtosis, there is a relatively large range of ratios of standard deviations for which the balanced allocation is preferred to the FNA.
We can further compute the $m$ necessary before the FNA outperforms the balanced allocation. Given the upper and lower bounds in Figure (ref), we smooth over the grid points using cubic interpolation. We then find the minimum $m$ at which the either bound attains the values in Table (ref). These are the “Exact" intervals presented in Table (ref). Here, we see that the necessary pilot sizes are between 25--50% of the full experiment. For the comparison of the treatment and marketing groups at 6 months, the necessary pilot size exceeds $2,000$, falling outside the set of grid points we explored. To complete the analysis, we use Proposition (ref) to obtain approximations of the necessary $m$. Specifically, we estimate $V$ using their sample analogs computed on the full experiment:
where $n_a$ is the number of units with treatment status $a$ in the full experiment and \(\overline{Y}(a) = \frac{1}{n_a} \sum_{A_i = a} Y_i\). We can then estimate $m$ by
Comparing the asymptotic intervals to the exact ones, we see that the former is far too optimistic for the fat-tailed DGP in ashraf2006tying. This is likely because the sub-Gaussian assumption is inappropriate for this data. Nonetheless, applying the asymptotic bounds to the case of the treatment vs marketing groups at 6 months, we find that a pilot size of 7,000 would be necessary for the FNA to outperform balanced randomization. This is nearly 4 times the size of the actual experiment.
In sum, our analysis suggests that the FNA would have performed poorly in the context of ashraf2006tying. Even though the outcomes of interest exhibit stronger heteroskedasticity, the fat tails of the outcome distributions also impedes the estimation of $\widetilde{p}$, so that ultimately, very large pilot sizes are needed for the FNA to outperform the balanced allocation in this example.
In this section, we use simulations to explore methods that can improve the performance of the FNA with small pilots. For definitions of the models, see Section (ref). We consider in turn (1) testing for homoskedasticity, (2) additive regularization and (3) exponential regularization. We conclude with a comparison of these results and speculate that exponential regularization is preferable to the other two methods. Additional simulation results can be found in Appendix (ref).
Since balanced randomization is preferred when outcomes are relatively homoskedastic, consider the following two-step procedure. First test for homoskedasticity in the pilot. If the test rejects, use the FNA in the main wave. Otherwise, use the balanced allocation instead.
Specifically, we will use the (linear) Wald test for the equality of variance, which has the following test statistic:
where
It is well known that $\tilde{W} \overset{d}{\to} N(0,1)$ as $m \to \infty$. We can therefore form a two-sided, level $\alpha$ test for homoskedasticity with the rejection rule:
where $\Phi(\cdot)$ is the CDF of the standard normal distribution. Under small pilot asymptotics, this test will not be consistent. However, the two-step procedure will be asymptotically equivalent to the FNA when pilots are large, since for any fixed level $\alpha$, the test will reject with probability approaching $1$ as $m \to \infty$. However, under such a “pre-testing procedure", the main-wave difference-in-means estimator will not be uniformly consistent.
{\sloppy Nonetheless, the procedure appears to perform well in practice. Figure (ref) presents the MSEs that obtain using first stage tests with $\alpha \in \left\{1\%, 5\%, 10\%, 20\%, 50\%\right\}$. When $\sigma(1)/\sigma(0) \geq 0.5$, decreasing $\alpha$ decreases MSE across all distributions. Conversely, when $\sigma(1)/\sigma(0) \leq 0.5$, decreasing $\alpha$ increases MSE across all distributions. The appropriate level of $\alpha$ then depends on researcher's preference for the performance of the method over the range of $\sigma(1)/\sigma(0)$.}
Rather than using a test to make a binary decision between the FNA and the balanced allocation, we can consider regularizing the FNA towards the balanced allocation. In particular, for a given tuning parameter $\nu$, let
In other words, we regularize $\tilde{\sigma}(1)$ by adding to it $\nu \tilde{\sigma}(0)$, and similarly for $\tilde{\sigma}(0)$. Pre-multiplying $\nu$ with $\tilde{\sigma}(1)$ and $\tilde{\sigma}(1)$ ensures that $\tilde{p}^\text{add}$ is invariant to their scale. Furthermore,
Figure (ref) presents results for $\nu \in \{0.01, 0.05, 0.1, 0.2, 0.5\}$. As we increase $\nu$, the MSE decreases for the homoskedastic region but increases for the heteroskedastic region.
Another method by which we can continuously regularize the FNA is as follows. For a given tuning parameter $\tau$, let
Observe that when $\tau = 1$, $\tilde{p}^{\text{add}}$ is FNA. For $0 \leq \tau < 1$, $\tilde{p}^{\text{exp}}$ regularizes the FNA towards the balanced allocation. In particular, for $\tau = 0$, $\tilde{p}^{\text{add}} = \frac{1}{2}$. Figure (ref) presents results for $\tau \in \{0.99, 0.95, 0.9, 0.8, 0.5\}$. As before, when we decrease $\tau$, the MSE decreases for the homoskedastic region but increases for the heteroskedastic region.
As the above sections demonstrate, each potential solution entails different trade-offs between MSE in the homoskedastic and heteroskedastic region. These trade-offs also vary depending on the tuning parameter chosen. In a crude attempt at comparing these methods, we consider the following. Choose tuning parameter values so that the MSEs of the methods are made as close as possible in the homoskedastic region. Then compare MSEs in the heteroskedastic region.
One such comparison is displayed in Figure (ref). Here, we see that having fixed performance in the homoskedastic region, exponential regularization seems to do a better job in the heteroskedastic region. This conclusion ostensibly extends to the Pareto distributions, as well as other choices of tuning parameters that keep the solutions appropriately comparable. We therefore speculate that exponential regularization is an effective method for improving the small sample performance of the FNA.
We study the properties of the Feasible Neyman Allocation (FNA) in an asymptotic framework for two-wave experiments that takes pilot size to be fixed as the size of the main wave tends to infinity. In this setting, the estimated allocation has error that is not negligible even in the limit. Our asymptotic model therefore corresponds more closely to the finite sample statistical problem in which pilots are small and the optimal allocation may be poorly estimated. We characterize the conditions under which the difference-in-means estimator has larger asymptotic variance compared to balanced randomization, where half of the main wave is assigned to treatment and the remainder to control. This happens when the potential outcomes are relatively homoskedastic with respect to treatment status or exhibit high kurtosis -- situations that may arise in practice. Our results suggest that the confluence of these factors -- small pilots, homoskedasticity, heavy-tails -- are disadvantageous to the FNA. Instead, researchers might find it useful to consider alternative procedures such as regularizing the FNA towards the balanced allocation.
We thank the editor, the associate editor and three anonymous referees for comments and suggestions that greatly improved the article. We are also grateful to Federico Bugni, Ivan Canay, Joel Horowitz, Dean Karlan, Max Tabord-Meehan and Chris Udry for helpful discussions. We report that there are no competing interests to declare.