Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
99,905 characters · 14 sections · 46 citation commands
Optimal Data Collection for Randomized Control Trials
This paper is motivated by the observation that empirical research in economics increasingly involves the collection of original data through laboratory or field experiments duflo2007, Banerjee:Duflo:11, BBR:11, List:11, List:Rasul:11, Hamermesh:13. This observation carries with it a call and an opportunity for research to provide econometrically sound guidelines for data collection.
We consider the decision problem faced by a researcher designing the survey for a randomized control trial (RCT). We assume that the goal of the researcher is to obtain precise estimates of the average treatment effect and/or a powerful t-test of the hypothesis of no treatment effect, using the experimental data.\footnote{Tetenov:2015 provides a decision-theory-based rationale for using hypothesis tests in the RCT and BCMS:2016 develop a theory of experimenters, focusing on the motivation of randomization among other things.} Data collection is costly and the researcher is restricted by a budget, which limits how much data can be collected. We focus on optimally trading off the number of individuals included in the RCT and the choice of covariates elicited as part of the data collection process.
There are, of course, other factors potentially influencing the choice of covariates to be collected in a survey for an RCT. For example, one may wish to learn about the mechanisms through which the RCT is operating, check whether treatment or control groups are balanced, or measure heterogeneity in the impacts of the intervention being tested. In practice, researchers place implicit weights on each of the main objectives they consider when designing surveys, and consider informally the different trade-offs involved in their choices. We show that there is substantial value to making this decision process more rigorous and transparent through the use of data-driven tools that optimize a well-defined objective. Instead of attempting to formalize the whole research design process, we focus on one particular trade-off that we think is of first-order importance and particularly conducive to data-driven procedures.
We assume the researcher has access to pre-experimental data from the population from which the experimental data will be drawn or at least from a population that shares similar second moments of the variables to be collected. The data set includes all the potentially relevant variables that one would consider collecting for the analysis of the experiment. The researcher faces a fixed budget for implementing the survey for the RCT. Given this budget, the researcher chooses the survey's sample size and set of covariates to optimize the resulting treatment effect estimator's precision and/or the corresponding t-test's power. This choice takes place before the implementation of the RCT and could, for example, be part of a pre-analysis plan in which, among other things, the researcher specifies outcomes of interest, covariates to be selected, and econometric techniques to be used.
In principle, the trade-offs involved in this choice involve basic economic reasoning. For each possible covariate, one should be comparing the marginal benefit and marginal cost of including it in the survey, which in turn, depend on all the other covariates included in the survey. As we discuss below, in simple settings it is possible to derive analytic and intuitive solutions to this problem. Although these are insightful, they only apply in unrealistic formulations of the problem.
In general, for each covariate, there is a discrete choice of whether to include it or not, and for each possible sample size, one needs to consider all possible combinations of covariates within the budget. This requires a solution to a computationally difficult combinatorial optimization problem. This problem is especially challenging when the set of potential variables to choose from is large, a case that is increasingly encountered in today's big data environment. Fortunately, with the increased availability of high-dimensional data, methods for the analysis of such data sets have received growing attention in several fields, including economics BCC:14. This literature makes available a rich set of new tools, which can be adapted to our study of optimal survey design.
In this paper, we propose the use of a computationally attractive algorithm based on the orthogonal greedy algorithm (OGA) -- also known as the orthogonal matching pursuit; see, for example, tropp2004greed and tropp2007signal among many others. To implement the OGA, it is necessary to specify the stopping rule, which in turn generally requires a tuning parameter. One attractive feature of our algorithm is that, once the budget constraint is given, there is no longer the need to choose a tuning parameter to implement the proposed method, as the budget constraint plays the role of a stopping rule. In other words, we develop an automated OGA that is tailored to our own decision problem. Furthermore, it performs well even when there are a large number of potential covariates in the pre-experimental data set.
There is a large and important body of literature on the design of experiments, starting with Fisher:1935. There also exists an extensive body of literature on sample size (power) calculations; see, for example, McConnell2015 for a practical guide. Both bodies of this literature are concerned with the precision of treatment effect estimates, but neither addresses the problem that concerns us. For instance, McConnell2015 have developed methods to choose the sample size when cost constraints are binding, but they neither consider the issue of collecting covariates nor its trade-off with selecting the sample size.
Both our paper and the standard literature on power calculations rely on the availability of information in pre-experimental data. The calculations we propose can be seen as a substantive reformulation and extension of the more standard power calculations, which are an important part of the design of any RCT. When conducting power calculations, one searches for the sample size that allows the researcher to detect a particular effect size in the experiment. The role of covariates can be accounted for if one has pre-defined the covariates that will be used in the experiment, and one knows (based on some pre-experimental data) how they affect the outcome. Then, once the significance level and power parameters are determined (specifying the type I and type II errors one is willing to accept), all that matters is the impact of the sample size on the variance of the treatment effect.
Suppose that, instead of asking what is the minimum sample size that allows us to detect a given effect size, we asked instead how small an effect size we could detect with a particular sample size (this amounts to a reversal of the usual power calculation). In this simple setting with pre-defined covariates, the sample size would define a particular survey cost, and we would essentially be asking about the minimum size of the variance of the treatment effect estimator that one could obtain at this particular cost, which would lead to a question similar to the one asked in this paper. Therefore, one simple way to describe our contribution is that we adapt and extend the information in power calculations to account for the simultaneous selection of covariates and sample size, explicitly considering the costs of data collection.
To illustrate the application of our method we examine two recent experiments for which we have detailed knowledge of the process and costs of data collection. We ask two questions. First, if there is a single hypothesis one wants to test in the experiment, concerning the impact of the experimental treatment on one outcome of interest, what is the optimal combination of covariate selection and sample size given by our method, and how much of an improvement in the precision of the impact estimate can we obtain as a result? Second, what are the minimum costs of obtaining the same precision of the treatment effect as in the actual experiment, if one was to select covariates and sample size optimally (what we call the “equivalent budget”)?
We find from these two examples that by adopting optimal data collection rules, not only can we achieve substantial increases in the precision of the estimates (statistical importance) for a given budget, but we can also accomplish sizeable reductions in the equivalent budget (economic importance). To illustrate the quantitative importance of the latter, we show that the optimal selection of the set of covariates and the sample size leads to a reduction of about 45 percent (up to 58 percent) of the original budget in the first (second) example we consider, while maintaining the same level of the statistical significance as in the original experiment.
To the best of our knowledge, no paper in the literature directly considers our data collection problem. Some papers address related but very different problems hahn2011we, List-et-al:11,Bhattacharya:Dupas:2012, McKenzie2012,Dominitz:Manski:16. They study some issues of data measurement, budget allocation or efficient estimation; however, they do not consider the simultaneous selection of the sample size and covariates for the RCTs as in this paper. Because our problem is distinct from the problems studied in these papers, we give a detailed comparison between our paper and the aforementioned papers in Section (ref).
More broadly, this paper is related to a recent emerging literature in economics that emphasizes the importance of micro-level predictions and the usefulness of machine learning for that purpose. For example, KLMZ:2015 argue that prediction problems are abundant in economic policy analysis, and recent advances in machine learning can be used to tackle those problems. Furthermore, our paper is related to the contemporaneous debates on pre-analysis plans which demand, for example, the selection of sample sizes and covariates before the implementation of an RCT; see, for example, Coffman:Niederle:15 and Olken:15 for the advantages and limitations of the pre-analysis plans.
The remainder of the paper is organized as follows. In Section (ref), we describe our data collection problem in detail. In Section (ref), we propose the use of a simple algorithm based on the OGA. In Section (ref), we discuss the costs of data collection in experiments. In Section (ref), we present two empirical applications, in Section (ref), we discuss the existing literature, and in Section (ref), we give concluding remarks. Appendices provide details that are omitted from the main text.
Suppose we are planning an RCT in which we randomly assign individuals to either a treatment ($D=1$) or a control group ($D=0$) with corresponding potential outcomes $Y_1$ and $Y_0$, respectively. After administering the treatment to the treatment group, we collect data on outcomes $Y$ for both groups so that $Y=DY_1 + (1-D)Y_0$. We also conduct a survey to collect data on a potentially very high-dimensional vector of covariates $Z$ (e.g. from a household survey covering demographics, social background, income etc.) that predicts potential outcomes. These covariates are a subset of the universe of predictors of potential outcomes, denoted by $X$. Random assignment of $D$ means that $D$ is independent of potential outcomes and of $X$.
Our goal is to estimate the average treatment effect $\beta_0 := E[Y_1-Y_0]$ as precisely as possible, where we measure precision by the finite sample mean-squared error (MSE) of the treatment effect estimator, and/or produce a powerful t-test of the hypothesis $H_0: \beta_0=0$. Instead of simply regressing $Y$ on $D$, we want to make use of the available covariates $Z$ to improve the precision of the resulting treatment effect estimator. Therefore, we consider estimating $\beta_0$ in the regression
where $(\alpha_0,\beta_0,\gamma_0')'$ is a vector of parameters to be estimated and $U$ is an error term. The implementation of the RCT requires us to make two decisions that may have a significant impact on the estimation of and inference on the average treatment effect:
Obviously, a large experimental sample size $n$ reduces the variance of the treatment effect estimator. Similarly, collecting more covariates, in particular strong predictors of potential outcomes, reduces the variance of the residual $U$ which, in turn, also improves the variance of the estimator. At the same time collecting data from more individuals and on more covariates is costly so that, given a finite budget, we want to find a combination of sample size $n$ and covariate selection $Z$ that leads to the most precise treatment effect estimator possible.
In this section, we propose a procedure to make this choice based on a pre-experimental data set on $Y$ and $X$, such as a pilot study or a census from the same population from which we plan to draw the RCT sample.\footnote{In fact, we do not need the populations to be identical, but only require second moments to be the same.} The combined data collection and estimation procedure can be summarized as follows:
We now describe the five steps listed above in more detail. The main component of our procedure consists of a proposal for the optimal choice of $n$ and $Z$ in Step 2, which is described more formally in Section (ref).
\setcounter{bean}{0}
In this subsection, we discuss some of conceptual and practical properties of our proposed data collection procedure.
\paragraph{Availability of Pre-Experimental Data.} As in standard power calculations, pre-experimental data provide essential information for our procedure. The availability of such data is very common, ranging from census data sets and other household surveys to studies that were conducted in a similar context as the RCT we are planning to implement. In addition, if no such data set is available, one may consider running a pilot project that collects pre-experimental data. We recognize that in some cases it might be difficult to have the required information readily available. However, this is a problem that affects any attempt to a data-driven design of surveys, including standard power calculations. Even when pre-experimental data are imperfect, such calculations provide a valuable guide to survey design, as long as the available pre-experimental data are not very different from the ideal data. In particular, our procedure only requires second moments of the pre-experimental variables to be similar to those in the population of interest.
\paragraph{The Optimization Problem in a Simplified Setup.} In general, the problem in (ref) does not have a simple solution and requires joint optimization problem over the sample size $n$ and the coefficient $\gamma$. To gain some intuition about the trade-offs in this problem, in Appendix C we consider a simplified setup in which all covariates are orthogonal to each other, and the budget constraint has a very simple form. In this case, the constraint can be substituted into the objective and the optimization becomes univariate and unconstrained. We show that if all covariates have the same price, then one wants to choose covariates up to the point where the percentage increase in survey costs equals the percentage reduction in the residual variance from the last covariate. Furthermore, the elasticity of the residual variance with respect to changes in sample size should equal the elasticity of the residual variance with respect to an additional covariate. If the costs of data collection vary with covariates, then this conclusion is slightly modified. If we organize variables by type according to their contribution to the residual variance, then we want to choose variables of each type up to the point where the percent marginal contribution of each variable to the residual variance equals its percent marginal contribution to survey costs.
\paragraph{Imbalance and Re-randomization.} In RCTs, covariates typically do not only serve as a means to improving the precision of treatment effect estimators, but also for checking whether the control and treatment groups are balanced. See, for example, Bruhn:McKenzie:09 for practical issues concerning randomization and balance. To rule out large biases due to imbalance, it is important to carry out balance checks for strong predictors of potential outcomes. Our procedure selects the strongest predictors as long as they are not too expensive (e.g. household survey questions such as gender, race, number of children etc.) and we can check balance for these covariates. However, in principle, it is possible that our procedure does not select a strong predictor that is very expensive (e.g. baseline test scores). Such a situation occurs in our second empirical application (Section (ref)). In this case, in Step 2, we recommend running the OGA a second time, forcing the inclusion of such expensive predictors. If the MSE of the resulting estimate is not much larger than that from the selection without the expensive predictor, then we may prefer the former selection to the latter so as to reduce the potential for bias due to imbalance at the expense of slightly larger variance of the treatment effect estimator.
An alternative approach to avoiding imbalance considers re-randomization until some criterion capturing the degree of balance is met (e.g., Bruhn:McKenzie:09, morgan2012gf,Morgan2015re and LDR:2016). Our criterion for the covariate selection procedure in Step 2 can readily be adapted to this case; however, the details are not worked out here. It is an interesting future research topic to fully develop a data collection method for re-randomization based on the modified variance formulae in morgan2012gf and LDR:2016, which account for the effect of re-randomization on the treatment effect estimator.
\paragraph{Expensive, Strong Predictors.} When some covariates have similar predictive power, but respective prices that are substantially different, our covariate selection procedure may produce a suboptimal choice. For example, if the covariate with the highest price is also the most predictive, OGA selects it first even when there are other covariates that are much cheaper but only slightly less predictive. In Section (ref), we encounter an example of such a situation and propose a simple robustness check for whether removing an expensive, strong predictor may be beneficial.
\paragraph{Properties of the Treatment Effect Estimator.} Since the treatment indicator is assumed independent of $X$, standard asymptotic theory of the treatment effect estimator continues to hold for our estimator (despite the addition of a covariate selection step). For example, it is unbiased, consistent, asymptotically normal, and adding the covariates $X$ in the regression in (ref) cannot increase the asymptotic variance of the estimator. In fact, inclusion of a covariate strictly reduces the estimator's asymptotic variance as long as the corresponding true regression coefficient is not zero. All these results hold regardless of whether the true conditional expectation of $Y$ given $D$ and $X$ is in fact linear and additive separable as in (ref) or not. In particular, in some applications one may want to include interaction terms of $D$ and $X$ Imbens:Rubin:2015. Finally, the treatment effect can be allowed to be heterogeneous (i.e. vary across individuals $i$) in which case our procedure estimates the average of those treatment effects.
\paragraph{An Alternative to Regression.} Step 4 consists of running the regression in (ref). There are instances when it is desirable to modify this step. For example, if the selected sample size $\hat{n}$ is smaller than the number of selected covariates, then the regression in (ref) is not feasible. However, if the pre-experimental sample $\mathcal{S}_{\rm pre}$ is large enough, we can instead compute the OLS estimator $\hat{\gamma}$ from the regression of $Y$ on $X_{\hat{\mathcal{I}}}$ in $\mathcal{S}_{\rm pre}$. Then use $Y$ and $Z$ from the experimental sample $\mathcal{S}_{\rm exp}$ to construct the new outcome variable $\hat{Y}_i^\ast := Y_i - \hat{\gamma}'Z_i$ and compute the treatment effect estimator $\hat{\beta}$ from the regression of $\hat{Y}_i^\ast$ on $(1,D_i)$. This approach avoids fitting too many parameters when the experimental sample is small and has the additional desirable property that the resulting estimator is free from bias due to imbalance in the selected covariates.
\paragraph{Multivariate Outcomes.} It is straightforward to extend our data collection method to the case when there are multivariate outcomes. Appendix G provides details regarding how to deal with a vector of outcomes when we select the common set of regressors for all outcomes.
In practice, the vector $X$ of potential covariates is typically high-dimensional, which makes it challenging to solve the optimization problem (ref). In this section, we propose a computationally feasible algorithm that is both conceptually simple and performs well in our simulations. In particular, it requires only running many univariate, linear regressions and can therefore easily be implemented in popular statistical packages such as STATA.
We split the joint optimization problem in (ref) over $n$ and $\gamma$ into two nested problems. The outer problem searches over the optimal sample size $n$, which is restricted to be on a grid $n\in\mathcal{N}:=\{n_0,n_1,\ldots,n_K\}$, while the inner problem determines the optimal selection of covariates for each sample size $n$:
To convey our ideas in a simple form, suppose for the moment that the budget constraint has the following linear form,
where $|\mathcal{I}(\gamma)|$ denotes the number of non-zero elements of $\gamma$. Note that the budget constraint puts the restriction on the number of selected covariates, that is, $|\mathcal{I}(\gamma)| \leq B/n$.
It is known to be NP-hard (non-deterministic polynomial time hard) to find a solution to the inner optimization problem in (ref) subject to the constraint that $\gamma$ has $m$ non-zero components, also called an $m$-term approximation, where $m$ is the integer part of $B/n$ in our problem. In other words, solving (ref) directly is not feasible unless the dimension of covariates, $M$, is small natarajan1995tr,Davis:1997jk.
There exists a class of computationally attractive procedures called greedy algorithms that are able to approximate the infeasible solution. See Temlyakov:2011 for a detailed discussion of greedy algorithms in the context of approximation theory. tropp2004greed, tropp2007signal, barron2008kk, Zhang:2009, Huang-et-al:2011, Ing:Lai:11, and sancetta2016, among many others, demonstrate the usefulness of greedy algorithms for signal recovery in information theory, and for the regression problem in statistical learning. We use a variant of OGA that can allow for selection of groups of variables (see, for example, Huang-et-al:2011).
To formally define our proposed algorithm, we introduce some notation. For a vector $v$ of $N$ observations $v_1,\ldots,v_N$, let $\|v\|_N := (1/N \sum_{i=1}^N v_i^2)^{1/2}$ denote the empirical $L^2$-norm and let $\mathbf{Y} := (Y_1,\ldots,Y_N)'$.
Suppose that the covariates $X^{(j)}$, $j=1,\ldots,M$, are organized into $p$ pre-determined groups $X_{G_1}, \ldots, X_{G_p}$, where $G_k\subseteq \{1,\ldots,p\}$ indicates the covariates of group $k$. We denote the corresponding matrices of observations by bold letters (i.e., $\mathbf{X}_{G_k}$ is the $N \times |G_k|$ matrix of observations on $X_{G_k}$, where $|G_k|$ denotes the number of elements of the index set $G_k$). By a slight abuse of notation, we let $\mathbf{X}_{k} := \mathbf{X}_{\{k\}}$ be the column vector of observations on $X_k$ when $k$ is a scalar. One important special case is that in which each group consists of a single regressor. Furthermore, we allow for overlapping groups; in other words, some elements can be included in multiple or even all groups. The group structure occurs naturally in experiments where data collection is carried out through surveys whose questions can be grouped in those concerning income, those concerning education, and so on. This can also occur naturally when we consider multivariate outcomes. See Appendix G for details.
Suppose that the largest group size $J_{\max} := \max_{k=1,\ldots,p} |G_k|$ is small, so that we can implement orthogonal transformations {\em within each group\/} such that $(\mathbf{X}_{G_j}' \mathbf{X}_{G_j})/N = \mathbf{I}_{|G_j|}$, where $\mathbf{I}_{d}$ is the $d$-dimensional identity matrix. In what follows, assume that $(\mathbf{X}_{G_j}' \mathbf{X}_{G_j})/N = \mathbf{I}_{|G_j|}$ without loss of generality. Let $|\cdot|_2$ denote the $\ell_2$ norm. The following procedure describes our algorithm.
\setcounter{bean}{0}
The algorithm above produces the selected sample size $\hat{n}$, the selection of covariates $\hat{\mathcal{I}}:= \hat{\mathcal{I}}_{\hat{n},k_{\hat{n}}}$ with $k_{\hat{n}}$ selected groups and $\hat{m} := m(\hat{n}):= |\hat{\mathcal{I}}_{\hat{n},k_{\hat{n}}}|$ selected regressors. Here, $\hat{\gamma}:=\hat{\gamma}_{\hat{n}}$ is the corresponding coefficient vector on the selected regressors $Z$.
In this section, we discuss the specification of the cost function $c(S,n)$ that defines the budget constraint of the researcher. In principle, it is possible to construct a matrix containing the value of the costs of data collection for every possible combination of $S$ and $n$ without assuming any particular form of relationship between the individual entries. However, determination of the costs for every possible combination of $S$ and $n$ is a cumbersome and, in practice, probably infeasible exercise. Therefore, we consider the specification of cost functions that capture the costs of all stages of the data collection process in a more parsimonious fashion.
We propose to decompose the overall costs of data collection into three components: administration costs $c_{\rm admin}(S)$, training costs $c_{\rm train}(S,n)$, and interview costs $c_{\rm interv}(S,n)$, so that
In the remainder of this section, we discuss possible specifications of the three types of costs by considering fixed and variable cost components corresponding to the different stages of the data collection process. The exact functional form assumptions are based on the researcher's knowledge about the operational details of the survey process. Even though this section's general discussion is driven by our experience in the empirical applications of Section (ref), the operational details are likely to be similar for many surveys, so we expect the following discussion to provide a useful starting point for other data collection projects.
We start by specifying survey time costs. Let $\tau_{j}$, $j=1,\ldots,M$, be the costs of collecting variable $j$ for one individual, measured in units of survey time. Similarly, let $\tau_0$ denote the costs of collecting the outcome variable, measured in units of survey time. Then, the total time costs of surveying one individual to elicit the variables indicated by $S$ are $$T(S) := \tau_0 + \sum_{j=1}^{M}\tau_{j}S_{j}.$$
A data collection process typically incurs costs due to administrative work and training prior to the start of the actual survey. Examples of such tasks are developing the questionnaire and the program for data entry, piloting the questionnaire, developing the manual for administration of the survey, and organizing the training required for the enumerators.
Fixed costs, which depend neither on the size of the survey nor on the sample size of survey participants, can simply be subtracted from the budget. We assume that $B$ is already net of such fixed costs.
Most administrative and training costs tend to vary with the size of the questionnaire and the number of survey participants. Administrative tasks such as development of the questionnaire, data entry, and training protocols are independent of the number of survey participants, but depend on the size of the questionnaire (measured by the number of positive entries in $S$) as smaller questionnaires are less expensive to prepare than larger ones. We model those costs by
where $\phi$ and $\alpha$ are scalars to be chosen by the researcher. We assume $0<\alpha<1$, which means that marginal costs are positive but decline with survey size.
Training of the enumerators depends on the survey size, because a longer survey requires more training, and on the number of survey participants, because surveying more individuals usually requires more enumerators (which, in turn, may raise the costs of training), especially when there are limits on the duration of the fieldwork. We therefore specify training costs as
where $\kappa(n)$ is some function of the number of survey participants.\footnote{It is of course possible that $\kappa$ depends not only on $n$ but also on $T(S)$. We model it this way for simplicity, and because it is a sensible choice in the applications we discuss below.} Training costs are typically lumpy because, for example, there exists only a limited set of room sizes one can rent for the training, so we model $\kappa( n) $ as a step function: \[ \kappa\left( n\right) =\left\{
\right. . \] Here, $\overline{\kappa}_{1}, \overline{\kappa}_{2}, \ldots$ is a sequence of scalars describing the costs of sample sizes in the ranges defined by the cut-off sequence $\overline{n}_{1}, \overline{n}_{2}, \ldots$.
Enumerators are often paid by the number of interviews conducted, and the payment increases with the size of the questionnaire. Let $\eta$ denote fixed costs per interview that are independent of the size of the questionnaire and of the number of participants. These are often due to travel costs and can account for a substantive fraction of the total interview costs. Suppose the variable component of the interview costs is linear so that total interview costs can be written as
where $T(S)$ should now be interpreted as the average time spent per interview, and $p$ is the average price of one unit of survey time. We employ the specification (ref) with (ref)--(ref) when studying the impact of free day-care on child development in Section (ref).
In many experiments, randomization is carried out at a cluster level (e.g., school level), rather than at an individual level (e.g., student level). In this case, training costs may depend not only on the ultimate sample size $n = c\, n_c$, where $c$ and $n_c$ denote the number of clusters and the number of participants per cluster, respectively, but on a particular combination ($c,n_c$), because the number of required enumerators may be different for different ($c,n_c$) combinations. Therefore, training costs (which now also depend on $c$ and $n_c$) may be modeled as
The interaction of cluster and sample size in determining the number of required enumerators and, thus, the quantity $\kappa(c,n_c)$, complicates the modeling of this quantity relative to the case without clustering. Let $\mu(c,n_c)$ denote the number of required survey enumerators for $c$ clusters of size $n_c$. As in the case without clustering, we assume that the training costs is lumpy in the number of enumerators used: \[ \kappa( c,n_c) :=\left\{
\right.. \] The number of enumerators required, $\mu(c,n_c)$, may also be lumpy in the number of interviewees per cluster, $n_c$, because there are bounds to how many interviews each enumerator can carry out. Also, the number of enumerators needed for the survey typically increases in the number of clusters in the experiment. Therefore, we model $\mu(c,n_c)$ as $$\mu(c,n_c) := \lfloor\mu_c(c)\cdot \mu_n(n_c)\rfloor,$$ where $\lfloor\cdot \rfloor$ denotes the integer part, $\mu_c(c) := \lambda c $ for some constant $\lambda$ (i.e., $\mu_c(c)$ is assumed to be linear in $c$), and $$\mu_n(n_c) := \left\{
\right..$$
In addition, while the variable interview costs component continues to depend on the overall sample size $n$ as in (ref), the fixed part of the interview costs is determined by the number of clusters $c$ rather than by $n$. Therefore, the total costs per interview become
where $\psi(c)$ is some function of the number of clusters $c$.
In randomized experiments, the data collection process often differs across blocks of covariates. For example, the researcher may want to collect outcomes of psychological tests for the members of the household that is visited. These tests may need to be administered by trained psychologists, whereas administering a questionnaire about background variables such as household income, number of children, or parental education, may not require any particular set of skills or qualifications other than the training provided as part of the data collection project.
Partition the covariates into two blocks, a high-cost block (e.g., outcomes of psychological tests) and a low-cost block (e.g., standard questionnaire). Order the covariates such that the first $M_{\rm low}$ covariates belong to the low-cost block, and the remaining $M_{\rm high}:=M-M_{\rm low}$ together with the outcome variable belong to the high-cost block. Let $$T_{\rm low}(S):= \sum_{j=1}^{M_{\rm low}}\tau_{j}S_{j}\qquad \text{and}\qquad T_{\rm high}(S):= \tau_0 + \sum_{j=M_{\rm low}+1}^{M}\tau_{j}S_{j}$$ be the total time costs per individual of surveying all low-cost and high-cost covariates, respectively. Then, the total time costs for all variables can be written as $T(S) = T_{\rm low}(S)+T_{\rm high}(S)$.
Because we require two types of enumerators, one for the high-cost covariates and one for the low-cost covariates, the financial costs of each interview (fixed and variable) may be different for the two blocks of covariates. Denote these by $\psi_{\rm low}(c,n_c) \eta_{\rm low} + c n_c p_{\rm low} T_{\rm low}(S)$ and $\psi_{\rm high}(c,n_c) \eta_{\rm high} + c n_c p_{\rm high} T_{\rm high}(S)$, respectively.
The fixed costs for the high-cost block are incurred regardless of whether high-cost covariates are selected or not, because we always collect the outcome variable, which here is assumed to belong to this block. The fixed costs for the low-cost block, however, are incurred only when at least one low-cost covariate is selected (i.e., when $\sum_{j=1}^{M_{\rm low}} S_j > 0$). Therefore, the total interview costs for all covariates can be written as
The administration and training costs can also be assumed to differ for the two types of enumerators. In that case,
We employ specification (ref) with (ref)--(ref) when, in Section (ref), we study the impact on student learning of cash grants which are provided to schools.
In this section, we re-examine the experimental design of Attanasio:2014sf, who evaluate the impact of access to free day-care on child development and household resources in Rio de Janeiro. In their dataset, access to care in public day-care centers, most of which are located in slums, is allocated through a lottery, administered to children in the waiting lists for each day-care center.
Just before the 2008 school year, children applying for a slot at a public day-care center were put on a waiting list. At this time, children were between the ages of 0 and 3. For each center, when the demand for day-care slots in a given age range exceeded the supply, the slots were allocated using a lottery (for that particular age range). The use of such an allocation mechanism means that we can analyze this intervention as if it was an RCT, where the offer of free day-care slots is randomly allocated across potentially eligible recipients. Attanasio:2014sf compare the outcomes of children and their families who were awarded a day-care slot through the lottery, with the outcomes of those not awarded a slot.
The data for the study were collected mainly during the second half of 2012, four and a half years after the randomization took place. Most children were between the ages of 5 and 8. A survey was conducted, which had two components: a household questionnaire, administered to the mother or guardian of the child; and a battery of health and child development assessments, administered to children. Each household was visited by a team of two field workers, one for each component of the survey.
The child assessments took a little less than one hour to administer, and included five tests per child, plus the measurement of height and weight. The household survey took between one and a half and two hours, and included about 190 items, in addition to a long module asking about day-care history, and the administration of a vocabulary test to the main carer of each child.
As we explain below, we use the original sample, with the full set of items collected in the survey, to calibrate the cost function for this example. However, when solving the survey design problem described in this paper we consider only a subset of items of these data, with the original budget being scaled down properly. This is done for simplicity, so that we can essentially ignore the fact that some variables are missing for part of the sample, either because some items are not applicable to everyone in the sample, or because of item non-response. We organize the child assessments into three indices: cognitive tests, executive function tests, and anthropometrics (height and weight). These three indices are the main outcome variables in the analysis. However, we use only the cognitive tests and anthropometrics indices in our analysis, as we have fewer observations for executive function tests.
We consider only 40 covariates out of the total set of items on the questionnaire. The variables not included can be arranged into four groups: (i) variables that can be seen as final outcomes, such as questions about the development and the behavior of the children in the household; (ii) variables that can be seen as intermediate outcomes, such as labor supply, income, expenditure, and investments in children; (iii) variables for which there is an unusually large number of missing values; and (iv) variables that are either part of the day-care history module, or the vocabulary test for the child's carer (because these could have been affected by the lottery assigning children to day-care vacancies). We then drop four of the 40 covariates chosen, because their variance is zero in the sample. The remaining $M=36$ covariates are related to the respondent's age, literacy, educational attainment, household size, safety, burglary at home, day care, neighborhood, characteristics of the respondent's home and its surroundings (the number of rooms, garbage collection service, water filter, stove, refrigerator, freezer, washer, TV, computer, Internet, phone, car, type of roof, public light in the street, pavement, etc.). We drop individuals for whom at least one value in each of these covariates is missing, which leads us to use a subsample with 1,330 individuals from the original experimental sample, which included 1,466 individuals.
\paragraph{Calibration of the cost function.} We specify the cost function (ref) with components (ref)--(ref) to model the data collection procedure as implemented in Attanasio:2014sf. We calibrate the parameters using the actual budgets for training, administrative, and interview costs in the authors' implementation. The contracted total budget of the data collection process was R\$665,000.\footnote{There were some adjustments to the budget during the period of fieldwork.}
For the calibration of the cost function, we use the originally planned budget of R\$665,000, and the original sample size of 1,466. As mentioned above, there were $190$ variables collected in the household survey, together with a day-care module and a vocabulary test. In total, this translates into a total of roughly 240 variables.\footnote{The budget is for the 240 variables (or so) actually collected. In spite of that, we only use 36 of these as covariates in this paper, as the remaining variables in the survey were not so much covariates as they were measuring other intermediate and final outcomes of the experiment, as we have explained before. The actual budget used in solving the survey design problem is scaled down to match the use of only 36 covariates.} Appendix B provides a detailed description of all components of the calibrated cost function.
\paragraph{Implementation.}
In implementing the OGA, we take each single variable as a possible group (i.e., each group consists of a singleton set). We studentized all covariates to have variance one. To compare the OGA with alternative approaches, we also consider LASSO and POST-LASSO for the inner optimization problem in Step 2 of our procedure. The LASSO solves
with a tuning parameter $\lambda > 0$. The POST-LASSO procedure runs an OLS regression of $Y_i$ on the selected covariates (non-zero entries of $\gamma$) in (ref). belloni2013, for example, provide a detailed description of the two algorithms. It is known that LASSO yields biased regression coefficient estimates and that POST-LASSO can mitigate this bias problem. Together with the outer optimization over the sample size using the LASSO or POST-LASSO solutions in the inner loop may lead to different selections of covariate-sample size combinations. This is because POST-LASSO re-estimates the regression equation which may lead to more precise estimates of $\gamma$ and thus result in a different estimate for the MSE of the treatment effect estimator.
In both LASSO implementations, the penalization parameter $\lambda$ is chosen so as to satisfy the budget constraint as close to equality as possible. We start with a large value for $\lambda$, which leads to a large penalty for non-zero entries in $\gamma$, so that few or no covariates are selected and the budget constraint holds. Similarly, we consider a very small value for $\lambda$ which leads to the selection of many covariates and violation of the budget. Then, we use a bisection algorithm to find the $\lambda$-value in this interval for which the budget is satisfied within some pre-specified tolerance.
\paragraph{Results.} Tables (ref) and (ref) summarize the results of the covariate selection procedures. For the cognitive test outcome, OGA and POST-LASSO select one covariate (“$|\hat{I}|$”),\footnote{For OGA, it is an indicator variable whether the respondent has finished secondary education, which is an important predictor of outcomes; for POST-LASSO, it is the number of rooms in the house, which can be considered as a proxy for wealth of the household, and again, an important predictor of outcomes.} whereas LASSO does not select any covariate. The selected sample sizes (“$\hat{n}$”) are 2,677 for OGA and POST-LASSO, and 2,762 for LASSO, which are almost twice as large as the actual sample size in the experiment. The performance of the three covariate selection methods in terms of the precision of the resulting treatment effect estimator is measured by the square-root value of the minimized MSE criterion function (“RMSE”) from Step 2 of our procedure. We focus on the MSE, but notice that gains in MSE translate into gains in the power of the corresponding t-test as discussed in Section (ref). The three methods perform similarly well and improve precision by about 25% relative to the experiment. Also, all three methods manage to exhaust the budget, as indicated by the cost-to-budget ratios (“Cost/B”) close to one. We do not put any strong emphasis on the selected covariates as the improvement of the criterion function is minimal relative to the case that no covariate is selected (i.e., the selection with LASSO). The results for the health assessment outcome are very similar to those of the cognitive test with POST-LASSO selecting one variable (the number of rooms in the house), whereas OGA and LASSO do not select any covariate.
To assess the economic gain of having performed the covariate selection procedure after the first wave, we include the column “EQB” (abbreviation of “equivalent budget”) in Tables (ref) and (ref). The first entry of this column in Table (ref) reports the budget necessary for the selection of $\hat{n}=$ 1,330 and all covariates, as was carried out in the experiment. For the three covariate selection procedures, the column shows the budget that would have sufficed to achieve the same precision as the actual experiment in terms of the minimum value of the MSE criterion function in Step 2. For example, for the cognitive test outcome, using the OGA to select the sample size and the covariates, a budget of R\$312,363 would have sufficed to achieve the experimental RMSE of $0.025285$. This is a huge reduction of costs by about 45 percent, as shown in the last column called “relative EQB”. Similar reductions in costs are possible when using the LASSO procedures and also when considering the health assessment outcome.
In Appendix F, we perform an out-of-sample evaluation by splitting the dataset into training samples for the covariate selection step and evaluation samples for the computation of the performance measures RMSE and EQB. The results are very similar to those in Tables (ref) and (ref).
Appendix D presents the results of Monte Carlo simulations that mimic this dataset, and shows that all three methods select more covariates and smaller sample sizes as we increase the predictive power of some covariates. This finding suggests that the covariates collected in the survey were not predicting the outcome very well and, therefore, in the next wave the researcher should spend more of the available budget to collect data on more individuals, with no (or only a minimal) household survey. Alternatively, the researcher may want to redesign the household survey to include questions whose answers are likely better predictors of the outcome.
In this subsection, we consider the study by Carneiro-et-al:14 who evaluate, using an RCT, the impact of school grants on student learning in Senegal. The authors collect original data not only on the treatment status of schools (treatment and control) and on student learning, but also on a variety of household, principal, and teacher characteristics that could potentially affect learning.
The dataset contains two waves, a baseline and a follow-up, which we use for the study of two different hypothetical scenarios. In the first scenario, the researcher has access to a pre-experimental dataset consisting of all outcomes and covariates collected in the baseline survey of this experiment, but not the follow-up data. The researcher applies the covariate selection procedure to this pre-experimental dataset to find the optimal sample size and set of covariates for the randomized control trial to be carried out after the first wave. In the second scenario, in addition to the pre-experimental sample from the first wave the researcher now also has access to the post-experimental outcomes collected in the follow-up (second wave). In this second scenario, we treat the follow-up outcomes as the outcomes of interest and include baseline outcomes in the pool of covariates that predict follow-up outcomes.
As in the previous subsection, we calibrate the cost function based on the full dataset from the experiment, but for solving the survey design problem we focus on a subset of individuals and variables from the original questionnaire. For simplicity, we exclude all household variables from the analysis, because they were only collected for 4 out of the 12 students tested in each school, and we remove covariates whose sample variance is equal to zero. Again, for simplicity, of the four outcomes (math test, French test, oral test, and receptive vocabulary) in the original experiment, we only consider the first one (math test) as our outcome variable. We drop individuals for whom at least one answer in the survey or the outcome variable is missing. This sample selection procedure leads to sample sizes of $N=2,280$ for the baseline math test outcome. For the second scenario discussed above where we use also the follow-up outcome, the sample size is smaller ($N=762$) because of non-response in the follow-up outcome and because we restrict the sample to the control group of the follow-up. In the first scenario in which we predict the baseline outcome, dropping household variables reduces the original number of covariates in the survey from 255 to $M=142$. The remaining covariates are school- and teacher-level variables. In the second scenario in which we predict follow-up outcomes, we add the three baseline outcomes to the covariate pool, but at the same time remove two covariates because they have variance zero when restricted to the control group. Therefore, there are $M=143$ covariates in the second scenario.
\paragraph{Calibration of the cost function.} We specify the cost function (ref) with components (ref)--(ref) to model the data collection procedure as implemented in Carneiro-et-al:14. Each school forms a cluster. We calibrate the parameters using the costs faced by the researchers and their actual budgets for training, administrative, and interview costs. The total budget for one wave of data collection in this experiment, excluding the costs of the household survey, was approximately \$192,200.
For the calibration of the cost function, we use the original sample size, the original number of covariates in the survey (except those in the household survey), and the original number of outcomes collected at baseline. The three baseline outcomes were much more expensive to collect than the remaining covariates. In the second scenario, we therefore group the former together as high-cost variables, and all remaining covariates as low-cost variables. Appendix B provides a detailed description of all components of the calibrated cost function.
\paragraph{Implementation.} The implementation of the covariate selection procedures is identical to the one in the previous subsection except that we consider here two different specifications of the pre-experimental sample $\mathcal{S}_{\rm pre}$, depending on whether the outcome of interest is the baseline or follow-up outcome.
\paragraph{Results.}
Table (ref) summarizes the results of the covariate selection procedures. Panel (a) shows the results of the first scenario in which the baseline math test is used as the outcome variable to be predicted. Panel (b) shows the corresponding results for the second scenario in which the baseline outcomes are treated as high-cost covariates and the follow-up math test is used as the outcome to be predicted.
For the baseline outcome in panel (a), the OGA selects only $|\hat{I}|=14$ out of the $145$ covariates with a selected sample size of $\hat{n}=3,018$, which is about 32% larger than the actual sample size in the experiment. The results for the LASSO and POST-LASSO methods are similar. As in the previous subsection, we measure the performance of the three covariate selection methods by the estimated precision of the resulting treatment effect estimator (“RMSE”). As in the previous section focus on the MSE, but notice that gains in MSE translate into gains in the power of the corresponding t-test as discussed in Section (ref). The three methods improve the precision by about 7% relative to the experiment. Also, all three methods manage to essentially exhaust the budget, as indicated by cost-to-budget ratios (“Cost/B”) close to one. As in the previous subsection, we measure the economic gains from using the covariate selection procedures by the equivalent budget (“EQB”) that each of the method requires to achieve the precision of the experiment. All three methods require equivalent budgets that are 7-9% lower than that of the experiment.
All variables that the OGA selects as strong predictors of baseline outcome are plausibly related to student performance on a math test:\footnote{Appendix E shows the full list and definitions of selected covariates for the baseline outcome.} They are related to important aspects of the community surrounding the school (e.g., distance to the nearest city), school equipment (e.g., number of computers), school infrastructure (e.g., number of temporary structures), human resources (e.g., teacher--student ratio, teacher training), and teacher and principal perceptions about which factors are central for success in the school and about which factors are the most important obstacles to school success.
For the follow-up outcome in panel (b), the budget used in the experiment increases due to the addition of the three expensive baseline outcomes to the pool of covariates. All three methods select no covariates and exhaust the budget by using the maximum feasible sample size of 6,755, which is almost nine times larger than the sample size in the experiment. The implied precision of the treatment effect estimator improves by about 47% relative to the experiment, which translates into the covariate selection methods requiring less than half of the experimental budget to achieve the same precision as in the experiment. These are substantial statistical and economic gains from using our proposed procedure.
\paragraph{Sensitivity Checks.} In RCT's, baseline outcomes tend to be strong predictors of the follow-up outcome. One may therefore be concerned that, because the OGA first selects the most predictive covariates which in this application are also much more expensive than the remaining low-cost covariates, the algorithm never examines what would happen to the estimator's MSE if it first selects the most predictive low-cost covariates instead. In principle, such selection could lead to a lower MSE than any selection that includes the very expensive baseline outcomes. As a sensitivity check we therefore perform the covariate selection procedures on the pool of covariates that excludes the three baseline outcomes. Panel (c) shows the corresponding results. In this case, all methods indeed select more covariates and smaller sample sizes than in panel (b), and achieve a slightly smaller MSE. The budget reductions relative to the experiment as measured by EQB are also almost identical to those in panel (b). Therefore, both selections of either no covariates and large sample size (panel (b)) and many low-cost covariates with somewhat smaller sample size (panel (c)) yield very similar and significant improvements in precision or significant reductions in the experimental budget, respectively.\footnote{Note that there is no sense in which need to be concerned about identification of the minimizing set of covariates. There may indeed exist several combinations of covariates that yield similar precision of the resulting treatment effect estimator. Our objective is highest possible precision without any direct interest in the identities of the covariates that achieve that minimum.}
As discussed in Section (ref), one may want to ensure balance of the control and treatment group, especially in terms of strong predictors such as baseline outcomes. Checking balance requires collection of the relevant covariates. Therefore, we also perform the three covariate selection procedures when we force each of them to include the baseline math outcome as a covariate. In the OGA, we can force the selection of a covariate by performing group OGA as described in Section (ref), where each group contains a low-cost covariate together with the baseline math outcome. For the LASSO procedures, we simply perform the LASSO algorithms after partialing out the baseline math outcome from the follow-up outcome. The corresponding results are reported in panel (d). Since baseline outcomes are very expensive covariates, the selected sample sizes relative to those in panels (b) and (c) are much smaller. OGA selects a sample size of 1,314 which is almost twice as large as the experimental sample size, but about 4-5 times smaller than the OGA selections in panels (b) and (c). In contrast to OGA, the two LASSO procedures do not select any other covariates beyond the baseline math outcome. As a result of forcing the selection of the baseline outcome, all three methods achieve an improvement in precision, or reduction of budgets respectively, of around 20% relative to the experiment. These are still substantial gains, but the requirement of checking balance on the expensive baseline outcome comes at the cost of smaller improvements in precision due to our procedure.
In Appendix F, we also perform an out-of-sample evaluation for this application by splitting the dataset into training samples for the covariate selection step and evaluation samples for the computation of the performance measures RMSE and EQB. The results are qualitatively similar to those in Table (ref).
In this section, we discuss related papers in the literature. We emphasize that the research question in our paper is different from those studied in the literature and that our paper is a complement to the existing work.
In the context of experimental economics, List-et-al:11 suggest several simple rules of thumb that researchers can apply to improve the efficiency of their experimental designs. They discuss the issue of experimental costs and estimation efficiency but did not consider the problem of selecting covariates.
hahn2011we consider the design of a two-stage experiment for estimating an average treatment effect, and proposed to select the propensity score that minimizes the asymptotic variance bound for estimating the average treatment effect. Their recommendation is to assign individuals randomly between the treatment and control groups in the second stage, according to the optimized propensity score. They use the covariate information collected in the first stage to compute the optimized propensity score.
Bhattacharya:Dupas:2012 consider the problem of allocating a binary treatment under a budget constraint. Their budget constraint limits what fraction of the population can be treated, and hence is different from our budget constraint. They discuss the costs of using a large number of covariates in the context of treatment assignment.
McKenzie2012 demonstrates that taking multiple measurements of the outcomes after an experiment can improve power under the budget constraint. His choice problem is how to allocate a fixed budget over multiple surveys between a baseline and follow-ups. The main source of the improvement in his case comes from taking repeated measures of outcomes; see Frison:Pocock:1992 for this point in the context of clinical trials. In the set-up of McKenzie2012, a baseline survey measuring the outcome is especially useful when there is high autocorrelation in outcomes. This would be analogous in our paper to devoting part of the budget to the collection of a baseline covariate, which is highly correlated with the outcome (in this case, the baseline value of the outcome), instead of just selecting a post-treatment sample size that is as large as the budget allows for. In this way, McKenzie2012 is perhaps closest to our paper in spirit.
In a very recent paper, Dominitz:Manski:16 proposed the use of statistical decision theory to study allocation of a predetermined budget between two sampling processes of outcomes: a high-cost process of good data quality and a low-cost process with non-response or low-resolution interval measurement of outcomes. Their main concern is data quality between two sampling processes and is distinct from our main focus, namely the simultaneous selection of the set of covariates and the sample size.
We develop data-driven methods for designing a survey in a randomized experiment using information from a pre-existing dataset. Our procedure is optimal in a sense that it minimizes the mean squared error of the average treatment effect estimator and maximizes the power of the corresponding t-test, and can handle a large number of potential covariates as well as complex budget constraints faced by the researcher. We have illustrated the usefulness of our approach by showing substantial improvements in precision of the resulting estimator or substantial reductions in the researcher's budget in two empirical applications.
We recognize that there are several other potential reasons guiding the choice of covariates in a survey. These may be as important as the one we focus on, which is the precision of the treatment effect estimator. We show that it is possible and important to develop practical tools to help researchers make such decisions. We regard our paper as part of the broader task of making the research design process more rigorous and transparent.
Some important issues remain as interesting future research topics. For example, we have assumed that the pre-experimental sample $\mathcal{S}_{\rm pre}$ is large, and therefore the difference between the minimization of the sample average and that of the population expectation is negligible. However, if the sample size of $\mathcal{S}_{\rm pre}$ is small (e.g., in a pilot study), one may be concerned about over-fitting, in the sense of selecting too many covariates. A straightforward solution would be to add a term to the objective function that penalizes a large number of covariates via some information criteria (e.g., the Akaike information criterion (AIC) or the Bayesian information criterion (BIC)).
{\singlespacing
\ifx\undefined\leavevmode\rule[.5ex]{3em}{.5pt}\ \fi \ifx\undefined\textsc \let\tmpsmall\tmpsmall\sc \fi
}
\setcounter{page}{1}