Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
97,647 characters · 15 sections · 39 citation commands
Identification and Estimation of Spillover Effects in Randomized Experiments
\abstract{I study identification, estimation and inference for spillover effects in experiments where units’ outcomes may depend on the treatment assignments of other units within a group. I show that the commonly-used reduced-form linear-in-means regression identifies a weighted sum of spillover effects with some negative weights, and that the difference in means between treated and controls identifies a combination of direct and spillover effects entering with different signs. I propose nonparametric estimators for average direct and spillover effects that overcome these issues and are consistent and asymptotically normal under a precise relationship between the number of parameters of interest, the total sample size and the treatment assignment mechanism. These findings are illustrated using data from a conditional cash transfer program and with simulations. The empirical results reveal the potential pitfalls of failing to flexibly account for spillover effects in policy evaluation: the estimated difference in means and the reduced-form linear-in-means coefficients are all close to zero and statistically insignificant, whereas the nonparametric estimators I propose reveal large, nonlinear and significant spillover effects. }
Keywords: spillover effects, treatment effects, causal inference, interference.
JEL codes: C10, C13, C14, C90.
\setcounter{page}{0}\thispagestyle{empty}
Spillover effects, which occur when an agent's actions or behaviors indirectly affect other agents' outcomes through peer effects, social interactions or externalities, are ubiquitous in economics and social sciences. A thorough account of spillover effects is crucial to assess the causal impact of policies and programs Abadie-Cattaneo_2018_Annurev,Athey-Imbens_2017_handbook. However, the literature is still evolving in this area, and most of the available methods for analyzing treatment effects either assume no spillovers or allow for them in restrictive ways, often without a precise definition of the parameters of interest or the conditions required to recover them.
This paper studies identification, estimation and inference for average direct and spillover effects in randomized controlled trials, and offers three main contributions. First, I provide conditions for nonparametric identification of causal parameters when the true spillovers structure is possibly unknown. Under the assumption that interference occurs within non-overlapping peer groups, I define a rich set of direct and spillover treatment effects based on a function, the treatment rule, that maps peers' treatment assignments and outcomes. Lemma (ref) links average potential outcomes to averages of observed variables when the posited treatment rule is possibly misspecified.
The second main contribution is to characterize the difference in means between treated and controls, and the coefficients from a reduced-form linear-in-means (RF-LIM) regression, two of the most commonly analyzed estimands when analyzing RCTs and spillover effects in general. Theorem (ref) shows that, in the presence of spillovers, the difference in means between treated and controls combines the direct effect of the treatment and the difference in spillover effects for treated and untreated units, and thus the sign of the difference in means is undetermined even when the signs of all direct and spillover effects are known. On the other hand, Theorem (ref) shows that a RF-LIM regression recovers a linear combination of spillover effects for different numbers of treated peers where the weights sum to zero, and hence some weights are necessarily negative. As a result, the coefficients from a RF-LIM regression can be zero even when all the spillover effects are non-zero. I then provide sufficient conditions under which the difference in means and the RF-LIM coefficients have a causal interpretation, that is, when they can be written as proper weighted averages of direct and/or spillover effects. I also propose a simple regression-based pooling strategy that is robust to nonlinearities and heterogeneity in spillover effects.
The third main contribution is to analyze nonparametric estimation and inference for spillover effects. In the presence of spillovers, the number of treatment effects to estimate can be large, and the probability of observing units under different treatment assignments can be small. Section (ref) provides general conditions for uniform consistency and asymptotic normality of the estimators of interest in a double-array asymptotic framework where both the number of groups and the number of parameters are allowed to grow with the sample size. This approach highlights the role that the number of parameters and the assignment mechanism play on the asymptotic properties of nonparametric estimators. More precisely, consistency and asymptotic normality are shown under two main conditions that are formalized in the paper: (i) the number of treatment effects should not be “too large” with respect to the sample size, and (ii) the probability of each treatment assignment should not be “too small”. These two requirements are directly linked to modeling assumptions on the potential outcomes, the choice of the set of parameters of interest and the treatment assignment mechanism. As an alternative approach to inference, the wild bootstrap is shown to be consistent, and simulation evidence suggests that it can yield better performance compared to the normal approximation in some settings.
The results in this paper are illustrated in a simulation study and using data from a randomized conditional cash transfer. The empirical results clearly highlight the pitfalls of failing to flexibly account for spillovers in policy evaluation: the estimated difference in means and RF-LIM coefficients are all close to zero and statistically insignificant, whereas the nonparametric estimators I propose reveal large, nonlinear and significant spillover effects.
This paper is related to a longstanding literature on peer effects and social interactions. A large strand of this literature has focused on identification of social interaction effects in parametric models. The most commonly analyzed specification is the linear-in-means (LIM) model, where a unit's outcome is modeled as a linear function of own characteristics, peers' average characteristics and peers' average outcomes Blume-etal_2015_JPE,Kline-Tamer_2019,Bramoulle-Djebbari-Fortin_2020_AnnuRev. Since Manski_1993_Restud's critique of LIM models, several strategies have been put forward to identify structural parameters Lee_2007_JoE,Bramoulle-Djebbari-Fortin_2009_JoE,Davezies-etal_2009_EJ,DeGiorgi-Pellizzari-Redaelli_2010_AEJ. All these strategies rely on linearity of spillover effects. In this paper, I consider an alternative approach that focuses on reduced-form casual parameters from a potential-outcomes perspective. Within this setup, I show that (reduced-form) response functions can be identified and estimated nonparametrically. While I do not consider identification of structural parameters in this paper, reduced-form parameters are inherently relevant, as they represent the causal effect of changing the peers' covariate values, which is generally more easily manipulable for a policy maker than peers' outcomes Goldsmith-Pinkham-Imbens_2013_JBES,Manski_2013_JBES. This is particularly true in my setup, where the covariate of interest is a treatment that is assigned by the policy maker. Furthermore, identification of reduced-form parameters can be thought of as a necessary ingredient for identifying structural models.
On the opposite end of the spectrum, Manski_2013_EJ and Lazzati_2015_QE study nonparametric partial identification of response functions under different restrictions on the structural model, the response functions and the structure of social interactions. My paper complements this important strand of the literature by considering a specific network structure in which spillovers are limited to non-overlapping groups, where the within-group spillovers structure is left unrestricted. This specific structure of social interactions allows me to obtain point (as opposed to partial) identification of causal effects without knowing the true mapping between treatment assignments an potential outcomes in a setting with wide empirical applicability. Furthermore, by focusing on this network structure I can analyze the effect of misspecifying the within-group spillovers structure, as discussed in Section (ref). Finally, I also complement this literature by providing a formal treatment of estimation and inference and showing validity of the wild bootstrap.
Another body of research has analyzed causal inference under interference from a design-based perspective in which potential outcomes are fixed and all randomness is due to the (known) treatment assignment mechanism Tchetgen-VdW_2012_SMMR,Ogburn-VanderWeele_2014_SS,Halloran-Hudgens_2016_CER. Generally, this literature focuses on two aggregate measures of treatment effects, the average direct effect and the average indirect effect, that average over peers' assignments under specific treatment assignment mechanisms such as completely randomized designs Sobel_2006_JASA or two-stage randomization Hudgens-Halloran_2008_JASA. In a related paper, Athey-Eckles-Imbens_2017_JASA derive a procedure to calculate finite-sample randomization-based p-values to test for the presence of spillover effects. My paper complements this literature in several ways. First, I focus on identifying and estimating the entire vector of spillover effects determined by the treatment rule, which can be seen either as the main object of interest, or as an ingredient to construct the aggregate summary measures of spillovers considered in the literature (see also Remark (ref)). Second, my identification results allow for general treatment assignment mechanisms. Third, I consider large-sample inference from a super-population perspective that allows me to disentangle the roles of the number of groups, the number parameters of interest and the treatment assignment mechanism in the performance of estimators and inferential procedures.
Finally, Moffit_2001, Duflo-Saez_2003_QJE, ,Hirano-Hahn_2010_EL and more recently Baird-etal_2018_Restat analyze the design of partial population experiments, where spillovers are estimated by exposing experimental units to different proportions of treated peers (or “saturations”). My paper complements this strand of the literature by analyzing identification under general experimental designs and by providing a formal treatment of the effect of the experimental design on inference, which formalizes the advantages of partial population designs. This fact is illustrated in Section (ref) and discussed in more detail in Section (ref) of the supplemental appendix.
The remainder of the paper is organized as follows. Section (ref) describes the setup and defines the parameters of interest. Section (ref) provides the main identification results. Section (ref) analyzes estimation and inference. Section (ref) provides a simulation study, and Section (ref) contains the empirical application. Section (ref) concludes. The proofs, together with additional results and discussions, are provided in the supplemental appendix.
As a motivating example, consider a program in which parents in low-income households receive a cash transfer from the government provided their children are enrolled in school and reach a required level of attendance. Suppose that this conditional cash transfer program is evaluated using a randomized pilot in which children are randomly selected to participate. There are several reasons to expect within-household spillovers from this program. On the one hand, the cash transfer may alleviate financial constraints that were preventing the parents from sending their children to school on a regular basis. The program could also help raise awareness on the importance of school attendance. In both these cases, untreated children may indirectly benefit from the program when they have a treated sibling. On the other hand, the program could create incentives for the parents to reallocate resources towards their treated children and away from their untreated siblings, decreasing school attendance for the latter. In all cases, ignoring spillover effects can severely underestimate the costs or the benefits of this policy.
Moreover, these alternative scenarios have drastically different implications on how to assign the program when scaling it up. In the first two situations, treating one child per household can be a cost-effective way to assign the treatment, whereas in the second case, treating all the children in a household can be more beneficial. An accurate assessment of spillovers is therefore crucial for the analysis and design of public policies.
Consider a random sample of groups indexed by $g=1,\ldots,G$, each with $n_g+1$ units, so that each unit $i$ in group $g$ has $n_g$ neighbors or peers and $1\le n_g<\infty$. I assume group membership is observable. Units in each group are assigned a binary treatment, and a unit's potential outcomes, defined below, can depend on the assignment of all other units in the same group. Using the terminology of Ogburn-VanderWeele_2014_SS, this phenomenon is known as direct interference. Interference is assumed to occur between units in the same group, but not between units in different groups.
The individual treatment assignment of unit $i$ in group $g$ is denoted by $D_{ig}$, taking values $d\in\{0,1\}$, and the vector of treatment assignments in each group is given by $\mathbf{D}_g=(D_{1g},\ldots,D_{n_g+1,g})$. For each unit $i$, $D_{jig}$ is the treatment indicator corresponding to unit $i$'s $j$-th neighbor, collected in the vector $\mathbf{D}_{(i)g}=(D_{1ig},D_{2ig},\ldots,D_{n_gig})$. This vector takes values $\mathbf{d}_g=(d_1,d_2,\ldots,d_{n_g})\in \mathcal{D}\subseteq \{0,1\}^{n_g}$.
A key element in this setup will be a function $h_0(\cdot)$ that summarizes how the vector $\mathbf{d}_g$ enters the potential outcome. More precisely, define a function or treatment rule: \[h_0:\mathcal{D}\to\mathcal{H}_0\] that maps $\mathbf{d}_g$ into some value $h_0(\mathbf{d}_g)$ of the same or smaller dimension, so that $\mathrm{dim}(\mathcal{H}_0)\le \mathrm{dim}(\mathcal{D})$. Following Manski_2013_EJ's terminology, for $h_0(\mathbf{d}_g)=\mathbf{h}_0$, I will refer to the tuple $(d,\mathbf{h}_0)$ as the effective treatment assignment, an element in the set $\{0,1\}\times\mathcal{H}_0$. The potential outcome for unit $i$ in group $g$ is denoted by the random variable $Y_{ig}(d,\mathbf{h}_0)$ where $\mathbf{h}_0=h_0(\mathbf{d}_g)\in\mathcal{H}^0$.
In what follows, $\mathbf{0}_ g$ and $\mathbf{1}_g$ will denote $n_g$-dimensional vectors of zeros and ones, respectively. Throughout the paper, I will assume that all the required moments of the potential outcomes are bounded. Unit-level direct effects are defined as differences in potential outcomes switching own treatment assignment for a fixed peer assignment $\mathbf{h}_0$, $Y_{ig}(1,\mathbf{h}_0)-Y_{ig}(0,\mathbf{h}_0)$. Unit-level spillover effects are defined as differences in potential outcomes switching peer assignments for a fixed own assignment $d$, $Y_{ig}(d,\mathbf{h}_0)-Y_{ig}(d,\mathbf{\tilde{h}}_0)$.
Given a vector of observed assignments $(D_{ig},\mathbf{D}_{(i)g})$, the observed outcome is given by $Y_{ig}(D_{ig},h_0(\mathbf{D}_{(i)g}))$ and can be written as:
\[Y_{ig}=\sum_{d\in\{0,1\}}\sum_{\mathbf{h}_0\in\mathcal{H}^0} Y_{ig}(d,\mathbf{h}_0)\mathbbm{1}(D_{ig}=d)\mathbbm{1}(h_0(\mathbf{D}_{(i)g})=\mathbf{h}_0).\]
To fix ideas, consider a household with three children, $n_g+1=3$. In this household, each kid has two siblings, with assignments $d_1$ and $d_2$, so $\mathbf{d}_g=(d_1,d_2)$. If the true treatment rule $h_0(\cdot)$ is the identity function, the potential outcome has the form $Y_{ig}(d,d_1,d_2)$ and hence each unit can have up to $2^{(n_g+1)}=8$ different potential outcomes. In this case, $Y_{ig}(1,0,0)-Y_{ig}(0,0,0)$ is the direct effect of the treatment when both of unit $i$'s siblings are untreated, $Y_{ig}(0,1,0)-Y_{ig}(0,0,0)$ is the spillover effect on unit $i$ of treating unit $i$'s first sibling, and so on. The average effect of assignment $(d,d_1,d_2)$ compared to $(\tilde{d},\tilde{d}_1,\tilde{d}_2)$ is thus given by $\mathbb{E}[Y_{ig}(d,d_1,d_2)]-\mathbb{E}[Y_{ig}(\tilde{d},\tilde{d}_1,\tilde{d}_2)]$. On the other hand, under an exchangeable treatment rule, the potential outcome can be written as $Y_{ig}(d,s)$ where $s=0,1,2$ is the number of treated siblings. The total number of different potential outcomes is $2(n_g+1)=6$, so exchangeability reduces the dimensionality of the effective assignments set from exponential to linear in group size.
I assume perfect compliance, which means that all units receive the treatment they are assigned to. I analyze the case of imperfect compliance in Vazquez-Bare_2020_IV. The data come from an infinite population of groups for which the researcher observes the outcomes at the unit level and the vector of treatment assignments in each group. Furthermore, by virtue of random assignment, potential outcomes are independent of treatment assignment. I formalize these features as follows.
Parts (a) and (b) in Assumption (ref) state that we observe a random sample of independent groups, and all groups with the same size are identically distributed. Part (c) indicates that units are identically distributed within each group, so that, for instance, $\mathbb{E}[Y_{ig}(d,\mathbf{h}_0)]$ is not indexed by $i$ (or $g$, given part (b)), and the same holds for other moments of the potential outcomes. Note that this condition does not prevent average potential outcomes (or other moments) to differ conditional on covariates. For example, if $X_{ig}$ denotes gender, the setup allows for $\mathbb{E}[Y_{ig}(d,\mathbf{h}_0)|X_{ig}=male]\ne \mathbb{E}[Y_{ig}(d,\mathbf{h}_0)|X_{ig}=female]$. Additionally, the identification results in the paper can be adapted to the case of non-identical distributions within group by switching focus from $\mathbb{E}[Y_{ig}(d,\mathbf{h}_0)]$ to $\sum_{i=1}^{n_g+1}\mathbb{E}[Y_{ig}(d,\mathbf{h}_0)]/(n_g+1)$. Finally, part (d) states that the treatment is randomly assigned and hence the vector of treatment indicators is independent of potential outcomes.
In practice, the true $h_0(\cdot)$ is usually unknown, and the researcher needs to posit a candidate $h(\cdot)$ that may or may not coincide with $h_0(\cdot)$. Given the lack of knowledge on the true assignment, a function $h(\cdot)$ that imposes fewer restrictions on the potential outcomes has a lower risk of misspecification. To formalize this idea, I introduce the following definition.
Intuitively, this means that $h(\cdot)$ gives a “cruder” summary of $\mathbf{d}_g$ (i.e. it discards more information) than $\tilde{h}(\cdot)$. In other words, a coarser function imposes more restrictions on the potential outcomes. For example, the exchangeable assignment $h(\mathbf{d}_g)=\mathbf{1}_g'\mathbf{d}_g$ is coarser than the identity function $\tilde{h}(\mathbf{d}_g)=\mathbf{d}_g$, and the reference group assignment $h(\mathbf{d}_g)=(d_1,d_2)$ is coarser than $\tilde{h}(\mathbf{d}_g)=(d_1,d_2,d_3,d_4)$.
The next section addresses identification of average potential outcomes when the true treatment rule $h_0(\cdot)$ is possibly unknown.
In what follows, let $\mathbf{H}_{ig}=h(\mathbf{D}_{(i)g})$ be the observed value of the chosen treatment rule, and let $\mathbf{H}^0_{ig}=h_0(\mathbf{D}_{(i)g})$. The following result links observed outcomes, potential outcomes and effective treatment assignments, and will be used in the upcoming theorems.
Different choices of the function $m(\cdot)$ lead to different estimands of interest. For example, setting $m(\cdot)=\mathbbm{1}(\cdot\le y)$ for some $y\in\mathbbm{R}$ yields $\mathbb{E}[m(Y_{ig})|D_{ig}=d,\mathbf{H}_{ig}=\mathbf{h}]=F_Y(y|D_{ig}=d,\mathbf{H}_{ig}=\mathbf{h})$ where $F_Y(\cdot|\cdot)$ is the conditional cdf of $Y_{ig}$. This choice of $m(\cdot)$ can be used to identify the distribution of potential outcomes. In what follows, unless explicitly stated, I will let $m(\cdot)$ be the identity function to reduce notation.
Lemma (ref) shows that the average observed outcome among units facing $D_{ig}=d$ and $\mathbf{H}_{ig}=\mathbf{h}$ averages the potential outcomes over all the assignments $\mathbf{h}_0$ that are consistent with $(D_{ig},\mathbf{H}_{ig})=(d,\mathbf{h})$, as long as the probability of $(d,\mathbf{h})$ is not zero.
To illustrate Lemma (ref), consider the previous example with three units and where $h_0(\cdot)$ is the identity function so the potential outcome has the form $Y_{ig}(d,d_1,d_2)$. Suppose we posit an exchangeable treatment rule $h(\mathbf{d}_g)=\mathbf{1}'_g\mathbf{d}_g$ and thus $\mathbf{H}_{ig}=S_{ig}=\sum_{j\ne i} D_{jg}$ which is a scalar counting how many of unit $i$'s peers are treated. By Lemma (ref), if $\mathbb{P}[D_{ig}=0,S_{ig}=1]>0$, $\mathbb{E}[Y_{ig}|D_{ig}=0,S_{ig}=1]$ equals a weighted average of $\mathbb{E}[Y_{ig}(0,1,0)]$ and $\mathbb{E}[Y_{ig}(0,0,1)]$, with weights given by the conditional probabilities of these different assignments.
In general, $(D_{ig},\mathbf{H}_{ig})=(d,\mathbf{h})$ may be consistent with many different effective assignments $\mathbf{h}_0$. When $h_0(\cdot)$ is coarser than $h(\cdot)$, however, the value of $\mathbf{h}_0$ is uniquely determined. In such cases, the second part of Lemma (ref) shows that $\mathbb{E}[Y_{ig}|D_{ig}=d,\mathbf{H}_{ig}=\mathbf{h}]$ identifies the value of the average potential outcome consistent with that assignment. For example, in the case of $n_g=2$, suppose that the true $h_0(\cdot)$ is exchangeable, so that outcomes have the form $Y_{ig}(d,s)$ with $s=0,1,2$, and suppose we posit $h(\mathbf{d}_g)=\mathbf{d}_g=(d_1,d_2)$. Setting $\mathbf{H}_{ig}=(1,1)$ implies that the sum of treated peers is equal to 2, and therefore $\mathbb{E}[Y_{ig}|D_{ig}=0,\mathbf{H}_{ig}=(1,1)]=\mathbb{E}[Y_{ig}(0,2)]$. In particular, this result implies that if $h(\cdot)$ is equal to $h_0(\cdot)$, $\mathbb{E}[Y_{ig}|D_{ig}=d,\mathbf{H}_{ig}=\mathbf{h}]=\mathbb{E}[Y_{ig}(d,\mathbf{h})]$.
The difference in means estimand $\beta_\mathsf{D}=\mathbb{E}[Y_{ig}|D_{ig}=1]-\mathbb{E}[Y_{ig}|D_{ig}=0]$, which compares the average observed outcomes between treated and controls, is arguably the most common estimand when analyzing randomized experiments. It is well known that, in the absence of spillovers, $\beta_{\mathsf{D}}$ equals the average treatment effect (ATE) when the treatment is randomly assigned. An estimate for the ATE can be calculated by estimating the model:
The following results characterizes the difference in means $\beta_{\mathsf{D}}$ in the presence of spillovers. In what follows, let $\mathbf{0}=h_0(\mathbf{0}_g)$.
Hence, the difference-in-means estimand equals the direct effect without treated peers $\mathbb{E}[Y_{ig}(1,\mathbf{0})-Y_{ig}(0,\mathbf{0})]$ plus the difference in weighted averages of spillover effects under treatment and under control. In general, the sign of this difference is undetermined, as it depends on the relative magnitudes of the average spillover effects on treated and controls. Thus, the difference in means can be larger than, smaller than or equal to the average direct effect $\mathbb{E}[Y_{ig}(1,\mathbf{0})-Y_{ig}(0,\mathbf{0})]$. In particular, if the spillover effects on treated units are equal to zero and the spillover effects on controls are of the same sign that the direct effect, the difference in means will underestimate the average direct effect without treated peers. This case matches the commonly invoked intuition that spillovers “contaminate” the control group.
Equation (ref) may give an incomplete assessment of the effect of a treatment because it completely ignores the presence of spillovers. When trying to explicitly estimate spillover effects, a common strategy is to estimate a reduced-form linear-in-means (RF-LIM) regression, which is given by:
where \[\bar{D}^{(i)}_g=\frac{1}{n_g}\sum_{j\ne i}D_{jg}.\] This is a regression of the outcome on a treatment indicator and the proportion of treated peers. In this specification, $\beta_\ell$ intends to capture a direct effect whereas $\gamma_{\ell}$ is seen as a measure of spillover effects, since it captures the average change in outcomes in response to a change in the proportion of treated neighbors. While the parameters $(\beta_\ell,\gamma_\ell)$ can be interpreted as linear projection coefficients, the following result shows that they do not have a causal interpretation in general.
Theorem (ref) characterizes the coefficients from the linear projection of $Y_{ig}$ into $D_{ig}$ and $\bar{D}^{(i)}_g$. The coefficient $\beta_\ell$ equals the difference in means minus an adjustment factor that depends on $\gamma_\ell$ and the relationship between treatment assignments within group. This parameter is similar to the one analyzed in Theorem (ref) and hence does not have a direct causal interpretation in general.
The coefficient $\gamma_\ell$ equals a linear combination of differences $\mathbb{E}[Y_{ig}|D_{ig}=d,S_{ig}=s]-\mathbb{E}[Y_{ig}|D_{ig}=d,S_{ig}=0]$ across all values of $s$ with weights $(\phi_0(s),\phi_1(s))_s$. Two factors obscure the causal interpretation of this coefficient. On the one hand, the magnitudes $\mathbb{E}[Y_{ig}|D_{ig}=d,S_{ig}=s]$ implicitly impose an exchangeable treatment rule that may be misspecified in general. The interpretation of these expectations is given in Lemma (ref). On the other hand, even if this treatment rule was correctly specified, so that $\mathbb{E}[Y_{ig}|D_{ig}=d,S_{ig}=s]-\mathbb{E}[Y_{ig}|D_{ig}=d,S_{ig}=0]=\mathbb{E}[Y_{ig}(d,s)-Y_{ig}(d,0)]$, these effects are combined using weights that can be negative. Indeed, it can be seen that $\sum_{s=0}^{n_g}\phi_d(s)=0$ for $d=0,1$ and hence some of the weights are necessarily negative (specifically, the ones corresponding to low values of $s$). Thus, $\gamma_\ell$ may be zero or negative when all the average spillover effects are positive or vice versa.
To illustrate the importance of these issues in an empirical setting, Section (ref) shows a case in which the estimates of $\beta_\ell$ and $\gamma_\ell$ are close to zero and not statistically significant, even when the estimated average spillover effects are all large and statistically significant when estimated nonparametrically.
The following result provides conditions under which the coefficients from a RF-LIM regression have a clear causal interpretation, that is, they can be written as proper weighted averages of direct and/or spillover effects.
The above result highlights two restrictions that the RF-LIM regression implicitly imposes on potential outcomes: (i) peers are exchangeable, so potential outcomes only depend on own treatment and the number of treated peers, and (ii) average spillover effects are linear in $s$ so that, for instance, the effect of having two treated peers is twice as large as the effect of having one treated peer. If these conditions hold, $\gamma_\ell$ recovers a weighted average of the effects of having all treated peers for treated and untreated units, where the weights are positive and sum to one. Hence, the RF-LIM regression is robust to some heterogeneity in spillover effects $\mathbb{E}[Y_{ig}(d,s)-Y_{ig}(d,0)]$ both over $s$ and over $d$, but suffers from potentially severe misspecification when spillover effects are nonlinear.
On the other hand, $\beta_\ell$ does not recover a causal effect in general. In the particular case in which $\kappa_1=\kappa_0$, that is, the average spillover effects are equal for treated and untreated units, $\beta_\ell$ equals the average direct effect with no treated peers $\mathbb{E}[Y_{ig}(1,0)-Y_{ig}(0,0)]$.
A straightforward way to make Equation (ref) more flexible is to include an interaction term between own treatment indicator and the proportion of treated peers:
where $\mathbb{E}[\xi_{ig}]=\mathbb{C}\text{ov}(D_{ig},\xi_{ig})=\mathbb{C}\text{ov}(\bar{D}^{(i)}_g(1-D_{ig}),\xi_{ig})=\mathbb{C}\text{ov}(\bar{D}^{(i)}_gD_{ig},\xi_{ig})=0$. The following result characterizes the coefficients from this specification.
According to this theorem, an interacted RF-LIM regression separates the spillover components $\mathbb{E}[Y_{ig}|D_{ig}=0,S_{ig}=s]-\mathbb{E}[Y_{ig}|D_{ig}=0,S_{ig}=0]$ and $\mathbb{E}[Y_{ig}|D_{ig}=1,S_{ig}=s]-\mathbb{E}[Y_{ig}|D_{ig}=1,S_{ig}=0]$. However, the issue of negative weights remains for each component, since $\sum_{s=0}^{n_g}\omega_d(s)=0$ for $d=0,1$ and hence some of the weights are necessarily negative.
Finally, the following result shows that when peers are exchangeable and spillover effects are linear, the interacted RF-LIM regression can recover all the causal parameters of interest.
According to this result, the coefficients from a correctly-specified interacted RF-LIM regression recover the average direct effect without treated peers and the spillover effects of having all peers treated, for treated and untreated units separately. Because of linearity, all the remaining spillover effects can be recovered by appropriately rescaling $\gamma^d_\ell$. For example, the average spillover effect from having one treated peer on an untreated unit is $\mathbb{E}[Y_{ig}(0,1)-Y_{ig}(0,0)]=\gamma^0_\ell/n_g$.
The previous sections provide conditions under which average direct and spillover effects can be nonparametrically identified by exploiting variation over own and peers' assignments. Because these magnitudes can be written as population averages, it is straightforward to construct their corresponding estimators based on sample cell means. The main challenge for estimation arises when groups are large. A large number of units per group (as in households with multiple family members or classrooms with a large number of students) requires estimating a large number of means in each of the cells defined by the treatment assignments. In such cases, the probability of observing some assignments can be close to zero and the number of observations in each cell may be too small to estimate the average potential outcomes.
For example, suppose the treatment is assigned as an independent coin flip with probability $p=1/2$. Under this assignment we would expect most groups to have about half its units treated, so when groups have, say, 10 units, 5 of them would be treated on average. The probability of observing groups with zero or all treated units, on the other hand, will be close to zero, and thus the average potential outcomes corresponding to these “tail assignments” will be hard to estimate precisely.
So far, the analysis has been done taking group size as fixed. When group size is fixed, small cells are a finite sample problem that disappears as the sample grows. To account for this phenomenon asymptotically, in this section I will generalize this setting and consider double-array asymptotics in which the group size is allowed to grow with the sample size. The goal is to answer the question of how large groups can be relative to the total sample size to allow for valid estimation and inference. The key issue to obtain consistency and asymptotic normality will be to ensure that the number of observations in all cells grows sufficiently fast as the sample size increases. This setup is not intended to model a population in which groups are effectively infinitely large, but as a statistical device to approximate the distribution of estimators in finite samples when the number of parameters can be “moderately” large, in a sense that will be made more precise in this section. The case with fixed group size is a particular case of this setting.
In this section I will assume that groups are equally sized, so that $n_g=n$. Recall that given a candidate treatment rule $h(\cdot)$ and $\mathbf{h}=h(\mathbf{d}_g)$, the effective treatment assignments are given by $(d,\mathbf{h}_g)\in\{0,1\}\times\mathcal{H}$. As formalized in Assumption (ref) below, $h(\cdot)$ is not assumed to equal the true treatment rule, but the true treatment rule $h_0(\cdot)$ has to be coarser than $h(\cdot)$ as specified in Definition (ref). To make the notation more compact, I will let $\mathcal{A}_n=\{0,1\}\times\mathcal{H}$ where the notation makes the dependence of this set on the group size explicit. The cardinality of this set is denoted by $|\mathcal{A}_n|$, which indicates the total number of parameters to be estimated. The observed effective treatment assignments will be $(D_{ig},\mathbf{H}_{ig})=\mathbf{A}_{ig}$, taking values $\mathbf{a}\in\mathcal{A}_n$, and $\mu(\mathbf{a})=\mathbb{E}[Y_{ig}|\mathbf{A}_{ig}=\mathbf{a}]$.
Because $\mathbf{A}_{ig}$ takes on a finite number of values, all the conditional means can be estimated jointly through the regression:
where $\mathcal{A}_{n,0}=\mathcal{A}_n\backslash \{\mathbf{a}_0\}$ and $\mathbf{a}_0$ is the baseline treatment assignment (typically, the assignment in which no unit is treated). Because this regression is fully saturated, by construction $\alpha=\mathbb{E}[Y_{ig}|\mathbf{A}_{ig}=\mathbf{a}_0]$ and $\beta_\mathbf{a}=\mathbb{E}[Y_{ig}|\mathbf{A}_{ig}=\mathbf{a}]-\mathbb{E}[Y_{ig}|\mathbf{A}_{ig}=\mathbf{a}_0]$. Hence, this regression can be seen as a nonparametric regression as it does not impose any functional form assumptions. Since all the coefficients are linear combinations of conditional means, it suffices to focus on the vector of means $\mu(\mathbf{a})$ to analyze the properties of the coefficient estimators.
Each treatment assignment mechanism determines a distribution $\pi(\cdot)$ over $\mathcal{A}_n$ where $\pi(\mathbf{a})=\mathbb{P}[\mathbf{A}_{ig}=\mathbf{a}]$ for $\mathbf{a}\in\mathcal{A}_n$. For example, when $\mathcal{A}_n=\{0,1\}$, if the treatment is assigned independently as a coin flip, $\pi(1)=\mathbb{P}[D_{ig}=1]=p$ and $\pi(0)=1-p$. Under the same assignment, with an exchangeable treatment rule, $\pi(d,s)=\mathbb{P}[D_{ig}=d,S_{ig}=s]={n \choose s}p^{s+d}(1-p)^{n+1-s-d}$. A key issue of this double-array asymptotic setup is that, since the size of the set $\mathcal{A}_n$ can increase with group size, the probabilities $\pi(\mathbf{a})$ can shrink towards zero for some (or all) assignments $\mathbf{a}\in\mathcal{A}_n$. The rate at which these probabilities decrease with the sample size is given by the experimental design. For instance, in the coin flip assignment just described, $\mathbb{P}[D_{ig}=0,S_{ig}=0]=(1-p)^{n+1}$ which decreases exponentially with $n$. Define: \[\underline{\pi}_n=\min\limits_{\mathbf{a}\in\mathcal{A}_n} \pi(\mathbf{a})\] which is the probability of the least likely treatment assignment. This probability, together with the total sample size, will determine the number of observations in the smallest assignment cell, that is, the number of observations available to estimate the “hardest” average potential outcome.
Let $\mathbf{A}_g=(\mathbf{A}_{1g},\ldots,\mathbf{A}_{n_g+1,g})$, $\mathbf{A}=(\mathbf{A}_1,\ldots,\mathbf{A}_G)$, and $\mathbf{Y}_g=(Y_{1g},Y_{2g},\ldots Y_{n_g+1,g})'$. I will assume the following.
Part (i) in Assumption (ref) states that the researcher has access to a sample of $G$ independent and identically distributed equally-sized groups. When groups have different sizes (for example, households with 3, 4 or 5 siblings), the analysis can be performed separately for each group size. Section (ref) of the supplemental appendix further discusses the case of unequally-sized groups. Part (ii) allows the posited treatment rule $h(\cdot)$ to be different from the true treatment rule, but requires it to be flexible enough to break the dependence between $Y_{ig}$ and $\mathbf{A}_{jg}$ conditional on $\mathbf{A}_{ig}$ for $j\ne i$. Part (iii) assumes that potential outcomes are independent within groups, and hence the only source of dependence between the observed outcomes is the assignment $\mathbf{A}_g$. This condition can be relaxed to arbitrary dependence structures when the group size is fixed. Together, conditions (ii) and (iii) imply that observed outcomes are independent conditional on the assignments. Importantly, note that these conditions do not restrict the correlation between treatment assignments $\mathbf{A}_{ig}$ and $\mathbf{A}_{jg}$ in any way. In fact, the effective treatment assignments are correlated by construction, since $\mathbf{A}_{ig}$ depends on $D_{jg}$ and vice versa.
Given a sample of $G$ groups with $n+1$ units each, let $\mathbbm{1}_{ig}(\mathbf{a})=\mathbbm{1}(\mathbf{A}_{ig}=\mathbf{a})$, $N_g(\mathbf{a})=\sum_{i=1}^{n+1}\mathbbm{1}_{ig}(\mathbf{a})$ and $N(\mathbf{a})=\sum_{g=1}^G N_g(\mathbf{a})$, so that $N_g(\mathbf{a})$ is the total number of observations receiving effective assignment $\mathbf{a}$ in group $g$ and $N(\mathbf{a})$ is the total number of observations receiving effective assignment $\mathbf{a}$ in the sample. The estimator for $\mu(\mathbf{a})$ is defined as:
Thus, the estimator for $\mu(\mathbf{a})$ is simply the sample average of the outcomes for observations receiving assignment $\mathbf{a}$, whenever there is at least one observation receiving this assignment.
The following assumption imposes some regularity conditions that are required for upcoming theorems. Let $\sigma^2(\mathbf{a})=\mathbb{V}[Y_{ig}|\mathbf{A}_{ig}=\mathbf{a}]$.
Then we have the following result.
Lemma (ref) says that, under condition (ref), the number of observations in the smallest cell will go to infinity, which implies that all the estimators are well defined asymptotically. Hence, condition (ref) formalizes the meaning of “large sample” in this context, and states that the number of groups has to be large relative to the total number of parameters and the probability of the least likely assignment. This expression can be interpreted as an invertibility condition for the design matrix of a linear regression model, in the specific case in which the regressors are mutually exclusive indicator variables. This requirement can be seen as a low-level condition that justifies the assumption of invertibility of the design matrix Cattaneo-Jansson-Newey_2018_JASA. When this condition does not hold, small cells may result in estimators with poor finite sample behavior and whose asymptotic distribution, if it exists, may be non-Gaussian. See Ma-Wang_2020_JASA for an example in the context of inverse-probability weighting estimators, which includes randomized experiments as a special case.
Next, let \[\hat{\sigma}^2(\mathbf{a})=\frac{\sum_{g=1}^G\sum_{i=1}^{n+1}(Y_{ig}-\hat{\mu}(\mathbf{a}))^2\mathbbm{1}_{ig}(\mathbf{a})}{N(\mathbf{a})}\mathbbm{1}(N(\mathbf{a})>0)\] be the variance estimator for each $\mathbf{a}$. Then we have the following result.
Equation (ref) shows that both the average outcome and variance estimators converge in probability to their true values, uniformly over treatment assignments, at the rate $\sqrt{\log|\mathcal{A}_n|/(G(n+1)\underline{\pi}_n)}$. The denominator in this rate can be seen as the minimum expected cell size, whereas the numerator is a penalty for having an increasing number of parameters. Equation (ref) bounds the difference between the distributions of the standardized average outcomes estimators and the standard normal distribution, uniformly over the treatment assignments. Under condition (ref), $G(n+1)\underline{\pi}_n\to \infty$, which gives asymptotic normality. Furthermore, this bound also reveals the rate at which the distribution of the standardized estimator approaches the standard normal, namely, $\sqrt{G(n+1)\underline{\pi}_n}$.
Another implication of Theorem (ref) is that the estimators $\hat{\mu}(\mathbf{a})$ have different convergence rates. More precisely, Lemma (ref) in the supplemental appendix and Markov's inequality imply that, for each $\mathbf{a}$, $|\hat{\mu}(\mathbf{a})-\mu(\mathbf{a})|=O_\mathbb{P}\left((G(n+1)\pi(\mathbf{a}))^{-1/2}\right)$ and thus the average outcomes for assignments whose probability decreases faster are estimated at a slower rate.
Finally, the condition that $|\mathcal{A}_n|=O(G(n+1)\underline{\pi}_n)$ requires that the number of parameters do not grow faster than the expected sample size in the smaller cell. Notice that when the number of parameters grows linearly, as in an exchangeable treatment rule, this condition is implied by Condition (ref). Section (ref) in the supplemental appendix provides sufficient conditions for these requirements under two different assignment mechanisms.
An alternative approach to perform inference in this setting is the bootstrap. Since the challenge for inference is that cells can have too few observations for the normal distribution to provide a good approximation, the wild bootstrap Shao_book can offer a more accurate approximation when groups are relatively large. One way to implement this type of bootstrap is to define weights $w_{ig}\in\{-1,1\}$ with probability $1/2$ independently of the sample. The bootstrap estimator for $\mu(\mathbf{a})$ is given by: \[\hat{\mu}^*(\mathbf{a})=\frac{\sum_{g=1}^G\sum_{i=1}^{n+1} Y^*_{ig}\mathbbm{1}_{ig}(\mathbf{a})}{N(\mathbf{a})}\] whenever the denominator is non-zero, where \[Y^*_{ig}\mathbbm{1}_{ig}(\mathbf{a})=(\bar{Y}(\mathbf{a})+(Y_{ig}-\bar{Y}(\mathbf{a}))w_{ig})\mathbbm{1}_{ig}(\mathbf{a})=(\bar{Y}(\mathbf{a})+\hat{\varepsilon}_{ig}w_{ig})\mathbbm{1}_{ig}(\mathbf{a})\] In what follows, $\mathbb{P}^*[\cdot]$ denotes a probability calculated over the distribution of $w_{ig}$, conditional on the sample, and $\mathbb{E}^*[\cdot]$ and $\mathbb{V}^*[\cdot]$ the expectation and variance calculated over $\mathbb{P}^*[\cdot]$. The validity of the wild bootstrap is established in the following theorem.
This theorem shows that the wild bootstrap can be used to approximate the distribution of the estimators as an alternative to the standard normal, which may not be accurate when cells have few observations. The performance of the wild bootstrap is illustrated in Section (ref) using simulation data.
This section illustrates the above results in a simulation setting. The outcome is binary and exchangeable, and generated by the following DGP: \[\mathbb{P}[Y_{ig}(d,s)=1]=\mu(d,s)=0.75+0.13 \times d + 0.12 \times(1-d)\mathbbm{1}(s>0)\] where the spillover effect on an untreated unit is equal to $0.12$ whenever at least one neighbor is treated, and zero for treated units. This DGP is based on the empirical application in Section (ref).
The simulations consider two assignment mechanisms. First, simple random assignment (SR), where treatment is assigned independently with $\mathbb{P}[D_{ig}=1]=0.5$. Second, two-stage randomization with fixed margins (2SR-FM), where in the first stage groups are assigned the total number of treated units between $t=0,1,2,\ldots,n+1$ with probability $q_t$, and then treated units are chosen randomly within each group. The probabilities $q_t$ are chosen so that sample sizes across all assignments are similar; see Section (ref) in the supplemental appendix for further details. Corollaries (ref) and (ref) in the supplemental appendix show that when peers are exchangeable, under SR, condition (ref) and the conditions in Theorem (ref) hold whenever $(n+1)/\log G\to 0$, whereas under 2SR-FM the conditions hold when $\log (n+1)/\log G\to 0$. Because the second condition is weaker, the normal approximation is expected to perform better for 2SR-FM as groups get larger.
The parameter of interest will be $\theta_0(n)=\mathbb{E}[Y_{ig}(0,n)]-\mathbb{E}[Y_{ig}(0,0)]$, which is the average spillover effect for an untreated unit with all peers treated. In this simulation, $\theta_0(n)=0.12$. This parameters can be seen as a “worst-case scenario” given that the probability of the assignment $(D_{ig},S_{ig})=(0,n)$ is one of the smallest (in fact, the smallest under 2SR-FM). The spillover effect estimator is the difference in cell means: \[\hat{\theta}_0(n)=\frac{\sum_{g=1}^{G}\sum_{i=1}^{n+1} Y_{ig}\mathbbm{1}_{ig}(0,n)}{N(0,n)}-\frac{\sum_{g=1}^{G}\sum_{i=1}^{n+1} Y_{ig}\mathbbm{1}_{ig}(0,0)}{N(0,0)}\] whenever $N(0,n)>1$ and $N(0,0)>1$, so that both the estimator and its standard error can be calculated. When at least one of the cells has one or zero observations, the estimator is undefined.
Table (ref) presents the results for a sample with 300 groups, for group sizes $n+1\in\{3,4,\ldots,8\}$. The upper panel shows the results under SR while the lower panel corresponds to the 2SR-FM assignment. In each panel, the first row gives the value of the condition to achieve consistency and asymptotic normality; intuitively, the closer this value is to zero, the better the normal approximation should be. The second and third rows show the bias and the variance of $\hat{\theta}_0(n)$, calculated over the values of the simulated estimates conditional on the estimate being well defined (i.e. when the cells have enough observations to calculate the estimator and its variance). Rows four to seven show the coverage rate and average length of a 95% confidence intervals based on the normal approximation and the wild bootstrap. The eighth row gives the proportion of the simulations in which the estimator or its standard error could not be calculated due to insufficient number of observations. Finally the last two rows show the average sample size in the two assignment cells of interest. Coverage rates are also shown in Figure (ref).
The simulations reveal that under simple randomization, the estimator performs well for $n\le 4$, with no bias and coverage rates very close to 95% for both the normal approximation and the wild bootstrap. When $n>4$, however, coverage rates decrease in both cases, more rapidly so for the normal confidence interval whose coverage rate drops to 88% for $n=7$. While the coverage of the bootstrap confidence interval also decreases, it remains closer to 95% compared to the normal approximation. The table also shows that under this assignment mechanism, the corresponding sample sizes decrease very rapidly in the relevant assignment cells. When $n=7$, each mean is calculated using 9 observations on average, and is undefined in about 32% of the simulations.
On the other hand, under two-stage randomization, both the normal and the wild bootstrap confidence intervals perform equally well and coverage remains very close to 95%. As shown in the last two rows, two-stage randomization ensures much larger sample sizes in the assignment cells of interest compared to simple randomization.
Table (ref) shows the same results for a sample with 600 groups. As expected, the estimator and confidence intervals show better performance compare to the case with $G=300$.
In this section I reanalyze the data from Barrera-Osorio-etal_2011_AEJ. The authors conducted a pilot experiment designed to evaluate the effect of a conditional cash transfer program, Subsidios Condicionados a la Asistencia Escolar, in Bogot{\'a}, Colombia. The program aimed at increasing student retention and reducing drop-out and child labor. Eligible registrants ranging from grade 6-11 were randomly assigned to treatment and control.\footnote{The experiment had two different treatments that varied the timing of the payments, but, following the authors, I pool the two treatment arms to increase the sample size. See Barrera-Osorio-etal_2011_AEJ for details.} The assignment was performed at the student level. In addition to administrative and enrollment data, the authors collected baseline and follow-up data from students in the largest 68 of the 251 schools. This survey contains attendance data and was conducted in the household. As shown in Table (ref), 1,594 households have more than one registered child (rows labeled 2 to 5), and since the treatment was assigned at the child level, this gives variation in the number of treated children per household. This can be seen in Table (ref), which shows that the number of treated children varies from 0 to 5.
I analyze direct and spillover effects restricting the sample to households with three registered siblings, which gives a total of 168 households and 504 observations. The outcome of interest is school attendance. Because groups are very small in this case, inference can be conducted using standard methods (see Remark (ref)).
I start by estimating the average direct and spillover effects exploiting variation in the number of treated siblings using the following regression:
Because this regression is saturated, it follows that: \[\tau=\mathbb{E}[Y_{ig} | D_{ig}=1,S_{ig}=0]-\mathbb{E}[Y_{ig} | D_{ig}=0,S_{ig}=0]\] and \[\theta_d(s)=\mathbb{E}[Y_{ig} | D_{ig}=d,S_{ig}=s]-\mathbb{E}[Y_{ig} | D_{ig}=d,S_{ig}=s].\] Lemma (ref) provides two alternative ways to interpret these estimands. If siblings are exchangeable, so that average potential outcomes take the form $\mathbb{E}[Y_{ig}(d,s)]$, then $\tau=\mathbb{E}[Y_{ig}(1,0)-Y_{ig}(0,0)]$ is the average direct effect of the treatment on a child with no treated siblings, whereas $\theta_d(s)=\mathbb{E}[Y_{ig}(d,s)-Y_{ig}(d,0)]$ is the average spillover effect of having $s$ treated siblings under own treatment status $d$. In this application, exchangeability may be a reasonable assumption if parents make schooling decisions based on how many of their children are treated (for example, to determine whether the cash transfer covers the direct and opportunity costs of sending their children to school), regardless of which of their children are treated.
On the other hand, if exchangeability does not hold, Lemma (ref) shows that the parameters from Equation (ref) combine weighted averages of average potential outcomes. For example, if siblings have a certain (possibly unknown) ordering so that the true average potential outcomes take the form $\mathbb{E}[Y_{ig}(d,\mathbf{d}_g)]$, then: \[\tau=\mathbb{E}[Y_{ig}(1,\mathbf{0}_g)-Y_{ig}(0,\mathbf{0}_g)]\] and \[\theta_d(s)=\sum_{\mathbf{d}_g:\mathbf{1}_g'\mathbf{d}_g=s}\mathbb{E}[Y_{ig}(d,\mathbf{d}_g)-Y_{ig}(d,\mathbf{0}_g)]\mathbb{P}[\mathbf{D}_{(i)g}=\mathbf{d}_g|D_{ig}=d,S_{ig}=s]\] which averages the spillover effects over the values of $\mathbf{D}_{(i)g}$ that are consistent with $S_{ig}=s$. Thus, Equation (ref) provides a way to summarize the direct and spillover effects of the program even when the true structure of the average potential outcomes is unknown. In addition, Tables (ref) and (ref) provide additional results that explore this issue further and provide a way to test exchangeability over different dimensions.
The estimates from Equation (ref) are shown in the third panel, “Full”, of Table (ref). These estimates suggest a positive direct effect of the treatment of 16.4 percentage points, significant at the 5 percent level, with almost equally large spillover effects on the untreated units. More precisely, the estimated effect on an untreated kid of having one treated sibling is 14.6 percentage points, while the effect of having two treated siblings is 14 percentage points. The hypothesis that $\theta_0(1)=\theta_0(2)$ cannot be rejected, which suggests some form of crowding-out: given that one sibling is treated, treating one more sibling does not affect attendance. These facts are consistent with the idea that, for example, the conditional cash transfer alleviates some financial constraint that was preventing the parents from sending their children to school regularly, or with the program increasing awareness on the importance of school attendance, since in these cases the effect occurs as soon as one kid in the household is treated, and does not amplify with more treated children.
On the other hand, spillover effects on treated children are much smaller in magnitude and negative. The fact that these estimates are negative does not mean that the program hurts treated children, but that treating more siblings reduces the benefits of the program. For example, the effect of being treated with two treated siblings, compared to nobody treated, can be estimated by $\hat{\tau}+\hat{\theta}_1(2)\approx 0.113$. Thus, a treated kid with two treated siblings increases her attendance in 11 percentage points starting from a baseline in which nobody in the household is treated.
In all, the estimates suggest large and positive direct and spillover effects on the untreated, with some evidence of crowding-out between treated siblings.\footnote{These empirical findings differ from those in Barrera-Osorio-etal_2011_AEJ, who find evidence of negative spillover effects. Their results are calculated over a different sample, since the authors focus on households with two registered children, whereas I consider households with three registered children. Differences in estimated direct and spillover effects may be due to heterogeneous effects across household sizes. For instance, in this sample, households with more registered children have lower income on average, so they may benefit differently from the program.}
\afterpage{
}
Using the results from the nonparametric specification as a benchmark, I now estimate the effects of the program using the difference in means analyzed in Section (ref). The left panel, “Diff. Means”, of Table (ref) shows the difference in means, calculated as the OLS estimator for $\beta_{\mathsf{D}}$ in Equation (ref). The results show that the difference in means is practically zero and not significant. Hence, by ignoring the presence of spillover effects, a researcher estimating the effect of the program in this way would conclude that the treatment has no effect.
This finding is due to the fact that the difference in means combines all the effects in the third panel into a single number, as shown in Theorem (ref). From Table (ref), the estimated spillover effects in this case are larger under control that under treatment, and have different signs, so $\hat{\theta}_1(s)-\hat{\theta}_0(s)<0$. Therefore, the spillover effects push the difference in means towards zero in this case.
Next, I estimate the effects using the RF-LIM regression analyzed in Section (ref). The estimates from Equation (ref) are given in the first column of the middle panel, “Linear-in-Means”, in Table (ref). The estimates reveal very small and statistically insignificant direct and spillover effects, substantially different from the results using Equation (ref).
Theorem (ref) and Corollary (ref) show that a RF-LIM regression implicitly imposes linearity of spillover effects and may suffer from misspecification when spillover effects are nonlinear. The estimates from the full nonparametric specification show that spillover effects are highly nonlinear in this case, which explains why the RF-LIM regression fails to recover these effects.
The second column in the RF-LIM panel presents the estimates from the interacted RF-LIM regression shown in Equation (ref). The results reveal that separately estimating the spillover effects for treated and controls mitigates the misspecification in this case, and the estimates are closer to the ones from the nonparametric specification, although the fact that 0.169 is not a weighted average of 0.146 and 0.14 suggests that some extrapolation bias remains due to the nonlinearity of spillover effects.
I now illustrate how to estimate pooled effects by averaging over the possible number of treated siblings (2 and 3 in this case). For this, I estimate the following regression: \[Y_{ig}=\alpha_p+\tau D_{ig}+\gamma_p^0\mathbbm{1}(S_{ig}>0)(1-D_{ig})+\gamma_p^1\mathbbm{1}(S_{ig}>0)D_{ig}+\nu_{ig}\] where \[\tau=\mathbb{E}[Y_{ig}|D_{ig}=1,S_{ig}=0]-\mathbb{E}[Y_{ig}|D_{ig}=0,S_{ig}=0]\] and \[\gamma_p^d=\sum_{s=1}^2\theta_d(s)\mathbb{P}[S_{ig}=s|D_{ig}=d,S_{ig}>0]\] where $\theta_d(s)$ is defined in Equation (ref) (see also Remark (ref)). From Table (ref) we can see that the estimated pooled spillover effects are $0.144$ for controls and $-0.045$ for treated, which, as expected, lie between the effects found with the saturated regression. These results illustrate how this type of pooling can provide a useful summary of spillover effects, which may be a feasible alternative when the total number of spillover effects is too large or cell sizes are small to estimate them separately.
Next, I illustrate how to relax the exchangeability assumption in two ways. First, I define an ordering between siblings by looking at differences (in absolute value) in ages, defining sibling 1 as the sibling closest in age and sibling 2 as the sibling farthest in age. Then, estimation is conducted by simply adding indicator variables for the possible different assignments. Table (ref) shows the estimates from this specification. The estimates reveal similar results to Table (ref), with a direct effect of 0.165, spillover effects on the untreated ranging from 0.133 to 0.14 and spillover effects on the treated ranging from -0.039 to -0.051.
In fact, exchangeability can be tested by assessing whether the spillover effects of siblings 1 and 2 are the same, which in this case amounts to testing equality of coefficients between rows 2 and 3, and between rows 5 and 6. The test statistic and corresponding p-value from this test are given in the last two rows of the table, where it is clear that exchangeability cannot be rejected in this case, although this could be due to low statistical power given the relatively small sample size.
Finally, I consider the case in which the effect of treated siblings depends on whether siblings are male or female, allowing for non-exchangeable siblings based on gender. The results are shown in Table (ref). In this table, $S_{ig}^m$ denotes the number of male treated siblings and $S_{ig}^f$ denotes the number of female treated siblings. The results are qualitatively similar, with some suggestive evidence of slightly larger spillover effects from female siblings. The hypothesis that the coefficients are the same cannot be rejected, although again the sample may be too small to draw precise conclusions about sibling exchangeability.
The findings in this paper offer several takeaways for analyzing spillover effects in randomized experiments. First, commonly-analyzed estimands such as the difference in means and RF-LIM coefficients implicitly impose strong assumptions on the structure of spillover effects, and are therefore not recommended as they generally do not have a causal interpretation. On the other hand, the full vector of spillover effects is identifiable whenever the experimental design generates enough variation in the number of treated units in each group and the researcher assumes a treatment rule that is flexible enough.
Second, while nonparametric estimation of all direct and spillover effects can give a complete picture of the effects of the treatment, it can be difficult to implement in practice when groups are large. As a guideline to determine in which cases spillover effects can be estimated nonparametrically, Theorem (ref) formalizes the notion of a “sufficiently large sample” in this context, and provides a way to assess the performance of the different types of treatment effect estimators depending on the number of groups, number of parameters of interest and treatment assignment mechanism. As an alternative, pooled estimands can recover weighted averages of spillover effects with known weights for which inference can be conducted under standard conditions.
The supplemental appendix discusses several important issues that can be further developed in future research. Sections (ref) and (ref) discuss extensions to unequal group sizes and the inclusion of covariates. The results in Section (ref) and the simulations in Section (ref) highlight the fact that the rate of convergence of the proposed estimators depend on the experimental design. This suggests that these results can be used to rank treatment assignment mechanisms, and this has implications for experimental design, as discussed in Section (ref).
The analysis in this paper leaves several open questions to be explored. One example is allowing for endogenous group formation. The identification results in this paper follow through when groups are endogenously formed, as long as they are formed before the treatment is assigned and their structure is not changed by the treatment (inference may require further assumptions to account for possible overlap between groups). On the other hand, when the structure of the group is affected by the treatment, the treatment can affect outcomes through direct effects, through spillover effects given the network, and through changing the network structure. In such cases, while it is possible to identify the “overall” effect Kline-Tamer_2019, random assignment of the treatment is generally not enough to separately point identify these different effects, and further assumptions are needed. On the other hand, the findings in this paper can be generalized to settings where the researcher does not have precise control on treatment take-up. In this direction, Vazquez-Bare_2020_IV analyzes instrumental variable methods that can be applied to RCTs with imperfect compliance, where treatment receipt is endogenous, or more generally in quasi-experimental settings.