Inference with a single treated cluster

Inference with a single treated cluster

\address{Department of Economics, University of Michigan, 611 Tappan Ave, Ann Arbor, MI 48109, USA. Tel.: +1 (734) 764-2355. Fax: +1 (734) 764-2769} \email{[email removed]} \urladdr{\href{https://umich.edu/ hagem}{umich.edu/ hagem}}

Introduction

Inference about the average effect of a binary treatment or policy intervention is often much more challenging than its estimation. For example, calculating a difference-in-differences estimate can be as simple as comparing the difference in average outcomes of individuals in a group before and after an intervention to the same differences in unaffected groups. The main challenge for inference is that individuals within each of these groups likely depend on one another in unobservable ways. Taking this dependence into account generally requires knowledge of an explicit ordering of the dependence structure within each group. While time-dependent data have a natural ordering, it may be difficult or impossible to credibly order cross-sectionally dependent data within states or villages. Researchers commonly try to sidestep this problem by splitting large groups into smaller clusters that are presumed to be independent in order to have access to standard inferential procedures based on cluster-robust standard errors or the bootstrap. Splitting states, villages, or other large groups into smaller clusters is often difficult to justify but necessary for most of the available inferential procedures because they achieve consistency by requiring the number of clusters to go to infinity. If a procedure is valid with a fixed number of clusters, it typically requires at least two treated clusters unless strong homogeneity conditions are satisfied. Numerical evidence by bertrandetal2004, mackinnonwebb2014, and others suggests that ignoring dependence and heterogeneity may lead to heavily distorted inference in empirically relevant situations. In both cases, the actual size of the test can exceed its nominal level by several orders of magnitude, i.e., nonexistent effects are far too likely to show up as highly significant.

In this paper, I introduce an asymptotically valid method for inference with a single treated cluster that allows for heterogeneity of unknown form. The number of observations within each cluster is presumed to be large but the total number of clusters is fixed. The method, which I refer to as a rearrangement test, applies to standard difference-in-differences estimation and other settings where treatment occurs in a single cluster and the treatment effect is identified by between-cluster comparisons. The key theoretical insight for the rearrangement test is that a mild restriction on some but not all of the heterogeneity in two samples of independent normal variables allows testing the equality of their means even if one sample consists of only a single observation. I prove that this is possible for empirically relevant levels of significance if the other sample consists of at least ten observations. The rearrangement test compares the data to a reordered version of itself after attaching a special weight to the sample with a single observation. The weights needed for most standard situations are tabulated in the paper and calculating additional weights is computationally simple. I also show that the weights remain approximately valid if the two samples of independent heterogeneous normal variables arise as a distributional limit. I exploit this result in the context of cluster-robust inference by constructing asymptotically normal cluster-level statistics to which the rearrangement test can be applied. The resulting test is consistent against all fixed alternatives to the null, powerful against $1/\sqrt{n}$ local alternatives, and does not require simulation or resampling.

Inference based on cluster-level estimates goes back at least to famamacbeth1973. Their approach is generalized and formally justified by ibragimovmueller2010, ibragimovmueller2016, who construct $t$ statistics from cluster-level estimates and show that these statistics can be compared to Student $t$ critical values. canayetal2014 obtain null distributions by permuting the signs of cluster-level statistics under symmetry assumptions. hagemann2019b permutes cluster-level statistics directly but adjusts inference to control for the potential lack of exchangeability. All of these methods allow for a fixed number of large and heterogeneous clusters but require several treated clusters. The rearrangement test complements these methods because it relies on the same type of high-level condition on the cluster-level statistics but is valid with a single treated cluster. Other methods that are valid with a fixed number of clusters are the tests of besteretal2014 and a cluster-robust version of the wild bootstrap cameronetal2008, djogbenouetal2019 analyzed by canayetal2018. However, these papers rely on strong homogeneity conditions across clusters that are not needed here.

Several approaches for inference have been developed specifically for difference-in-differences estimation. conleytaber2011 provide a method that is valid with a single treated cluster and infinitely many control clusters under strong independence and homogeneity conditions that justify an exchangeability argument. fermanpinto2019 extend this approach to situations where the form of heteroskedasticity is known exactly. Another extension by ferman2020 allows for spatial correlation while maintaining conleytaber2011's exchangeability condition. The rearrangement test differs from these methods because it is not limited to models estimated by difference in differences, does not rely on exchangeability conditions, and allows for completely unknown forms of heterogeneity. Other approaches due to mackinnonwebb2019b, mackinnonwebb2019a use randomization (permutation) inference for difference-in-differences estimation and other models with few treated clusters. They test “sharp” fisher1935 nulls under randomization hypotheses and asymptotics where the number of clusters is eventually infinite. In contrast, the present paper is able to test conventional nulls in a setting with finitely many clusters.

The remainder of the paper is organized as follows: Section (ref) proves several new results on normal random vectors with independent, heterogeneous entries after a specific transformation and introduces the rearrangement test. Section (ref) establishes the asymptotic validity of the test in the presence of finitely many heterogeneous clusters when only one cluster received treatment and discusses several examples. Section (ref) illustrates the finite sample behavior of the new test in simulations and in data used by garthwaiteetal2014, who analyze the effects of a large-scale disruption of public health insurance in Tennessee. Section (ref) concludes. The appendix contains auxiliary results and proofs.

I will use the following notation. $1\{A\}$ is an indicator function that equals one if $A$ is true and equals zero otherwise. Limits are as $n\to\infty$ unless noted otherwise and $\leadsto$ denotes convergence in distribution.

Inference with heterogenous normal variables

In this section, I construct a test for the equality of means of two samples of independent heterogeneous normal variables where one sample consists of only a single observation. The other sample has finitely many observations. I show that the test has power while controlling size (Theorem (ref)) and remains approximately valid if this two-sample problem characterizes the large sample distribution of a random vector of interest (Proposition (ref)).

Consider $q$ independent variables $X_{0,1},\dots, X_{0,q}$ with $X_{0,k}\sim N(\mu_0,\sigma_k^2)$ for $1\leqslant k\leqslant q$. Independently, there is an additional variable $X_1 \sim N(\mu_1, \sigma^2)$. I interpret this as a two-sample problem with “control” sample $X_{0,1},\dots, X_{0,q}$ and “treatment” sample $X_1$, although all of the following still applies if these roles are reversed. The objective is to test the null hypothesis of equality of means,

without knowledge of $\mu_0, \sigma, \sigma_1,\dots,\sigma_q$ and without assuming that these quantities can be consistently estimated. I account for the uncertainty about $\mu_0$ by recentering the data $X = (X_1, X_{0,1},\dots, X_{0,q})$ with $\bar{X}_0 = q^{-1}\sum_{k=1}^q X_{0,k}$ to define

for some known weight $w\in(0,1)$ that will be chosen shortly. If $X_1-\bar{X}_0 >0$, the $1+w$ increases $X_1-\bar{X}_0$ and $1-w$ decreases $X_1-\bar{X}_0$. If $X_1-\bar{X}_0 < 0$, these effects are reversed. The idea underlying the test is that if the decreased version of $X_1-\bar{X}_0$ is still large in comparison to $X_{0,1} - \bar{X}_0,\dots, X_{0,q} - \bar{X}_0$, then this size difference is unlikely to be only due to heterogeneity in $\sigma^2, \sigma_1^2,\dots,\sigma_q^2$ but provides evidence that $\mu_1$ and $\mu_0$ are in fact not equal. I show below that $w$ gives precise probabilistic control over this comparison. In particular, choosing $w$ appropriately allows me to construct a test whose size can be bounded at a predetermined significance level.

Before defining the test statistic, I first introduce some notation. For a given vector $s\in\mathbb{R}^d$, let $s_{(1)}\leqslant\cdots\leqslant s_{(d)}$ be the ordered entries of $s$. Denote by $s\mapsto s^\triangledown = (s_{(d)},\dots, s_{(1)})$ the operation of rearranging the components of $s$ from largest to smallest. The test uses $S(X,w)$ and its rearranged version $S(X,w)^\triangledown$ in the difference-of-means statistic

to define the test function

The test, which I refer to as rearrangement test, rejects if $\varphi (X,w) = 1$ and does not reject otherwise. As stated, the test is against the alternative of a positive treatment effect, $H_1\colon \mu_1 > \mu_0$. For a test against $H_1\colon \mu_1 < \mu_0$, simply use $\varphi (-X, w)$. These alternatives can be combined to provide a two-sided test. I describe the exact implementation below equation (ref) ahead. Also note that the first difference of means in (ref) simplifies to $T(S(X,w))=X_1-\bar{X}_0$ but $T(S(X,w)^\triangledown)$ is in general a complicated function of $w$.

Intuitively, the rearrangement test can be interpreted as a permutation test that treats $S = S(X,w)$ as if it were the data and uses the second largest permutation statistic of $T(S)$ as critical value $c$. If $T(S) > c$, then the only possibility left is that $T(S)$ equals its largest permutation statistic. For the difference of means $T(S)$, that statistic must be $T(S^\triangledown)$ and therefore $T(S) > c$ is equivalent to $\varphi (X,w) = 1$. Because $S$ is being permuted and not $X$, this also explains why it is sensible to write $T(S(X,w))$ instead of $X_1-\bar{X}_0$ in the definition of the test function (ref). A classical permutation test would then use an exchangeability condition on $S$ to determine the size of the test. Even though the $S$ constructed here is far from exchangeable, I will show that this test has power while controlling size at a predetermined level. Instead of relying on exchangeability, the results here depend on the joint normality of $X$ combined with the location and scale invariance property $\varphi (X,w) = \varphi ((X-\mu_0 1_{q+1})/\sigma,w)$, where $1_{q+1}$ is a ($q+1$)-vector of ones. The location invariance is forced by the recentering of $X$ with $\bar{X}_0$ and effectively removes $\mu_0$ from the list of nuisance quantities. The scale invariance is ensured by the specific choices of $T$ and $\varphi$. It reduces the dimensionless unknowns $\sigma, \sigma_1,\dots,\sigma_q$ to the more tractable ratios $\sigma_1/\sigma,\dots,\sigma_q/\sigma$.

I start with the analysis of size and power, and connect these results with the situation where $X = (X_1, X_{0,1},\dots, X_{0,q})$ is an asymptotic approximation later on. I assume that the variances $\sigma_k^2$ of the $X_{0,k}$, $1\leqslant k\leqslant q$, are bounded away from zero by some $\text{\b{$\sigma$}}^2>0$ for all but one $k$. This avoids a trivial and in practice easily recognizable situation where some of the $X_{0,k}$ are exactly equal. I also restrict the variance $\sigma^2$ of $X_1$ to be bounded above by some $\bar{\sigma}^2<\infty$ because letting $\sigma\to\infty$ in $\varphi(X,w)$ would have the same effect as setting all $\sigma_k^2$ equal to zero. Under the null hypothesis, the distribution of $\varphi (X,w)$ is then determined by the unknown value of \[\lambda \in \Lambda \coloneqq \{ (\mu_0, \sigma, \sigma_1,\dots, \sigma_q) \in\mathbb{R}\times (0,\infty)^{q+1} : \sigma \leqslant \bar{\sigma} \text{ and } \sigma_k \geqslant \text{\b{$\sigma$}} \text{ for all $k$ but one} \}.\] Under the alternative, the distribution of $\varphi (X,w)$ also depends on the treatment effect $\delta = \mu_1 - \mu_0$. I write ${\mathord \mathrm{E}}_{\lambda, \delta}$ and ${\mathord P}_{\lambda, \delta}$ to emphasize this dependence but occasionally drop subscripts to prevent clutter.

My strategy is to first bound the null rejection probability ${\mathord \mathrm{E}}_{\lambda, 0} \varphi (X,w)$ uniformly in $\lambda\in\Lambda$ by a smooth function of the weight $w$. I can then find a $w$ to make the bound exactly equal to the desired significance level to guarantee size control. The bound is also a function of the number of control observations $q$ and the maximal relative heterogeneity $\varrho = \bar{\sigma}/\text{\b{$\sigma$}}$ of treated and untreated observations. The parameter $\varrho$ is user chosen and has a simple interpretation: it restricts how much more variable $X_1$ can be relative to the $X_{0,k}$ when one of the $\sigma_k$ equals zero and the remaining $\sigma_k$ are all equal to the lower limit $\text{\b{$\sigma$}}$. This is the worst-case scenario for the test because $X_1$ is then likely to be very large on accident in comparison to the $X_{0,k}$. In that scenario, a $\varrho$ of 5 simply means that the variance of $X_1$ can be up to $5^2 = 25$ times larger than the variances of all but one of the $X_{0,k}$ and “infinitely more variable” than the remaining $X_{0,k}$. There are no restrictions on how much less variable $X_1$ can be than $X_{0,1},\dots,X_{0,q}$ and, in particular, $\bar{\sigma}/\text{\b{$\sigma$}}$ can be less than one.

The following theorem is the main theoretical result of the paper. It establishes the existence of a size bound that is valid for a fixed number of control observations $q$ and fully accounts for the uncertainty about the parameters in $\Lambda$. The theorem also shows that the test has power against the alternative $H_1\colon \mu_1 > \mu_0$. Results in the other direction follow by considering ${\mathord \mathrm{E}}_{\lambda,-\delta}\varphi(-X, w)$ instead of ${\mathord \mathrm{E}}_{\lambda,\delta}\varphi(X, w)$. The discussion immediately below focuses on the implications of the theorem. I address some of its technical aspects towards the end of this section. Let $\Phi$ and $\phi$ denote the normal distribution and density functions, respectively.

The theorem implies that the rearrangement test controls size, i.e., \[ \sup_{\lambda \in \Lambda} {\mathord \mathrm{E}}_{\lambda, 0} \varphi (X, w) \leqslant \alpha, \] whenever $q$, $w$, and $\varrho$ are such that $\xi_q(w,\varrho) \leqslant \alpha$ for the desired significance level $\alpha$. The bound $\xi_q(w,\varrho)$ has several properties that make this possible. In particular, it is monotonically increasing in $\varrho$ and decreasing in $q$. The reason for the monotonicity is that if $X_1$ can be more variable than $X_{0,1},\dots,X_{0,q}$, then the burden of proof to show “$\mu_1 > \mu_0$” as opposed to “$\mu_1 = \mu_0$ with a large realization of $X_1$” becomes necessarily higher. A large $q$ can ameliorate this effect somewhat because it removes uncertainty about $\mu_0$. The bound also tends to be decreasing in $w \in [0,1]$ because the integral generally dominates the other components, but can increase slightly in some situations.

This is illustrated in Figure (ref), where $w\mapsto \xi_q(w,\varrho)$ (solid lines) is essentially decreasing over the entire domain except for $\varrho=2$ and $w\geqslant .85$. Most importantly, it can be seen that $w\mapsto \xi_q(w,\varrho)$ decreases enough to dip below the desired significance level $\alpha = .05$ (dashed line) for all values of $\varrho$. As $q$ increases (not shown), $w\mapsto \xi_q(w,\varrho)$ is pushed towards zero but the shape of the function does not change meaningfully with $q$. The $w$ at which $\xi_q(w,\varrho)=\alpha$ is generally unique for most empirically relevant $\alpha$ and does not exist in some extreme situations. This can be seen in Figure (ref), where $w\mapsto \xi_q(w,\varrho)$ crosses $\alpha = .05$ only once for each $\varrho$ but, for example, $\xi_q(w,\varrho) = .6$ is never attained.

Theorem (ref) also provides information about the interplay between $w$ and the test under the alternative. In particular, it shows that the rearrangement test has power against $H_1 : \mu_1 > \mu_0$ for every $w\in (0,1)$ but the power declines sharply at $w=1$. I therefore explore the behavior of the test with $w$ near $1$ further in the following result. It provides a lower bound on the power of the test for fixed $\delta$.

The bound shows that the test exhibits a standard relationship between the signal $\delta$ and the noise components $\sigma_1,\dots,\sigma_q$. Power is low if the signal relative to $\sigma$ is weak or the noise in the control group relative to $\sigma$ is strong. The latter relationship is in contrast to Theorem (ref), where small $\sigma_k$ relative to $\sigma$ were problematic. In addition, the bound also clarifies that $w$ dampens $\delta$ through the function $w\mapsto (1+w)/(1-w)$, which is arbitrarily large for $w$ sufficiently close to $1$. A $w$ very close to $1$ can therefore drown out a large treatment effect even if the noise coming from the control observations is mild. (The role of the supremum is simply to find the best possible balance for a given set of parameters.) It is also worth noting that the bound is tight enough to converge to $1$ as $\delta \to \infty$ and to $0$ as $w \to 1$.

Because the $w$ that satisfies $\xi_q(w,\varrho) = \alpha$ is not necessarily unique and because Proposition (ref) suggests that power against the alternative $H_1 : \mu_1 > \mu_0$ for $w$ near one can be low, it is sensible to choose the smallest feasible $w$, denoted by

in the definition of the rearrangement test function for a test of size $\alpha$,

The test $\varphi_\alpha$ also depends on $\varrho$ but this is suppressed here to prevent clutter. Table (ref) lists values of $w_q(\alpha,\varrho)$ for common choices of $\alpha$ as a function of $\varrho$ and $q$. They guarantee

The list is not exhaustive and additional values can be easily calculated by numerical integration. An R command that performs the calculations can be found at https://hgmn.github.io/rea.

Table (ref) shows that the rearrangement test is available in a wide variety of situations depending on the desired significance level and tolerance for heterogeneity. For instance, a test with a 10% significance level is already available with $q=10$ control observations. A 5% level test becomes available at $q=15$, a 1% level test at $q=20$, and for $q\geqslant 25$ there are essentially no restrictions to the level and underlying heterogeneity. This provides two avenues for implementation:

The first option is similar in spirit to the ubiquitous staigerstock1997 rule of thumb for weak instruments, where an $F$ statistic larger than 10 corresponds to a tolerance for an at most 10% bias (as defined in stockyogo2001) in the instrumental variables estimator relative to least squares. The second option takes the form of a “robustness check.” It has a meaningful interpretation because a result that is robust to a tenfold larger standard deviation in the treated observation relative to the control sample is more credible than a result that only survives a twofold difference in standard deviation. This second option leaves it up to the reader to decide whether the results are convincing.

The test decision itself is simple. Choose $w = w_q(\alpha,\varrho)$ from Table (ref) for a given number of control observations $q$, desired significance level $\alpha$, and maximal tolerance for heterogeneity, e.g., $\varrho = 2$. For this $w$, compute $S = S(X, w)$ as in (ref) and reorder the entries of $S$ from largest to smallest to obtain $S^\triangledown$. For an $\alpha$-level test of $\mu_1 = \mu_0$, reject in favor of $\mu_1 > \mu_0$ if $T(S) = T(S^\triangledown)$ as defined in (ref). For a one-sided test with level $\alpha$ against $\mu_1 < \mu_0$, reject if $T(-S) = T((-S)^\triangledown)$. For a two-sided test with level $2\alpha$, reject in favor of $\mu_1 \neq \mu_0$ if either $T(S) = T(S^\triangledown)$ or $T(-S) = T((-S)^\triangledown)$. The “robustness check” increases $\varrho$ until the null hypothesis can no longer be rejected against the desired alternative. The test decision is monotonic in $\varrho$, i.e., if $\varrho' > \varrho$ lead to the same test decision, then the decision does not change for any value between $\varrho$ and $\varrho'$. An R command that implements the test and the robustness check for any choice of $\varrho$ is available at \href{https://hgmn.github.io/rea}{https://hgmn.github.io/rea}.

I now turn to a discussion of some technical aspects of the size bound $\xi_q(w,\varrho)$ that forms the theoretical underpinning for the rearrangement test. The bound, defined in (ref), has three components with simple interpretations: The $1/2^{q+1}$ removes an unlikely event ($X_1 < \mu_0$, $X_{0,1} < \mu_0, \dots, X_{0,q} < \mu_0$ at the same time) from consideration. This forces a monotonicity property over the complement of this event and allows tightly bounding an oracle version of the problem where $\mu_0$ replaces $\bar{X}_0$ in (ref). This bound is the integral in (ref). The minimization problem then adjusts for the fact that the data are centered by $\bar{X}_0$ instead of the unknown $\mu_0$. The minimizer does not have closed form but is easily found numerically.\footnote{In particular, at $t = 1/q$, $\Phi(\sqrt{q-1} w t )^{q-1} + 2\Phi(- qt ) < \Phi(1/\sqrt{q})^{q-1} + 2\Phi(- 1) < 1$ for $q > 2$. Because $\Phi(\sqrt{q-1} w t )^{q-1} + 2\Phi(- qt ) \geqslant 1$ at $t\in \{0,\infty \}$, the minimization problem always has an interior solution. This also implies that the bound as a whole is a smooth function of $w$ and $\varrho$.} Taken together, $\xi_q(w_q(\alpha, \varrho),\varrho)$ can therefore be roughly viewed as a tight bound for a high-probability event plus two small adjustments. I use Table (ref) to illustrate the relative size of these adjustments. In the table, empty cells correspond to situations where there is either no $w$ such that $\xi_q(w, \varrho) = \alpha$ or more than $\alpha/2$ of $\xi_q(w_q(\alpha, \varrho),\varrho)$ is taken up by the non-tight parts of the bound. Cells in italics are settings where between $\alpha/2$ and $\alpha/10$ of the bound are taken up by the non-tight parts. The lack of tightness in the remaining cells is less than $\alpha/10$. For these cells $\sup_{\lambda \in \Lambda} {\mathord \mathrm{E}}_{\lambda,0} \varphi_\alpha (X)$ approximately equals $\alpha$. As the table shows, $\xi_q(w_q(\alpha, \varrho),\varrho)$ is an essentially tight bound for $\sup_{\lambda \in \Lambda} {\mathord \mathrm{E}}_{\lambda,0} \varphi_\alpha (X)$ for $q\geqslant 30$. The bound is also nearly tight for values of $q$ as small as 15 as long as $\varrho$ is not too large. I return to a discussion of this aspect of the rearrangement test in Example (ref) (ahead), where I illustrate the size of the test numerically.

Finally, before concluding this section, I show that the rearrangement test remains approximately valid for random vectors $X_n$ converging in distribution to the random vector $X = (X_1, X_{0,1},\dots, X_{0,q})$ described in Theorem (ref). The reason is that ${\mathord \mathrm{E}} \varphi(X_n, w)$ and ${\mathord \mathrm{E}} \varphi(X, w)$ eventually coincide whenever $X$ has independent entries and a smoothly distributed first entry. The $X$ in Theorem (ref) easily satisfies these conditions, which makes $\varphi_\alpha(X_n)$ asymptotically an $\alpha$-level test.

I use Theorem (ref) and Proposition (ref) in the next section to construct a simple method for inference with a single treated cluster. Section (ref) shows how the rearrangement test performs in Monte Carlo experiments.

Inference with a single treated cluster

In this section, I use a single high-level condition to extend the rearrangement test introduced in the previous section to a test about a scalar parameter in research designs with a finite number of large, heterogeneous clusters where only a single cluster received treatment. I then outline how these results can be applied in empirical practice.

Suppose data from $q+1$ large clusters (e.g., states, industries, or villages observed over one or more time periods) are available. Data are dependent within clusters but independent across clusters. The exact form of dependence is unknown and not presumed to be estimable. An intervention took place during which one cluster received treatment and and $q$ clusters did not. The quantity of interest is a treatment effect or an object related to it that can be represented by a scalar parameter $\delta$. Because the entire cluster was treated, this parameter is only identified up to a location shift $\theta_0$ within the treated cluster and therefore only the left-hand side of \[ \theta_1 = \theta_0 + \delta \] can be identified from this cluster. If the treated cluster would have behaved similarly to the untreated clusters in the absence of an intervention, then $\theta_0$ can be identified from each untreated cluster. Pairwise comparison then identifies $\delta$.

The identification strategy outlined in the preceding paragraph is the idea behind differences in differences---arguably the most popular identification strategy in modern empirical research---and a variety of other models. The goal of this section is to use the rearrangement test to provide a generic method for testing the hypothesis \[H_0\colon \delta = 0, \] or, equivalently, $H_0\colon \theta_1 = \theta_0$. I achieve this by obtaining an estimate $\hat{\theta}_{1}$ of $\theta_1$ and estimates \smash{$\hat{\theta}_{0,1},\dots, \hat{\theta}_{0,q}$} of $\theta_0$ so that \[ \hat{\theta}_n = (\hat{\theta}_{1}, \hat{\theta}_{0,1} \dots, \hat{\theta}_{0,q}) \] is approximately a vector of independent but potentially heterogeneous normal variables that can be used as if it were the data vector $X$ from Section (ref).

The following example explains how to construct $\hat{\theta}_n$ in a simple situation. I discuss construction of $\hat{\theta}_n$ for difference in differences towards the end of this section.

I will now show that the cluster-level statistics $\hat{\theta}_n$ can be used together with the results in the previous section to perform a consistent test as the sample size $n$ grows large. The test is not limited to parameters estimated by least squares. Instead, consistency relies on the condition that a centered and scaled version of some estimate $\hat{\theta}_n$ converges to a $(q+1)$-dimensional normal distribution,

where $\mathchoice {\raisebox{.0em}{ $\overset{\theta}{\leadsto}$ }} {\raisebox{-.15em}{ $\overset{\raisebox{-.25em}{\scriptsize$\theta$}}{\leadsto}$ }} {} {}$ denotes weak convergence under $\theta = (\theta_1,\theta_0)$. For fixed $\theta$, the display can be interpreted as $\sqrt{n}(\hat{\theta}_{1} - \theta_1, \dots,\hat{\theta}_{0,1} - \theta_0, \dots, \hat{\theta}_{0,q} - \theta_0)\leadsto N(0, \operatorname*{diag}(\sigma, \sigma_1, \dots, \sigma_q))$ to include the case that one of the $\sigma_1,\dots, \sigma_q$ may be zero as in Theorem (ref).

A key feature of condition (ref) is that the $\sigma$ and $\sigma_1,\dots, \sigma_q$ are not assumed to be known or estimable by the researcher. This is important for applications because consistent variance estimation generally requires knowledge of an explicit ordering of the dependence structure within each cluster. While time-dependent data are automatically ordered, it may be difficult or impossible to infer or credibly assume an ordering of the data within states or villages. In contrast, (ref) can be established under weak (short-range) dependence conditions that only require existence of a potentially unknown ordering for which the dependence of more distant units decays sufficiently fast. machkouriaetal2013 present convenient moment bounds and limit theorems for this situation. For more results in this direction, see also besteretal2014 and references therein. In general, the convergence in (ref) also implicitly requires the number of observations in all clusters to grow with the sample size $n$. However, the clusters are not required to have similar or even identical sizes. Another noteworthy feature of condition (ref) is the diagonal covariance matrix of the limiting distribution. It is the only independence condition that is imposed on the clusters.

I now show that under the joint convergence (ref), a rearrangement test that uses $\hat{\theta}_n$ is asymptotically of level $\alpha$ with a single treated cluster and a fixed number of control clusters. The test $\varphi_\alpha(\hat{\theta}_n)$, as defined in (ref), has power against all fixed alternatives $\theta_1 = \theta_0 + \delta$ with $\delta > 0$ and local alternatives $\theta_1 = \theta_0 + \delta/\sqrt{n}$ converging to the null. In the latter situation, $\theta_0$ is fixed and $\theta = (\theta_0 + \delta/\sqrt{n}, \theta_0)$ implicitly depends on $n$. The convergence in (ref) is then a statement about an entire sequence $(\theta_0 + \delta/\sqrt{n}, \theta_0)$ instead of a single point. Results for alternatives with $\delta < 0$ follow from the same result by considering $\varphi_\alpha(-\hat{\theta}_n)$. These tests can be combined into a two-sided test that has power against fixed and local alternatives from either direction. Algorithm (ref) at the end of this section shows how this can be implemented.

I now discuss how the high-level condition (ref) can be verified in an application. The specific example I use is difference-in-differences estimation but the arguments presented here apply more broadly. See also canayetal2014 and hagemann2019b for similar types of arguments in other models. For simplicity, I focus on (ref) under the null hypothesis $H_0 : \theta_1 = \theta_0$.

Before concluding this section, I present a brief summary of how the rearrangement test can be implemented in practice. By Theorem (ref), the following procedure provides an asymptotically $\alpha$-level test in the presence of a finite number of large clusters when only a single cluster received treatment. The test is computationally simple and does not require simulation or resampling, can be two-sided or one-sided in either direction, is able to detect all fixed alternatives, and is powerful against $1/\sqrt{n}$-local alternatives. Recall that $\varrho$ here measures how much more variable the estimate from the treated cluster $\smash{\hat{\theta}_1}$ can be relative to the second-least variable control cluster estimate $\hat{\theta}_{0,k}$. A $\varrho$ of $5$ means that the (asymptotic) variance of $\smash{\hat{\theta}_1}$ can be up to $5^2 = 25$ times larger. There is no restriction on how much less variable $\smash{\hat{\theta}_1}$ can be than any of the other estimates and $\smash{\hat{\theta}_1}$ can be infinitely more variable than the least variable control cluster. (See also the discussion above Theorem (ref).)

This test can also be used as a “robustness check” if inference was originally performed with a method designed for a finer level of clustering, e.g., at the county level instead of the state level. In that case Algorithm (ref) can illustrate how well the results of the original test hold up if there is dependence across counties. As I point out in Section (ref), one could start at $\varrho = 0$ or $\varrho = 1$ and increase $\varrho$ until the null hypothesis can no longer be rejected. This is informative because a result that holds up to a potentially $\varrho^2 = 25$ times larger variance is more credible than a result that only holds if $\varrho^2 = 1$, i.e., if $\hat{\theta}_1$ cannot be more variable than all but one $\hat{\theta}_{0,k}$. If the rearrangement test is used in difference-in-differences models in conjunction with the popular conleytaber2011 test, it is important to note that $\varrho^2 = 1$ still allows for substantial heterogeneity whereas the \citetalias{conleytaber2011} test presumes full homogeneity across clusters.

An R command that implements Algorithm (ref) and the robustness check for any choice of $\varrho$ is available at \href{https://hgmn.github.io/rea}{https://hgmn.github.io/rea}. The next section shows how the rearrangement test performs in simulations and an application.

Numerical results

This section explores the finite-sample behavior of the rearrangement test in two experiments. Example (ref) compares the rearrangement test to the widely used conleytaber2011 test in the two-way fixed effects model with clusters. Example (ref) applies the rearrangement test as a robustness check for the results of garthwaiteetal2014. The discussion focuses on one-sided tests to the right but the results apply more generally.

Conclusion

I introduce a generic method for inference about a scalar parameter in research designs with a finite number of large, heterogeneous clusters where only a single cluster received treatment. This situation is commonplace in difference-in-differences estimation but the test developed here applies more generally. I show that the test asymptotically controls size and has power in a setting where the number of observations within each cluster is large but the number of clusters is fixed. The test combines independent, approximately Gaussian parameter estimates from each cluster with a weighting scheme and a rearrangement procedure to obtain its critical values. The weights needed for most empirically relevant situations are tabulated in the paper. The critical values are computationally simple and do not require simulation or resampling. The test is highly robust to situations where some clusters are much more variable than others. Examples and an empirical application are provided.