Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
84,203 characters · 22 sections · 88 citation commands
Inference for Synthetic Controls via Refined Placebo Tests
Synthetic control methods were first introduced by Abadie and Gardeazabal abadie2003economic to analyze the effects of terrorism in the Basque Country on the economy of the region. Since then, the method has developed into a powerful and popular tool in comparative case studies and program evaluation with very few treated units abadie2010synthetic, cavallo2013catastrophic, billmeier2013assessing, abadie2015comparative, castillo2015tourism, gobillon2016regional, peri2019labor, donohue2019right, ben2021augmented, abadie2021using, sun2023using. We focus on applications of the synthetic control framework in panel data settings with one treated unit. Even in applications with multiple treated units, this setting is still relevant if the researcher wants to assess effect heterogeneity.
Suppose we observe data with $N$ units, $T$ time periods, where the $I$-th unit is treated from time $T_0+1,\dots,T$. The synthetic control (SC) method models the treated unit by a synthetic control, a convex combination of the other control units which resembles the treated unit as closely as possible according to pre-treatment outcomes and covariates. That is, we seek a vector of weights $W = (w_1,\dots,w_{I-1},w_{I+1},\dots,w_N)$ such that $w_{j} \geq 0$ for all $j$ and $\sum_{j \neq I}^{N} w_j = 1$, which corresponds to a weighted average of control units. The method chooses the weights $W$ by minimizing the norm
where $X_I \in \bR^p$ are covariates, or pre-treatment outcomes, for the treatment unit and $X_0 \in \bR^{p \times (N-1)}$ is a matrix whose rows are covariates for the control units. Finally, $V \in \bR^{p \times p}$ is a matrix which weights the different covariates, which may be either pre-specified or learned from the data abadie2011synth. Importantly, in many flagship applications of the method, there is one treated unit and only a small number of control units. Moreover, the outcomes are typically measured over a short time horizon. For example in abadie2003economic, $N = 17, T = 43, T_0 = 15$ and in abadie2010synthetic, $N = 39, T = 31, T_0 = 19$. See Appendix (ref) for a review of 31 papers that apply the synthetic control method, from which it is clear that applications to small datasets are very common.
By modelling the control counterfactuals for unit $I$, the SC method can be used to make inference on the sample average treatment effect on the treated. There are two popular approaches to inference for synthetic controls. The first imposes factor-model type structure on the outcomes, and does asymptotic inference. Many works in the literature, such as xu2017generalized, amjad2018robust, athey2021matrix, arkhangelsky2021synthetic, ben2021augmented, cattaneo2021prediction, ferman2021synthetic, cattaneo2022uncertainty, ben2022synthetic, imbens2023identification, shen2023same, sun2023using, chernozhukov2018t, li2020statistical, impose this assumption to study properties of the synthetic control estimator under the assumption that $N$ or $T$ is large. Most of these methods even require both the number of treated units and the number of post-treatment periods to be large. While various asymptotic approaches to inference exist, we focus on settings where small sample sizes do not justify the assumptions behind asymptotic inference. Furthermore, the asymptotic approaches often simplify the synthetic control procedures substantially to obtain provable guarantees. For example, they often do not consider a data-driven weight matrix $V$ that proves to be crucial in practice abadie2021using.
The second approach is the design-based framework, where the data are seen as fixed, and the treatment unit $I$ is assumed to be uniformly assigned from $1$ to $N$. The placebo test of abadie2003economic, abadie2010synthetic is an example; it was further extended in doudchenko2016balancing, firpo2018synthetic, bottmer2021design, shaikh2021randomization. The uniform assignment assumption holds by design in experimental settings doudchenko2019designing, jones2019synthetic, abadie2021synthetic. Even in observational settings, as argued by bottmer2021design, the assumption is “a natural starting point for many analyses, often after some adjustment for observed covariates. In particular, many of the analyses of SC (synthetic control) methods explicitly or implicitly refer to units being comparable". The assumption could be more tenable after adjusting for covariates, as bottmer2021design suggest. The assumption can also be justified in quasi-experimental settings arkhangelsky2021double, borusyak2020non, borusyak2024design. Examples such as dupont2015happened, coffman2012hurricane, where synthetic controls are used to analyze the impact of natural disasters, provide another setting for this assumption. The rate of natural disasters at a particular location can be estimated from past data, and is plausibly random over small geographic locations. In this paper, we consider a design-based framework for inference.
The standard inference procedure in the design-based framework is the placebo test. It was introduced in abadie2003economic, abadie2010synthetic as a diagnostic tool related to permutation inference and Fisher's randomization test. The procedure was formalized and expanded upon in firpo2018synthetic to make inference under the assumption that $I$ is uniformly assigned. It creates a $p$-value via the following recipe. For every unit $i$, create its synthetic control using all other units and obtain predictions $\hat{Y}_{it}$ for all $t > T_0$. Then measure the goodness-of-fit by comparing $\hat{Y}_{it}$ with the observed outcomes through any summary statistic $R_i$, such as the widely used RMSPE statistic defined in (ref). Finally, the exact placebo $p$-value is defined as
The standard argument for permutation tests shows that $p_{\text{{\tiny exact-placebo}}}$ is a valid $p$-value.
\paragraph{Small sample issues of the placebo test.} The $p$-value obtained from such tests are coarse: they take values in the grid \sloppy $\set{\frac{1}{N}, \frac{2}{N},\dots,1}.$ When $N$ is relatively small like in the premier applications of the synthetic control method abadie2003economic,abadie2010synthetic, abadie2015comparative or those examples in Appendix (ref), the smallest value $1/N$ of the $p$-value may give statistical significance, while the $2/N$ may switch this conclusion. This low level of granularity hinders the application of the placebo test and related inference methods in practice. In some applications, the value of $1/N$ may be too large to even reject at a pre-specified level $\alpha$. For example, when $\alpha = 0.05$, $\alpha < 1/N$ in both abadie2003economic and abadie2015comparative. Due to a desire to control size at traditional levels like $\alpha \in \set{0.01,0.05,0.10}$ and generally small datasets, $N\alpha < 3$ is typical for a variety of applications. This includes state/province/prefecture-level analysis abadie2010synthetic, castillo2015tourism, dupont2015happened, donohue2019right, hankins2020finally, ben2021augmented, ben2022synthetic, district-level analysis within a state sun2023using, and manufacturer-level analysis fremeth2013making. Even in country-level or county/city-level analysis where the donor pool appears to be large, the number of comparable control units may be small due to restrictions based on geographical proximity coffman2012hurricane, billmeier2013assessing, potential spillovers dupont2015happened, ando2015dreams, economic prosperity abadie2015comparative, market conditions peri2019labor,boes2012effect, and eligibility newiak2017evaluating. On the other hand, it is a common practice to rule out control units with poor pre-treatment fits for the placebo test abadie2010synthetic, almer2012effect, which can result in a fair drop in the sample size.
More importantly, the placebo test is powerless in small data settings where $\alpha < 1/N$. For each unit $i$ and time $t$, let $(Y_{it}(1), Y_{it}(0))$ denote the pair of potential outcomes had the unit $i$ been treated and not treated at time $t$, respectively. Further, let $\cl{Y} = \set{(Y_{it}(1),Y_{it}(0)): i \in [N], t \in [T]}$ denote the realizations of all potential outcomes and assume they take distinct values almost surely. Then given $I$ is uniformly assigned, $p_{\text{{\tiny exact-placebo}}}$ is distributed uniformly on the set $\set{1/N,2/N,\dots,1}$ under the sharp null $H_0: Y_{It}(1) = Y_{It}(0)$ for all $t$. Thus, the conditional Type-I error is \[ \Prob_{H_0}(p_{\text{{\tiny exact-placebo}}} \leq \alpha | \cl{Y}) = \frac{\floor{ N\alpha}}{N}. \] Averaging over $\mathcal{Y}$, the unconditional Type-I error is \[ \Prob_{H_0}(p_{\text{{\tiny exact-placebo}}} \leq \alpha) = \frac{\floor{ N\alpha}}{N}. \] As a result, when $\alpha < 1/N$, the placebo test has zero size. Proposition (ref) extends this argument to show that the placebo test has no power in this setting against a natural class of alternatives.
Can we develop a valid $p$-value which has power in these $\alpha < 1/N$ settings? One useful remedy is to define an approximate version of $p_{\text{{\tiny exact-placebo}}}$ by not comparing with the treated unit itself:
Clearly, its conditional and unconditional Type-I errors are both $(\floor{N\alpha} + 1) / N$, which is strictly larger than $\alpha$. The textbook solution to resolve this dilemma (e.g., romano2005testing) is to use a randomized $p$-value $p_{\text{{\tiny exact-placebo}}} - U/N$ where $U$ is an independent draw from a uniform distribution on $[0, 1]$. While this randomized $p$-value has an exact Type-I error control and allows to reject the null even when $\alpha < 1/N$, it offers a cheap way of $p$-hacking -- the researcher can simply choose a random seed that generates a large $U$ without revealing its value. Taking this strategic behavior to the extreme, the randomized $p$-value is no different from the approximate placebo $p$-value defined in (ref). Aside from the $p$-hacking incentive, the approximate $p$-value defined in (ref) rescaled by $N/(N-1)$ is sometimes used implicitly in the sense that significance is claimed when $p_{\text{{\tiny app-placebo}}} = 0$, i.e., the treated unit has the highest summary statistic.
For the aforementioned reasons, the approximate placebo is unsatisfactory. Is there any alternative we may use? Unfortunately, no conditionally finite-sample-valid non-randomized $p$-value has power here, as Proposition (ref) in Appendix (ref) shows. However, this does not imply unconditional impossibility. In applications of synthetic controls, it is reasonable to treat potential outcomes as random. Thus, it suffices to seek an unconditionally-valid p-value. One way to get useful inference is to reject if the $p$-value $p(I) \leq 1/N$, and hope that $\Prob_{H_0}(p(I) \leq 1/N | \cl{Y}) = 0$ for some $\cl{Y}$. Averaged over different datasets $\cl{Y}$, we may hope for an unconditionally valid $p$-value. For example, when $1/2N < \alpha < 1/N$, if $\Prob_{H_0}(p(I) \leq \alpha | \cl{Y})$ takes values $0$ and $1/N$ both with probability $1/2$, then the unconditional Type-I error $\Prob_{H_0}(p(I) \leq \alpha) = 1/2N < \alpha$. Neither the exact nor the approximate placebo test has this property because their conditional Type-I errors are both constant regardless of $\mathcal{Y}$. Thus, we cannot achieve this goal by modifying the standard permutation test.
We present the Leave-Two-Out (LTO) placebo test, which gives up conditional validity for a shot at unconditional validity. The LTO placebo test is a non-randomized approximate $p$-value which can be seen as a leave-two-out extension of the usual placebo test, with a user-specified statistic. The LTO placebo test has several advantages over the approximate placebo inference method.
In addition, we discuss a weighted variant of the LTO placebo test under non-uniform assignment probabilities, a sensitivity analysis for the procedure, other versions of LTO placebo tests, and leave-$r$-out extensions for $r > 2$.
In this section, we consider a semisynthetic simulation using the California proposition 99 example of abadie2010synthetic, one of the flagship applications of the synthetic control method. The dataset records cigarette sales, prices, and various other regional attributes in 39 US states, over the years 1970 to 2000. In abadie2010synthetic, the data are used in the synthetic control procedure to analyze the effect of Proposition 99, a 1988 California state law increasing cigarette taxes, on smoking. In this example, $N = 39, T = 30, T_0 = 19.$
We construct a semisynthetic version of the data, where California is removed from the dataset. A subsample of $N = 30$ units is randomly chosen from the remaining set of $38$ units. In the smaller dataset, a state is chosen uniformly at random to be the `treated' unit. For the treated unit $I$, we posit a uniform treatment effect $\tau$, and we update the observed outcomes $Y_{I,t}$ by adding $\tau$. A range of values of $\tau$ are tested, corresponding roughly to multiples of $(0,-1,-2,-3)$ of the standard deviation of the outcome variable (cigarette sales). When $\tau = 0$ we are calculating the Type-I error of the procedures. Further detail on the setup of this simulation can be found in Section (ref).
In the first setting, we take $\alpha = 0.02$ $(\alpha < 1/N)$. We run the approximate placebo test, its randomized version, and the LTO placebo test in order to compare their Type-I error and power. See Section (ref) for more detail. The result of this comparison are shown in Figure (ref). Several observations can be made regarding empirical performance. Firstly, the Type-I errors of the LTO $p$-values are on average smaller than $1/N$, and is below $\alpha$ in each simulation. This means that for a majority of the Monte Carlo resamples of the dataset, the empirical Type-I error was zero. Thus, the LTO procedure is unconditionally valid. On the other hand, the Type-I error of the approximate placebo test is always fixed. We do not present the exact placebo test because it is powerless. Simultaneously, the power of the LTO placebo test is better than that of the approximate placebo test when the effect size is large. When the effect size is intermediate, however, the approximate placebo test has higher power, but not by much. In Section (ref), we propose a more powerful test, which has nearly identical power as the approximate placebo test in this setting, by modifying the LTO p-value for any given $\alpha$; see Figure (ref). Lastly, the randomized placebo test controls Type-I error at the desired level, but has much smaller power than the other two methods. Further it is susceptible to adverse incentives to $p$-hack.
In the second setting we take $\alpha = 0.05 \,\, (\alpha \geq 1/N)$. We compare the exact placebo test, the randomized placebo test, the approximate placebo test, and the LTO placebo test; see the right side of Figure (ref). We can largely draw the same conclusions as with the $\alpha < 1/N$ setting. While the power gap between the LTO placebo test and approximate placebo test is larger for intermediate effect sizes, a more powerful LTO placebo test developed in Section (ref) reduces about half of the gap; see Figure (ref) for a full comparison. We again observe that the LTO procedure is unconditionally valid. In most Monte Carlo resamples, the Type-I error is in fact zero. Notably, the unconditional Type-I error of the LTO placebo test is even lower than that of the exact placebo test, which is below $\alpha$. Moreover, the power of the LTO placebo test is consistently larger than that of the exact placebo test. In this case, the randomized placebo test interpolates between the exact and approximate placebo tests and controls Type-I error at the desired level. See Section (ref) for a detailed description of the numerical experiment, and results for similar datasets.
\paragraph{Real Data Analysis.} We replicate the analysis of California Proposition 99's effect on smoking using both the placebo $p$-value and our LTO method. Here, $N = 39$ with one treated unit. To highlight a sensitivity analysis procedure we develop for the LTO $p$-value, we conduct inference with $\alpha = 0.05$. The LTO $p$-value obtained was $0.024$ while the exact placebo $p$-value was $0.026$, which exactly equals $1/N$. The approximate placebo $p$-value was $0$.
In this setting, we can also inspect how sensitive the conclusion of non-significance is with respect to the equal weights assumption. We apply a Rubin-Rosenbaum style sensitivity analysis procedure to this example, in order to assess the sensivity of inference to the equal weights assumption; see Section (ref) for more details. The sensitivity analysis relies on a weighted version of the LTO $p$-value, which is also an approximate $p$-value under non-uniform treatment probabilities $\pi_i$ for unit $i$. Fixing a $\Gamma,$ we constrain the non-uniform propensities $\pi_i$ to sum to $1$ and satisfy $\pi_i \in [\frac{1}{\Gamma N}, \frac{\Gamma}{N}]$ (similarly to rosenbaum1983assessing), and calculate the largest possible value of the weighted $p$-value for propensities in this constraint. Finally, we search for the smallest value of $\Gamma$ such that the conclusion of significance is overturned.
As the reported LTO $p$-value is $0.024$, we want to see how large $\Gamma$ can be before the conclusion of significance is possibly overturned. The output is shown in Figure (ref). From the figure, $\Gamma$ is around $1.4$ before the maximum possible $p$-value is greater than $0.05$. This suggests the conclusion of significance is robust under moderate confounding.
Consider a design-based setup similar to that of the placebo test. We observe a panel $Y_{it}$ of $N$ units, $T$ time periods, where the $I$th unit is treated from time $T_0+1,\dots,T$. In potential outcomes, the data is given as: \[
. \] We assume that the treatment index $I$ is uniformly assigned across the $N$ units. Synthetic control methods estimate the average treatment effect on the treated, $\tau_{I,t} = Y_{I,t}(1) - Y_{I,t}(0)$, by using the synthetic control $\hat{Y}_{I,t}$ to estimate the control counterfactual. Following the literature abadie2010synthetic, abadie2015comparative, firpo2018synthetic, the primary inference task of interest is to test the sharp null hypothesis
Extensions to sharp nulls of the form $H_0: Y_{i,t}(1) = Y_{i,t}(0) + \tau_{i,t}$ for known quantities $\tau_{i,t}, i =1,\dots,N,$ and $t \geq T_0 + 1$ can also be handled.
Our LTO placebo test procedure works as follows:
By approximate $p$-value, we mean that the LTO $p$-value satisfies the following Type-I error guarantee.
The proof of Theorem (ref) is presented in Appendix (ref). It is a (non-trivial) consequence of the following more general result. We sketch its proof at the end of the section.
Theorem (ref) guarantees that $p_{{\scriptscriptstyle\mathrm{naive-LTO}}}$ never has worse conditional Type I error than $p_{\text{{\tiny app-placebo}}}$, for a wide range of parameters. Specifically, by Theorem (ref) (b), when $\alpha = 0.05$, it implies that the LTO $p$-value has the same theoretical guarantee with the placebo $p$-value for any $N \le 79$. This covers a wide range of settings where the synthetic control method is applied; see Appendix (ref). By Theorem (ref) (c), we can easily check if the naive LTO placebo test has the same Type-I error guarantee as the approximate placebo test for any $N$ and $\alpha$. In particular, for all $6 < N < 200$, this equivalence holds for $\alpha \in \{0.01, 0.02\}$. For $\alpha = 0.05$, it holds except for $N \in \{80, 100, 120, 139, 140, 159, 160, 179, 180, 198, 199\}$. To further gain some insight for larger $\alpha$, we plot the Type-I error bound in Theorem (ref) for $N = 17$ in Figure (ref). The LTO p-value offers guarantees that are comparable to or slightly stronger than those of the approximate placebo test for all practical levels, i.e., when $\alpha \le 0.25$. Lastly, when $N$ is large, the bound is close to $\frac{3 - \sqrt{9 - 12\alpha}}{2}$, which is very close to $\alpha$ when $\alpha$ is small. For $\alpha = 0.05$, the right hand side is around $0.0508$. For $\alpha = 0.1,$ the bound comes out to $0.104.$
On the other hand, as suggested by our empirical results in Section (ref), the actual Type-I error of the LTO $p$-value can be strictly lower than the bound while the bound for the placebo $p$-value is tight. Combined with the improved granularity and improved power documented in Sections (ref) and (ref), $p_{{\scriptscriptstyle\mathrm{LTO}}}$ provides a compelling alternative to $p_{\text{{\tiny app-placebo}}}$ for randomization inference.
\paragraph{Proof Sketch of Theorem (ref).} The proof of the coverage guarantee relies on analyzing combinatorial structures related to tournaments. We present the proof here in order to introduce the main ideas of the analysis; it is an interesting extension of the proof of coverage in the Jackknife+ barber2021predictive. Let us recall the notation \[ \bb{I}(k \succ i,j) := \mathbf{1}\set{R_{i,j,k;k} > \max(R_{i,j,k;i}, R_{i,j,k;j})}. \] Call a unit $k$ strange if \[ \sum_{\substack{i,j \in[N] \setminus k \\ i \neq j}} \bb{I}(k \succ i,j) \geq (1-\alpha) (N-1)(N-2). \] Intuitively, the LTO $p$-value can be seen as the normalized rank of the unit $I$ in a tournament where matches are held between every triple of distinct units $i,j,k$. The winner of the match is the unit with the largest residual. We call a unit strange if it wins too many matches, quantified above. In such a tournament, not everyone can win too often; following this intuition, we upper bound the number of strange units. \\
Let $\cl{S}$ denote the set of strange units, and let $s := |\cl{S}|$. Summing the definition of strangeness on both sides, we see that \[ \sum_{k \in \cl{S}} \sum_{i,j \in [N], i\neq j} \bb{I}(k \succ i,j) \geq (1-\alpha) s (N-1)(N-2). \] The sum on the left hand side may be split into three terms:
where $i \neq j\neq k$ is shorthand notation for mutually distinct $i, j, k$. For the first sum, note by renaming the labels that $\sum_{i,j,k \in \cl{S}, i\neq j\neq k} \bb{I}(k \succ i,j) = \sum_{i,j,k \in \cl{S}, i\neq j\neq k} \bb{I}(i \succ j,k) = \sum_{i,j,k \in \cl{S}, i\neq j\neq k} \bb{I}(j \succ k,i).$ Thus \[ \text{(I)} \leq \sum_{i,j,k \in \cl{S}, i\neq j\neq k} \frac{1}{3}\left(\bb{I}(k \succ i,j) + \bb{I}(i \succ j,k) + \bb{I}(j \succ k,i) \right) \leq s(s-1)(s-2)/3. \] Crucially, we use the fact that there can be at most one winner in a triple comparison: $\bb{I}(k \succ i,j) + \bb{I}(i \succ j,k) + \bb{I}(j \succ k,i) \leq 1$.\footnote{We allow for zero winners in the triple comparison if there are ties in the residuals.} For the second sum, swap the naming of the $i,k$ labels and use the bound $\bb{I}(k \succ i,k) + \bb{I}(i \succ k,j) \leq 1$ to see that \[ \text{(II)} \leq 2\sum_{i,k \in \cl{S}, i\neq k, j \in \cl{S}^c} \frac{1}{2}\left(\bb{I}(k \succ i,k) + \bb{I}(i \succ k,j) \right) \leq s(s-1)(N - s). \] Finally, in the last sum, we use the naive bound $\text{(III)} \leq s (N-s)(N-1-s).$
Combining these bounds and working through algebra, we arrive at a quadratic inequality in the variable $\beta := s/N$: \[ \frac{\beta^2}{3} - \beta(1 - \frac{1}{N}) - \frac{4}{3N^2} + \frac{1}{N} + \alpha(1 - \frac{1}{N})(1- \frac{2}{N}) \geq 0. \] We can solve this by quadratic formula, yielding that, for any $\alpha < 2/3$,
To conclude, under uniform treatment we have $\Prob(p_{{\scriptscriptstyle\mathrm{naive-LTO}}} \leq \alpha) = \Prob(I \in \cl{S}) = \beta.$ The bound in equation (ref) shows that $p_{{\scriptscriptstyle\mathrm{naive-LTO}}}$ is an approximate $p$-value. A full proof with further computational details may be found in Appendix (ref).
Notice that in defining the notion of strange unit, we have great flexibility in choosing the threshold. Call a unit $k$ $c$-strange if \[ \sum_{\substack{i,j \in[N] \setminus k \\ i \neq j}} \bb{I}(k \succ i,j) \geq (1-\alpha - c) (N-1)(N-2). \] This results in a new $p$-value $p_{{\scriptscriptstyle\mathrm{naive-LTO}}} - c$. By Theorem (ref), \[\Prob(p_{{\scriptscriptstyle\mathrm{naive-LTO}}} - c \leq \alpha) = \Prob(p_{{\scriptscriptstyle\mathrm{naive-LTO}}}\leq \alpha + c)\leq \frac{\floor{N f(N,\alpha+c)}}{N}.\] As a result, for any $c > 0$ such that \[\frac{\floor{N f(N,\alpha+c)}}{N} = \frac{\floor{N f(N,\alpha)}}{N},\] the adjusted $p$-value $p_{{\scriptscriptstyle\mathrm{naive-LTO}}} - c$ has the same bound on the rejection probability as the naive LTO $p$-value $p_{{\scriptscriptstyle\mathrm{naive-LTO}}}$ but is strictly more powerful than the latter, since it is strictly smaller. This motivates the improved LTO placebo $p$-value.
The power improvement is enabled by the discrete Type-I error bound. Since the improved LTO placebo test has the same Type-I error bound as the naive one, Theorem (ref) applies to $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha)$. As a result, $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha)$ has the same theoretical guarantee as $p_{\text{{\tiny app-placebo}}}$ when $N\alpha < 4(1 - 2/(N-1)(N-2))$. Note that $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha)$ depends on $\alpha$ and hence, unlike $p_{{\scriptscriptstyle\mathrm{naive-LTO}}}$, cannot be viewed as a $p$-value. It can only be used for testing $H_0$ at level $\alpha$.
While the expression of $c(N, \alpha)$ is complicated, we can easily compute it by a binary search. For the Proposition 99 dataset, $N = 39$, $c(N, 0.02) = 0.006$ and $c(N, 0.05) = 0.002$. For the application in abadie2003economic, $N = 17$ and $c(N, 0.05) = 0.0125$.
\paragraph{Further power improvement by enforcing decision monotonicity.}
One undesirable feature of $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha)$ is that the procedure is not decision-monotonic in $\alpha.$ That is, if the test based on $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha)$ rejects at some level $\alpha$, the test using $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha')$ may not reject at an $\alpha'$ with $\alpha' > \alpha$. This is due to the step-like behavior of the Type-I error upper bound.
We can enforce decision monotonicity and also improve power of the procedure by considering instead \[ I_\alpha = \max_{\alpha' \leq \alpha} \mathbf{1}\set{p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha') \leq \alpha'}, \] rejecting at level $\alpha$ if $I_\alpha = 1$. Clearly this procedure is more powerful than rejecting only when $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha) \leq \alpha$. The procedure also controls Type-I error. To see this,
The last line may be written
where $g(\alpha) = \operatorname{argmax}_{\alpha' \leq \alpha} \alpha' + c(N, \alpha') - \delta.$ The latter is equal to $\Prob(p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(g(\alpha)) \leq g(\alpha)) \leq \frac{\floor{N f(N,g(\alpha))}}{N} \leq \frac{\floor{N f(N,\alpha)}}{N}$, which is the same Type-I error bound as using just $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha).$ We do not use this procedure in our simulations, leaving this method for theoretical interest.
The existing literature on synthetic controls is vast: see the survey article abadie2021using for a review. Accordingly, there has been much work addressing inference for synthetic controls. Among the first inferential procedures for synthetic controls was the placebo test, first described in abadie2003economic, and formalized by Firpo and Possebom firpo2018synthetic into a permutation test. They work in the same design-based framework where the probability of treatment is uniform. Several extensions are made, including to a non-equally weighted inference procedure which is useful for sensitivity analysis.
Other works have also considered the design-based setup and randomization or permutation-based inference. See bottmer2021design for a design-based analysis of the synthetic controls procedure under a similar uniform treatment assumption, and also a random treatment time assumption. shaikh2021randomization gives a related randomization inference procedure in a staggered adoption setting where there are multiple treated units, under the additional assumption that the time at which each unit adopts the treatment that follows a Cox proportional hazards model. In general, design-based inference proves to be a powerful tool for panel data analysis arkhangelsky2021double,rambachan2020design, abadie2020sampling, athey2022design, roth2023efficient,xu2023difference.
Inference procedures have also been developed assuming underlying factor model structure and related time series assumptions on the data. chernozhukov2021exact develop an inference procedure assuming a regularity condition on the residual process in time. The paper contains an extensive discussion on the various choices for creating the counterfactual or synthetic control. Another line of research cattaneo2021prediction, cattaneo2022uncertainty provides prediction intervals via non-asymptotic concentration bounds, under stationarity and weak dependence time series assumptions on the data. See also hahn2017synthetic, amjad2018robust, arkhangelsky2021synthetic, arkhangelsky2023large, li2020statistical, chernozhukov2018t for related asymptotic analyses of the synthetic-control type estimators under time series assumptions.
For many of these results, consistency of the estimator and the resulting inference procedure is shown in the limit as the number of controls and units goes to infinity. Although many of these works offer finite sample bounds, it is difficult to contextualize these results when the applications of synthetic controls are usually small $N$, small $T$ settings. Furthermore, many of these asymptotic results require simplifications on the synthetic control procedure to carry out analysis. The LTO placebo test can use other counterfactual models beyond the standard synthetic control method, and we require no conceptual simplifications to do inference. As a result, the LTO procedure may be used with generalizations of the original synthetic control procedure such as those outlined in doudchenko2016balancing, amjad2018robust, chernozhukov2021exact, arkhangelsky2021synthetic, gunsilius2023distributional. Furthermore, it can be applied with non-synthetic-control methods developed for inference with few clusters; see Section (ref) for further discussion.
Outside of experimental settings, the assumption of uniform treatment is questionable. To handle non-uniform treatment assignments, we first propose a weighted analog of the LTO $p$-value. Suppose that the probability of treating unit $k$ is given by $\pi_k, k =1,\dots,N$. In some applications, these probabilities can be estimated. For example, when assessing the effect of earthquakes cavallo2013catastrophic, dupont2015happened, the probability of earthquake in each region can be estimated via statistical seismology vere2005statistical. Then the following weighted approximate $p$-value generalizes the LTO $p$-value of Section (ref).
Note that $p_{{\scriptscriptstyle\mathrm{w-LTO}}}(\pi) = p_{{\scriptscriptstyle\mathrm{naive-LTO}}}$ when $\pi_i = 1/N$ for all $i$. We prove the following upper bound for the Type-I error of $p_{{\scriptscriptstyle\mathrm{w-LTO}}}(\pi)$ as the value of an integer programming problem. The proof is deferred to Appendix (ref).
When $\pi_i \equiv 1/N$, it can be shown via tedious algebra that Theorem (ref) recovers Theorem (ref). In the general non-uniform case, while the bound is given by a complicated nonconvex integer programming problem, it can be solved approximately by off-the-shelf solvers when $N$ is small, e.g., $N < 50$.
When assignment probabilities are not controlled by the researcher, it is useful to have procedures to analyze sensitivity of our approach to the uniform assignment assumption. The goal of sensitivity analysis is to answer the question: by how much would our statistical conclusions change if the treatment were non-uniform? Phrased another way, by how much would the treatment probabilities have to change to overturn a conclusion of statistical significance? Firpo and Possebom firpo2018synthetic give an approach to do sensitivity analysis for the exact placebo test under non-uniform treatments. We can extend their approach to the naive LTO placebo test using the weighted LTO placebo $p$-values as the building block. The procedure is as follows.
By Theorem (ref), it remains to further add the constraint (ref) into the integer programming problem. By carefully analyzing it, we end up with a much simpler result. The proof is presented in Appendix (ref).
When $\Gamma = 1$, Theorem (ref) recovers Theorem (ref). When $N$ is large, the bound has the same limit $\frac{3 - \sqrt{9-12\alpha}}{2}$ as obtained for the $\Gamma = 1$ setting. Unlike Theorem (ref), we cannot sharpen it by taking advantage of the discreteness of the Type-I error because $\pi_i$ can take a range of values.
\paragraph{Computational Details.} The optimization problem in step 2) may pose challenges. To optimize this problem, we use the following trick. Fixing a $\Gamma$, let $B_\Gamma$ be the set of $\pi$ satisfying the constraints in Equation (ref). We only need to check if $\max_{\pi \in B_\Gamma} p_{{\scriptscriptstyle\mathrm{w-LTO}}}(\pi)$ is greater than $\alpha$. This is equivalent to checking that \[ \max_{\pi \in B_\Gamma} \sum_{\substack{j,k \neq I \\ j \neq k}} \pi_j \pi_k\mathbb{I}\set{I \not\succ j,k} - \alpha\left((1-\pi_I)^2 - \sum_{l\neq I} \pi_l^2\right) > 0 \] The optimization problem can be written as a simple quadratic program. Namely, we must check whether \[ \max_{\pi \in B_\Gamma} \pi^\top A\pi + 2\alpha \pi^\top e_I > \alpha \] where $A := (G + \alpha I_N -2\alpha e_{I}e_I^\top)$, with $G \in \bR^{N \times N}$ is a zero-one matrix with entries $G_{jk} = \mathbb{I}\set{I \not\succ j,k},$ whenever $j$ and $k$ do not equal $I$ and zero otherwise. Here $e_I$ is the standard basis vector for index $I.$ If the goal is to calculate $\max_{\pi \in B_\Gamma} p_{{\scriptscriptstyle\mathrm{w-LTO}}}(\pi)$, we can do a binary search over $c \in [0,1]$ that satisfy $\max_{\pi \in B_\Gamma} p_{{\scriptscriptstyle\mathrm{w-LTO}}}(\pi) > c$, and apply the same trick as above. \\
The objective is not necessarily concave as $A$ is not guaranteed to be positive semidefinite. As it turns out, the resulting optimization problem is a fundamental NP-hard problem known as quadratic nonconvex programming with box constraints. In practice, non-convex quadratic programming problems are solved using heuristics such as branch-and-bound techniques. As the dimensionality of the problems that we are solving is rather small, we expect off-the-shelf solvers should do well on this problem. Moreover, whenever the computed LTO $p$-value is small, the matrix $G + I_N - 2e_{I}e_I^\top$ is usually rather sparse, which would help with computation.\\
Thus far we have explored theoretical properties of the LTO placebo tests. In this section, we compare empirical properties of the LTO placebo tests on semisynthetic examples based on the California Proposition 99 on tobacco consumption abadie2010synthetic, the Basque Country terrorism on local economic prosperity abadie2003economic, and the German Reunification on West Germany's GDP. abadie2015comparative. All simulation results rely on the R package Synth abadie2011synth, and may be found at \url{https://github.com/tsudijon/LeaveTwoOutSCI}.
The structure of the semisynthetic simulations is the same for each dataset. Firstly, we subsample $N$ units (smaller than the size of the original data) from each dataset, excluding the original treated unit. As described briefly in the introduction, on each subsample of the data, a unit is chosen uniformly at random to be treated. For each treated unit $I$, we posit a time-invariant treatment effect $\tau$, which we add on to the observed outcomes $Y_{I,t}$ following $T_0$. A range of values of $\tau$ are used, corresponding approximately to $(0,-1,-2,-3)$ times the standard deviation of the outcome variable measured at $T_0.$ For each subsample, several methods are compared in terms of power: the exact placebo, approximate placebo, randomized placebo, the naive LTO, and powered LTO.
The setting for $\tau=0$ amounts to comparing the Type-I errors for all methods. We compare the methods in two regimes: one in which $\alpha < 1/N$ and one in which $\alpha \geq 1/N$. In the $\alpha < 1/N$ regime case we omit the results for the exact placebo, which will be powerless in this setting. Each method uses the widely-adopted RMSPE statistic defined in (ref), and the synthetic controls are constructed in ways which follow the original analyses of each dataset. Finally, we average the powers and Type-I errors for all methods over 20 Monte Carlo subsamples of the original dataset, and report the resulting mean. This is an estimate of the unconditional Type-I error and power of each method.
Broadly, we observe the following empirical results. The LTO procedures demonstrate higher power in large signal to noise ratios, compared against the three placebo $p$-values. In intermediate signal to noise ratios, the naive LTO procedure often has lower or comparable power than the placebo procedures, but the powered LTO fares better. In the $\alpha < 1/N$ setting, we observe that the unconditional Type-I errors of the LTO methods are below $\alpha$ in all cases, whereas that of the approximate placebo $p$-value is always $1/N$. This suggests that the upper bound on the conditional Type-I error established in Theorem (ref) is not tight, and in practice, the LTO procedures often satisfy Type-I error constraints. The randomized placebo test does control the Type-I error, but its power is quite poor relative to the other procedures.
\paragraph{California Proposition 99.} Before showing semisynthetic results for other datasets, we give further details for the experiments done in Section (ref). In Section (ref), only the naive LTO procedure was compared with the placebo. A full comparison with both versions of the LTO procedure can be found in Figure (ref). When constructing the synthetic controls, we matched on several covariates: average retail price of cigarettes, log per capita income, the proportion of the population age 15–24, and per capita beer consumption. We use the mean of these variables over the 1980–1988 period, along with lagged smoking consumption in 1975, 1980, and 1988. This replicates the specification of abadie2010synthetic.
On the left, the powered LTO procedure is unconditionally valid in this experiment, while providing a power boost over the naive procedure. This is enough to make the power comparable to that of the approximate placebo, for intermediate signal to noise ratios. The power gain for larger effect sizes is smaller. The same observations can be seen on the right. While the powered LTO procedure increases Type-I error, its unconditional Type-I error is still less than that of the exact placebo. It is thus unconditionally valid in this experiment. It provides a significant power boost, however. Both LTO methods have larger power than the exact placebo for even modest effect sizes. In this setting, the approximate placebo violates the Type-I error constraint.
\paragraph{Basque Country Terrorism.} Next, we consider a similar setup using a dataset of Spanish regional GDP studied in abadie2003economic. In this setting, we take $\alpha = 0.05, N = 15$ as an example of inference in the $\alpha < 1/N$ setting. When constructing all synthetic controls, we imitate the choice of covariates found in the running Basque data example of abadie2011synth. The results are found in Figure (ref).
Remarkably, the conclusions identified for the Proposition 99 data carry over to this dataset. As before, the LTO procedures demonstrate much lower unconditional Type-I error than the upper bound advertises, while at the same time, they exhibit higher power than the inexact placebo test in higher signal to noise ratios. It is also interesting in this setting that the LTO procedures do well in intermediate signal to noise ratios.
In the right side of Figure (ref), we run the same experiment in when $\alpha = 0.1$; the same observations are broadly true. In this setting, the unconditional Type-I error of the LTO procedures are similar to that of the exact placebo (implying unconditional validity), while the power is much higher in for even moderately large effect sizes. The power of the LTO procedures even matches that of the approximate placebo test, which unconditional Type-I error control here.
\paragraph{German Reunification.}
In abadie2015comparative, the synthetic control method is applied to understand the effects of German reunification on the GDP of West Germany, using 16 other OECD countries as potential controls. The simulation takes multiple size $N = 14$ subsamples of the original data, without West Germany, and compares the power of the placebo and LTO method versus effect size.
Figure (ref) shows the results for two choices of $\alpha$. Both settings were constructed with the RMSPE statistic. On the left, $\alpha = 0.05$. In this comparison, the power gain of the LTO procedure is evident at higher signal to noise ratios. In this regime, the unconditional Type-I error of the LTO placebo test is much closer to that of the placebo test, and unfortunately, all procedures with non-zero power must violate conditional validity, per Proposition (ref). However, Theorem (ref) guarantees that the Type-I error will never be worse than that of the placebo test.
On the right, the Type-I errors of the LTO methods are essentially the same as that of the exact placebo, indicating that both methods are unconditionally valid in this setting. It is interesting to notice the power gain of both LTO methods. At intermediate signal to noise ratios, the naive LTO has a similar power to the exact placebo, but for larger signals, the power advantage is considerable. The powered LTO method is considerable more powerful in the intermediate effect size case, and is far more powerful than the exact placebo.
It is worth revisiting the classic case studies of the synthetic control method in abadie2003economic, abadie2010synthetic, abadie2015comparative using the LTO and placebo inference procedures. For each use of the LTO $p$-value, we analyze the sensitivity of the conclusion to the equal weights assumption. For all sensitivity analyses, we use Gurobi 10.0 gurobi to solve the nonconvex quadratic optimization problem for optimizing $p_{{\scriptscriptstyle\mathrm{w-LTO}}}(\pi)$. A similar sensitivity analysis for the exact and approximate placebo test may be done using the results of firpo2018synthetic, which we do not replicate here.
In Section (ref), we already saw the application of the LTO placebo test and standard placebo tests for the Proposition 99 example of abadie2010synthetic, along with the sensitivity analysis for the LTO placebo test. In this section, we replicate the original analyses of abadie2015comparative, abadie2003economic, using the exact and approximate placebo $p$-value, the naive LTO placebo test, and the powered LTO placebo test. Focusing on the German Reunification dataset abadie2015comparative, the exact and approximate placebo $p$-value are both too large to reject the null hypothesis. By contrast, both LTO p-values are below $\alpha$. The correction used by the powered LTO $p$-value is decent. Since $p_{{\scriptscriptstyle\mathrm{naive-LTO}}}\le \alpha$, we can perform sensitivity analysis to calculate the smallest $\Gamma$ that overturns the conclusion. The output is shown in Figure (ref). From the figure, $\Gamma$ is around $1.1$ before the maximum possible $p$-value is greater than $0.05$. Overall, the sensitivity curve does not increase too steeply as a function of $\Gamma$.
For the Basque country terrorism dataset of abadie2003economic, we repeat the exercise to analyze the effect of terrorism on GDP of the Basque country. Here, $N = 17$ with the onset of the treatment being $1970.$ We conduct inference with $\alpha = 0.05,$ which is less than $1/N.$ Surprisingly in this example, we obtained a LTO $p$-value of $0.67$; the placebo $p$-value gave a similar non-significant result at $0.41.$ We expected much smaller $p$-values. One possible explanation is that in prior analyses of the Basque country terrorism dataset abadie2011synth, several regions were removed if they had poor fit for the treatment period. We do not incorporate these adjustments in the LTO inference procedure. Because the LTO $p$-value is non-significant, in practice a sensitivity analysis procedure would not be run. It is still interesting to see how the curve of maximum weighted $p$-values grows with $\Gamma$. Figure (ref) shows the output for this example and the previous one.
We discuss several extensions of the LTO placebo test. The first defines the notion of strange point in a different way, while the second leaves $r$ points out at a time.
Instead of labeling a unit as strange if it wins too many matches, we can consider a unit to be strange if its rank across all matches is too large. This leads to a variant of the procedure which has similar coverage guarantees as the original Jackknife+ procedure; see Theorem 1 of barber2021predictive. Define $\rank\set{I; I,i,j} := \mathbb{I}\set{I \succ j} + \mathbb{I}\set{I \succ i},$ where in this context, $\set{I \succ j} : = \set{R_{I,i,j;I} > R_{I,i,j;j}}$ and similarly for $\set{I \succ i}$. We can define a $p$-value \[ p_{{\scriptscriptstyle\mathrm{rs-LTO}}} := \frac{1}{(N-1)(N-2)}\sum_{\substack{i,j \in [N]\setminus I\\ i \neq j}} (2 - \rank\set{I; I,i,j}), \]
Using a similar argument as in Theorem (ref), we derive an upper bound on the Type-I error of $p_{{\scriptscriptstyle\mathrm{rs-LTO}}}$. The proof is presented in Appendix (ref).
The bound is equal to or $1/N$ larger than that of $p_{{\scriptscriptstyle\mathrm{naive-LTO}}}$ and $p_{{\scriptscriptstyle\mathrm{powered-LTO}}}(\alpha)$. In particular, when $\alpha < 1/(N-1)$, all three choices have Type-I error controlled at $1/N$, as shown in Theorem (ref). However, it is easy to see that $2 - \rank\set{I; I,i,j}\ge \mathbb{I}\{I\not\succ i,j\}$ and thus $p_{{\scriptscriptstyle\mathrm{w-LTO}}}\le p_{{\scriptscriptstyle\mathrm{rs-LTO}}}$ almost surely. Thus, $p_{{\scriptscriptstyle\mathrm{rs-LTO}}}$ is less favorable because it always gives more conservative inference.
We may consider an analogous procedure for the placebo test where $r$-tuples of data points are left out. The analysis of the method in this setting clarifies the essential features of the LTO case.
Extending the LTO procedure (Remark (ref)), we consider a tournament between all $N$ units where matches are held for every $(r+1)$ units. For each match among $\{i_0, i_1, \ldots, i_r\}$, we leave out all participants and construct synthetic controls for each of them. The score $R_{\{i_0, i_1, \ldots, i_r\}; i_j}$ of participant $i_j$ is defined via the same form as (ref). For example, we can calculate the RMSPE by using $[N]\setminus\{i_0, i_1, \ldots, i_r\}$ as the donor pool. We say $i_0$ wins a match among $\{i_0, i_1, \ldots, i_r\}$, denoted by $i_0 \succ i_1, \ldots, i_r$, if $R_{\{i_0, i_1, \ldots, i_r\}; i_0} > \max_{j\in [r]}R_{\{i_0, i_1, \ldots, i_r\}; i_j}$. Extending the LTO p-value, define
Anagolously, we can also define the rank-sum LRO p-value:
where $\rank\set{I; I, i_1, \ldots, i_r}= \sum_{j=1}^{r}\mathbb{I}\{I\succ i_j\}$ and, in this context, $I\succ i_j$ iff $R_{\{I, i_1, \ldots, i_r\}; I} > \max_{j\in [r]}R_{\{I, i_1, \ldots, i_r\}; i_j}$. Note that they recover $p_{{\scriptscriptstyle\mathrm{LTO}}}$ and $p_{{\scriptscriptstyle\mathrm{rs-LTO}}}$ when $r = 2$. Above, the normalization factor $2/r$ is chosen to make the Type-I error to converge to $\alpha$ when $N$ tends to infinity, as evidenced by Theorem (ref) below.
The following two results generalize Theorem (ref) and Theorem (ref) to $p_{{\scriptscriptstyle\mathrm{LRO}}}$ and $p_{{\scriptscriptstyle\mathrm{rs-LRO}}}$, respectively. Both proofs are presented in Appendix (ref). Via tedious algebra, we can show that they recover Theorem (ref) and Theorem (ref) when $r = 2$.
We propose a novel leave-two-out placebo test for the synthetic control method under uniform assignments. Compared to the standard placebo test based on permutation tests, the LTO placebo test has two advantages. First, it has the same theoretical conditional Type-I error guarantee for practical $(N, \alpha)$ pairs; further, it often achieves a strictly lower unconditional Type-I error in practice, unlike the standard placebo test for which the guarantee is tight. In particular, this permits valid inference even in the challenging $\alpha < 1/N$ case. Second, we find empirically that it has higher power for moderately large signal-to-noise ratio and comparable power otherwise. In most of our experimental settings, the powered-LTO placebo test almost Pareto-improves the standard placebo test.
While the design assumption is strong, it does not require large sample size and/or long time series, which are crucial for outcome-based approaches but uncommon in applications of the synthetic control method. Moreover, unlike most outcome-based approaches, the LTO placebo test does not artificially simplify the important steps, like weight matrix estimation and model selection, in the construction of synthetic controls. On the other hand, we propose a weighted variant of the LTO placebo test and a sensitivity analysis to account for non-uniform assignments. Below we discuss a few other related issues.
The placebo tests discussed in previous sections all exploit variation in which units are treated. An alternative approach is to exploit the variation of treatment timing. In the literature, in-time placebo tests are often performed by choosing a pre-treatment period, pretending it to be the time when the treatment occurs, and running the same synthetic control procedure to eyeball check if the treated unit differs from its synthetic control between the fake and actual treatment time fremeth2013making, abadie2015comparative,castillo2017causal, cole2020impact, chattopadhyay2023did. To formalize it into a rigorous statistical inference procedure, we can follow firpo2018synthetic verbatim by assuming the treatment time $T_0$ is uniform in $[t_1, t_2]$ for some $1\le t_0 < t_1 \le T$. Then the standard and LTO placebo tests can both be directly applied in this setting. For in-time placebo tests, the small sample issue is arguably more pronounced because it is typically only reasonable to choose a small time window $[t_1, t_2]$. Based on the promising performance of LTO placebo tests in unit-level placebo tests when the sample size is small, we expect they continue to excel for in-time placebo tests, though we leave the formal investigation for future work.
In previous sections we focus on the sharp null (ref). Following firpo2018synthetic, chernozhukov2021exact, it is straightforward to test any postulated effect vector $\theta\in \mathbb{R}^{T-T_0}$: \[H_0(\theta): \tau_{I, T_0+1} = \theta_1, \tau_{I, T_0+2} = \theta_2, \ldots, \tau_{I, T} = \theta_{T-T_0}.\] In fact, by subtracting $\theta_j$ from $Y_{I(T_0+j)}(1)$, we are back to the case of sharp null. Phrased differently, we can redefine the RMSPE $R_{i,j,I;I}(\theta)$ for the treated unit within the triple $\{i,j,I\}$ by replacing $Y_{It}$ by $Y_{It} - \theta_{t-T_0}$ for $t=T_0+1, \ldots, T$ and reject $H_0(\theta)$ if $p_{{\scriptscriptstyle\mathrm{naive-LTO}}}(\theta)\le \alpha$, where
Note that the other RMSPEs remain the same because the synthetic control weights only depend on pre-treatment outcomes which are unchanged.
By inverting the test, we can obtain a confidence region for $(\tau_{I, T_0+1}, \tau_{I, T_0+2}, \ldots, \tau_{I, T})$: \[\mathcal{C}_{1-\alpha} = \left\{\theta\in \mathbb{R}^{T-T_0}: p_{{\scriptscriptstyle\mathrm{naive-LTO}}}(\theta) > \alpha\right\}.\] When the RMSPE is used, by (ref), there exists $A_{ij}\in \mathbb{R}^{T-T_0}$ and $b_{ij}\in \mathbb{R}$ for $i,j\in [N]\setminus I$ such that \[\mathcal{C}_{1-\alpha} = \left\{\theta\in \mathbb{R}^{T-T_0}: \sum_{i,j\in [N]\setminus I}\mathbb{I}\set{\|\theta - A_{ij}\|_2^2 \le b_{ij}} > \alpha (N-1)(N-2)\right\}.\] Each indicator gives a ball and $\mathcal{C}_{1-\alpha}$ is an union of intersections of these balls that lie in at least $\alpha (N-1)(N-2)$ balls. By Theorem (ref), \[\Prob\Big((\tau_{I, T_0+1}, \tau_{I, T_0+2}, \ldots, \tau_{I, T})\in \mathcal{C}_{1-\alpha}\Big)\ge 1 - \frac{\floor{N f(N, \alpha)}}{N}.\]
Power analysis is challenging in our setting as it typically relies on large sample approximation that tends to be uninformative when $N$ and $T$ are small, particularly given the complexity of the synthetic control procedure. In this section, we make an attempt to explain the empirically observed power improvement over the approximate placebo test.
In the classical asymptotic regime where $N$ tends to infinity, a test is consistent if its power converges to $1$ for any fixed alternative. Qualitatively, consistency ensures that the null hypothesis will be rejected with high probability as the problem becomes sufficiently easy. We migrate the notion of consistency to the fixed-$N$ setting by defining a test to be consistent if the power converges to $1$ as the signal strength under the alternative grows to infinity. Consistency serves as a minimal requirement for a desirable test. We show that, if $\alpha < 1/N$, the LTO placebo test is uniformly consistent, whereas the approximate placebo test is not. In fact, the latter may have nearly zero power even with an arbitrarily large signal strength.
A proof may be found in Section (ref).
The theory of LTO placebo tests only relies on the uniform assignment assumption but not the choice of summary statistics. Thus, it can be applied beyond synthetic control applications to any problem with the same structure. Here, we discuss one instance in the context of inference with few clusters bertrand2004much, donald2007inference, ibragimov2010t, conley2011inference, bester2011inference, imbens2016robust, ibragimov2016inference,canay2017randomization, mackinnon2018wild, hagemann2019placebo, canay2020use, hagemann2020inference, canay2021wild, hagemann2023permutation, webb2023reworking; see also the surveys by cameron2015practitioner and mackinnon2023cluster. With a few exceptions that assumes normality of errors (e.g., donald2007inference), existing approaches with theoretical guarantees rely on large clusters in order for the within-cluster test statistics to be close to normally distributed (e.g., ibragimov2010t, ibragimov2016inference, hagemann2020inference) or at least close to be symmetrically distributed (e.g., canay2017randomization, canay2021wild, hagemann2023permutation) -- these distributional assumptions are hard to justify in most applications of synthetic controls. Other procedures work well empirically when the cluster sizes are moderate, e.g., $\ge 30$ cameron2008bootstrap, though the theory still requires the sample size tends to infinity in each cluster. Moreover, to do inference on treatment effects, many methods that can accommodate small clusters require at least two conley2011inference, hagemann2019placebo or $\log (1/\alpha)$ canay2017randomization treated clusters if the total number of clusters is below $1/\alpha$.
In sum, when there is a single treated cluster, which is common in applications, existing methods rely on either large-sample approximation, or that the number of control clusters is at least $1/\alpha - 1$. In this case, if the data comes from a cluster randomized experiment su2021model, lu2023design or if the researcher is willing to assume uniform assignment, the LTO procedure can be applied to test Fisher's sharp null. We leave the details for future research.
Classical randomization inference fisher1937design, romano1989bootstrap, romano1990behavior, ding2017paradox exploits exchangeability, or group invariance more generally, by comparing the test statistic for the observed data with those for permuted or group-transformed data. Despite a century of development, nearly all randomization inference procedures are variants of this generic form with different choices of test statistics or group of transformations. The LTO placebo tests, or the more general leave-$r$-out tests, are new types of randomization inference that move beyond classical permutation or rank-based inference.
Our tests generalize the Jackknife+ barber2021predictive, which also differs from classical randomization inference. It is an extension of conformal prediction vovk2005algorithmic and the Jackknife procedure wu1986jackknife to construct prediction intervals with guaranteed coverage under exchangeability. Let $Z_1 = (X_1, Y_1), \ldots, Z_N = (X_N, Y_N)$ be exchangeable observations where $X_i$ denotes a set of covariates and $Y_i$ denotes the outcome. The goal is to construct a prediction interval $\hat{C}(X_{N+1})$ such that $\Prob(Y_{N+1} \in \hat{C}(X_{N+1}))\ge 1-\alpha$ assuming $Z_{N+1} = (X_{N+1}, Y_{N+1})$ is exchangeable with $\{Z_1, \ldots, Z_N\}$. Jackknife+ essentially inverts a leave-one-out test: \[p(Y_{N+1}) = \frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\set{R_{i,N+1;N+1}\le R_{i, N+1; i}},\] where $R_{i,j;j} = |Y_j - \hat{\mu}_{-i,-j}(X_j)|$ where $\hat{\mu}_{-i,-j}(\cdot)$ can be any model fit on $(Z_k)_{k\in [N]\setminus \{i, j\}}$. However, Jackknife+ suffers from the same granularity issue as the standard placebo tests because it can only take multiples of $1/N$.
Another exception is the prediction inference of model transfer errors by andrews2022transfer. They consider a setting with data from $N$ exchangeable training domains (e.g., countries), denoted as $Z_1, \ldots, Z_N$. For a given economic or machine learning model and a pair of domains $Z_i, Z_j$, let $e(Z_i, Z_j)$ denote the accuracy (e.g., the mean square error) of the model fit on the $i$-th domain and evaluated on the $j$-th domain. Let $Z_{N+1}$ denote an unobserved target domain that is exchangeable with $\{Z_1, \ldots, Z_N\}$. The goal is to construct a prediction interval for the random transfer error $e(Z_I, Z_{n+1})$ where $I$ is chosen from $[N]$ uniformly at random. Their interval is constructed based on quantiles of $\{e(Z_i, Z_j): i,j\in [N], i\neq j\}$ even if they are no longer exchangeable as in classical randomization inference due to the bivariate structure.