Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
64,624 characters · 11 sections · 77 citation commands
Splash! Robustifying Donor Pools for Policy Studies
\begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }} \@thanks \endgroup \setcounter{footnote}{0} \let \begingroup \def\fnsymbol{footnote}{\fnsymbol{footnote}} \def\hbox to 0pt{$^{\@thefnmark}$\hss}{\hbox to 0pt{$^{\@thefnmark}$\hss}} \vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\@endauthor\rm\hss \vskip \interauthorskip \@startauthor{\@endauthor\rm\hss \vskip \interauthorskip \@startauthor} \@startauthor \@author \@endauthor}} \@thanks \endgroup \setcounter{footnote}{0} \let\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\@endauthor\rm\hss \vskip \interauthorskip \@startauthor{\@endauthor\rm\hss \vskip \interauthorskip \@startauthor} \@startauthor \@author \@endauthor}}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax\relax \let\vbox{\hsize\textwidth \linewidth\hsize \vskip \beforetitskip {
} \vskip \aftertitskip {\def\unskip\enspace{\rm and}\enspace{\unskip\enspace{\rm and}\enspace} \def\it{\it} \def\sc{\sc} \def\bf{\bf} \def\rm\hss \vskip \interauthorskip \bf{\rm\hss \vskip \interauthorskip \bf} \bf \@author }}\relax \gdef\@thanks\gdef\@author\gdef\@title\let\thanks\relax
Twenty years ago, synthetic control methods (SC/SCM) were developed by Abadie2003 for causal inference in public policy/economic settings as an extension of the difference-in-differences method (DiD). DiD and other methods commonly make some variant of the parallel trends assumption for imputing the counterfactual. Parallel trends posits the average of the untreated units is a good proxy for the counterfactual of the treated unit. However, this assumption may be hard to defend in may empirical settings augdd. Thus, SCM posits a modification to DiD: that a weighted average of certain untreated unit (donors) is a better proxy for the counterfactual than an average of all of them. The original synthetic control method (OSC) does this via convex hull restrictions, which prevents extrapolation bias and encourages a sparse set of donors getting weight Abadie2015,Abadie2021. This makes SCM to be more transparent in quantifying which units contribute to the counterfactual. For this reason and others, SCM has grown in use as a tool for policy analysts scotlandsc,Abadie2015, Calderon2015,Cole2020,Marinello2021,Bulle2022,Cohn2022, bullinger2021child,sun2019money.
This study is intended to aid policy analysts who are using SCM in the selection of a donor pool. The validity of SCM is predicated on the donor pool (or the set of untreated units) used. The donor pool should ideally be as similar as possible to the treated unit in the pre-intervention period. These units should also not have undergone similar shocks/interventions. This way, policy analysts can be more confident that any deviation from the observed series is the result of the policy change instead of other confounders. But while the not treated condition can be fairly straightforward, composing a donor pool that is meaningfully similar to the treated unit, particularly in a policy context, usually involves some subjective researchers decisions. For example, suppose a school institutes a new mathematics curriculum: should donors be restricted based on socioeconomic status– if so, what should the threshold be (i.e., including only communities with median annual incomes greater than \$70,000 a year)? If so, what in practical terms makes this cutoff so different from \$80,000 a year? Should we only use public schools? The same district? Should we only include schools in the same state, bordering states, or even districts in bordering counties from adjacent states? What if a mix of donor types (i.e., states to regions) made better sense as suggested by SHI2023512? Should we impose size restrictions, and if so, what would it even mean empirically to be similarly sized? In the West Germany case for example, Abadie2015 note that interpolation can occur “especially if the donor pool contains units with characteristics that are very different from those of the [treated] unit", and thus opted to restrict their donor pool to 16 similar nations to West Germany. In addition to questions about similarity, SCMs might be theoretically invalid when we have \(N>>T\) Abadie2021a,Amjad2018, SHI2023512,mcclelland2022update because they may cause interpolation biases. Abadie2022 note that the risk of interpolation bias is high when \(N>>T\) because the SCM may fit to idiosyncratic values of the donor time series. Interpolation is also possible when we have outlier units or substantial volatility of the time series. Particularly in a modern world where policy analysts increasingly leverage large \(N\) datasets with more granularity than countries or states, researchers more frequently encounter settings where interpolation is a much bigger threat, given the levels of dimensionality Yang2021,Alonso2020,He2022,ocp,yu2022revitalising,dave2022political,scotlandsc.\footnote{Alonso2020 have \(\mathcal{T}_0=9\) to \(J=32\), Yang2021 have \(\mathcal{T}_0=22\) to \(J=278\), dave2022political have \(\mathcal{T}_0=11\) to \(J=150\), He2022 have \(\mathcal{T}_0=10\) to \(J=327\), yu2022revitalising have \(\mathcal{T}_0=15\) to \(100 <J <500\), ocp have \(\mathcal{T}_0=18\) to \(J=64\), and scotlandsc have \(\mathcal{T}_0=20\) to \(J=345\). We do not suggest that these analyses are invalid, rather, we only highlight recent cases of analysts encountering \(N>T\) settings. }
For a visual example of a setting where we have noisy outcomes and potentially irrelevant donors, consider (ref) derived from the data of hotelpaper, who study the causal impact of Barcelona's hotel moratorium on the daily price of hotel rooms throughout the city. The authors have 82 hotel markets across Europe to compare Barcelona to.
We plot the weekly average of hotel room prices. Even though we have a long pre-intervention time series, it is still clear that many units have prices far higher or lower than Barcelona. Clearly, we have a complex and noisy time series, with some units exhibiting considerable variance in their weekly prices compared to Mediterranean cities or Barcelona itself. This is a setting where interpolation is possible. By comparison, the original case studies which used SCM-- or the panel data approach (PDA), SCM's cousin-- had much fewer donors Abadie2003, Abadie2010, Abadie2015, hsiaopda. Whatever the potential source of interpolation bias, the common way of preventing it is to “judiciously" select a donor pool Abadie2021. Thus, a central question for researchers using SCM is “Which units belong in the donor pool in the first place?". Specifically, Abadie2022 advise analysts to “trim" their donors to be similar to the values of the treated unit in the pre-intervention period-- and this is where our complications begin. While in practice donor trimming based on outcome/covariate differences is very common fundment, xinj,olysport,aosla,GRIER2023174, little formalized guidance for donor trimming exists in the policy context. Thus, trimming is typically based on largely qualitative logic, as different authors will ultimately have different standards on what exactly constitutes `too different' for many given metrics.\footnote{Methods such as trimming via standard deviation distances have been proposed by mcclelland2022update, but the author offers no theoretical justification for this metric, and it is unclear how such a procedure would be valid for larger datasets.}
However, what if simple trimming is not straightforward: we might ask how to trim down the donor pool from (ref). What restrictions upon the predictors or other criteria should we use to get a more feasible donor pool-- do restrictions on predictors like climate, unemployment, or geographic factors matter? In hotelpaper, the authors have 82 donors, but only a small subset of them are cities located on the Mediterranean Sea. Should we use only Mediterranean donors, or is using cities in France and Spain sufficient? Unfortunately, we cannot even try to answer these questions because we do not know which donors are which-- Booking.com anonymized them to preserve differential privacy. Furthermore, no covariates about the donors were provided. Here, we simply can't make such judgements about what the “right" set of donors is a priori, as there's no predictors to other metrics to start from. Given these constraints (in this example and in many real policy datasets), we argue that leaving the selection of the donor pool to subjective judgment or mere human intuition can compromise the robustness of the results. When machine learning experts extended the SCM toolkit, they did so in part to address problems from high-dimensions, missing data, noise, and corrupted data. Some work even addresses interpolation biases mcclelland2022update, Kellogg2021, where LASSO or nearest neighbor estimators are used as bias-correction techniques. Amjad2018 develop a robust SCM, employing singular value decomposition and Bayesian methods to deal with problems such as lack of covariates, missing data, and noisy outcome series, while generating confidence intervals. Later, this approach was extended to a tensor/multiple outcome framework Amjad2019.
Even though machine-learning methods have been quintessential in developing new/different objective functions and optimizations for SCM, a curiosity is that not much work exists on formalized methods for donor trimming as a concept.\footnote{Work does exist on covariate selection, for example: One extension by VivesiBastida2022 employs LASSO to select a sparse set of optimal predictors of the outcome from a large set of irrelevant ones. Statistical learning aside, we also have methods proposed by cherrypick to mitigate biases from which covariates to use/pick.} Our paper addresses this gap for public policy analysis. In particular, from the title of our paper, we delve further into how one might further `robustify' an SCM estimator. The term comes from pcr who emphasize the value of “first find[ing] a low-dimensional embedding of the data before fitting a prediction model". pcr use principal component analysis (PCA) to estimate the low-dimensional embedding of their donors, and then use Ridge regression to impute the counterfactual. We extend the idea of `robustification' when applied to donor trimming: we begin by finding the low-rank structure functional data representation of the pre-intervention outcomes. We then select the most comparable donors to a treated unit. We conclude by fitting a prediction model on the low-rank representation of these most comparable donors. The sketch above highlights how our algorithmic methods to select donors differs from simple use of intuition. Specifically, we compare two algorithms: one method we use is functional principal component analysis (\verb|fPCA|) and the other is the forward selection panel data approach (\verb|fsPDA|), both recently presented by Bayani2021,SHI2023512.\footnote{In the first draft of this paper, we considered the pairwise deviation minimization method proposed by yu2022revitalising. However, this method has no formalized stopping mechanism, which would complicate comparisons. We leave this, and other methods, to future work.} The former is a 5 step method which first estimates the low-rank structure of the pre-intervention time-series and clusters over them to trim down the donor pool, while the latter uses a simple OLS algorithm to pre-process the donors. Both approaches are completely based on pre-intervention outcome trends instead of matching on many predictors, thereby removing one potential source of subjectivity to used to compare the units. Additionally, they are valid in cases where we have many more donors than pre-intervention time periods. Thus, they can alleviate the taxing step of needing to trim down a massive donor pool to a set of relevant units. Our paper is motivated by the following empirical applications:
We first compare both trimming methods to OSC in synthetic studies. Then, we implement them in the aforementioned empirical case studies. We presume some familiarity with the SCM and PDA approaches: for a detailed review, see Hollingsworth2020, scmvpda or WAN2018121, Abadie2021. Before we continue, we introduce notation. Denote scalars, vectors, and matrices as \(x\), \(\mathbf{x}\), and \(\mathbf{X}\). Let \(\mathbb{R}\) denote the real number space. Let \(\|\cdot \|_{1}\) and \(\|\cdot \|_{2}\) denote the vector norms \(L_1\) and \(L_2\) respectively. We are concerned with three sets of units: \(\mathcal{N}\coloneqq \bigcup\{ \widetilde{\mathcal{N}}_{0},\mathcal{N}_{0},\mathcal{N}_{1}\}\). Power set \(\mathcal{N}\) is the universe of units, indexed by \(j\). Subsets \(\mathcal{N}_{0}\) and \(\mathcal{N}_{1}\) are (un)treated units, defined respectively as \(\mathcal{N}_{0}\coloneqq \{I+1,\ldots,J\}\) and \(\mathcal{N}_{1}\coloneqq \{0,\ldots,I\}\). \(\widetilde{\mathcal{N}}_{0}\subset \mathcal{N}_{0}\) is the set of units selected by either \verb|fsPDA| or \verb|fPCA-SYNTH|. We observe a time series for each unit indexed by \(t \in \left(1, T\right) \cap \mathbb{N}\). Partition our time series into sets \(\mathcal{T}\coloneqq \mathcal{T}_{0} \cup \mathcal{T}_{1}\) with their own cardinalities, where \(\mathcal{T}_{0}\coloneqq \{1\ldots T_0 \}\) is pre-intervention periods and \(\mathcal{T}_{1}\coloneqq \{T_0+1\ldots T \}\) denotes post-intervention periods. We also observe outcomes \(\mathbf{Y}\coloneqq \left (y_{jt}\right)_{j \in \mathcal{N},t \in \mathcal{T}} \in \mathbb{R}^{\left(J+1 \right)\times T }\). Treated unit outcomes are \(\mathbf{y}_0 \coloneqq [y_{0t} \ldots y_{0T}]^{\top}\) and the outcomes for donors are a \(T \times J\) matrix \(\mathbf{Y}_{\mathcal{N}_0} \coloneqq [y_{1t} \ldots y_{JT}]^{\top}\). An indicator \(d \in \{0,1\}\) is the treatment. We assume a linear factor model as a DGP, where \(y^{0}_{jt}\) and \(y^{1}_{jt}\) represent the status of the units being untreated or treated.
Here, \(y^{0}_{jt}\) represents the DGP for all units prior to treatment. If the unit is untreated, we presume the pre-intervention factor model holds. Also, this model prohibits spillover effects of the treatment. Here \(\boldsymbol{\mu}_{j} = \bigr[\mu_{j1},\ldots \mu_{jr}\bigr]^{\top}\) and \(\boldsymbol{\delta}_{t} = \bigr[\delta_{1t},\ldots \delta_{rt}\bigr]^{\top}\) are \(r \times 1\) factor loadings and common factors, \(\boldsymbol{\theta}_{t}\) are \(1 \times k\) coefficients for \(k \times 1\) \(\mathbf{Z}_{j}\) predictor effects, and \(\boldsymbol{\epsilon}_{jt}\) represents Gaussian noise Gaillac2023. The fundamental difficulty of causal inference is that we observe \(y_{jt} = d_{jt} y_{jt}^1 + (1 - d_{jt}) y_{jt}^0\), where we cannot simultaneously observe a unit being treated and untreated-- thus, our goal is to impute the counterfactual \(\hat{y}_{jt} \). Our treatment effect is simply \(\tau_{jt} \coloneqq y_{jt}-\hat{y}_{jt}\), where we are typically interested in \(\{\tau_{0T_{0+1}}, \ldots, \tau_{0T_1}\}\). We also make two additional assumptions
Namely, we presume a \(J \times 1 \) vector of unit weights \(\boldsymbol{\omega}^{\ast} = \left(w_{1} \ldots w_{J} \right) \) will generate a SC, \(\hat{y}_{jt} \coloneqq \boldsymbol{\omega}^{\ast}\mathbf{y}_{\mathcal{N}_{0}}^{\prime}\), such that the counterfactual approximates the observed trajectory of the treated unit in the pre-intervention period. These weights should also match on the common factors which drive the factor loadings from (ref).
Note that the factor loadings embedded in the donor outcomes are a good proxy to match on the pre-intervention time series of the treated unit [for more on this point, see Hollingsworth2020. However, because we do not observe these loadings, we use the time variant outcomes of donors to match on these loadings. We presume these common factors are alpha mixing, or that there is weak correlation between them such that severe serial correlation won't affect our results, and that there is enough variation in the common factors to match on the unobserved factor loadings. The trimming methods are meant to select some optimal subset of donors, \(\widetilde{\mathcal{N}}_{0} \subset \mathcal{N}_{0}\), such that we maximize the similarities of the donor pool to the treated unit, thereby automating the trimming process. Our causal estimand and diagnostic of interest are:
or, the post-intervention average treatment effect on the treated and the pre-intervention root-mean squared prediction error.
First, we review the trimming algorithms. Then, we review the SC and PDA approaches to estimation at a broad level.
Forward selection was recently studied by SHI2023512, li2022novel as a viable method of donor trimming for causal inference. Here is a short sketch of it:
where $R$ is the stopping time hyperparameter to be tuned. SHI2023512 used a modified BIC MBICPaper to estimate this hyperparameter:
In this method, we basically regress the outcome variable of the treated unit on the outcome variable of all donor members over the pre-intervention time period. We then store the \(R^{2}\) statistics from each estimation, and retain the unit in \(\widetilde{\mathcal{N}}_{0}\) with maximum \(R^{2}\). We then repeat this process again, until there's no significant improvement in the prediction.
Aside from linear regression or Euclidean distance as advocated by Abadie2022, one viable method to extract a donor pool is \(k\)-means clustering olukanmi2020rethinking. \(k\)-means clustering finds cluster centers indexed by \(i\) \(\lbrace \mathbf{C} = c_{1},c_{2},\ldots ,c_{k}\rbrace\), defined as
such that \({\bigcup }_{i = 1}^{k} y_{j} = \mathbf{Y}_{t \in \mathcal{T}_0}\) and \(y_{i} \cap y_{j} = \emptyset\). Suppose \(\mathbf{G}=[g_{tk}]_{N\times C}\) is an indicator matrix, where \(g_{tk} \in \{0,1\}\) denoting, \(\forall j \in N\) if \(y_{t}\) belongs in a given cluster. The solution to k-means clustering minimizes the within-cluster variance
Note the maximum number of clusters allowed is a hyperparameter in the K-means algorithm. To avoid manually choosing of the number of clusters, we use Silhouette statistics to ascertain the optimal number of clusters ROUSSEEUW198753, computed via
After the clustering, we keep the cluster with the treated unit. The new donor pool is \(\widetilde{\mathcal{N}}_{0} = C \setminus \{c_i \mid j = 0\}\), or all of the units in the \(i\)th cluster except for the treated unit. Some papers in recent years have adopted this exact approach to trim donors LEE2021103617,rudholm2022does. However, clustering on raw data in policy or economic settings can be very misleading, particularly when dealing with \(N>>T\) or \(T>>N\) settings. The reason for this is that there may be outlier units (as a whole unit or at specific points in time), noise, and many irrelevant units-- what's more, standard clustering over the raw data might detect incidental similarities between units (and cause overfitting). We are interested in the broader temporal correlations between the treated unit and donor units clushdrev, and do not wish to fit to noise. So, to address this problem, the first step of \verb|fPCA-SYNTH| is to reduce the data's dimensionality via functional Principal Component Analysis (fPCA) before clustering.
One of the main limitations of standard PCA as used in pcr is that “the data points on each curve are assumed to be independent of each other, but in reality it is known that any point on a continuous time-series is correlated with the data points that precede and follow that point” fpcasport. Thus, a main advantage of fPCA over standard PCA is that accounts for temporal correlation within the data points WARMENHOVEN2021110106 via functional basis splines while reducing the data dimensionality. Hence, the functional PCA scores preserve the temporal correlation among unit observations. Once we have the low-dimensional representation of the outcomes matrix, we may now k-means cluster to extract our donor pool. We would like to highlight some differences between these algorithms: forward selection is a greedy parametric algorithm in that it seeks to maximize the \(R^{2}\) statistic to predict the treated pre-intervention series, and has no method of discarding donors once they've been selected. Thus while forward selection performs regularization, it may suffer from overfitting biases as discussed in li2022novel, who solves the overfitting probem by using difference-in-differences. In contrast, \verb|fPCA-SYNTH| is non-parametric in nature. Because the clustering is done over the fPCA scores, there is less of a concern for overfitting.
Broadly, SCMs and PDAs follow a minimization of the treated unit and the pre-intervention donor matrix
The SC from any selected pool is denoted as \(\hat{y}_{jt} \coloneqq \boldsymbol{\omega}^{\ast}\mathbf{y}_{\widetilde{\mathcal{N}}_{0}}^{\prime}\). OSC solves
where the weights lie on the simplex \(\Delta^{J-1}\) and are constrained to be within \(\mathbb{I} \coloneqq [0,1] \subset \mathbb{R}\). Practically, constraints force the weights to not allow predictions outside of the donor pool the user supplies. Oftentimes for OSC, analysts use additional predictors to improve pre-intervention fit, where a matrix of importance weights are computed
However, the addition of covariates does not by itself mitigate interpolation biases in the large \(N\) setting Abadie2021a, or if there are irrelevant donors/noisy outcomes.
With \verb|fPCA-SYNTH|, the donors are already correlated with the treated unit over time via the clustering. However, we still wish to find a low-rank representation of the donor-pool to estimate the counterfactual with.\footnote{See Appendix A for more detail on low-rank estimation as well as additional technical details.} To do this, we use Robust PCA (RPCA) scores. RPCA scores are obtained via the solution to the following optimization
where \(\mathbf{L}\) and \(\mathbf{S}\) are signal and noise matrices of the pre-intervention time-series. \(\mathbf{L}\) and \(\mathbf{S}\) are computed via the Alternating Direction Method of Multipliers with the augmented Lagrangian, where the former matrix emphasizes the pre-intervention temporal patterns and the latter shrinks the influence of the \(\mathbf{S}\) matrix with each iteration. We then use \(\mathbf{L}\) (again in the pre-intervention period) to estimate the unit weights for the SC, as this matrix represents the patterns of the donors in the pre-intervention period. Since the units are already correlated, we no longer need to use the convex hull restriction from OSC. RPCA-SYNTH solves the weights using convex cone restrictions
We expect the estimation of \verb|fPCA-SYNTH| to be less sensitive to outliers and noise compared to OSC, since it does not formally partition the observations using low-rank techniques. Forward Selection PDA estimates the counterfactual as follows using OLS.
It uses its optimal donors as obtained from its donor trimming algorithm.
The goal of the simulations and empirical analyses are to illustrate the utility of algorithmic donor trimming. Thus, while we also study the treatment effect and pre-treatment fit, our simulations have the additional demand of picking the optimal donor pool. We meet this challenge with Gaussian Processes (GP). GPs define a distribution over functions. More formally, GPs are collections of random variables \(\mathbf{x} \in \mathbb{R}^d\) where a finite set \(p(f(\mathbf{x}_1),\ldots,f(\mathbf{x}_N))\) is derived from a joint Gaussian distribution \(p(\mathbf{f} \lvert \mathbf{x}) = \mathcal{N}(\mathbf{f} \lvert \boldsymbol\mu, \mathbf{K})\). A GP is completely specified by its mean function $m(x)$ (defining its time trend) and covariance function $k(x, x')$ (defining temporal shape/smoothness). $x$ and $x'$ are points in the input space, denoted as \(f(x) \sim \mathcal{GP}(m(x),k(x,x'))\). GPs are versatile approaches to generating complex panel data distributions which might involve non-linear relationships and non-stationary patterns. The final set of processes/donor pools looks like
Kernel functions are used to define the covariance function in GPs. The first is the Radial Basis Function (RBF) Kernel: \(k(x, x^{\prime})= \text{exp}\left(- ||x - x^{\prime}||^2 \mathbin{/}2l^2 \right)\), to define the mean. The second is the Constant Kernel which scales the overall amplitude of the function \( k(x, x') = \kappa\). We use the Exponential Sine Squared Kernel to model the periodicity in the time series: \(k(x, x^{\prime}) = \text{exp}\left(-2 * \text{sin}^2\left(\left[\pi ||x - x^{\prime}|| \mathbin{/}p \right]\mathbin{/}l^2\right)\right)\). The white kernel models the white noise, denoted as: \( \sigma_n^2 \text{ if } x \equiv x^{\prime}, 0 \mbox{ otherwise.}\) We generate a synthetic high-dimensional dataset of 160 units. We generate two donor pools, each with 80 units per pool. In the treated pool, we select one unit to be treated. (ref) shows the treated pool of units (the blue pool) and a set of much noisier, irrelevant donors (the grey pool). We generate these data across 40 time periods, with the hypothetical intervention occurring at \(t=30\). For simplicity, we do not assign a treatment effect, or an ATT of 0 in this case. We report the pre and post-intervention Root-Mean Squared Errors for each algorithm, and discuss the donors selected by each approach.
The fPCA method finds 112 units to be the most relevant to the treated unit. The SC is constructed of only 5 units (all of which are from the relevant data generating process). The forward selection method performs much worse, selecting 23 donors, 9 of which are from the unrelated donor pool. OSC selects 11 units, 5 of which are irrelevant. fsPDA and the original synthetic control method suggest ATTs of 1.7 and -1.4 respectively. \verb|fPCA-SYNTH| has an ATT of 0.093 (the lowest in absolute value), even though it fits the worst of the methods(1.9 vs 1.4 and 0 for OSC and fsPDA, respectively). Following Amjad2018, we may use the post-intervention RMSE as a metric for forecasting quality. Precisely, we take the ratio of the pre and post RMSE for each method
The ratio for \verb|fPCA-SYNTH| is 0.92, for OSC is 3, and for forward selection is 566. Of these, fsPDA has the highest ratio because it clearly overfits the preintervention period, so as a result we have in poor out of sample performance. OSC has the next highest (also having the second best pre-intervention fit) ratio, suggesting that it also overfits to the pre-intervention period. The simulation results highlight the benefits of preprocessing and denoising: in situations where there are linear trends (as we use here), noise, and many donors to choose from, the OSC won't match on the units which are most representative of the target unit's data generating process. Having methods which are robust to outliers, trendy time series, and noise are of use to policymakers because they offer safeguards against overfitting biases which can draw the wrong conclusions.
First we revisit the West Germany example by Abadie2015, who study how West Germany reunifying with its Eastern counterpart affected the GDP per capita of West Germany. Before we continue though, we must lay some essential ground rules for this exercise: this study will not be a perfect replication of Abadie2015 simply because the exact outcome they use (as far as we are aware) is not available.Abadie2015 use “GDP per Capita (PPP, 2002 USD)" from OECD National Account statistics. As far as we know, the only data OECD keeps now for GDP per Capita is for 2011 dollars. In absence of the original dataset with all of the excluded donors, we use the newest data available from the Madison Project bolt2020maddison. The authors begin with 23 OECD nations, discarding Luxembourg and Iceland due to their small and `peculiar' economies. They also discard Turkey due to small GDP per Capita relative to the other OECD nations. Finally, the authors discard Ireland, Finland, Sweden, and Canada as they all experienced structural shocks to their economies in the 1990s, finally limiting their donor pool to 16 OECD nations. Abadie2015 write, however, their results are robust to the inclusion of Ireland, Finland, Sweden, and Canada. So, we replicate these analyses using the original 16 donors, while also adding in the 4 nations Abadie2015 excluded (we even add Turkey), giving us 21 donors.\footnote{Note that our results, with the exception of using all 23 donors, are also robust to including either Iceland or Luxembourg (in addition to Turkey).} We do not include predictors in the OSC estimator (the original paper used seven predictors). As before, we first estimate the counterfactual using \verb|fPCA-SYNTH| and OSC (we omit the forward selection algorithm for clarity). (ref) plots the observed trajectory of West Germany, and we overlay the four counterfactuals we estimate. In the plot, “OECD Donors" refers to the original 16 donors used by Abadie2015. The “Full OECD Donors" means Ireland, Finland, Sweden, Canada, and Turkey were included.
We can see that the weights selected by \verb|fPCA-SYNTH| are Australia, Netherlands and Norway [all of which were also selected into the donor pool in Bayani2021]. When we include all of the donors, it weighs Austria, Netherlands, Norway, United Kingdom, and United States. Notice that all of these with the exception of the United States were also included as donors in Bayani2021. The imputed counterfactual is also robust to the exclusion of the United States. OSC paints a different picture: substantively, the results also suggest that the GDP per capita would otherwise be higher, however, the effect sizes are double those of \verb|fPCA-SYNTH|. Furthermore, the OSC results are sensitive to the inclusion of the other irrelevant donors, even matching on Turkey (which Abadie2015 note has a particularly low GDP per Capita). While it's true that \verb|fPCA-SYNTH| includes the United States, RPCA algorithm used by \verb|fPCA-SYNTH| guards against the drastic changes in effect sizes.
The results, even in medium dimensions, illustrate the utility of a donor trimming algorithm. OSC is sensitive to changes in the donor pool, and \verb|fPCA-SYNTH| is less so, even getting the same effect size while including the United States, which was not included in the cluster in Bayani2021.
In July 2015, the city of Barcelona, Spain implemented a moratorium for the building of new hotels, ostensibly because over-tourism had deleterious social and economic impacts to the surrounding environment. hotelpaper study the counterfactual of Barcelona's average prices for hotel rooms, using daily pre-intervention data from January 2011 to June 30th 2015 to construct the counterfactual. Here, we estimate the causal impacts of Barcelona's hotel moratorium. It is tempting to follow Abadie2015 and restrict our donors to Mediterranean cities. However, Barcelona's average price is almost always higher than the average of Mediterranean donors, so it is possible (if not likely) that Barcelona could be better approximated by some non-Mediterranean donors. In other words, it is an arbitrary assumption we make that only Mediterranean cities would be the ideal donor set- given how cities like London and others are in the donor pool, we elect to use the universe of units instead of simply limiting our units to the Mediterranean. Our outcome is the mean price of hotel rooms in each city, normalized to 100. Here, \(j=0\) is Barcelona and \(\mathcal{N}_{0}=82 \) represents the universe of donors. While data from hotelpaper are in daily format, we use the average weekly price as our outcome. The observation window is from week 1 of January 2011 to week 31 (August 7th) 2017.
After applying fPCA to all pre-intervention weeks, the cluster for Barcelona retained 14 donors, or \(\widetilde{\mathcal{N}}_{0}=13\). Of this subset, 9 donors, or 64%, were Mediterranean cities. For the weights, 4 donor units were given weight. Of these, 3 were from the Mediterranean Sea donor pool. For \verb|fs|, 11 donors were chosen. Of those 11, 8 were Mediterranean. For OSC, 3 of the 7 donors were Mediterranean. Notice how these methods, particularly the PCA and fs method, appeal very nicely to intuition. Since hotelpaper already hypothesized Mediterranean Donors were more likely to be similar to Barcelona, the fact fPCA and forward-selection both choose donor pools with majority Mediterranean donors reassures us the methods are picking up on donors most related to Barcelona. Equally, the fact we get this result without explicitly adjusting for other predictors of room prices is doubly reassuring for the validity of both methods. In contrast, while OSC has good fit, most of its donors are not Mediterranean. Because SCM is justified on a latent variable model, the counterfactual must also match on unobserved confounding, instead of simply fitting to the pre-intervention period or covariate balance. Since we would imagine that at least many of the weighed units would be Mediterranean, this suggests the algorithmic approaches are more theoretically justified than OSC in setting where there are many irrelevant units. Note, that all methods used here agree in terms of the direction of the average treatment effects on the treated: the moratorium seems to have raised the prices of hotel rooms, on average.
Another important point when evaluating the estimators is robustness to changes in the intervention date, to see how well the SC tracks the observed series. In this case, we change the intervention date to week 27 of 2011, a full 4 years before the intervention. Here, \verb|fPCA-SYNTH| assigns two weights of the 19 units it selects as optimal, one of which is Mediterranean (OSC gets the same unit). OSC's weights in this case are actually 57% Mediterranean. Noticeably, the forward selection method has the most drastic change in the ATT, with the sign flipping completely, and departing far from the observed time series, meeting it only at the end of the time series. By comparison, \verb|fPCA-SYNTH|'s Post to Pre RMSE ratio changes the least compared to the other two methods. Its synthetic Barcelona tracks the observed Barcelona until the intervention date, even though the effect size differs from the first intervention date. The fact that its predictive capacity in terms of Post/Pre Ratio changes so little, being so close to the start of the time series, while retaining a similar proportion of theoretically relevant donors suggests that it is more robust to changes in the model than the OSC method.
Globally, sugar sweetened beverage (SSB) taxes, like other forms of excise taxes, have become popular economic policies which are intended to increase public health benefits without simply banning consumption of certain goods. Industry argues SSB taxes will have negative effects on employment, since now fewere people will be able to work in their warehouses/industries that are predicated on SSB revenue. So, the causal evaluation of SSB taxes on consumption of SSBs and secondary economic impacts has become popular among economists and policy analysts [e.g., BSCSSB]. Marinello2021 study the causal impact of San Francisco's SSB tax on various forms of employment, finding a very small impact on employment levels. We replicate their analyses using the estimators presented so far. The authors limit their donor pool to 62 urban counties around the United States, using SCM to evaluate the causal impact. They also have 62 months of pre-intervention data, with the intervention beginning on January 1, 2018. Precisely, our outcome is the total number of people employed in each city/county equivalent. The full dataset in Marinello2021 consists of 93 donors. So, we keep all 93 donors. Another key deviation from our analysis is that unlike Marinello2021 who use many covariate predictors, we use none.
The weights assigned by \verb|fPCA-SYNTH| are: 'Riverside County': 0.096, 'San Benito County': 4.264, 'Denver County': 0.049, 'Hillsborough County': 0.138, 'St. Tammany Parish': 1.924, 'Kings County': 0.171, 'Wake County': 0.122, 'Hays County': 1.005, 'Kendall County': 2.187. For context, Riverside and San Benito counties are located in Combined Statistical Areas in California. Wake, Denver, Hillsborough, and Kings counties are home to Raleigh/Durham, Denver, Tampa, and Seattle, all major major American cities. For forward selection, the OLS coefficients are 'Travis County': 8690.0, 'St. Tammany Parish': 0.788, 'Hartford County': 2.57, 'Baltimore city': -0.715, 'Charles County': 0.71, 'Placer County': 3.42, 'District of Columbia': 0.829, 'Essex County': -0.231. Regarding the OSC, we'd like to highlight a mistake we made during the writing of this paper: at first, we'd mistakenly coded the intervention as being in February 2018 instead of January. The pre-intervention RMSE for OSC in this case was 163.725 and the post-intervention error was 208.755. In other words, the line did not fit the pre-intervention period at all. When we corrected this, the pre-RMSE is now 17.867 and post-error is 37.213. While a marked improvement from the specification with the miscoded intervention, we highlight, as others have, how the OSC can be quite sensitive to small tweaks in the model, suggesting its estimates are less robust. Most notably, this also suggests the covariates the authors included originally (7 predictors, 5 county specific and 2 industry specific) can be quintessential to estimating counterfactuals with OSC. For a more realistic in-time placebo test, we report a 6 month placebo test, where we move the intervention to June of 2017 instead of 2018.
As we can see, all of the other methods (especially \verb|fPCA-SYNTH|) are quite robust to a longer backdate in the treatment, whereas OSC is not. Note that this does not mean the original analyses are wrong or incorrect. Our methods still obtain the main result, that the effect of the SSB tax on employment was quite small. We only echo the comments of others Bayani2021, Amjad2018 who remark that that the need for covariates to obtain convergence and good fit is a limitation of the original method. Particularly for empirical applications, this need for auxillary predictors can sometimes not be at all trivial. Thus, public policy analysts might prefer methods which both automate the selection of a donor pool while obtaining similar effects. Now we turn to the estimates with the actual intervention date. \verb|fsPDA| detects a -0.813% effect, and \verb|fPCA-SYNTH| finding a 0.065% average impact on employment. The fact that these methods (without predictors) reach qualitatively identical conclusions of the original paper, while at the same time selecting mostly large metropolitan areas as donors suggests that in many cases (such as Washington DC or San Benito County), donor trimming methods or additional advances to the standard SCM may be of use to policy analysts for their robust properties, particularly when there are multiple donors and little covariate information to use for the standard SCM.
In this study, we delve into the significance of algorithmic methods for selecting the optimal donor pool for SCM. The challenge of accurately identifying an appropriate control group stands at the forefront of causal inference, given that the control group's validity is the basis for the causal studies. The importance of donor pool selection is amplified within the SCM context due to the classic model's vulnerability to interpolation bias, particularly when we have many donors and/or trendy/noisy outcome time-series. Policy analysts oftentimes employ domain expertise or Euclidean distance measures to select donors: our research underscores the necessity of adopting modern machine learning approaches for this task. Put simply, theory is not a substitute for formal algorithmic analysis: if we think that most of the relevant donors should come from a given pool of units, then evidence from unsupervised algorithms should suggest this.
Particularly as the use of big data to solve policy problems that require causal inference grows grimmer2015we, so too should the causal methodologies used to do analysis. Particularly when analysts are faced with many potential donors, algorithmic selection of a donor pool is useful to researchers who use SCMs. We think our results should leave policy analysts and empirical researchers with a few major takeaways. First, researchers can use these methods (and others such as dynamic time-warping) when the research context provides lots of degrees of freedom with regard to donor pool selection, or if a particular set of SCM results is sensitive to the composition of the donor pool. So, if leave-one-out analyses are not robust, one viable method would be to use clustering or another method to extract the optimal donors and re-run original SCM analyses. Algorithmic donor trimming reduces the role of arbitrary research decisions.
Our work has limitations, too. Because policy research increasingly substitutes synthetic control for DiD designs cunninghamsub2023, having a better understanding of the exact conditions under which an algorithm selects a valid set of donors is useful. In this paper, we use Gaussian Processes and three empirical cases-- however, formal finite sample analyses of the asymptotic properties of k-means clustering or similar algorithms may expand this untapped area of the literature. Given recent finite sample studies of the \(k\)-means algorithms and SCMs zhang2022asymptotics,ASYMSCM,zeitler2023nonparametric, more formalized asymptotic study of algorithmic donor trimming methods may be warranted.
\printbibliography