Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
85,141 characters · 21 sections · 92 citation commands
The Augmented Synthetic Control Method
\thispagestyle{empty} \pagenumbering{gobble}
\pagenumbering{arabic}
\def\spacingset#1{ {#1}} \spacingset{1}
\onehalfspacing
The synthetic control method (SCM) is a popular approach for estimating the impact of a treatment on a single unit in panel data settings with a modest number of control units and with many pre-treatment periods Abadie2003,AbadieAlbertoDiamond2010,Abadie2015. The idea is to construct a weighted average of control units, known as a synthetic control, that matches the treated unit's pre-treatment outcomes. The estimated impact is then the difference in post-treatment outcomes between the treated unit and the synthetic control. SCM has been widely applied --- the main SCM papers have over 4,000 citations --- and has been called “arguably the most important innovation in the policy evaluation literature in the last 15 years” athey2017state.
A critical feature of the original proposal, not always followed in practice, is to use SCM only when the synthetic control's pre-treatment outcomes closely match the pre-treatment outcomes for the treated unit Abadie2015. When it is not possible to construct a synthetic control that fits pre-treatment outcomes well, the original papers advise against using SCM. At that point, researchers often fall back to linear regression. This allows better (often perfect) pre-treatment fit, but does so by applying negative weights to some control units, extrapolating outside the support of the data.
We propose the augmented synthetic control method (ASCM) as a middle ground in settings where excellent pre-treatment fit using SCM alone is not feasible. Analogous to bias correction for inexact matching rubin1973adjust, abadie2011bias, ASCM begins with the original SCM estimate, uses an outcome model to estimate the bias due to imperfect pre-treatment fit, and then uses this to de-bias the estimate. If pre-treatment fit is good, the estimated bias will be small, and the SCM and ASCM estimates will be similar. Otherwise, the estimates will diverge, and ASCM will rely more heavily on extrapolation.
Our primary proposal is to augment SCM with a ridge regression model, which we call Ridge ASCM. We show that, like SCM, the Ridge ASCM estimator can be written as a weighted average of the control unit outcomes. We also show that Ridge ASCM weights can be written as the solution to a modified synthetic controls problem, targeting the same imbalance metric as traditional SCM. However, where SCM weights are always non-negative, Ridge ASCM admits negative weights, using extrapolation to improve pre-treatment fit. The regularization parameter in Ridge ASCM directly parameterizes the level of extrapolation by penalizing the distance from SCM weights. By contrast, (ridge) regression alone, which can also be written as a modified synthetic controls problem with possibly negative weights, allows for arbitrary extrapolation and possibly unchecked extrapolation bias.
We relate Ridge ASCM's improved pre-treatment fit to a finite sample bound on estimation error under several data generating processes, including a simple linear model and the linear factor model often invoked in this setting AbadieAlbertoDiamond2010. Under a linear model, improving pre-treatment fit directly reduces bias, and the Ridge ASCM penalty term negotiates a bias-variance trade-off. Under a latent factor model, improving pre-treatment fit again reduces bias, though there is now a risk of over-fitting. The penalty term also directly parameterizes this trade-off. Thus, choosing the hyperparameter will be important for practice; we propose a cross-validation procedure in Section (ref).
Finally, we also describe how the Augmented SCM approach can be extended to incorporate auxiliary covariates other than pre-treatment outcomes. We first propose to include the auxiliary covariates in parallel to the lagged outcomes in both the SCM and outcome models. We also propose an alternative when there relatively few covariates, extending a suggestion from Doudchenko2017: first residualize both the pre- and post-treatment outcomes against the auxiliary covariates, then fit Ridge ASCM on the residualized outcome series. We show that this controls the estimation error under a linear factor model with auxiliary covariates.
We demonstrate the properties of Augmented SCM both via calibrated simulation studies and by using it to examine the effect of an aggressive tax cut in Kansas in 2012 on economic output, finding a substantial negative effect. Overall, we see large gains from ASCM relative to alternative estimators, especially under model mis-specification, in terms of both bias and root mean squared error. We implement the proposed methodology in the augsynth package for R, available at \href{https://github.com/ebenmichael/augsynth}{https://github.com/ebenmichael/augsynth}.
The paper proceeds as follows. Section (ref) briefly reviews related work. Section (ref) introduces notation and the SCM estimator. Section (ref) gives an overview of Augmented SCM. Section (ref) gives key numerical results for Ridge ASCM. Section (ref) bounds the estimation error under a linear model and a linear factor model, the standard setting for SCM. Section (ref) extends the ASCM framework to incorporate auxiliary covariates and alternative outcome models. Section (ref) reports on extensive simulation studies as well as the application to the Kansas tax cuts. Finally, Section (ref) discusses some possible directions for further research. The appendix includes all of the proofs, as well as additional derivations and technical discussion, including a discussion of inference.
SCM was introduced by Abadie2003 and AbadieAlbertoDiamond2010, Abadie2015 and is the subject of an extensive methodological literature; see abadie2019synthreview and samartsidis2019assessing for recent reviews. We briefly highlight some relevant aspects of this literature.
A first set of papers adapts the original SCM proposal to allow for more robust estimation while retaining the simplex constraint on the weights. Robbins2017, Doudchenko2017, Abadie_LHour, minard2018dispersion incorporate a penalty on the weights into the SCM optimization problem, building on a suggestion in Abadie2015. gobillon2016regional and hazlett2018trajectory explore dimension reduction strategies and other data transformations that can improve the performance of the subsequent estimator. See also bilinski2020goldilocks, who propose a cross-validation procedure for selecting the number of lagged outcomes to include.
A second set of papers relaxes various constraints imposed in the original SCM problem. In particular, Doudchenko2017 relax the SCM restriction that control unit weights be non-negative, arguing that there are many settings in which negative weights would be desirable. amjad2018robust propose an interesting variant that combines negative weights with a pre-processing step. powell2018imperfect instead allows for extrapolation via a Frisch-Waugh-Lovell-style projection, which similarly generalizes the typical SCM setting. Doudchenko2017 and ferman2018revisiting both propose to incorporate an intercept into the SCM problem, which we discuss in Section (ref). There have also been several other proposals to reduce bias in SCM, developed independently and contemporaneously with ours. Abadie_LHour also propose bias correcting SCM using regression, but focus on bias due to interpolation rather than poor pre-treatment fit. kellogg2020combining propose using a weighted average of SCM and matching, trading off interpolation and extrapolation bias. Finally, arkhangelsky2018synthetic propose the Synthetic Difference-in-Differences estimator, which can be seen as a special case of our proposal with a constrained outcome regression.
Finally, there have been several recent proposals to use outcome modeling rather than SCM-style weighting in this setting. These include the matrix completion method in athey2017mcp, the generalized synthetic control method in Xu2017, and the combined approaches in hsiao2018panel. We explore the performance of select methods, both in isolation and in combination with SCM, in Section (ref).
We consider the canonical SCM panel data setting with $i = 1, \ldots, N$ units observed for $t = 1, \ldots, T$ time periods; for the theoretical discussion below, we will consider both $N$ and $T$ to be fixed. Let $W_i$ be an indicator that unit $i$ is treated at time $T_0<T$ where units with $W_i=0$ never receive the treatment. We restrict our attention to the case where a single unit receives treatment, and follow the convention that this is the first one, $W_1 = 1$; see benmichael2019multisynth for an extension to multiple treated units. The remaining $N_0=N-1$ units are possible controls, often referred to as donor units in the SCM context. To simplify notation, we limit to one post-treatment observation, $T=T_0+1$, though our results are easily extended to larger $T$.
We adopt the potential outcomes framework neyman1923, rubin1974 and invoke SUTVA, which assumes a well-defined treatment and excludes interference between units rubin1980; the potential outcomes for unit $i$ in period $t$ under control and treatment are $Y_{it}(0)$ and $Y_{it}(1)$, respectively. The observed outcomes are then:
We next assume that control potential outcomes are generated as a fixed component $m_{it}$ plus mean zero, additive noise $\varepsilon_{it}$ drawn from some distribution $P(\cdot)$, $$Y_{it}(0) = m_{it} + \varepsilon_{it}.$$ The treated potential outcome is then $Y_{it}(1) = Y_{it}(0) + \tau_{it}$, where the treatment effects $\tau_{it}$ are the key estimands, and are fixed parameters. The treatment effect of interest is thus $\tau = \tau_{1T} = Y_{1T}(1) - Y_{1T}(0)$.
In Section (ref), we consider special cases where $m_{it}$ is a linear function of lagged outcomes or where $m_{it}$ is a linear factor model; in the Appendix, we also consider the case where $m_{it}$ is a linear model with Lipshitz deviations from linearity. We make two assumptions about the distribution of the noise terms $\varepsilon_{it}$. First, we assume that treatment assignment $W_i$ is ignorable given $m_{it}$; specifically that the noise terms in the post-treatment time periods $\bm{\varepsilon}_T = (\varepsilon_{1T},\ldots,\varepsilon_{NT})$ are mean-zero and uncorrelated with treatment assignment,
where the expectation is taken with respect to the noise term at the post-treatment time $T$, $\bm{\varepsilon}_T$. As a result, the noise terms for the treated and control units do not systematically deviate from each other. We discuss this in more detail in the context of our application in Section (ref). Second, for the theoretical results in Sections (ref) and (ref), we assume that the error terms $\varepsilon_{it}$ are independent (across units and over time) sub-Gaussian random variables with scale parameter $\sigma$. See, for example, chernozhukov2017exact for an extended discussion of this general setup in panel data settings.
To emphasize that pre-treatment outcomes serve as covariates in SCM, we use $X_{it}$, for $t\leq T_0$, to represent pre-treatment outcomes; we use the terms pre-treatment fit and covariate balance interchangeably. With some abuse of notation, we use $\bm{X}_{0\cdot}$ to represent the $N_0$-by-$T_0$ matrix of control unit pre-treatment outcomes and $\bm{Y}_{0T}$ for the $N_0$-vector of control unit outcomes in period $T$. With only one treated unit, $Y_{1T}$ is a scalar, and $\bm{X}_{1\cdot}$ is a $T_0$-row vector of treated unit pre-treatment outcomes. The data structure is then:
The Synthetic Control Method imputes the missing potential outcome for the treated unit, $Y_{1T}(0)$, as a weighted average of the control outcomes, $\bm{Y}_{0T}'\bm{\gamma}$ Abadie2003, AbadieAlbertoDiamond2010, Abadie2015. Weights are chosen to balance pre-treatment outcomes and possibly other covariates. We consider a version of SCM that chooses weights $\bm{\gamma}$ as a solution to the constrained optimization problem:
where the constraints limit $\bm{\gamma}$ to the simplex $\Delta^{N_0} = \{\bm{\gamma} \in \ensuremath{\mathbb{R}}^{N_0} \mid \gamma_i \geq 0 ~\forall i, ~\sum_i\gamma_i = 1\}$, and where $\bm{V_x} \in \ensuremath{\mathbb{R}}^{T_0 \times T_0}$ is a symmetric importance matrix and $\|\bm{V_x}^{1/2} (\bm{X}_{1\cdot} - \bm{X}_{0\cdot}'\gamma)\|^2_2 \equiv (\bm{X}_{1\cdot} - \bm{X}_{0\cdot}'\gamma)' \bm{V_x} (\bm{X}_{1\cdot} - \bm{X}_{0\cdot}'\bm{\gamma})$ is the 2-norm on $\mathbb{R}^{T_0}$ after applying $\bm{V_x}^{1/2}$ as a linear transformation. To simplify the exposition and notation below, we will generally take $\bm{V_x}$ to be the identity matrix. The simplex constraint in Equation (ref) ensures that the weights will be sparse and non-negative; AbadieAlbertoDiamond2010, Abadie2015 argue that enforcing this constraint is important for preserving interpretability.
Equation (ref) modifies the original SCM proposal in two ways. First, Equation (ref) excludes auxiliary covariates; we re-introduce them in Section (ref). Second, Equation (ref) penalizes the dispersion of the weights with hyperparameter $\zeta \geq 0$, following a suggestion in Abadie2015. Possible penalization functions include an elastic net penalty Doudchenko2017, an entropy penalty Robbins2017, and a penalty based on a measure of pairwise distance Abadie_LHour. The choice of penalty is less central when weights are constrained to be on the simplex, but becomes more important below when we relax this constraint Doudchenko2017.
We can view the SCM optimization problem in Equation (ref) as an approximate balancing weights estimator Zubizarreta2015,hirshberg2019minimax. As with all balancing estimators, a central question is what quantity to balance. Following the recent methodological literature Doudchenko2017, ferman2018revisiting, Equation (ref) directly optimizes for the pre-treatment fit, minimizing the (possibly weighted) imbalance of pre-treatment outcomes between the treated unit and the weighted control mean. In Section (ref), we argue that this a natural quantity to target under both linearity and a latent factor model. Many choices are possible, however, and we can easily modify Equation (ref) to balance other summary measures and functions of the lagged outcomes; see, for example, gobillon2016regional, amjad2018robust, and hazlett2018trajectory.
When the treated unit's vector of lagged outcomes, $\bm{X}_{1\cdot}$, is inside the convex hull of the control units' lagged outcomes, $\bm{X}_{0\cdot}$, the SCM weights in Equation (ref) achieve perfect pre-treatment fit, and the resulting estimator has many attractive properties, including a bias bound established by AbadieAlbertoDiamond2010. Due to the curse of dimensionality, however, achieving perfect (or nearly perfect) pre-treatment fit is not always feasible with weights constrained to be on the simplex ferman2018revisiting. When “the pre-treatment fit is poor or the number of pre-treatment periods is small,” Abadie2015 recommend against using SCM. Even if the pre-treatment fit is excellent, AbadieAlbertoDiamond2010, Abadie2015 propose extensive placebo checks to ensure that SCM weights do not overfit to noise. Thus, the conditional nature of the analysis is critical to deploying SCM, excluding many practical settings. Our proposal enables the use of (a modified) SCM approach in many of the cases where SCM alone is infeasible.
We now show how to modify the SCM approach to adjust for poor pre-treatment fit. Let $\hat{m}_{iT}$ be an estimator for the post-treatment control potential outcome $Y_{iT}(0)$. The Augmented SCM (ASCM) estimator for $Y_{1T}(0)$ is:
where weights $\hat{\gamma}_i^\text{scm}$ are the SCM weights defined above. Standard SCM is a special case, where $\hat{m}_{iT}$ is a constant. We will largely focus on estimators that are a function of pre-treatment outcomes, $\hat{m}_{iT} \equiv \hat{m}(\bm{X}_i)$, where $\hat{m}:\ensuremath{\mathbb{R}}^{T_0} \to \ensuremath{\mathbb{R}}$.
Equations (ref) and (ref), while equivalent, highlight two distinct motivations for ASCM. Equation (ref) directly corrects the SCM estimate, $\sum \hat{\gamma}_i^\text{scm} Y_{iT}$, by the imbalance in a particular function of the pre-treatment outcomes $\hat{m}(\cdot)$. Intuitively, since $\hat{m}$ estimates the post-treatment outcome, we can view this as an estimate of the bias due to imbalance, analogous to bias correction for inexact matching rubin1973adjust, abadie2011bias. In this form, we can see that SCM and ASCM estimates will be similar if the estimated bias is small, as measured by imbalance in $\hat{m}(\cdot)$. In independent work, Abadie_LHour also consider a bias-corrected estimator of this form, though their paper focuses on reducing bias due to interpolation rather than extrapolation.
Equation (ref), by contrast, is analogous to standard doubly robust estimation robins1994estimation, which begins with the outcome model but then re-weights to balance residuals. This is also comparable in form to the generalized regression estimator in survey sampling Cassel1976, Breidt2017, which has been adapted to the causal inference setting by, among others, Athey2016 and Hirshberg2018. We discuss a connection to inverse propensity score weighting in Appendix (ref).
While this setup is general, the choice of estimator $\hat{m}$ is important both for understanding the procedure's properties and for practical performance. We give a brief overview of two special cases: (1) when $\hat{m}$ is linear in the pre-treatment outcomes; and (2) when $\hat{m}$ is linear in the comparison units. Ridge regression is an important example that is linear in both pre-treatment outcomes and comparison units; we explore this estimator further in Sections (ref) and (ref).
First, consider a model that is linear in pre-treatment outcomes, $\hat{m}(\bm{X}) = \hat{\eta}_0 + \bm{\hat{\eta}} \cdot \bm{X}$. The augmented estimator (ref) is then:
Pre-treatment periods that are more predictive of the post-treatment outcome will have larger (absolute) regression coefficients and so imbalance in these periods will lead to a larger adjustment. Thus, even if we do not a priori prioritize balance in any particular pre-treatment time periods (via the choice of $\bm{V_x}$), the linear model augmentation will adjust for the time periods that are empirically more predictive of the post-treatment outcome. As we show in Section (ref), the ridge-regularized linear model is an important special case in which the resulting augmented estimator is itself a penalized synthetic control estimator. This allows for a more direct analysis of the role of bias correction.
Second, consider an outcome model that is a linear combination of comparison units, $\hat{m}(\bm{X}) = \sum_{W_i = 0}\hat{\alpha}_i(\bm{X}) Y_{iT}$, for some weighting function $\hat{\alpha}:\ensuremath{\mathbb{R}}^{T_0} \to \ensuremath{\mathbb{R}}^{N_0}$. Examples include $k$-nearest neighbor matching and kernel weighting as well as other “vertical” regression approaches athey2017mcp. The augmented estimator (ref) is itself a weighting estimator that adjusts the SCM weights:\footnote{We thank an anonymous reviewer for suggesting this presentation.}
Here the adjustment term for unit $i$, $\hat{\gamma}^\text{adj}_i$, is the imbalance in a unit $i$-specific transformation of the lagged outcomes that depends on the weighting function $\alpha(\cdot)$. While $\hat{\gamma}^\text{scm}$ are constrained to be on the simplex, the form of $\hat{\gamma}^\text{adj}$ makes clear that the overall weights can be negative.
There are many special cases to consider, though we do not explore them in depth. One is the linear-in-lagged-outcomes model with equal coefficients, $\hat{\eta}_t = \frac{1}{T_0}$, which estimates a fixed-effects outcome model as $\hat{m}(\bm{X}_i) = \bar{X}_i$. The corresponding treatment effect estimate adjusts for imbalance in all pre-treatment time periods equally, and yields a weighted difference-in-differences estimator:
An augmented estimator of this form has appeared as the de-meaned or intercept shift SCM Doudchenko2017, ferman2018revisiting.\footnote{In these proposals, the SCM weights balance the residual outcomes $X_{it}-\bar{X}_i$ rather than the raw outcomes $X_{it}$. We further consider balancing residuals in Section (ref).} See also arkhangelsky2018synthetic, who extend this to weight across both units and time.
In Section (ref) we conduct a simulation study to inspect the performance of a range of estimators, including other penalized linear models, such as the LASSO, flexible machine learning models, such as random forests, and panel data methods, such as fixed effects models and low-rank matrix completion methods Xu2017, athey2017mcp.
We now inspect the algorithmic and numerical properties for the special case where $\hat{m}(\bm{X}_i)$ is estimated via a ridge-regularized linear model, which we refer to as Ridge Augmented SCM (Ridge ASCM). With Ridge ASCM, the estimator for the post-treatment outcome is $\hat{m}(\bm{X}_i)=\hat{\eta}_0^\text{ridge} + \bm{X}_{i}^{\prime}\bm{\hat{\eta}}^\text{ridge}$, where $\hat{\eta}_0^\text{ridge}$ and $\bm{\hat{\eta}}^\text{ridge}$ are the coefficients of a ridge regression of control post-treatment outcomes $\bm{Y}_{0T}$ on centered pre-treatment outcomes $\bm{X}_{0\cdot}$ with penalty hyper-parameter $\lambda^\text{ridge}$:\footnote{Similar to the synthetic controls problem, we can regularize time periods differently with a generalized ridge penalty $\bm{\eta}' \bm{\Lambda} \bm{\eta}$ using an importance matrix $\bm{\Lambda}$. Following the typical case with diagonal elements, the generalized ridge penalty reduces to separate regularization on each time period.}
The Ridge Augmented SCM estimator is then:
We first show that Ridge ASCM is a linear weighting estimator as in Equation (ref). Unlike augmenting with other linear weighting estimators, when augmenting with ridge regression the implied weights are themselves the solution to a penalized synthetic control problem, as in Equation (ref). Using this representation, we show that when the treated unit lies outside the convex hull of the control units, Ridge ASCM improves the pre-treatment fit relative to SCM alone by allowing for negative weights and extrapolating away from the convex hull.
Allowing for negative weights is an important departure from the original SCM proposal, which constrains weights to be on the simplex. Ridge regression alone also allows for negative weights, and may have negative weights even when the treated unit is inside of the convex hull. In contrast, Ridge ASCM directly penalizes distance from the sparse, non-negative SCM weights, controlling the amount of extrapolation by the choice of $\lambda^{\text{ridge}}$, and only resorts to negative weights if the treated unit is outside of the convex hull.
We now express both Ridge ASCM and ridge regression alone as special cases of the penalized SCM problem in Equation (ref). The Ridge ASCM estimate of the counterfactual is the solution to (ref), replacing the simplex constraint with a penalty $f(\gamma_i) = \left(\gamma_i - \hat{\gamma}^\text{scm}_i\right)^2$ that penalizes deviations from the SCM weights.
When the treated unit is in the convex hull of the control units --- so the SCM weights exactly balance the lagged outcomes --- the Ridge ASCM and SCM weights are identical. When SCM weights do not achieve exact balance, the Ridge ASCM solution will use negative weights to extrapolate from the convex hull of the control units. The amount of extrapolation is determined both by the amount of imbalance and by the hyperparameter $\lambda^\text{ridge}$. When SCM yields good pre-treatment fit or when $\lambda^\text{ridge}$ is large, the adjustment term will be small and $\bm{\hat{\gamma}}^\text{aug}$ will remain close to the SCM weights.
We can similarly characterize ridge regression alone as a solution to a penalized SCM problem where the penalty term, $f(\gamma_i) = \left(\gamma_i - \frac{1}{N_0}\right)^2$, penalizes the variance of the weights. Other penalized linear models, such as the LASSO or elastic net, do not have this same representation as a penalized SCM estimator.
For ridge regression alone, the hyperparameter $\lambda^{\text{ridge}}$ controls the variance of the weights rather than the degree of extrapolation from the simplex. Thus, in order to reduce variance, the ridge regression weights might still be negative even if the treated unit is inside of the convex hull and SCM achieves perfect fit.
Figure (ref) visualizes these results in two dimensions. Figure (ref) shows the distribution of control units and the treated unit outside the convex hull, along with the weighted average of control units using the ridge and Ridge ASCM weights, varying $\lambda^\text{ridge}$. We see that ridge regression alone begins (for large $\lambda^\text{ridge}$) at the center of the control units, while Ridge ASCM begins at the SCM solution; both move smoothly towards an exact fit solution as $\lambda^\text{ridge}$ is reduced. Figure (ref) shows the distance of the ridge and Ridge ASCM weights from the simplex as $\lambda^\text{ridge}$ varies. Ridge ASCM weights begin at the SCM solution, which is on the boundary of the simplex, then extrapolate outside the convex hull. In contrast, ridge regression weights begin in the center of the simplex (i.e., uniform weights), but then leave the simplex (i.e., some negative weights) before the corresponding weighted average is outside of the convex hull. Over this range, marked as red in Figure (ref), it is possible to achieve the same level of balance with non-negative weights, but ridge regression uses negative weights in order to reduce the variance. Eventually, as $\lambda^{\text{ridge}} \to 0$, both ridge and Ridge ASCM use negative weights to achieve perfect balance; the weight vectors are different, however, with the Ridge ASCM vector closer to the simplex.
As this example makes clear, both ridge and Ridge ASCM extrapolate from the support of the data to improve pre-treatment fit relative to SCM alone. Ridge ASCM, however, is equivalent to SCM weights when SCM achieves exact pre-treatment fit. When achieving excellent pre-treatment fit with SCM is possible, Abadie2015 argue that we should prefer SCM weights over possibly negative weights: a slight balance improvement is not worth the extrapolation and the loss of interpretability. In this case, the Ridge ASCM weights will be close to the simplex, while the ridge regression weights may be quite far away. When this is not possible, however, and SCM has poor fit, some degree of extrapolation is critical; Ridge ASCM allows the researcher to directly penalize the amount of extrapolation in these cases.\footnote{See king2006dangers for a discussion of extrapolation in constructing counterfactuals. As they note: “If we learn that a counterfactual question involves extrapolation, we still might wish to proceed if the question is sufficiently important, but we would be aware of how much more model dependent our answers would be.”}
Just as the hyper-parameter $\lambda^\text{ridge}$ parameterizes the level of extrapolation, it also parameterizes the level of improvement in pre-treatment fit over the SCM solution. Because we are removing the non-negativity constraint and allowing for extrapolation outside of the convex hull, the pre-treatment fit from Ridge ASCM will be at least as good as the pre-treatment fit from SCM alone, i.e., $\|\bm{X}_{1} - \bm{X}_{0\cdot}'\bm{\hat{\gamma}}^\text{aug}\|_2 \leq \|\bm{X}_{1} - \bm{X}_{0\cdot}'\bm{\hat{\gamma}}^\text{scm}\|_2$. We can exactly characterize the pre-treatment fit of Ridge ASCM using the singular value decomposition of the matrix of control outcomes.
From Equation (ref), we see that the pre-treatment imbalance for Ridge ASCM weights is smaller than that of SCM weights by at least a factor of $\frac{\lambda}{d_m^2 + \lambda}$. Thus, Ridge ASCM will achieve strictly better pre-treatment fit than SCM alone, except in corner cases where pre-treatment fit will be equal. For example, these will be equal when the pre-treatment SCM residual $\bm{X}_1 - \bm{X}_{0\cdot}'\hat{\bm{\gamma}}^\text{scm}$ is orthogonal to the lagged outcomes of the control units $\bm{X}_{0\cdot}$. From Equation (ref), we see that the relationship for pre-treatment imbalance fit between SCM and ridge regression is less clear; ridge regression penalizes deviations from uniformity, rather than deviations from SCM weights.
We now relate Ridge ASCM's improved pre-treatment fit to improved estimation error under several data generating processes. We first consider the case where the post-treatment outcome is linear in the pre-treatment outcomes, and then extend to the canonical linear factor model considered by AbadieAlbertoDiamond2010. In the appendix, we also consider a more general formulation, including when the outcome model is approximately linear, with Lipshitz deviations from linearity.
Under a linear model, improving pre-treatment fit directly reduces bias, and the Ridge ASCM penalty term negotiates a bias-variance trade-off. Under a latent factor model, improving pre-treatment fit again reduces bias, though there is now a risk of over-fitting. The penalty term also directly parameterizes this trade-off. Thus, choosing the hyper-parameter $\lambda^\text{ridge}$ is important in practice. In Section (ref), we describe a cross-validation hyper-parameter selection procedure that builds on the in-time placebo check in Abadie2015. In Section (ref) we explore these trade-offs through simulations, finding substantial gains to augmentation via ridge regression in a variety of settings.
We first illustrate the key balancing idea in the simple case where the post-treatment outcome is a linear combination of lagged outcomes plus additive noise:
A special case of this setup is an auto-regressive process of order $K \leq T_0$. Here we consider the pre-treatment outcomes $\bm{X}_i$ fixed and so the randomness in the post-treatment outcome $Y_{iT}(0)$ is due to the noise term $\varepsilon_{iT}$. As in Section (ref), we assume that the $N$ units' noise terms are independent, mean zero sub-Gaussian random variables with scale parameter $\sigma$ that are uncorrelated with treatment assignment $W_i$.
We consider a generic weighting estimator with weights $\bm{\hat{\gamma}}$ that are independent of the post-treatment outcomes $Y_{1T},\ldots,Y_{NT}$; both SCM and Ridge ASCM take this form. Under linearity, the difference between the counterfactual outcome $Y_{1T}(0)$ and the weighting estimator $\hat{Y}_{1T}(0)$ decomposes into: (1) systemic error due to imbalance in the lagged outcomes $\bm{X}$, and (2) idiosyncratic error due to the noise in the post-treatment period:
With this setup, a weighting estimator that exactly balances the lagged outcomes $\bm{X}$ will eliminate all systematic error. Furthermore, if $\bm{\beta}$ is sparse, then it suffices to balance only the lagged outcomes with non-zero coefficients; for example, under an AR($K$) process, $(\beta_1,\ldots,\beta_{T_0 - K - 1}) = 0$, it is sufficient to balance only the first $K$ lags.
If the weighting estimator does not perfectly balance the pre-treatment outcomes $\bm{X}$, there will be a systematic component of the error, with the magnitude depending on the imbalance. Below we construct a finite sample error bound for Ridge ASCM (and for SCM, the special case with $\lambda^\text{ridge}=\infty$). This bound on the estimation error holds with high probability over the noise in the post-treatment period $\bm{\varepsilon}_T$.
Proposition (ref) shows the finite sample error of Ridge ASCM weights is controlled by the imbalance in the lagged outcomes and the $L^2$ norm of the weights; Lemma (ref) in the Appendix gives a deterministic bound for $\|\bm{\hat{\gamma}}^{\text{aug}}\|_2$. See Athey2016 for analogous results on balancing weights in high dimensional cross-sectional settings.
In the special case that SCM weights have perfect pre-treatment fit, ASCM and SCM weights will be equivalent, and the estimation error will only be due to variance of the weights and post-treatment noise. When SCM weights do not achieve perfect pre-treatment fit, Ridge ASCM with finite $\lambda < \infty$ extrapolates outside the convex hull, improving pre-treatment fit and thus reducing bias. This is subject to the usual bias-variance trade-off: The second term in (ref) is increasing in the $L^2$ norm of the weights, which will generally be larger for ASCM than for SCM. The hyperparameter $\lambda$ directly negotiates this trade off.
We now consider the case where control potential outcomes are generated according to a linear factor model. Similar to the linear case above, when pre-treatment fit is poor, improving the pre-treatment fit by extrapolating away from the convex hull reduces the bias, with a possible bias-variance tradeoff. Unlike in the linear setting, however, there is additional possible error due to balancing noisy pre-treatment outcomes rather than the underlying factors. This raises the risk of over-fitting, though it can be reduced by appropriate choice of the extrapolation penalty. Specifically, in settings where the noise is high, ASCM can increase error relative to SCM alone. SCM limits over-fitting in this high-noise case by constraining weights to the simplex, although SCM alone is likely to perform poorly here regardless.
Following the setup in AbadieAlbertoDiamond2010, we assume that there are $J$ unknown, latent time-varying factors $\bm{\mu}_t=\{\mu_{jt}\} \in \ensuremath{\mathbb{R}}^{T}$, $j=1,\dots,J$, with $\max_{jt} |\mu_{jt}| \leq M$, where $J$ will typically be small relative to $N$ and $T_0$. In addition, each unit has a vector of unknown factor loadings $\bm{\phi}_i \in \ensuremath{\mathbb{R}}^J$. We consider both the time-varying factors $\bm{\mu}_t$ and the unit-varying factor loadings $\bm{\phi}_i$ to be non-random quantities and will consider a fixed $N$ and $T_0$. In this setup, the control potential outcomes are weighted averages of these factors plus additive noise:
where again the randomness in $Y_{it}(0)$ is only due to the noise term $\varepsilon_{it}$. We assume that $\varepsilon_{it}$ are independent, mean zero sub-Gaussian random variables with scale parameter $\sigma$ that are uncorrelated with treatment assignment $W_i$. Slightly abusing notation, we collect the pre-intervention factors into a matrix $\bm{\mu} \in \ensuremath{\mathbb{R}}^{T_0 \times J}$, where the $t$\textsuperscript{th} row of $\bm{\mu}$ contains the factor values at time $t$, $\bm{\mu}_t^{\prime}$. Following bai2009panel and Xu2017, we further assume that the factors are orthogonal and normalized, i.e., that $\frac{1}{T_0}\bm{\mu}'\bm{\mu} = \bm{I}_{J}$.
Under this model, the finite-sample error of a weighting estimator depends on the imbalance in the latent $\bm{\phi}$ and a noise term due to the noise at time $T$:
Balancing the observed pre-treatment outcomes $\bm{X}$ will not necessarily balance the latent factor loadings $\bm{\phi}$. Following AbadieAlbertoDiamond2010, we show in the appendix that, under Equation (ref), we can decompose the imbalance term as:
where $\bm{\varepsilon}_{i(1:T_0)} = (\varepsilon_{i1},\ldots,\varepsilon_{iT_0})$ is the vector of pre-treatment noise terms for unit $i$. The first term is the imbalance of observed lagged outcomes and the second term is an approximation error arising from the latent factor structure. With $\sigma=0$ and $J$ small, the approximation error is zero, and it is possible to express $Y_{iT}(0)$ as a linear combination of $Y_{it}(0)$, $t=1,\dots,T_0-1$, recovering the linear model (ref). However, with $\sigma>0$ we cannot write the period-$T$ outcome as a linear combination of earlier outcomes plus independent, additive error.
With this setup, we can bound the finite-sample error in Equation (ref) for Ridge ASCM weights (and for SCM weights as a special case). This bound is with high probability over the noise in all time periods $\varepsilon_{it}$, and accounts for the noise in the pre- and post-treatment outcomes separately.
Theorem (ref) shows that, relative to the linear case in Proposition (ref), there is an additional source of error under a latent factor model: approximation error due to balancing lagged outcomes rather than balancing underlying factors. In particular, it is now possible that a control unit only receives a large weight because of idiosyncratic noise, rather than because of similarity in the underlying factors. See arkhangelsky2018synthetic and Ferman2019 for asymptotic analogues of this finite sample bound.
In the special case where SCM achieves perfect pre-treatment fit, considered by AbadieAlbertoDiamond2010, the ASCM and SCM weights are equivalent and the error is only due to post-treatment noise and the approximation error. The bound in Theorem (ref) accounts for the worst case scenario where the control unit with the largest weight is only similar to the treated unit due to idiosyncratic noise. The approximation error, and thus the bias, converges to zero in probability as $T_0 \to \infty$ under suitable conditions on the factor loadings $\bm{\mu}_t$ ferman2018revisiting. Intuitively, as we observe more $X_{it}$ --- and can exactly balance each one --- we are better able to match on the index $\bm{\phi}_i \cdot \bm{\mu}_t$ and, as a result, on the underlying factor loadings.\footnote{We show in the supplementary material that with dependent errors the probability of the worst-case error additionally scales with the maximum eigenvalue of the covariance matrix. Dependence leads to a more complicated error structure overall; we leave a thorough analysis of this to future work. }
Without exact balance, Theorem (ref) shows that a long pre-period may not be enough to control the error due to imbalance. This emphasizes the critical role that conditioning on excellent pre-treatment fit plays in ensuring that SCM alone yields reasonable estimates of the counterfactual.
When perfect pre-treatment fit is not feasible with SCM, Ridge ASCM with $\lambda < \infty$ will extrapolate outside the convex hull, where the hyper-parameter $\lambda$ controls the amount of extrapolation. As in the linear case, extrapolation reduces the error due to imbalance in lagged outcomes. Unlike in the linear case, however, this could possibly lead to over-fitting to the noisy lagged outcomes. While some extrapolation can be helpful on net, the optimal amount will depend on the synthetic control fit and the amount of noise. Figure (ref) illustrates this using SCM weights from the empirical example we discuss in Section (ref), where pre-treatment fit is relatively poor. For each value of $\sigma$, the figure plots the sum of the imbalance, SCM approximation error, and excess approximation error terms in the bound in Theorem (ref). At each noise level, a small amount of extrapolation leads to a smaller error bound, but as $\lambda$ shrinks there is a point where further extrapolation leads to over-fitting and eventually to a worse error bound than without extrapolation. The risk of overfitting is greater when the noise is particularly large, though even here a sufficiently regularized ASCM estimate has lower error bound than SCM alone (represented as the $\lambda \to \infty$ bound). When noise is less extreme, the benefits of augmentation are larger and the optimal amount of regularization shrinks.
It is worth noting that Theorem (ref) gives a worst-case bound. In Section (ref) we inspect the typical performance of the Ridge ASCM estimator via extensive simulation studies and find that gains to pre-treatment fit through augmentation outweigh increased approximation error in a range of practical settings, including when noise is very large.
We propose a cross-validation approach for selecting $\lambda$ inspired by the in-time placebo check proposed by Abadie2015. Let $\hat{Y}_{1k}^{(-t)} = \sum_{W_i=0}\hat{\gamma}^\text{aug}_{i(-t)} Y_{ik}$ be the estimate of $Y_{1k}$ where time period $t$ is excluded from fitting the estimator in (ref). Abadie2015 propose to compare the difference $Y_{1t} - \hat{Y}_{1t}^{(-t)}$ for some $t \leq T_0$ as a placebo check. We can extend this idea to compute the leave-one-out cross validation MSE over time periods:
We can then choose $\lambda$ to minimize $CV(\lambda)$ or follow a more conservative approach such as the “one-standard-error” rule hastie2009elements. This proposal is similar to the leave-one-out cross validation proposed by Doudchenko2017, who select hyperparameters by holding out control units and minimizing the MSE of the control units in the post-treatment time $T$. Finally, only excluding time period $t$ might be inappropriate for some outcome models, e.g. the linear model in Section (ref). In these settings we can extend the procedure to exclude all time periods $\geq t$ when estimating $\hat{\gamma}_{(-t)}^\text{aug}$. For related proposals, see kellogg2020combining and bilinski2020goldilocks.
Thus far, we have focused exclusively on lagged outcomes as predictors. We now consider the case where there are also a small number of auxiliary covariates $\bm{Z}_i \in \ensuremath{\mathbb{R}}^K$ for unit $i$. These auxiliary covariates may include summaries of lagged outcomes or time-varying covariates such as the pre-treatment mean $\bar{X}_i$. Let $\bm{Z}_{0\cdot} \in \ensuremath{\mathbb{R}}^{N_0 \times K}$ denote the matrix of covariates for the donor units. We assume that the covariates are centered, $\bar{\bm{Z}}_{0\cdot} = \mathbf{0}$.
These auxiliary covariates can be incorporated both into the balance objective for SCM and into the outcome model used for augmentation in ASCM. For example, we can extend SCM to choose weights to solve
where $\Delta^{N_0}$ is the $N_0$-simplex. Similarly, we can augment these weights with an outcome model $\hat{m}(\bm{X}_i, \bm{Z}_i)$ that is a function of both the lagged outcomes and auxiliary covariates. For example, we can extend Ridge ASCM to choose $ \hat{m}(\bm{X}, \bm{Z})=\hat{\eta}_0 + \bm{X}'\hat{\bm{\eta}}_x + \bm{Z}'\hat{\bm{\eta}}_z$ and fit via ridge regression:
Both this SCM criterion and augmentation estimator incorporate user-specified weights that determine the importance of balancing each set of covariates (Equation (ref)) or the amount of regularization for each set of coefficients (Equation (ref)). There are many potential choices for these weights. We discuss two, appropriate to different settings depending on the number of auxiliary covariates.
A sensible default when the dimension of the auxiliary covariates is moderate is to incorporate the lagged outcomes $\bm{X}$ and the auxiliary covariates $\bm{Z}$ equally in Equations (ref) and (ref), setting $\theta_x = \theta_z = 1$ and $\lambda_x = \lambda_z = \lambda^\text{ridge}$, after standardizing auxiliary covariates and lagged outcomes to have the same standard deviation. With this setup the numerical and algorithmic results in Section (ref) apply for the combined vector of lagged outcomes and auxiliary covariates, $(\bm{X}_i, \bm{Z}_i) \in \ensuremath{\mathbb{R}}^{T_0 + K}$. In particular, Ridge ASCM is again a penalized SCM estimator that adjusts the synthetic control weights that solve optimization problem (ref) to achieve better balance by extrapolating outside of the convex hull.
An alternative approach when the dimension of the auxiliary covariates is small relative to $N$ (i.e., $K \ll N$) is to fit a regression model that regularizes the lagged outcome coefficients $\bm{\eta_x}$ but does not regularize the auxiliary covariate coefficients $\bm{\eta_z}$ (i.e., set $\lambda_z = 0$). Lemma (ref) below writes the resulting augmented estimator as its corresponding penalized SCM optimization problem, with weights that perfectly balance the auxiliary covariates. This has two key implications. First, since the auxiliary covariates $\bm{Z}$ are exactly balanced regardless of the balance that the SCM weights achieve alone, we can exclude them from the optimization problem (ref). Second, as we show below, the pre-treatment fit on the lagged outcomes depends on how well the SCM weights balance the residualized lagged outcomes $\check{\bm{X}}$. This suggests modifying Equation (ref) to balance $\check{\bm{X}}$ rather than the lagged outcomes $\bm{X}$, which leads to the two-step procedure: (1) residualize the pre- and post-treatment outcomes on the auxiliary covariates $\bm{Z}$; and (2) estimate Ridge ASCM on the residualized outcomes. This two-step procedure follows from a related proposal in Doudchenko2017.
Comparing to the numerical results in Section (ref), Lemma (ref) shows that the two-step approach penalizes extrapolation from the convex hull in the residualized space $\check{\bm{X}}$, rather than in the lagged outcomes themselves. In essence, by residualizing out the auxiliary covariates $\bm{Z}$ the two-step approach allows for a possibly large amount of extrapolation in the auxiliary covariates, while carefully penalizing extrapolation in the part of the lagged outcomes that is orthogonal to the covariates.
In the Appendix, we consider the performance of this estimator when the outcomes follow a linear factor model with covariates for the special case where $\lambda^\text{ridge} \to \infty$ and the weights $\hat{\bm{\gamma}}^\text{cov}$ do not extrapolate from the convex hull after residualization. We find that, if $K$ is small relative to $N_0$, exactly balancing a small number of auxiliary covariates and targeting imbalance in the residuals $\check{\bm{X}}$ can lead to decreased error due to pre-treatment fit, with only a small increase in approximation error and error due to post-treatment noise. However, with larger numbers of auxiliary covariates, the approach that incorporates auxiliary covariates in parallel to lagged outcomes would be more appropriate.
We now turn to simulations and an empirical illustration. First, we conduct extensive simulation studies to assess the performance of different methods, finding substantial gains from ASCM. We then use our approach to examine the effect of an aggressive tax cut on economic output in Kansas in 2012.
We now present simulation studies calibrated to our empirical illustration in Section (ref). Specifically, we use the Generalized Synthetic Control Method Xu2017 to estimate a factor model with three latent factors based on the series of log GSP per capita ($N=50$, $T_0=89$). We then simulate outcomes using the distribution of estimated parameters and model selection into treatment as a function of the latent factors; see Appendix (ref) for additional details. We also present results from three additional DGPs, each calibrated to estimates from the same data: (1) the factor model with quadruple the standard deviation of the noise term, (2) a unit and time fixed effects model, and (3) an autoregressive model with 3 lags.
We explore the role of augmentation using simple outcome estimators. For each DGP, we consider five estimators: (1) SCM alone, (2) ridge regression alone, (3) Ridge ASCM, (4) fixed effects alone, and (5) SCM augmented with fixed effects (i.e., de-meaned SCM), as shown in Equation (ref). Figure (ref) shows the Monte Carlo estimate of the absolute bias as a percentage of the absolute bias for SCM, with one panel for each simulation DGP; Appendix Figure (ref) shows the corresponding estimator root mean squared error (RMSE).
There are several takeaways. First, augmenting SCM with a ridge outcome regression reduces bias relative to SCM alone --- without conditioning on excellent pre-treatment fit --- in all four simulations. This underscores the importance of the recommendation in AbadieAlbertoDiamond2010, Abadie2015 to use SCM only in settings with excellent pre-treatment fit.\footnote{AbadieAlbertoDiamond2010, Abadie2015 also strongly recommend incorporating auxiliary covariates, weighted by their predictive power, into the procedure, noting that this is important for further reducing bias. For simplicity, the simulations do not include auxiliary covariates.} Under the baseline factor model and the fixed effect model, the ridge augmentation greatly reduces bias, by more than 75% in the factor model simulation and over 90% in the fixed effects simulation. In the AR(3) model and in the factor model with greater noise, the gains to augmentation relative to SCM are more limited. Second, Ridge ASCM has slightly lower bias than ridge regression alone across all of the simulation settings. Third, when the fixed effects estimator is incorrectly specified, combining SCM with a fixed effects estimator has much lower bias than either method alone. And even when the fixed effects estimator is correctly specified, de-meaned SCM has similar bias to the (correctly specified) fixed effects approach. Finally, Appendix Figure (ref) shows that in all simulations ASCM has lower RMSE than SCM, as the large decrease in bias more than makes up for the slight increase in variance.
Complementing the worst-case analysis in Section (ref), we now consider how the typical performance of augmentation relates to the amount of extrapolation and the quality of the original SCM fit. Figure (ref) shows the bias and RMSE as a function of $\lambda$ for the primary factor model simulation, conditional on the quartile of SCM fit. Larger values of $\lambda$ (and hence smaller adjustments) are to the left, with the left-most points in the plots representing SCM. First, as expected, Augmented SCM substantially reduces bias regardless of SCM pre-treatment fit. However, the gains are more modest when the SCM fit is in the best quartile: in this case the bias is non-monotonic in $\lambda$ and there is some optimal choice of $\lambda$ that minimizes the bias. Second, it is possible to under-regularize with ASCM, as evident in the RMSE achieving a minimum for an intermediate value of $\lambda$. Furthermore, when pre-treatment fit is good, augmentation with too-small $\lambda$ leads to higher RMSE than SCM alone.
Next, we evaluate alternative outcome models for use in ASCM. For each DGP we consider SCM augmented with (1) LASSO, (2) a random forest, (3) CausalImpact Brodersen2015, (4) matrix completion using MCPanel athey2017mcp and (5) fitting the factor model directly with gsynth Xu2017. We compare ASCM to the pure outcome models as well as pure SCM. Figure (ref) shows the absolute bias for these methods, again as a percentage of the absolute bias for SCM alone; Appendix Figure (ref) shows the RMSE. We broadly see the same results as with Ridge ASCM. In our simulations, augmenting SCM almost always reduces the bias relative to SCM (unconditional on good pre-treatment fit) with some models improving SCM more than others. Additionally, in nearly every case ASCM also has lower bias than outcome modeling alone. Appendix Figures (ref) and (ref) show the bias and RMSE when SCM fit is in the top quintile. As before, conditioned on good SCM pre-treatment fit the gains to augmentation with flexible outcome models are more limited, except with the oracle gsynth estimator.
Overall we find that SCM augmented with a penalized regression model has consistently good performance across data generating processes. Due to this performance and the method's relative simplicity, we therefore recommend augmenting SCM with penalized regression as a reasonable default in settings where SCM alone has poor pre-treatment fit. In particular, we suggest using ridge regression; among the other benefits, Ridge ASCM allows the practitioner to diagnose the level of extrapolation due to the outcome model.
In 2010, Sam Brownback was elected governor of Kansas, having run on a platform emphasizing tax cuts and deficit reduction rickman2018two. Upon taking office, he implemented a substantial personal income tax cut, both lowering rates and reducing credits and deductions. This is a valuable test of “supply side” models: Brownback argued that the tax cuts would increase business activity in Kansas, generating economic growth and additional tax revenues that would make up for the static revenue losses. Kansas' subsequent economic performance has not been impressive relative to its neighbors; however, potentially confounding factors include a drought and declines in the locally important aerospace industry. Finding a credible control for Kansas is thus challenging, and SCM-type models offer a potential solution.
We estimate the effect of the tax cuts on log gross state product (GSP) per capita using the second quarter of 2012 --- when Brownback signed the tax cut bill into law --- as the intervention time. Results are consistent using outcomes scales other than the standard normalization of log GSP per capita (see Appendix (ref)). We use four primary estimators: (1) SCM alone fit on the entire vector of lagged outcomes, (2) Ridge ASCM, (3) Ridge ASCM including auxiliary covariates in parallel to lagged outcomes and (4) Ridge ASCM on residualized outcomes, as in Section (ref). \footnote{The covariates we include are the pre-treatment averages of (1) log state and local revenue per capita, (2) log average weekly wages, (3) number of establishments per capita, (4) the employment level, and (5) log GSP per capita. For the augmented estimator on the lagged outcomes we select the hyperparameter $\lambda^\text{ridge}$ as the largest $\lambda$ within one standard error of the $\lambda$ that minimizes the cross-validation placebo fit $CV(\lambda)$; see Section (ref). Appendix Figure (ref) plots $CV(\lambda)$. When including the auxiliary covariates we use the minimal $\lambda$.} These estimators rely on the ignorability assumption in Equation (ref); substantively, this assumes that post-treatment shocks for Kansas will be the same as for other states in expectation. This also rules out unobserved confounders that affect both post-treatment shocks and the decision to enact the Brownback tax cut bill.
Figure (ref), known as a “gap plot”, shows the difference between Kansas and its synthetic control before and after the passage of the tax cuts along with 95% point-wise confidence intervals computed via the conformal inference procedure from chernozhukov2017exact; see Appendix (ref). Appendix (ref) shows additional results, including estimates plotted on the raw outcome scale and results with alternative estimators. First, SCM alone achieves fairly poor pre-treatment fit; the synthetic control exceeds the treatment unit by two to four percent in 2004--2005, on the same scale as the estimated average post-treatment effect of a 3 percent decrease. This lack of pre-treatment fit should make us wary of the validity of the SCM effect estimates, and suggests that there may be gains to augmentation. Augmenting SCM with ridge regression indeed improves pre-treatment fit, especially in the mid 2000s. To better understand this, we can inspect the ridge regression coefficients for lagged outcomes (see Appendix Figure (ref)), which put the most weight on the two most recent years. Adding auxiliary covariates and augmenting further improves both pre-treatment fit and balance on the covariates; see Figure (ref). Finally, balancing the auxiliary covariates via residualization also improves pre-treatment fit. Overall, the estimated impact is consistently negative for all four approaches, with weaker evidence that the effect persists to the end of the observation period.
To check against over-fitting, Figure (ref) shows in-time placebo estimates for the Ridge ASCM estimator with covariates, with placebo treatment times in the second quarter of 2009, 2010, and 2011. We estimate placebo effects that are near zero with all three placebo treatment times. Appendix Figures (ref) and (ref) show the corresponding placebo estimates for SCM alone and Ridge ASCM without covariates.
Figure (ref) shows the covariate balance for the four estimators. While SCM and Ridge ASCM achieve excellent fit for the pre-treatment average log GSP per capita, neither estimator achieves good balance on the other covariates, most notably the average employment level across the quarters of the pre-period. In contrast, including the auxiliary covariates into both the SCM and ridge optimization problems greatly improves the covariate balance, and --- by design --- residualizing on the auxiliary covariates perfectly balances them. Moreover, Ridge ASCM on residualized outcomes achieves very good pre-treatment fit on the lagged outcomes as shown in Figure (ref).
Finally, Figure (ref) shows the weights on donor units for SCM and Ridge ASCM as well as SCM and Ridge ASCM weights when including covariates jointly with the lagged outcomes. Appendix Figure (ref) shows the SCM and Ridge ASCM weights fit on the lagged outcomes after residualizing out the auxiliary covariates. Here we see the minimal extrapolation property of the ASCM weights. The SCM weights are zero for all but six donor states. The Ridge ASCM weights are similar but deviate slightly from the simplex, with only Louisiana receiving a meaningful negative weight. In addition, the Ridge ASCM weights retain some of the interpretability of the SCM weights. For the donor units with positive SCM weight, Ridge ASCM places close to the same weight. For the majority of those with zero SCM weight, Ridge ASCM also places a close to zero weight, with relatively few donor units with non-negligible negative weight. By contrast, Appendix Figure (ref) shows the weights from ridge regression alone: many of the weights are negative and the weights are far from sparse. Including auxiliary covariates changes the relative importance of different states by adding new information, but the minimal extrapolation property remains.
SCM is a popular approach for estimating policy impacts at the jurisdiction level, such as the city or state. By design, however, the method is limited to settings where excellent pre-treatment fit is possible. For settings when this is infeasible, we introduce Augmented SCM, which controls pre-treatment fit while minimizing extrapolation. We show that this approach controls error under a linear factor model and propose several extensions, including to incorporate auxiliary covariates.
There are several directions for future work. The most immediate is to explore robust inferential methods for settings where pre-treatment SCM fit is imperfect. In Appendix (ref), we outline how to apply the conformal inference approach of chernozhukov2017exact to Augmented SCM, as well as a possible modification based on the jackknife+ approach of Barber2019. More work is needed, however, to understand the performance of this in more general settings. We could also extend this to allow for sensitivity analysis that directly parameterizes departures from, say, the linear factor model, as in recent approaches for sensitivity analysis for balancing weights soriano2019sensitivity.
A second area for future inquiry is the application of the ASCM framework to settings with multiple treated units. For instance, there are different approaches in settings when all treated units are treated at the same time: some papers propose to fit SCM separately for each treated unit Abadie_LHour, while others simply average the units together kreif2016examination, Robbins2017. The situation is more complicated with staggered adoption, when units take up the treatment at different times dube2015pooling, donohue2017right. We explore this extension in benmichael2019multisynth.
A third potential extension is to more complex data structures, such as applications with multiple outcomes series for the same units (e.g., measures of both earnings and total employment in minimum wage studies); hierarchical data structures with outcome information at both the individual and aggregate level (e.g., students within schools); or discrete or count outcomes.
\singlespacing
\singlespacing