Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
57,300 characters · 23 sections · 39 citation commands
Forecasting Algorithms for Causal \ Inference with Panel Data
\onehalfspacing
Conducting causal inference with panel data is a core challenge in social science research. Consider a panel of states, one of which (the treated state) adopts a new policy. What is the effect of this policy on some outcome of interest? The question can be cast as a prediction problem: the (potential) untreated outcomes for the treated state can be observed in the time periods prior to the new policy's adoption, but not afterwards. Meanwhile, the potential untreated outcomes for the control states can be observed both before and after the new policy's adoption. Identifying the causal effect of the policy entails estimating what the outcome would have been in the treated state during time periods after the policy's adoption, had the new policy not been adopted holland1986. Alternative tools for causal inference in panel data settings can be understood as different methods for imputing these counterfactual outcomes on the basis of the data that is observed: namely, data from the treated state prior to the policy's adoption, and from the control states in time periods both before and after the policy's adoption.
In this paper, we draw on new advances in the time series forecasting literature to improve the accuracy of the imputations employed for causal inference in panel data settings. Over the past few years, the forecasting literature has proposed a number of deep neural architectures that have significantly improved predictive capabilities over older models. However, these models are designed to be applied to data for single time series, i.e., predicting future values of a unit based on that same unit's past values. For causal inference with panel data, this is an important limitation because single-unit time series models do not incorporate information from the time series of control unit outcomes to help estimate missing values for the treated unit. We overcome this limitation by incorporating the time series of outcomes for control units into the forecasting model for the treated unit as additional features. That is, to impute the potential untreated outcome in the treated unit following the treatment, we feed into the model the outcome from the control states during the same time period -- effectively casting contemporaneous outcomes of the control states as “leading indicators” for the potential untreated outcome of the treated state.
Among the different deep architectures for forecasting models, we focus on the neural basis expansion analysis for time series (N-BEATS) algorithm oreshkin2019. N-BEATS is a deep neural architecture designed to predict future values in a time series on the basis of past values. The algorithm has been shown to perform well in a range of forecasting tasks. The key innovation we rely on is a recent adaptation of N-BEATS that incorporates time series other than the one being predicted as additional features olivares2019; in the panel data setting, this innovation allows us to use N-BEATS to predict unobserved values of the treated unit on the basis of prior values of the treated unit as well as contemporaneous values of the control units. Because our proposed approach essentially involves using the N-BEATS algorithm to estimate a synthetic (i.e., predicted) untreated outcome for the treated state during the post-treatment period, we refer to it as Synthetic N-BEATS (“SyNBEATS”).
Although the N-BEATS algorithm has been shown to excel at a range of forecasting tasks, an important concern is whether its performance will be as strong when applied to the relatively small panel data sets typically employed in social science research. With limited data, simpler methods like synthetic controls (SC) or two-way fixed effects (TWFE) may yield more reliable causal estimates. To assess the suitability of SyNBEATS for causal inference with panel data, we compare it to existing alternatives across two canonical panel data settings. Specifically, we contrast performance in data that has been used to estimate the effect of a cigarette sales tax in California abadie2010 and the German reunification on the West Germany economy abadie2015. In both of these settings, we find that SyNBEATS outperforms canonical methods such as SC and TWFE estimation. In addition, we compare the performance of these models in estimating the impact of simulated events on abnormal returns in publicly traded firms baker2020machine. In this setting, where historical values would not be expected to provide much information about future outcomes, SyNBEATS only marginally improves performance relative to the other estimators. We also compare SyNBEATS to two recent proposed causal inference methods for panel data settings: matrix completion (MC) mcathey2021 and synthetic difference-in-differences (SDID) sdid. In the three settings we consider, we find that SyNBEATS generally achieves comparable or better performance compared to synthetic difference-in-differences, and significantly outperforms matrix completion. We further investigate the factors that shape the relative performance of SDID and SyNBEATS through a range of simulations. Finally, we unpack SyNBEATS' strong comparative performance, and find it stems from both model architecture and SyNBEATS' efficient use of time-series data in informing its predictions.
Our findings build on a growing literature applying machine learning tools to casual inference. In addition to the new methods discussed above, two recent papers similar in spirit to ours include muhlbach2021tree, which adapts tree-based methods to synthetic control estimators, and poulos2021rnn, which applies a recurrent neural network model to predict treated units' outcomes based on pre-treatment observations. As discussed below, our approach differs from these in that SyNBEATS forms its counterfactual prediction directly using both pre-treatment values of the treated unit and post-treatment values of the untreated units.
The remainder of the paper proceeds as follows: Section (ref) describes SyNBEATS and related estimators. Sections (ref) and (ref) compare SyNBEATS to common and recent estimators, respectively. Section (ref) explores the source of SyNBEATS' performance. Section (ref) concludes.
This section describes our setup, introduces SyNBEATS, and describes alternative panel data estimators.
We consider a panel data setting with $N$ units across $T$ time periods. One unit ($N$) adopts a new policy that takes effect for every period after $T_0<T$. We use $W_{it} \in \{0,1\}$ to indicate whether a unit has adopted the policy, so that $W_{it} = 1$ for $i=N$ and $t> T_0$, and $W_{it} = 0$ otherwise. We are interested in the causal effect of the policy on some outcome of interest $Y_{it}$. Let $Y_{it}(1)$ and $Y_{it}(0)$ denote the (potential) outcomes of interest for unit $i$ in period $t$ corresponding to $W_{it}=1$ and $W_{it}=0$, respectively. For any unit and time period, the researcher observes only one potential outcome: $$Y_{it} = W_{it} Y_{it}(1) + (1-W_{it}) Y_{it}(0)$$ The econometric challenge, illustrated in Figure (ref)(a), is to impute the missing counterfactual outcomes, $Y_{Nt}(0)$ for $t>T_0$, in order to estimate the average treatment effect on the treated: $$\tau^{ATT}= E[Y_{it}(1) - Y_{it}(0) | W_{it} = 1]$$
Many methods have been proposed to conduct causal inference in panel data settings. One popular approach is the synthetic control (SC) estimator abadie2003, abadie2010 , which constructs a synthetic counterfactual for the treated unit in the post-treatment periods by finding weights on relevant control unit values such that the synthetic unit fits the treated unit values well in the pre-treatment period. doudchenkoimbens2016 expand upon the SC literature and show these methods can be viewed as leveraging the structure of the data vertically across units (Figure (ref)(b)). In particular, the weights on the control units can be interpreted as being derived from a regression of treated unit outcomes on control outcomes in the pre-treatment period:
where the weights are non-negative and sum to 1, and $\theta_0$ is typically constrained to be 0.
Although less common in panel data estimation, another approach is to assume that temporal patterns are stable across units to exploit the structure of the data horizontally (Figure (ref)(c)). In the panel data setting, this “unconfoundedness” approach extrapolates treated unit counterfactual values in $t>T_0$ by using weights obtained from regressing the control outcomes in $t>T_0$ on those in prior periods. Formally, with the number of treated units indexed by $\mathcal{I}$, we can view the weights as the solution to the following minimization problem:
Whereas the methods discussed thus far rely exclusively on either vertical or horizontal information, two-way fixed effects (TWFE) estimators simultaneously exploit stable patterns over time and across units. In the simplest case with no covariates, the TWFE estimate for the treatment effect is obtained from:
Interactive fixed effects and more generally factor models extend this literature to allow for richer unobserved heterogeneity bai2002, bai2003. These more general models assume the data generating process is given by a linear function of observed covariates and an unobserved low-rank matrix plus noise, allowing for arbitrary interactions beyond simple additive effects. Without covariates and with binary treatment, the outcomes can be described as $$\boldsymbol{Y} = \boldsymbol{L} + \boldsymbol{W} \odot \boldsymbol{\tau} + \boldsymbol{\epsilon}$$ where $\boldsymbol{L}$ represents the target estimand we wish to recover. In the factor model literature, $ \boldsymbol{L}:= \boldsymbol{UV}^T$ with loadings $\boldsymbol{U}$ and factors $\boldsymbol{V}$.
A recent estimator for $\boldsymbol{L}$ proposed by mcathey2021 draws on the matrix completion (MC) literature. MC aims to recover missing entries in a data matrix that are missing at random, in contrast to the “block” missingness present in panel data causal inference settings (Figure 1(a)). mcathey2021 incorporate the intuition of TWFE into matrix completion by solving the standard low-rank matrix completion problem but keeping fixed effects explicitly unregularized.
Another recent method simultaneously incorporating both vertical and horizontal information is the synthetic difference-in-difference (SDID) estimator. First, it employs a SC-type approach to reweight the unexposed control units, creating a parallel trend. Second, SDID applies a difference-in-differences type analysis on this reweighted panel. In contrast to standard difference-in-differences, SDID allows time period weights to vary to accommodate the possibility that more recent periods are more informative. With both sets of weights, the estimator recovers the treatment effect by solving: $$(\hat{\tau}^{SDID}, \hat{\mu},\hat{\alpha},\hat{\delta}) = \text{arg min}_{\tau,\mu,\alpha,\delta} \left\{ \sum_{s=1}^N\sum_{t=1}^T ( Y_{st} - \mu -\alpha_s - \delta_t - \mathbf{1}_{\{s=N\}} \mathbf{1}_{\{t > T_0 \}} \tau)^2 \hat{\omega}_i \hat{\lambda}_t \right\}$$
We cast the challenge of causal inference in the panel data setting as a supervised learning problem and propose drawing on both horizontal and vertical information to inform the prediction of the (missing) untreated potential outcomes for the treated unit in the post-treatment time period. Specifically, we propose a neural network architecture to determine prediction weights across time and units, depicted in Figure (ref).
Model training is based on the mean squared error loss over the pre-intervention period when $Y_{Nt}$ is fully observed. To form predictions for these values at every time step, the model uses the contemporaneous control outcomes at $t$ and the lagged treated unit's outcomes from $t-n_{\text{lag}}$ to $t-1$, where $n_{\text{lag}}$ denotes the number of lagged outcomes used. Because $n_{\text{lag}}$ is constant across all training examples, we define $\underline{t} := t-n_{\text{lag}}$ for convenience. Formally, at every $t = 1,\ldots,T_0$, our model predicts the observed $Y_{Nt}$ with $\{Y_{it} \}_{i=1}^{N-1}$ and $\{Y_{Ns}\}_{s=\underline{t}}^{t-1}$. The model, $m: \mathbb{R}^{(N-1)+(n_\text{lag})} \to \mathbb{R}$, is trained via gradient descent methods to minimize loss based on the following loss function:\footnote{The loss function assumes a one-period-ahead prediction objective, which is what we implement in practice. The method can also support multiple-period-ahead predictions by averaging over the loss in subsequent future periods.}
In a standard feedforward neural network, each training input $\boldsymbol{x} \in \mathbb{R}^{d_0}$ is fed through multiple layers, with each layer $\ell$ generating an intermediate output $\boldsymbol{h}^{[\ell]} \in \mathbb{R}^{d_\ell}$.\footnote{For ease of notation, $\ell = 1,\ldots,L$ denotes which layer is of interest.} This relationship is governed by $$\boldsymbol{h}^{[\ell]} = f^{[\ell]}(\boldsymbol{W}^{[\ell]}\boldsymbol{h}^{[\ell-1]} + \boldsymbol{b}^{[\ell]})$$ Here, $\boldsymbol{W}^{[\ell]} \in \mathbb{R}^{d_\ell \times d_{\ell-1}}$ and $\boldsymbol{b}^{[\ell]} \in \mathbb{R}^{d_\ell} $ represent weight and bias matrices akin to standard linear regression. In the first layer, $\boldsymbol{h}^{[0]} \equiv \boldsymbol{x}$. The function $f(\cdot)$ is taken here to be $f(x) = \max\{0,x\}$, so that $f$ explicitly introduces non-linearity into the network to allow it to learn non-linear relationships in the data. After an arbitrary number of layers $L$ with intermediate outputs $\boldsymbol{h}^{[1]}, \ldots, \boldsymbol{h}^{[L]}$, the model yields a final output $\boldsymbol{\hat{y}} \in \mathbb{R}^{d_y}$. This in turn is compared to the observed outcome $\boldsymbol{y} \in \mathbb{R}^{d_y}$ and a standard loss function (e.g. MSE) is used to compute the overall loss. As each $\boldsymbol{h}^{[\ell]}$ depends on all components of $\boldsymbol{h}^{[\ell-1]}$ at each layer, we call such a network fully connected. Consistent with other nonparametric estimation methods, a closed-form solution does not exist generically, and the parameters must be estimated via gradient descent methods. In neural networks, this numerical method proceeds in a familiar manner: the gradient of the loss function propagates backward through the network via the chain rule to update all weights $\boldsymbol{W}^{[\ell]}$ and $\boldsymbol{b}^{[\ell]}$ until the weights converge.\footnote{Without any assumptions or additional structure on the problem, a global minimum is not guaranteed, but in practice even local minima may perform well in RMSE compared to standard econometric models.}
In contrast to a standard feed-forward neural network, the N-BEATS architecture is compartmentalized into different blocks. Given a time series input $\boldsymbol{x} \in \mathbb{R}^{d_0}$, where $d_0$ is the length of the lookback period used to forecast, each block $k$ adopts this standard neural network structure but generates not only a forecast $\boldsymbol{\hat{y}} \in \mathbb{R}^{d_y}$ with $d_y$ as the length of the forecast horizon, but also a {backcast} $\boldsymbol{\hat{x}} \in \mathbb{R}^{d_0}$. In its original implementation, $\boldsymbol{x}$ is composed of historical values of the time-series being predicted (e.g., lagged outcomes of the treated unit). olivares2019 modifies this approach to allow for predictions using information from additional (covariate) time series. As shown in Figure (ref), the model makes its prediction in each block via a set of shared fully connected layers that branch into two separate forecast and backcast layers. For each block $k$, the shared fully connected layers yield intermediate outputs $\boldsymbol{h}^{[1]}_k,\ldots, \boldsymbol{h}^{[L]}_k$ as before. Next, $\boldsymbol{h}^{[L]}_k$ is split into $\boldsymbol{h}^f_k$ and $\boldsymbol{{h}}^b_k$, which serve as the final intermediate outputs for the forecast and backcast, respectively. The predictions in each block are finally given by:
where $g^f(\cdot)$ and $g^b(\cdot)$ are linear projection operators.
The backcast at block $k$, $\boldsymbol{\hat{x}}_{k}$, is fed to the next block $k+1$ as the new input:
The forecast at block $k$, $\boldsymbol{\hat{y}}_{k}$, is added iteratively to the forecasts at all other previous blocks:
In other words, this block structure breaks down the signal from input $\boldsymbol{x}$ step-by-step, with each $\boldsymbol{\hat{x}}_k$ representing the signal leftover that could not be explained by the previous neural networks in blocks $1, \ldots, k$. Although in theory an arbitrarily large, standard feed-forward neural network should be able to flexibly mine all input signal and achieve arbitrarily low error, in practice these alternatives struggle to do so within reasonable computational limits.
SyNBEATS largely follows oreshkin2019 and olivares2019 in its model architecture. The sole modification is that we augment the feature set to include outcomes for contemporaneous control units, in addition to historical values of the treated unit (i.e., the time series being predicted). To do so, we concatenate the contemporaneous controls and the lagged outcomes into a single vector. Each training example at time step $t$ is thus represented as: $$\boldsymbol{x}_t := [\{Y_{it}\}_{i=1}^{N-1}, \{Y_{Ns}\}_{s=\underline{t}}^{t-1}]$$ This is then is fed through the model architecture to predict $\boldsymbol{y}_t := [\{Y_{Ns}\}_{s=t}^{\overline{t}}]$, where $\overline{t}-t+1$ refers to the number of future periods being predicted.
In our implementation of SyNBEATS, we set $\overline{t}=t$. To predict beyond a single post-treatment period, we iteratively use the predicted value of the preceding period as an input to the algorithm; for example, in order to predict $\widehat{Y}(0)_{s,t+1}$, we use $\widehat{Y}(0)_{st}$ as an input.
As described above, SyNBEATS entails training a supervised model to predict the untreated potential outcome for the treated unit based on prior (untreated) values of the treated unit and contemporaneous values of the control units:
For SyNBEATS to accurately estimate the average treatment effect on the treated, $\tau$, the relationship between the label, $Y_{Nt}$, and the features, $\{Y_{it}(0) \}_{i=1}^{N-1}, \{Y_{Ns}(0)\}_{s=\underline{t}}^{t-1}$ during the post-period must be learnable from data observed during the pre-period. A sufficient condition for this to hold is that the distribution of outcomes is stationary and the timing of treatment is exogenous. Similar conditions are commonly imposed to justify the use of synthetic controls and related estimators.
That being said, due to the neural network architecture underlying the estimator, it is difficult to provide asymptotic guarantees on SyNBEATS's behavior. Indeed, highly flexible estimators are generally biased due to overfitting and regularization; the same is true for commonly employed estimators such as synthetic controls. In some applications, researchers nonetheless may be particularly interested in the consistency of the estimator. One method for obtaining a consistent SyNBEATS estimator is via bias-correction ttest, viviano. Under this approach, the researcher evaluates the estimator's bias using a subset of the pre-treatment sample. This can then be subtracted from the SyNBEATS-estimated treatment effect to yield an asymptotically consistent estimate. Under standard event study assumptions (exogeneity of treatment timing, stationarity of outcomes, etc.), we can adapt the proof from viviano to show that this bias-adjusted version of SyNBEATS yields a consistent treatment effect estimate. We develop this estimator and compare its performance to our baseline SyNBEATS estimator in Section (ref) below.
In this section, we compare the performance of SyNBEATS to two popular methods for causal inference in panel settings: synthetic controls and two-way fixed effects estimation. We analyze these methods in three data settings.
In each setting, we collect outcome data $Y_{it}$ for a balanced panel of $i\in\{1,2,...,N\}$ untreated units and $t\in\{1,2,..,T_0, T_0+1, ..., T\}$ periods. As discussed in Section 2.1, the econometric challenge is to estimate the counterfactual untreated outcome $Y_{Nt}(0)$ for unit $N$ when $t>T_0$. We compare alternative estimators for this term by analyzing the prediction error. To do so, we conduct placebo exercises assuming $Y_{Nt}(0)$ is unobserved (by masking actually observed outcomes) for $t > T_0$ and predicting $Y_{Nt}(0)$ (out-of-sample) using other data from the panel that would be available to the researcher if the unit really had been treated (i.e. if $Y_{Nt}(0)$ really was unobserved). We consider two types of validation exercises, involving: (1) the prediction error obtained from untreated control units in the post-treatment period, and (2) the prediction error for the treated unit during the pre-treatment period. By comparing estimates of $Y_{Nt}(0)$ derived from alternative methods to the true $Y_{Nt}(0)$, we can evaluate the performance of the various methods in estimating $\tau^{ATT}$.
This subsection briefly discusses our implementation of SyNBEATS, SC, and TWFE.
\subparagraph{Synthetic Controls} We implement the SC method as proposed by abadie2010. As recommended in the subsequent literature, we use pre-treatment outcomes to assess a control state's suitability for inclusion in the SC cavallo2013, doudchenkoimbens2016.
\subparagraph{Two-Way Fixed Effects} We consider a standard TWFE regression model without time-varying covariates. For consistency with the other methods we consider, we allow the treatment effect to vary by post-treatment year, and estimate the following model:
Unlike the other approaches considered here, we estimate the TWFE model on both the pre-treatment values of $Y_{st}(0)$ as well as the post-treatment values of $Y_{st}(1)$. The model's implied prediction of the potential untreated outcome in a given post-treatment year can be derived from the estimated treatment effect in the corresponding year:
\subparagraph{SyNBEATS} We implement our SyNBEATS algorithm as described in Section (ref). We use the original hyperparameters as described by oreshkin2019, as they have shown to yield high performance in a large number of time series settings and our applications have limited data available for hyperparameter tuning.\footnote{Specifically, our architecture includes 30 stacks, a single block with 4 layers and a layer width of 256. We use the Adam optimizer with default settings, including a learning rate of 0.001.} For the same reason, we set $n_{\text{lag}} = 1$. Although in principle $n_{\text{lag}}$ is a hyperparameter that can be tuned to improve performance, for convenience we treat it as fixed.
In 1988, California implemented Proposition 99, raising the state tax on cigarettes from 10 cents to 35 cents per pack. abadie2010 apply the SC method to this setting to estimate the effect of the tax on cigarette sales. Comparing the real California to its synthetic counterpart, they find that the tax reduced cigarette sales in the years following its adoption.
Below, we perform two exercises to assess the performance of SyNBEATS in this setting. The period of observation in the data set ranges from 1970 to 2000. The post-treatment period begins in 1989. Our outcome of interest is per capita cigarette sales (in packs), which we observe at the state-year level. Following abadie2010, we take as our control group the 38 states that did not adopt tax increases during this period.
In our first exercise, we exclude California (the true treated state). Instead, we assume that one other state from the control group (the “pseudo-treated” state) has been treated with a cigarette sales tax in 1989. We mask the post-1988 cigarette sales of the pseudo-treated state and apply each of the alternative causal inference methods. For each method, we assess the prediction error for the (actually untreated) outcome in the pseudo-treated state in the post-treatment period. This exercise will yield a valid assessment of the prediction errors for California in 1989 to the extent that, for each method we consider, the 1989 prediction error for California is drawn from the same distribution as the 1989 prediction error for the control states.
We repeat this process for the 38 control states, iteratively defining each control state as the pseudo-treated state. We first focus on the prediction error for the post-treatment year (1989). This yields 38 prediction errors for each estimation method -- one for each pseudo-treated state. To summarize the performance of a method, we compare the root mean squared error (RMSE) of the predictions across control states.
We also evaluate each model based on a longer-term, 5-year prediction window (1985--1989). In this case, each state will have five prediction errors, one for each post-treatment period. For the longer-term predictions, we calculate mean squared error based on the prediction errors in each pseudo-treated state over each post-treatment year.
Because the out-of-sample prediction error determines the accuracy of the estimated treatment effect, we compare the various estimation methods along this dimension. The model predictions are visualized in Figure (ref), and the resulting distribution of prediction errors is summarized in Table (ref). Among the methods we consider, SyNBEATS yields the most accurate prediction, with an RMSE of 3.59, 54% improvement over the second-best alternative of SC. Similarly, for longer-term predictions, SyNBEATS yields the best performance, with an RMSE of 8.17, which is a 27% improvement over SC as the second-best alternative.
In this second exercise, we maintain California as the treated state, but we (counterfactually) assume that Proposition 99 took effect during some year prior to 1989 (the “pseudo-treated year”). We iteratively define as our pseudo-treated year each year between 1975 and 1988. To assess longer-term predictions, we also consider 5-year pseudo-post-treatment periods in the same calendar year window (i.e., 1975--1979 through 1984--1988). In each iteration, we use data from years prior to the pseudo-treated year to estimate the predictive models.\footnote{The one exception is the TWFE model, which, as described above, we estimate using data that includes pseudo-treated years.} This exercise will yield a valid assessment of the prediction errors for California in 1989 to the extent that, for each method we consider, California's prediction error in 1989 is drawn from the same distribution as its prediction error for earlier years.
As shown in Table (ref), SyNBEATS outperforms other traditional estimators in their short-term predictions, improving the RMSE by 31% compared to the second-best alternative (SC). To further facilitate a direct comparison of short- to long-term predictions for each estimator, in Figure (ref), we contrast predictions in the first year after treatment to those obtained in the fifth year after treatment. Unsurprisingly, all estimators perform worse for longer-term predictions, with the performance difference between SyNBEATS and SC closing as well. Focusing on the full five-year window, SyNBEATS slightly outperformed SC.
In this subsection, we apply SyNBEATS and the other estimators we consider to estimate the effect of Proposition 99 on cigarette sales in California. Specifically, we train the model to predict cigarette sales per capita in California during years prior to 1989, and apply the model out-of-sample to predict California cigarette sales per capita in 1989 onward. Our estimated treatment effects correspond to the difference between these predictions and the actual, observed cigarette sales per capita in California during these years. The results of this exercise are displayed in Figure (ref). Using SyNBEATS, we estimate that Proposition 99 reduced cumulative cigarette sales per capita by a total of 192 packs in the ten years following its passage. Over the same time horizon, the other estimators imply reductions of 207 packs (SC) and 292 packs (TWFE). In relative terms, the SyNBEATS estimate corresponds to a reduction in cigarette sales of 22% (p=0.03).\footnote{As suggested above, we use randomization inference and assume California was randomly treated. Specifically, our p-value is derived from a placebo test over pseudo-treated units. We estimate a percentage reduction in smoking that is as large as California's in only one of the 38 placebo states we consider.}
The Berlin Wall fell on November 9, 1989. A year later, on October 3, 1990, West Germany and East Germany officially reunited, effectively marking the end of the Cold War. In a canonical study, abadie2015 apply the SC method to examine the causal effect of the reunification on the economy of West Germany, using 16 other countries as control units. They find that the reunification reduced West Germany's per capita GDP by about 1,600 USD per year in the period of 1990-2003.
Again, we perform two exercises to assess the accuracy of alternative causal inference methods in this setting. First, we iteratively predict the 1990 GDP of each country in the control group. Second, we iteratively predict West Germany's GDP from 1963 to 1989. In this setting as well, SyNBEATS consistently outperforms the other methods we consider, both with respect to short-term and long-term predictions. It also outperforms these alternatives with respect to both predicting pseudo-treatment country outcomes in the true treatment year (Figure (ref)) and in predicting true treatment country outcomes in pseudo-treatment years (Figure (ref)).
As above, we apply SyNBEATS and the other estimators we consider to estimate the effect of the German reunification on West Germany's GDP. Specifically, we train the model to predict the GDP in West Germany during years prior to 1990, and apply the model out-of-sample to predict West Germany GDP in 1990 onward. Our estimated treatment effects correspond to the difference between these predictions and the actual, observed GDP in West Germany during the post-treatment years. The results of this exercise are displayed in Figure (ref). Using SyNBEATS, we estimate that reunification reduced GDP per capita in West Germany by a total of \$17,766 in the 13 years after it occurred. This corresponds to a decrease of 5% (p=0.05).\footnote{As above, this p-value is derived from a placebo test over untreated units; only one of the 16 placebo countries we consider has an estimated percentage GDP reduction as large as West Germany's.} In contrast, over the same time horizon, SC implies a GDP reduction of \$15,116. The two-way fixed effects estimator suggests an average increase in GDP per capita of \$9,577.
In this exercise, we rely on data used in baker2020machine to predict abnormal stock returns for simulated events. This exercise is well suited for assessing the performance of the various estimators for use in financial event study analyses -- i.e., estimates of the causal effect of a shock (such as securities litigation or a merger announcement) on stock prices. The data set includes returns for firms with a share price above \$5 between 2009 and 2019. It also contains 10,000 randomly selected, unique firm-level pseudo-events (i.e., the events do not correspond to anything that would be expected to systematically affect the firms' stock price). For each firm-level event, we use returns data for the 250 trading days prior to the event to predict the returns on the event date. As in baker2020machine, for each firm, our pool of control units contains all firms with the same four-digit SIC industry code; if there are fewer than eight such firms, we include peers with the same three-digit SIC industry code.
Using 100 randomly selected pseudo-events from this data, we compare the predictions of each estimator for the stock price on the day after the simulated event (Table (ref)). As in the previous settings, SyNBEATS tends to generate the most accurate predictions, although here the performance gains are more modest. One potential explanation for why SyNBEATS does not improve performance as dramatically in this setting is that there is limited information in prior stock performance that can be used to predict future stock performance.
So far, we have provided evidence that SyNBEATS outperforms SC and TWFE, two commonly employed methods by social scientists for causal inference in panel data settings. In this section, we compare SyNBEATS to two recently proposed estimators, matrix completion (MC) and synthetic difference-in-differences (SDID), described in Section (ref).
We begin by comparing the performance of SyNBEATS to MC and SDID using the three empirical applications described in Section (ref). To implement MC, we implement the version of the estimator introduced in mcathey2021, which has been modified to allow for the estimation of causal effects in panel data settings.
To implement SDID, we follow sdid and their implementation in the associated code release. In particular, we first construct the level-shifted synthetic control with the unit weights described in Section (ref). Next, a weighted DID analysis is performed, where the pre-treatment average value of the treated unit and its level-shifted synthetic control are both given as weighted averages determined by the SDID time weights. The post-treatment average values are just simple averages, and a standard DID analysis is then applied. We note that SDID yields an average treatment effect on the treated unit over the entire post-treatment period of interest. To obtain predictions over varying treatment horizons, we recover the per-period treatment effects by conducting the last DID analysis step on each value in the post-treatment period rather than the simple average over all periods.
To compare SyNBEATS with MC and SDID, we replicate the analyses in the previous section. The results are presented in Table (ref). SyNBEATS dramatically outperforms MC in each analysis we consider with the data sets corresponding to Proposition 99 and the German Reunification. With respect to SDID, the results are more nuanced: SyNBEATS performs better in 6 out of the 8 analyses we consider using these two datasets, but SDID out-performs SyNBEATS in the remaining two. Notably, the performance gains of SyNBEATS compared to SDID are smaller than with the other estimators we consider. Finally, in predicting stock returns, SyNBEATS yields the lowest RMSE, but all three of the estimators yield comparable performance -- again, consistent with the hypothesis that time series forecasts are unlikely to greatly improve the predictive power in this context.
Our findings thus far suggest that SyNBEATS consistently outperforms TWFE, SC, and MC. In contrast, the results were more mixed with respect to SDID: SyNBEATS achieved a lower RMSE than SDID for 7 out of the 9 evaluations we considered, whereas SDID yielded lower prediction error for 2 of the 9 evaluations.
To shed additional light on the factors shaping the relative performance of SyNBEATS and SDID, we next conduct a simulation exercise. We consider a data generating process (DGP) via a linear factor model of the form studied in bai2009, gobillonmagnac2016, xu_2017, adapted to allow for variation in the relative importance of fixed and interactive effects in generating the outcomes (see notes to Appendix Figure (ref) for details).
In our simulations, we try to recover the null effect and vary three parameters, each time comparing the performance of SyNBEATS to SDID. The parameters we vary were selected to capture a range of panel data settings that empirical researchers may confront in practice.
We simulate data for each parameter choice 100 times and estimate outcomes over 4 periods. Model performance is evaluated as the average RMSE over all simulations and post-treatment periods. Results are depicted in Appendix Figure (ref).
As shown in the figure, SyNBEATS consistently outperforms SDID under a variety of parameter choices. However, the relative advantage of SyNBEATS decreases (and even reverses) for shorter pre-treatment periods and fewer controls. This is consistent with the assumption that the greater flexibility in the functional form of SyNBEATS requires more training data to generate reliable estimates. Further, the relative advantage of SyNBEATS decreases as the relative importance of fixed effects increases. As we confirm with additional simulations (Appendix Figure (ref)), SDID more consistently outperforms SyNBEATS if the fixed effects strongly dominate the data-generating process. This is consistent with the functional form assumptions of SDID being well-suited for that scenario.
Overall, our results suggest that SyNBEATS appears more accurate if researchers have sufficient training data. However, if training data is sparse, or if the data generating process is known to be dominated by unit fixed effects, SDID may yield lower prediction errors.
In this section, we investigate the source of SyNBEATS' strong observed performance relative to alternative estimators. SyNBEATS differs from other estimators in two important ways: (1) its use of both horizontal and vertical information to inform its imputation, and (2) its use of the N-BEATS residual block architecture. But, unlike many common estimators, SyNBEATS is not guaranteed to be consistent; it could be that its performance gains in terms of RMSE stem from providing concentrated, but potentially biased estimates. We explore each of these possibilities in turn.
To assess the importance of combining both horizontal and vertical information, we compare SyNBEATS to alternative estimators that, although comparable in their architecture, have access to limited information. The first estimator uses a pure forecasting algorithm that does not take into account the information from control states. Instead, it uses data only on pre-treatment outcomes of the treated unit. This approach is exactly the implementation of N-BEATS as originally proposed by oreshkin2019. The second estimator removes lagged outcomes of the treated unit and instead uses only information from the control states, in a similar way to synthetic controls. To implement it, we adapt the N-BEATS architecture by stripping out the treated unit's lags, translating the problem into one with no time-series dependence. We maintain the same residual block structure, using control unit outcomes to predict the treated unit's outcome in each period, with the assumption that the relationship between units is stable across time.
Appendix Table (ref) compares the performance of our baseline SyNBEATS estimator to these variants. Unsurprisingly, SyNBEATS performs the best, but the performance degradation from excluding horizontal information is dramatically larger than from excluding vertical information. This suggests that much of the performance gains in SyNBEATS might be traced back to its ability to efficiently learn the time series structure of the treated unit's outcomes. Without vertical information, SyNBEATS performs similar to or slightly better than TWFE or SC, whereas it performs substantially worse than competing estimators for the longer-term predictions. These results suggest that incorporating the post-treatment information from the control units becomes increasingly important as the prediction horizon grows and the accuracy of the forecast degrades.
To investigate the importance of the model architecture upon which SyNBEATS relies, Appendix Table (ref) compares SyNBEATS to alternative methods for imputing counterfactual treated unit outcomes based on vertical and horizontal information. SyNBEATS substantially outperforms “off-the-shelf” neural network and random forest models. A possible explanation is that SyNBEATS' residual structure allows for efficient mining of the signal without much data, in a similar manner to boosted trees. In a highly overparametrized regime where the number of model parameters far exceeds training data available, SyNBEATS' residual-based architecture allows the model to make full use of the limited data available and converge quickly in a setting where other flexible estimation methods typically suffer. In contrast, the performance gains are present, but less dramatic, when SyNBEATS is compared to a simple linear model. We thus conclude that SyNBEATS' strong observed performance is due to both its model architecture and its ability to use this architecture to exploit information from horizontal and vertical sources efficiently.
Finally, the performance gains from SyNBEATS could stem from a sacrifice of consistency in favor of providing concentrated, but biased estimates. Due to potential misspecification arising from flexible estimation, a researcher interested in a consistent estimator cannot simply optimize for mean squared error alone. To explore this idea, we implement a correction procedure in which we estimate and then adjust for the incurred bias.
Recall that SyNBEATS estimates the counterfactual control outcomes $\hat{Y}_{Nt}(0)$ and consequently the treatment effect as:
The bias-corrected SyNBEATS estimator employs SyNBEATS twice:first to obtain a treatment effect estimate, and second to obtain a bias estimate for adjustment. We begin by splitting the pre-treatment period equally. Using the first half of the pre-treatment data, $t=1,\ldots, \lfloor T_0 /2 \rfloor$, as the training set, we train SyNBEATS to construct counterfactuals $\hat{Y}_{Nt,\text{pre}}(0)$ for the second half of the pre-treatment data, $t=\lfloor T_0/2 \rfloor +1, \ldots, T_0$. This yields an estimate of the estimator's bias. We then subtract this estimated bias from the original SyNBEATS (trained on the full pre-treatment period) estimate:
Both the bias incurred by the SyNBEATS estimator and the estimated bias in the second half of the pre-treatment period converge in probability to the same limit. Thus, the difference between the two vanishes asymptotically.
viviano prove that under the following set of canonical event study assumptions, such a procedure results in consistent estimates. Formally, the distribution of outcomes $(Y_{Nt}(0), Y_{1t}, \ldots, Y_{N-1,t})$ is stationary, and the treatment timing is exogenous. Together these ensure that the empirical distributions of the pre- and post-treatment outcomes are invariant across time periods. Similar conditions are implied in the Synthetic Controls literature abadie2010, sdid, ttest which treat $T_0$ as deterministic (and hence, exogenous). Next, one imposes standard $\beta$-mixing conditions viviano, so that the observations are asymptotically independent. Lastly, the treated unit's potential outcome under treatment can be decomposed as $Y_{Nt}(1) = \mu_{Nt} + \epsilon_{Nt}$, where $\mu_{Nt}<\infty$ is a possibly non-stationary expectation, and $\epsilon_{Nt}$ is an additive stationary shock, with $\mathbbm{E}[\epsilon_{Nt}] = 0$.
We now assess the importance of consistency guarantees by comparing the performance of this bias-corrected estimator to our baseline (non-bias-corrected) SyNBEATS estimator. We illustrate the empirical effects of bias correction in the California Proposition 99 and West Germany Reunification datasets. We revisit the setting of our exercise with pseudo-treated, where we restrict the data to time periods before the actual treatment occurs. In the California Proposition 99 dataset, we iteratively predict the per-capita cigarette sales from 1975 to 1988; in the West Germany Reunification dataset, we iteratively predict West Germany's GDP from 1965 to 1989.\footnote{This time window differs slightly from our pseudo-treated years exercise above (1963-1989), as we split the pre-treatment period in half in the bias-adjustment method but still require a non-zero number of observations for training in the bias estimation.} The error distribution of the SyNBEATS estimator and the bias-adjusted SyNBEATS estimator is shown in Figure (ref).
For this bias-correction procedure to be meaningful, the gains in bias must be large relative to the increase in variance. Across both datasets, we see that applying the the bias-correction shifts the mean error closer to zero, with a larger (percentage) change in the Prop 99 setting. In the Prop 99 setting, this reduction in bias comes at the cost of a substantial increase in variance, but in the West Germany setting, applying the bias correction leads to both slightly lower bias and slightly lower variance (Table (ref)).\footnote{One explanation for this is finite sample noise, coupled with the already strong performance of the original SyNBEATS estimator. As our estimator minimizes mean squared error, the very modest improvement in bias and marginal reduction in variance may be a result of overall lower mean squared error due to randomness in the training process. }
Researchers interested in implementing SyNBEATS will have to decide the extent to which asymptotic unbiasedness is important for their application. For example, if a researcher further decides to tune the SyNBEATS model via time-series cross-validation techniques, overfitting and regularization biases may be exacerbated, in which case a bias-correction may be valuable. Overall, our results in this subsection suggest that combining SyNBEATS with bias-correction techniques can reduce the bias of estimated treatment effects in empirical applications, but in some cases this can come at a substantial reduction in precision.
This paper introduced the SyNBEATS algorithm as a tool for causal inference in panel data settings. In the applications we considered, SyNBEATS consistently yielded lower prediction errors compared to commonly employed estimators (SC and TWFE), as well as comparable or stronger performance than more recently developed approaches (MC and SDID). Supplemental analyses suggest that model architecture and the use of both vertical and horizontal information contribute to SyNBEATS' strong performance. Overall, compared to alternative estimators, SyNBEATS may yield more accurate estimates of the causal effect of policies of interest in a range of realistic panel-data settings.
Our focus has been on developing SyNBEATS and assessing the accuracy of its performance on real data; we have not focused on quantifying the uncertainty of the resulting estimates. In settings where researchers require more conventional standard errors, the approaches that have been proposed for SC-type estimators can be readily adapted to SyNBEATS. For example, the researcher can apply SyNBEATS to placebo treatments obtained by permuting the treatment year or treatment unit abadie2010,shaikh2021randomization. Alternatively, instead of modeling the assignment mechanism directly, one can quantify uncertainty in model prediction using conformal inference methods to obtain finite sample coverage guarantees conformalinf. Lastly, viviano show how one can exploit stationarity and use the block-bootstrap procedure politis1994stationary to construct valid $t$-statistics under a sharp null.
\setcounter{page}{1}