Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
108,621 characters · 15 sections · 62 citation commands
\pagenumbering{Alph} \long\def\symbolfootnote[#1]#2{\begingroup\footnote[#1]{#2}\endgroup}
\setcounter{page}{1} \pagenumbering{arabic}
In social science, we are often interested in the effects of policy interventions on aggregate entities to evaluate previous, understand current, or counsel future policies. The aggregate units may be firms, organizations, geographic areas, etc. Data often stem from observational studies. The task of estimating such effects has been heavily studied, and various methods apply to different data available (for reviews, see, e.g., Imbens2009 and Abadie2018). One approach is to compare the treated unit to a control unit not exposed to the event. One of the first examples is Card1990, who uses Southern US cities as a comparison group to estimate the effect of an unanticipated Cuban migratory influx in Miami. However, the design of a comparative case study faces certain challenges.
First, it is not always transparent how specific control units are chosen, and the appropriate control may be chosen ex-post. Running several regressions may lead to publication bias due to multiple comparisons franco2014publication. Second, many of the current methods to evaluate policies are based on regressions that try to maximize the pre-treatment fit, which may not generalize well out-of-sample. The situation illustrates the classical bias-variance trade-off, where methods are often chosen to minimize bias rather than balancing bias for variance to minimize the mean-squared prediction error. We argue that the problem underlying synthetic controls is in fact a prediction problem, and approaches should be designed specifically to accurately predict the outcome of the treated unit post-treatment in a counterfactual state absent from the treatment. This is especially useful if pre-treatment inference is not a goal in itself (for a discussion on recasting economic problems as prediction problems, see, e.g., Kleinberg2015). Third, the standard approach to comparative case studies is to specify a linear functional form to capture the relationship between the treated unit and the control units. This may be restrictive if we are trying to answer questions for which no economic model exists. In addition, the standard approach does not take nonlinearities, especially interactions, into account except those explicitly modeled by the researcher. If the process that generates the outcomes for the treated unit in the pre-treatment period is nonlinear in the control outcomes, the resulting bias may be severe.
abadie2010synthetic solve the first challenge by relying on the ideas of Abadie2003. In the presence of a single treated unit and several control candidates, synthetic controls form a set of weights such that the weighted average of the control units approximately matches the treated unit in the pre-treatment period. The same weights are then channeled to the post-treatment period to estimate a synthetic control group that constitutes the counterfactual state of the world, in which the treated unit was not exposed to the treatment. By restricting weights to be non-negative and sum to one, the method already reduces the risk of overfitting, but ultimately, however, the method is focused on pre-treatment fit (and inference), which may not generalize out of sample. Doudchenko2016 modify the synthetic controls by allowing for a more transparent and flexible regularization, where weights may be negative and are not required to sum to one. In particular, weights are estimated using the elastic net estimator which, compared to ordinary least squares, tends to shrink the weights towards zero and set some of them exactly to zero. Especially in moderately-high dimensions, this approach has shown promise in forecasting studies. Also, the selection property by zeroing out some weights has attractive interpretations as it allows researchers to pinpoint which control units that have no explanatory power when forming the counterfactual outcome.
Both of the above methods, however, specify a linear model that is not capable of automatically detecting nonlinearities between treated and control units. In particular, we expect many low-order interactions of the control outcomes to be informative in explaining the outcomes of the treated unit. For instance, consider the empirical application in abadie2010synthetic regarding cigarette sales in the US. While the sales in California may be modeled as a weighted average of the sales in New York and Florida given a common cigarette consumption pattern along the coasts, a decrease in sales in New York may be associated with an even bigger decrease in California given a low period of sales in Florida. This could happen if the people of California see themselves as trendsetters in regards to health; when people in both New York and Florida are reducing their cigarette consumption, people of California want to reduce their consumption even further. Note that, in the context of forecasting, it is becoming natural to include interactions and higher-order terms when relying on regularization-based estimators---in contrast to synthetic controls. However, important interactions and higher-order terms can be difficult to anticipate ex-ante. The kitchen sink approach would be to include all higher-order terms up to a pre-specified order, e.g., to the third order. This approach quickly faces its own problems, since even with 10 control units, all terms up to third-order would count $\binom{10+3}{3}-1=285$, which is infeasible to handle for most parametric estimators given a reasonable amount of observations. Thus, if nonlinearities are deemed important or the researcher does not have the domain knowledge required to specify an economic (parametric) model, it might be more appropriate to apply a flexible nonparametric prediction method.
Nonparametric approaches to estimating treatment effects do exist in the econometric toolbox. Indeed, Athey2016, Wager2018, and Athey2019 also rely on ideas from machine learning to study heterogeneous treatment effects using nonparametric models. They propose various modifications of the random forests algorithm of Breiman2001. Moreover, their methods are most suitable when a large set of both observations and covariates is available as they focus on heterogeneous treatment effects, whereas we focus on average treatment effects. As another example, Hartfort2017 use deep neural nets for counterfactual prediction. We find, however, that many applications in social science and ours included do not enjoy the luxury of having sufficiently large datasets available to apply (deep) neural nets.
We recast the problem of estimating a synthetic control as the problem of predicting one, similarly to both Doudchenko2016 and Athey2019 who also advocate for powerful prediction methods. This way, we do not have to rely on linear, parametric models that potentially misspecify the true underlying model. We choose a popular method from the machine learning literature, namely the random forests algorithm, which handles interactions and other nonlinearities automatically. In this way we obtain tree-based synthetic control method as an alternative, particularly suited for applications where the researcher prefers accurate post-treatment predictions over the ability to do pre-treatment inference, and when the empirical question is not guided by any economic model that can justify specific assumptions on the statistical model. This method also allows us to consider all potential controls in the donor pool transparently; in particular, if some control units do not contribute to explaining the treated unit, the method is flexible enough to leave them out. Furthermore, our method naturally captures nonlinearities between treated and control units ---features which alternatively should have been captured by including a (potentially large) number of interactions and higher-order terms. We provide theoretical guarantees for our method as we establish asymptotic unbiasedness and consistency of the random forests predictions as well as consistency of a corresponding estimator of the average treatment effect.
We showcase the tree-based synthetic control method by estimating the effect of relocating the US embassy from Tel Aviv to Jerusalem on the number of weekly conflicts in Israel and Palestine. It is beyond our interest to judge the particular political decision, rather we propose a method to estimate its impact. We use conflict data from December 28, 2015, to November 3, 2018, for Israel and Palestine as well as for 11 of the remaining countries in the Middle East as controls. The data are provided by the Armed Conflict Location & Event Data Project raleigh2010introducing. Our results indicate that the weekly number of conflicts has increased by 26 incidents on average after the relocation was announced on December 6, 2017, until November 3, 2018. This corresponds to more than doubling the number of conflicts. This application highlights the need for nonlinear methods. For instance, when we seek to understand which periods are similar in terms of the level of conflict, it is difficult to consider conflict levels in Iraq and Saudi Arabia separately without an interaction between them. Imagine some violent and frequent conflicts in the South of Iraq in a given period. The regime of Saudi Arabia may react by increasing the appearance of police forces in major cities, and as a result, the number of conflicts falls. If such interactions matter for the conflict level in Israel and Palestine, we would incur an omitted variable bias by leaving them out. We use the recently proposed conformal inference test by Chernozhukov2017b to formally justify our results. The increase is statistically significant at a 1% level.
The proposed method uses the pre-treatment periods to estimate the relationship between the treated and control units and it imposes this relationship in the post-treatment period, similarly to abadie2010synthetic and Doudchenko2016. Our model for the conditional expectation, the canonical random forests regression model, have proved successful in many applications (see, e.g., Montgomery2018 for a recent paper in political science, or Ng2019 in IO). Further, variants of random forests have already been employed in the treatment effects literature either directly Athey2016,Wager2018,Athey2019 or indirectly Chernozhukov2017a,Chernozhukov2018. Common to these papers is that they rely on the unconfoundedness assumption and assume there is a relationship between outcomes for a given unit over time (estimated by regressing control unit outcomes in treated periods on lagged outcomes) that is stable across units. In contrast, the synthetic control literature assumes there is a relationship between different units (estimated by regressing treated unit outcomes on control outcomes) that is stable over time. Our approach falls into the latter. Intuitively, for each period where the treated unit is treated, our model locates a few corresponding pre-treatment periods based on the control units and uses the average of the pre-treatment outcomes of the treated unit as a counterfactual prediction in the post-treatment period. Stated differently, our model aggregates the pre-treatment periods into similar subgroups based on the control units. Then, it computes the average of the outcomes of the treated unit in each of the subgroups. In the post-treatment period, the model remembers how to group the periods and assigns the corresponding pre-treatment average to each of the periods. This gives an estimate of the potential outcome for the treated unit in the absence of the treatment. Having an estimate for all periods after the intervention, we compute the average of the differences between the estimate and the actual outcome, similarly to Chernozhukov2018ate.
Using the application, we compare our proposed method to state-of-the-art methods on several metrics. We show that our method performs at least on par with competing approaches, while enjoying the benefits of being more off-the-shelf. In particular, we impose fewer assumptions on the relationship between the treated unit of interest and the units in the donor pool, which may become beneficial when no economic model exists to guide the researcher. When this relationship is indeed linear, our method is still able to recover it, although the standard methods may be more efficient. All methods considered agree on the magnitude of the treatment effect.
The rest of the paper is organized as follows. Section (ref) introduces the framework underlying synthetic controls and the tree-based synthetic method is formally presented. In addition, this section presents theoretical guarantees. Section (ref) considers the context of Israel and Palestine and presents the results alongside several robustness checks. Section (ref) compares our method to state-of-the-art econometric methods. Section (ref) concludes. All proofs are deferred to the Appendix.
We consider $N+1$ cross-sectional units observed in $T$ periods and assume without loss of generality that only the first unit is exposed to the treatment, leaving $N$ units as controls\footnote{We will use treatment and intervention interchangeably.}. At any given time $t = 1,\dots, T$, we group the observations as $(X_t,Y_t)$ where $X_t\in \mathcal{X}\coloneqq \mathcal{X}_1\times \cdots \times \mathcal{X}_N$ consists of the $N$ control units and $Y_t\in \mathbb{R}$ is the treated unit. Here $\mathcal{X}_i \subseteq \mathbb{R}$ reflects the support of the $i$th control unit. We assume that the treatment occurs at time $T_{0}<T$, leaving the first $T_{0}$ periods as the pre-treatment period. In many applications, ours included, the treatment may have an effect before implementation via announcement or anticipation, and $T_{0}$ should be redefined accordingly. We assume implicitly that the treatment does not affect the outcome for the control units (cf.\ (ref) and (ref) of Appendix (ref)). For a thorough discussion on this assumption, see e.g. Rosenbaum2007. Let $Y_{t}^{0}$ denote the potential outcome that would be observed for the treated unit at time $t$ in absence of treatment, and similarly, let $Y_{t}^{1}$ denote the potential outcome that would be observed if exposed to the intervention. In particular, we have that
We define $\tau_{t}=Y_{t}^{1}-Y_{t}^{0}$ as the effect of the intervention at time $t$ for the treated unit. Assuming that both $(X_t,Y^0_t)$ and $(Y^1_t)$ are stationary processes, the average treatment effect (ATE) can thus be characterized as
We remark that, under the mild assumption of ergodicity of the sequences $(Y^0_t)$ and $(Y^1_t)$, a simple consistent estimator of $\tau$ is given by the difference in averages before and after treatment, that is,
This estimator is based solely on $Y_1,\dots, Y_T$ and does not make use of information carried in the control units $X_t$ for $t=1,\ldots,T$. Alternatively, to make use of the information carried in the control units, we rely on a prediction-based estimator of $\mathbb{E}[Y^0_1]$ where the unobserved (no intervention) outcomes $Y^0_t$, $t=T_0+1,\dots, T$ are imputed by means of regression. The simple unbiased estimate often turns out to be of limited practical value as it fails to deliver a precise estimate of the average treatment effect (for a related discussion in the context of randomized control trials, see, e.g., Deaton2018). Another benefit of our approach relative to the simple average is that it unlocks estimates of $\tau_t$ for all $t=T_0+1,\ldots,T$, which are intrinsically interesting to study as the effects may increase or decrease over time.
Regression imputation is a well-known strategy in the context of missing data (see, e.g., musil2002comparison,shao2002sample). The basic idea behind this method is that, with $f(x) = \mathbb{E}[Y^0_1\mid X_1=x]$ being the regression function, $f(X_t)$ delivers a “good” proxy of $Y^0_t$ for $t=T_0+1,\dots, T$. Note that $X_{t}$ could include covariates other than the control units as long as they are not affected by the intervention. For instance, we would not be able to include stock market indicators for Israel and Palestine. For simplicity, however, we follow abadie2010synthetic and focus on using the control units. Also note that $f(X_t)$ has the same distribution as $\mathbb{E}[Y_1^0\mid X_1]$ and not $Y^0_1$. However, since their means coincide, this is not an issue in the estimation of the ATE. The resulting estimator $\hat{\tau}$ of $\tau$ that we will employ in the empirical application is simply obtained by replacing the second average in (ref) by the average over predictions of $Y^0_{T_0+1},\dots, Y^0_T$, i.e.,
where $\hat{\tau}_t = Y^1_t - \hat{f}(X_t)$, where $\hat{f}$ is a suitable estimator of $f$.
The framework outlined above coincides with the idea of abadie2010synthetic to the extent that we also use regressions to estimate the relationship between treated and control units in the pre-treatment period and assume that the estimated relationship continues into the post-treatment period. However, to estimate the ATE using in (ref), we choose a popular and flexible estimator, $\hat{f}$, for the regression function called random forests regression Breiman2001. Using random forests as an estimator for $f$ and under suitable regularity assumptions, we prove in the following that $\hat{\tau}$ is a consistent estimator of $\tau$ in the sense that
However, we begin by providing some intuition behind how random forests work in the context of the tree-based synthetic control method.
The cornerstone of random forests is a single regression tree. Regression trees are obtained by recursively partitioning the input space (i.e., the possible values of the control units) into cleverly chosen subsets (nodes), and then they output a constant value for all inputs within the same terminal node (also called a leaf). Specifically, any (post-treatment) outcome of the control units belongs exactly to one particular leaf, and to form the counterfactual prediction for the treated unit, the model uses the average pre-treatment outcome of the treated unit based on the corresponding outcomes of the control units falling into the same leaf. The recursive partitioning is formed in such a way that the leaf associated to a given outcome can be identified by asking a sequence of questions such as “Is the outcome of the $i$th control unit above $10$?”, “Is it below $20$?”, “Is the outcome of the $j$th control above $15$?”, etc. Indeed, starting from the entire input space, the procedure works by selecting a node to split, a split direction (which of the $N$ control units to ask a question about), and a split position (the level to exceed or stay below). The node to split is called the parent node of the resulting two subnodes, which are also called child nodes. The random forests estimate is simply obtained by averaging over $B$ regression trees, which differ due to exogenous randomness injected in the recursive partitioning procedure.
Figure (ref) shows an example related to our application. In the example, we divide the weekly level of conflicts in Israel and Palestine at each period $t\leq T_{0}$ into bins based on the weekly level of conflicts in Saudi Arabia and Iraq. Given an observation of the weekly level of conflicts in Saudi Arabia and Iraq at a new point in time, say $t'>T_{0}$, we decide which of the four leaves that $t'$ belongs to. As an example, suppose this observation ends up in the first leaf, Subgroup 1. Our prediction of the weekly level of conflicts in Israel and Palestine is then the average of all observations that fall into Subgroup 1 in the pre-treatment period. Hence, the outcomes for Saudi Arabia and Iraq enter only in the stratification and, thus, the approach also allows the inclusion of other covariates.
To continue the example above, a possible data-generating process (DGP) that fits nicely to this tree-based framework would be
where $Y_{\cdot,t}$ denotes the conflict level in period $t<T_{0}$, $IP$ abbreviates Israel-Palestine, $SA$ Saudi Arabia, and $IR$ Iraq. An additive model that does not explicitly take the interaction into account would suffer from omitted variable bias. On the other hand, the random forests model requires no such knowledge, since it will automatically detect the (unknown) functional form of the regression function as long as $(Y_{SA,t},Y_{IR,t})$ are included as control units---this will be shown in the theoretical results below.
Let $\mathcal{D}_{T_0} = \{(X_1,Y^0_1),\dots, (X_{T_0},Y^0_{T_0})\}$ denote the data from the pre-treatment period used to build the estimator $\hat{f}$. In the following we argue that, under suitable assumptions, various types of random forests are consistent estimators of $f$. Then, we establish consistency of the tree-based synthetic control method in the context ATE estimation. Details of the presented results, as well as their proofs, can be found in Appendix (ref).
In line with davis2020rf and wager2015adaptive we consider a subclass of random forests, which we call $(\alpha,k,m)$-forests, parameterized through $\alpha \in (1/2)$, $k\geq 1$, and $m \geq 2k$. The triple $(\alpha,k,m)$ indicates that the trees of the forest obey the following rules:
We will now attach some intuition to (ref)--(ref). The first rule (ref) ensures that leaves are not too large. Since we also impose the (rather technical) rule (ref) that splits cannot be too “unbalanced”, it effectively means that we require a large number splits before getting to a leaf (i.e., many questions should be asked about the outcome of the control units before the associated leaf can be identified). Concerning (ref), this is imposed to ensure that sample averages within leaves stabilize such that they are not too far from their theoretical counterparts. Finally, (ref) implies that many splits will be placed across any of the $N$ directions of the input space; in other words, the partition associated to the given tree is fine in all directions.
\paragraph*{Consistency of random forests regression} Consistency of similar forests was established in wager2015adaptive in the setting where $(X_1,Y^0_1),\dots, (X_{T_0},Y^0_{T_0})$ are assumed to be i.i.d., and davis2020rf considered an autoregressive setup, $X_t = (Y^0_{t-1},\dots, Y^0_{t-p})$, when $Y^0_t$ is a $p$-th order Markov chain. None of these settings, however, are suited for our application. Therefore, in Appendix (ref), we formally introduce the assumptions needed on the data-generating process to establish Theorem (ref) (see Assumption (ref)). Most of the assumptions are not restrictive from a practical point of view and are mainly imposed for the sake of simplicity and to avoid other, less transparent, assumptions. Note that we assume that the sequence $(X_t,Y^0_t)_{t\in \mathbb{Z}}$ is stationary and has exponentially decaying strong mixing coefficients. This is classical when proving asymptotic results, particularly when the results contain information on convergence rates as is the case in Lemma (ref) (in such situation, ergodicity is not sufficient). It is satisfied for a wide range of stationary processes; e.g., ARMA processes, Markov chains, and other short-memory time series are included in this setting. We are now ready to formulate our consistency result for $(\alpha, k ,m)$-forests.
While it is always of particular interest to know about convergence rates as well, we note that this depends heavily on the rate of the random forests estimator $\hat{f}$. Such results are very difficult to prove and exist only in idealized settings---in particular, results are only available in the case of independent observations and the restrictions on the random forests algorithm are often rather strict and unrealistic in practice.
\paragraph*{Consistency of the tree-based synthetic control method} We can now apply Theorem (ref) to prove consistency of $\hat{\tau}$ as given in (ref). We will, however, need a slightly stronger assumption than Assumption (ref). In particular, we assume that the sequence $(X_t,Y_t^0)_{t\in \mathbb{Z}}$ has exponentially decaying $\beta$-mixing coefficients, which is imposed to be able to estimate certain expectations. Many stationary processes satisfy the $\beta$-mixing condition as well; e.g., ARMA and Markov processes. In addition, we assume that the sequence $(Y^1_t)_{t\in \mathbb{Z}}$ is ergodic, and $\mathbb{E}[\vert Y^1_t\vert ]<\infty$, which is needed for the sample average $\sum_{t=T_0+1}^TY^1_t$ to converge to $\mathbb{E}[Y^1_1]$ (and for the latter to be well-defined and finite). The full set of assumptions needed to establish Theorem (ref) is provided in Appendix (ref) (see Assumption (ref)). We can now formulate our consistency result for tree-based synthetic control methods.
Monday afternoon December 6, 2017, the US President fulfilled a major campaign promise by announcing the relocation of the embassy from Tel Aviv to Jerusalem, which took place on May 14, 2018. Many international media reported intensively on the move that broke with decades of US policy by recognizing Jerusalem as the capital of Israel, although former US presidents have also been commenting on the relocation. For instance, Bill Clinton supported recognizing Jerusalem as the capital and the principle of moving the embassy there. George W. Bush said before taking office that he intended to move the embassy, and Barack Obama spoke of Jerusalem as the capital of Israel that ought to remain undivided. However, the former presidents all consistently signed waivers to postpone the move.
The relocation should be viewed as the most recent event in the ongoing Israeli-Palestinian conflict, dating back to the mid-20th century in which the Jewish immigration and the sectarian conflict in Mandatory Palestine between Jews and Arabs took place. In 1948, the establishment of the State of Israel alongside the State of Palestine was proclaimed, and US President at the time Harry S. Truman recognized the new nation. Since 1967, Israel has held all of the pre-war cities of West and East Jerusalem, and in addition, the Gaza Strip has been under Israel's control. Ever since, several wars have been fought between the Arab countries and Israel, and a permanent solution is still to be found. For a complete review and analysis of the Israeli-Palestinian conflict, see Frisch2004 and Eriksson2018.
We use daily country-level panel data in the period December 28, 2015, to November 3, 2018, on conflicts reported by the Armed Conflict Location & Event Data Project raleigh2010introducing. The conflicts cover riots, protests, strategic development, remote violence, violence against civilians, various types of battles, and headquarter or base establishments. We consider the aggregate of all conflicts and leave the disaggregating for further research. The data consist of multiple daily observations which we aggregate into weekly observations to smooth the daily variations. We have no other data on a daily or weekly frequency. The treated countries considered are Israel and Palestine, which we aggregate into one treated unit to take into account the interdependency of the two countries Arnon2001.\footnote{We sometimes refer to Israel and Palestine as Israel-Palestine.} Aggregating them into one treated unit rather than having one of them, say Israel, as a potential control is necessary to meet the assumption of no interference between units. One may be interested in the effects on Israel and Palestine separately, leaving out the other country completely to avoid interference. Another reason to aggregate Israel and Palestine into one is because several of the reported conflicts happen at the border between the two countries, which favors the aggregation.
An interesting hypothesis is whether the conflicts in Palestine accelerate earlier than the conflicts in Israel. However, this is hard to measure, as the conflicts in both countries may be initiated by people from either place, making it difficult to disentangle the effect in Israel from the effect in Palestine. As we are interested in the overall effect in the area, we aggregate the countries for now and leave the other hypothesis for future research.
The control countries we consider are all the remaining countries in the Middle East but Syria and Iran, which include Bahrain, Iraq, Jordan, Kuwait, Lebanon, Oman, Qatar, Saudi Arabia, Turkey, United Arab Emirates, and Yemen, giving us a total of 11 control countries. The data coverage for Syria starts from January 2017, and instead of restricting our sample to begin here, we choose to exclude Syria. We also exclude Iran because of its involvement in the Israeli-Palestinian conflict and its relation to the US, which make it too difficult to justify the assumption of no inference between units (see Buonomo2018 for an analysis of the Iran-US relation).
In fact, if we compare the trends in the weekly level of conflicts in Iran and Israel-Palestine before and after the move of the embassy, the co-movement is clear. We document the trends in the weekly number of conflicts for all countries in the Middle East except Syria in Appendix (ref). The pre-intervention period covers 101 weeks, starting December 28, 2015, and ending December 3, 2017, just before the announcement. The post-intervention period begins on December 4, 2017, and ends on November 3, 2018, leaving 48 weeks for estimating the average level of conflicts in Israel and Palestine in the counterfactual situation where the US embassy is not relocated. Summary statistics for the weekly number of conflicts across the Middle East countries are provided in Table (ref).
Further, we show the distribution of the weekly number of conflicts in Israel-Palestine in both the pre-treatment and post-treatment period in Figure (ref). It follows from Figure (ref) that the distribution is shifted to the right in the post-treatment period, which tentatively suggests that violent weeks tend to occur more often in the post-treatment period. Last, Figure (ref) shows the level of conflicts over time in Israel-Palestine as well as the average of the remaining countries. As noted in Section (ref), under mild assumption of ergodicity of the sequences $(Y^0_t)$ and $(Y^1_t)$, a simple before-after comparison is sufficient to identify the average treatment effect, which in this case would be 23.5 weeks. This simple yet unbiased estimate is roughly in line with the results we show next.
Our application is motivated by Figure (ref), showing the weekly number of conflicts in Israel-Palestine over the entire sample period. The two vertical lines indicate the date when the relocation of the US embassy was announced and the date of the actual move, respectively, and also, we plot the average of the remaining countries. A couple of observations are worth noting. First, visual inspection suggests that the average weekly number of conflicts in Israel-Palestine has in fact increased subsequent to the announcement. In contrast, the average number of weekly conflicts over the remaining countries in the Middle East does not appear to follow the same upward shift after the announcement. We formalize this shortly.
Second, the volatility of the weekly number of conflicts in Israel-Palestine seems much higher after the announcement, supporting the histogram in Figure (ref). This has important economic implications, as it indicates that conflicts tend to cluster and that misfortunes never come singly. Considering the conflicts more closely, for instance analyzing the degree of violence in the clusters, is interesting, but we postpone this for future research.
Finally, note the large spike in the average number of conflicts across the remaining countries in the Middle East around July 2016. Specifically, the week with the highest average number of conflicts runs from July 18 to July 24, which is just after the military coup was attempted in Turkey on July 15 against state institutions, including the government and President Erdo\u{g}an. During the coup, more than 2,100 people were injured and over 300 were killed. This rare event shows up in the estimation for some methods that are exposed to outliers.
Figure (ref) displays the weekly number of conflicts for Israel-Palestine and its estimated counterpart during the period December 28, 2015, to November 3, 2018. The observed level of conflicts in Israel-Palestine is closely followed by the estimated counterpart in the entire pre-intervention period until the move was announced on December 4, 2017. This suggests that the time periods before the announcement can be grouped together into homogeneous subgroups based on the level of conflicts in the neighboring countries, and for these subgroups of time periods, the level of conflicts in Israel and Palestine is relatively constant. In fact, the average of the observed weekly number of conflicts in the pre-intervention period is 25.32, whereas the estimated counterpart is 25.41, indicating an accurate fit on average. Note that the estimated counterpart to Israel-Palestine is always closer to the average level of weekly conflicts instead of capturing the spikes to the fullest extent. The is an attractive feature of the averaging that happens in our model as the model implicitly becomes conservative.
Altogether, we take this as evidence that the tree-based synthetic control method can be used to predict a counterfactual Israel-Palestine, which provides a sensible approximation to the true level of conflicts that would have occurred in that region in absence of the move. Thus, we next use the tree-based synthetic control method to estimate the average treatment effect of moving the embassy.
We estimate the effect of the relocation of the US embassy for each of the 48 weeks after the announcement as the difference between the observed level of conflicts in Israel-Palestine and its counterfactual analog. The differences follow as the discrepancies between the two lines in the shaded area of Figure (ref). Immediately after the move is announced, both the observed and counterfactual level of conflicts increase, but to very different degrees, and in fact the observed level of weekly conflicts in Israel and Palestine reaches its maximum level across the entire sample within the first week of the announcement.
For the rest of the post-announcement period, the observed level of conflicts sees a higher base level with distinctly conflict-ridden weeks, whereas the counterfactual Israel-Palestine maintains the lower base level from the pre-announcement period. Specifically, the average of the observed number of weekly conflicts in the post-intervention period is 48.88, whereas the estimated counterpart is 22.78, indicating a significant difference. This suggests that the relocation of the embassy has a numerically positive effect on the level of conflicts in Israel and Palestine, meaning that the level generally increases in the entire post-announcement period.
We assess the weekly estimates of the impact directly in Figure (ref), where we plot the differences between the observed and estimated number of weekly conflicts in Israel and Palestine. Figure (ref) unveils the same story as Figure (ref). The gap of approximately zero on average in the pre-intervention period indicates that the tree-based synthetic control method is able to approximate well the true level of conflicts, albeit very fluctuating. To be precise, the average difference between the observed and estimated weekly number of conflicts in the pre-intervention period is only -0.09. This is obviously a heuristic as average discrepancy does not per se capture accurate fit, and this number should considered jointly with Figure (ref).
Using all 48 weeks after the announcement, our results show that the level of conflicts in Israel and Palestine is increased by an average of more than 26 incidents per week, which corresponds to an increase of approximately 103%. The estimated average effect is associated with a bootstrapped standard error of 2.67 using 10,000 block bootstrap samples with block length equal to 3. That is, the 95% bootstrap confidence interval of the weekly increase is between 20.88 and 31.36. This translates into a percentage point change between roughly 82-124%. We acknowledge that the confidence interval is rather wide, which is not surprising due to the volatility in the number of conflicts across weeks. The results are insensitive to the choice of block length.
Naturally, the assumption of no interference between the treated and control units can be violated in several ways in the context of analyzing the effect of moving the US embassy. The Israeli-Palestinian conflict is an issue in all of the region, and the ties between the countries are complex to understand. For instance, we choose to exclude Iran in the sample, because the Iranian government has played an active role in the conflict. The results with and without Iran are, however, not significantly different, because the tree-based synthetic control method averages over the number of conflicts in Israel-Palestine and uses only the neighboring countries, i.e., the controls, to stratify the time periods.
This feature of the method makes it more robust to the potential violations compared to methods that base the estimates on the outcomes for the control units. Further, the average weekly number of conflicts across all control countries does not differ between the pre- and post-intervention period. In particular, the average over the control countries in the pre-intervention period is 32.80, whereas the same figure is 30.82 in the post-intervention period. The small difference is likely to be driven by the coup attempt in Turkey.
The placebo tests we review shortly reveal that no other relevant country experienced the same effect of the relocation of the US embassy. Last, the conformal inference test in Section (ref) provides evidence that our model is correctly specified and that the increase is statistically significant. Taken altogether, it is our judgment that the potential violations do not appear to be severe in this context.
We want to assess how much our results are driven by mere chance. If we are able to obtain estimated effects of the same magnitude for the control countries as for Israel-Palestine by relabeling treatment and control unit, we would not be able to interpret our analysis as providing any significant effects. To make inference about the effect of the embassy relocation, we follow the strategy outlined in abadie2010synthetic, Bertrand2004, and Abadie2003 and run placebo tests.
Placebo tests re-do the original analysis, but switch the roles between the treated unit and a randomly chosen control unit, the rationale being that using the control unit not exposed to treatment should lead to an estimated effect of approximately zero. By applying the tree-based synthetic control method individually to all the countries in the donor pool, we can therefore evaluate the significance of our analysis. We expect one of two outcomes. If the placebo tests deliver estimates of the average effect of similar magnitude as for Israel-Palestine, we cannot rightfully interpret our results as evidence for a significant effect. If, on the other hand, that none of the placebo tests for the countries in which the US embassy was not moved lead to similar estimated effects, then we take this as evidence that our tree-based analysis documents a significant effect of moving the US embassy in terms of an increased level of conflicts. One important condition, however, is that the pre-intervention fit to the weekly number of conflicts is precise for the country in question when we run the placebo test.
To assess the significance of our estimates, we perform a series of placebo test for which we create a counterfactual state of the world. That is, we iteratively treat each control country in the remaining Middle East as if it had experienced a move of the US embassy at exactly the same time as the move in Israel, while we also reassign both Israel and Palestine to the control group. In each iteration, we apply tree-based controls to the respective country to estimate the impact of the fictive embassy move on the weekly number of countries. The series of placebo tests gives us a distribution of differences between the observed and estimated number of conflicts over the countries.
Figure (ref) plots the differences in the observed and estimated number of conflicts for all the placebo analyses and the original analysis. The blue line shows the case for Israel-Palestine, reproducing Figure (ref). The other lines show the same differences estimated by the tree-based synthetic control method, but for each of the 11 control countries in the donor pool. Figure (ref) indicates that the tree-based synthetic control method provides an accurate fit in the pre-intervention period for Israel and Palestine as well as for most of the control countries.
In particular, the pre-intervention root mean squared prediction error (RMSPE) for Israel-Palestine is 5.77, where RMSPE is computed as the root average of the squared differences between the observed and estimated weekly number of conflicts. The pre-intervention median RMSPE for the control countries is 1.71. This should not be taken as evidence that the ability to fit the pre-intervention is higher for the control countries than for Israel-Palestine. In fact, mean RMSPE over the control countries is 9.51, indicating that a few control countries stand out in terms of high RMSPE, while for most control countries, we achieve a very low RMSPE. This is supported by Figure (ref) from which it is apparent that the pre-intervention fit is very imprecise for some countries.
The country with the worst fit is Turkey with an RMSPE of 61.88. This result, however, is not surprising due to the attempted military coup in 2016 that led to an extreme spike in the number of conflicts. As this coup attempt was, of course, unanticipated, the conflict situation in the other countries was normal, and therefore, no statistical method would be able to capture this outlier. Similar problems arise for Iraq and Yemen, which are the countries with the overall highest variation in the weekly number of conflicts. This high variation makes it difficult for the tree-based synthetic control method, and likely any other method, to produce a valid fit in the pre-intervention period without imposing too much flexibility. As a result, the RMSPE for Turkey, Iraq, and Yemen are all more than double that of Israel-Palestine and any other control country.
To handle the countries for which the tree-based synthetic control method gives a poor fit, we follow an argument provided in abadie2010synthetic as they encounter the same issue for some of the states. If the tree-based synthetic control method had failed to deliver a reasonable fit to the observed weekly level of conflicts in the pre-intervention period for Israel-Palestine, we would treat the lack of fit as evidence that the estimated increase in the weekly number of conflicts in the post-intervention period was arbitrary and not caused by the move of the US embassy. Analogously, we cannot take into account the estimated effects in the post-intervention period for Turkey, Iraq, and Yemen when assessing the degree of chance in our results for Israel-Palestine.
Consequently, we provide another version of Figure (ref) in which we have excluded the placebo tests for Turkey, Iraq, and Yemen. This effectively corresponds to removing countries for which the RMSPE is more than double the one for Israel-Palestine. Figure (ref) provides the restricted version of Figure (ref) from which we have excluded Turkey, Iraq, and Yemen. The median RMSPE over the remaining countries in the Middle East drops to 0.35, and the corresponding mean drops to 1.37.
Removing the countries for which the tree-based synthetic control method would be ill-advised tells a clear message. The largest estimated effect on the weekly number of conflicts in the post-intervention period is to be found for Israel-Palestine. More precisely, while the average estimated effect for Israel-Palestine is 26.12 in the post-intervention period, the corresponding figure over the placebo tests is 1.38. For the pre-intervention period, the estimated gaps are -0.09 and -0.02, respectively.
The use of placebo tests as a mode of inference for synthetic controls is heavily debated (see, e.g., Hahn2017). We emphasize that placebo tests evaluate significance relative to a benchmark distribution (here, a uniform distribution) for the given assignment mechanism in the data. This permutation-based test is conditional on the sample and exploits the randomness induced by the (placebo) assignment mechanism. In contrast, sample-based tests are conditional on the assignment mechanism and exploit the randomness in the DGP. But because the sample mechanism is not well-defined and the sample is in fact the population in the cross section (all countries in the Middle East are included), sample-based tests are often complicated in this settings abadie2019using.\footnote{We thank Alberto Abadie for pointing this out.}
We consider another approach to assessing the significance of our results, namely computing ratios of post-/pre-intervention measures both for Israel-Palestine and the control countries. As abadie2010synthetic, we compute the ratios in terms of RMSPE. Arguably, the advantage of comparing ratios relative to post-intervention gaps is that we do not necessarily have to exclude ill-fitting placebo runs in an iterative way as demonstrated by figures (ref) and (ref). For instance, although the RMSPE for Turkey is the highest across all in the pre-intervention period, it is similarly high in the post-intervention period, and the ratio will be more robust to this.
The only countries with a higher ratio of post-/pre-intervention RMSPE than Israel-Palestine are Jordan and Oman. This observation, however, does not cause much concern when we take into account the gaps in both periods. For Jordan, the pre-intervention gap between the observed and estimated weekly number of conflicts is -0.02, whereas the same figure is 0.46 in the post-intervention period. Likewise, the figures for Oman are -0.00 and 0.06, respectively. Thus, the high ratios of post/pre-intervention RMSPE for the two countries are likely driven by a few very conflict-ridden weeks after the intervention.
In addition to the ratios of post-/pre-intervention RMSPE used in abadie2010synthetic, we also compute the ratios of post-/pre-intervention mean absolute error (MAE) between the observed and estimated weekly number of conflicts. Using either the ratio of post-/pre-intervention RMSPE or MAE has different advantages. RMSPE penalizes large errors more than MAE, but MAE is more interpretable. We provide both ratios for each country in Table (ref), in which we also provide the respective pre- and post-intervention measures. Note from Table (ref) than Oman is the only country with a higher ratio of post-/pre-intervention MAE than Israel-Palestine. In absolute terms, again, the result for Oman is not too disturbing for our analysis.
We consider one last approach to draw inference about our results. Recall that our proposed method as well as the other methods considered relies on cross-sectional regressions. Whenever the joint distribution of the data is not well-approximated by cross-sectional regressions, the model will provide a poor global fit in the sense that not all $N$ controls will fit the model, which is exactly the case in our application as well as in abadie2010synthetic. In this situation, Chernozhukov2017b propose an exact and robust conformal inference method along with an associated validity test. In the following, we rely on the validity test of the needed assumptions rather than going into the theoretical aspects.
The method requires only a good local instead of a good global fit, as it relies solely on a suitable model for the treated unit and it focuses on the time-series dimension. Essentially, the procedure postulates a null trajectory and tests the sharp null hypothesis $\mathcal{H}_{0}:\tau_t=\tau_t^{o}$ for $t=T_0+1,\ldots,T$. For the test to be valid, the estimator of the counterfactual outcome for the treated unit needs to be consistent and stable and be able to provide residuals that are exchangeable.
To assess the plausibility of the key assumptions, Chernozhukov2017b provide placebo specification tests. The conditions result in non-asymptotic validity of the test, meaning that the $p$-value is approximately unbiased in size Chernozhukov2017b. The proposed inference method is valid for stationary and weekly dependent data.
We are interested in testing the hypothesis that the trajectory of the policy effects in the post-treatment is zero. Hence, our main hypothesis is
The test statistic $S$ is based on the $\left(\left(T-T_{0}\right)\times1\right)$ vector of residuals of our model $\hat{u}_t$ for $t=T_0+1,\ldots,T$. The test statistic is then defined by
where we set $q=1$. To compute $p$-values, the test relies on two different sets of permutations, the i.i.d permutations denoted $\Pi_{\text{i.i.d}}$ and the moving block permutations denoted $\Pi_{\rightarrow}$. The moving block permutations are necessary if the sequence of residuals exhibits serial dependence. The $p$-value is estimated as $\hat{p}=1-\hat{F}\left(S\left(\hat{u}_{T_0+1},\ldots,\hat{u}_{T}\right)\right)$, where
To assess the validity of the assumptions underlying the test, the first step is to perform a placebo specification test. Based on the outlined procedure, the idea is to test the null hypothesis that
for a given $\kappa\geq1$ based on pre-treatment data. The null hypothesis (ref) is true if the underlying assumptions are correct. Thus, rejecting the null provides evidence against a correct specification. For proofs and additional details, we refer to Chernozhukov2017b.\footnote{Note that Chernozhukov2017b also provide a test for the average effect over time. However, this requires the total number of periods to be much larger than the post-treatment periods, which is not the case in our application.}
We begin the analysis by testing the underlying assumptions of our proposed method, i.e., consistency, stability, and exchangeability of the residuals. We apply both i.i.d. permutations and the moving block permutations. We use $\kappa=10$ and randomly sample 10,000 elements from the set of all permutations with replacement for the i.i.d. permutations. The resulting $p$-values follow from Table (ref). All $p$-values from both permutation schemes are above 60% and most of them are above 80%, and thus, we fail to reject the null hypothesis. This serves as evidence for a correct model specification. We further see that the $p$-values differ slightly between the i.i.d. permutations and the moving block permutations, where the $p$-values tend to be lower using moving block permutations. This provides evidence for some serial dependence in the residuals.
Next, we turn to our main hypothesis in (ref). We consider again both the i.i.d. permutations with 10,000 random samples as well as the moving block permutations. The $p$-value based on the i.i.d. permutations is 0.000, whereas the $p$-value based on the moving block permutations is 0.007. We reject the null hypothesis in both cases given both $p$-values are below 1%, providing evidence that the trajectory of the policy effects from the embassy relocation is different from zero. The formal test results thus appear to be in agreement with the other inference results provided in this section.
In Section (ref), we provide evidence that the decision to move the US embassy from Tel Aviv to Jerusalem has resulted in a significant increase in the weekly number of conflicts in Israel and Palestine. We assess the robustness of our results in several ways, including performing formal inference tests, conducting a series of placebo runs, and evaluating the fit on different measures such as ratios of post-/pre-intervention RMSPE and MAE. In this section, we compare the tree-based synthetic control method to three state-of-the-art methods in the econometric literature. We begin by introducing the methods.
abadie2010synthetic also consider a version of (ref), but assume linearity of $f$ in $X_{t}$. In particular, abadie2010synthetic assume that there exists a set of perfect weights $\omega^{*}=\left(\omega_{1}^{*},\ldots,\omega_{N}^{*}\right)'$ such that $\left\langle \omega^{*},X_{t}\right\rangle =Y_{t}$ for $t=1,\ldots,T_0\le T_{0}$. Considering $Y_{t}-\left\langle \omega^{*},X_{t}\right\rangle $, abadie2010synthetic prove that its mean is approximately zero under standard conditions, which suggests using $\hat{\tau}_{t}=Y_{t}-\left\langle \omega^{*},X_{t}\right\rangle $ as an estimator for $\tau_{t}$ in periods $t>T_{0}$. The weights are then estimated by
which in practice can estimated by constrained least squares. The synthetic control method is mainly tailored for empirical settings with relatively more time periods than control units, i.e., $T_0 \gg N$.
Doudchenko2016 propose a regularized extension to synthetic controls, namely the elastic net estimator. The optimization problem is similar to (ref) but adds a regularization term to the objective function with inspiration from shrinkage estimation. Let $\left(\lambda,\alpha\right)\in\mathbb{R}\times\mathbb{R}$ be a given pair of hyperparameters to be tuned, and let $\mu\in\mathbb{R}$ be an intercept, capturing the possibility that the outcomes for the treated unit are systematically different from the other units. Then, Doudchenko2016 propose to estimate the weights by
Note that (ref) neither requires zero intercept, weights summing to one, nor non-negative weights. The elastic net estimator enjoys the selection property known from Lasso by the $\ell_{1}$-penalty term Tibshirani1996,Zou2005. Essentially, some weights are likely to be zeroed out, meaning that some control units are not predictive of the treated unit.
Both the synthetic control and the elastic net estimator may be viewed as cross-sectional regressions in which the outcome of the treated unit is regressed on the outcomes of the control units in the pre-treatment period. Assuming stability over time, the cross-sectional pattern is then carried over into the post-treatment period, based on which the counterfactual outcome for the treated unit is predicted using the control units. This form of regression in causal panel data models is known as vertical regressions, a term coined by Athey2018matrix. The (almost) symmetric formulation is known as horizontal regressions, where the post-treatment outcomes are regressed on the pre-treatment outcomes using only the control units. This time-series approach estimates a relationship which is then applied to the treatment unit assuming stability across units and requires $N\gg T$. It is not a symmetric problem because the order of $T$ matters in contrast to the order of $N$.
However, both methods have a disadvantage in cases with $T\approx N$ as they do not fully exploit the panel structure by running either cross-sectional or time-series regressions. A recent approach to causal panel data models that takes both sources of variation into account is the matrix completion method by Athey2018matrix, treating $Y_{t}^{0}$ for $t>T_{0}$ as missing. We are now ready to compare the methods introduced.
First, we apply the synthetic control method, serving as a baseline model. Then, we apply the regularized counterpart, i.e., the elastic net estimator. The matrix completion method combines elements from vertical and horizontal regressions, and it is the last method we include.
Figure (ref) shows the observed and estimated number of weekly conflicts in Israel-Palestine for all four methods, and two features of the methods are noticeable. First, the fit in the pre-intervention period gives an idea of the ability to approximate the weekly level of conflicts in Israel-Palestine, which is highly fluctuating. The synthetic control method, the elastic net estimator, and the matrix completion method are comparable in terms of pre-intervention fit, the matrix completion method being marginally in the lead. The reason the elastic net estimator performs slightly better compared to the synthetic control method is likely because the elastic net is less restrictive when estimating weights. None of the comparison methods, however, are able to approximate the weekly level of conflicts in the pre-intervention period as well as the tree-based control method.
Second, the variation in the estimated counterfactuals in the post-intervention period hints at the degree of overfitting, particularly, if there is no or limited variation. If a given method fits only to noise in the pre-treatment period, the post-treatment predictions will be roughly constant because the associated noise do not match the fitted noise. Both the elastic net estimator and the tree-based synthetic control method appear to deliver reasonable variation in the estimates. They are able to fit the shape and pattern, but not the level of the observed conflicts. The ability to fit shape but not level is exactly what leads us to estimate a significant effect of the embassy move.
In contrast, the estimates by the synthetic control method and the matrix completion method have little variation and are closely centered around the average weekly number of conflicts in the pre-intervention period. This is a sign of overfitting.\footnote{We thank Stefan Wager for pointing this out.} However, given the data available and in particular the number of control units, this is not surprising. Recall that the matrix completion method combines elements from vertical and horizontal regressions. For the horizontal part, it tries to fit the post-intervention outcomes to the pre-intervention outcomes using only 11 control countries. As the number of weeks is much greater than the number of control countries, it is not surprising that vertical regressions do perform better.
Figure (ref) conveys the same insights as Figure (ref), but instead of showing the observed and estimated number of weekly conflicts separately, it displays the differences between the two. Considering the differences instead of actuals provides an easier approach to evaluating pre-intervention fit. Again, a good ability to approximate the pre-intervention level of conflicts corresponds to differences closely around zero. As apparent in Figure (ref), the tree-based synthetic control method delivers the best pre-intervention fit, followed by the matrix completion method, the elastic net estimator, and the synthetic control method. It is, however, impossible to assess the overfitting indicated by little post-intervention variation from Figure (ref).
From Figure (ref) and (ref), we have argued that the tree-based synthetic control method performs at least as well as state-of-the-art methods. Supporting this, Table (ref) provides the various measures that follow from the figures. In particular, we compute the RMSPE and MAE in the pre-intervention period for all the methods considered. Both measures capture the ability to approximate the observed weekly level of conflicts in Israel-Palestine. The tree-based synthetic control method outperforms all other methods on these metrics. We also report the standard deviation of the estimated number of weekly conflicts in the counterfactual Israel-Palestine absent of the embassy move. The elastic net estimator is the only comparison method that delivers higher variation than the tree-based synthetic control method. The matrix completion method delivers almost no variation in the estimates.
Evaluating the degree of overfitting by computing standard errors is clearly insufficient. One final approach to simultaneously assessing the ability of the methods to approximate the weekly number of conflicts in Israel-Palestine and the degree of overfitting is to repeat the analysis, but hold out a subsample of the pre-intervention period and compute the RMSPE and MAE on this subsample. The hold-out sample serves as a test sample, but in contrast to the post-intervention period, we observe $Y_{t}^{0}$ as if the intervention has not yet occurred. This allows us to evaluate the predictive ability. Specifically, we hold out the last 10% of the observations in the pre-intervention period, resulting in an estimation sample and a validation sample. Then, we re-run all methods on the estimation sample.
For the methods that require tuning of hyperparameters, namely the tree-based synthetic control method, the elastic net estimator, and the matrix completion method, we further split the estimation sample using an 80/20% split as in the original analysis. We use the 20% to select the hyperparameters rather than selecting hyperparameters on the full estimation sample. For the synthetic control method, we use the whole estimation sample to estimate the weights for each country as it does not require any hyperparameters. Having estimated all parameters, we apply all the methods to the validation sample for which we know the true outcome and compute RMSPE and MAE.
Table (ref) shows the results of the hold-out sample approach. The elastic net estimator performs best in terms of both metrics, followed by the tree-based synthetic control method, the synthetic control method, and lastly the matrix completion method. Our suspicion that the matrix completion method overfits as seen in Figure (ref) appears to be confirmed. We emphasize that this is not an objection to the method, but rather a result of the structure of the data, namely $T\gg N$. The elastic net estimator performs very well on the validation sample, and in fact better than evaluated on the entire pre-intervention period.
Normally, we would take this as a sign of underfitting, but as we run more than 20 different specifications of the elastic net estimator in the pre-intervention period, it is more likely caused by the validation sample being too small. The tree-based control performs comparably in the validation sample as in using the entire pre-intervention period, which indicates that neither overfitting nor underfitting takes place. Being a nonparametric method, however, it requires more data and, the fact that we only estimate the hyperparameters using roughly 70% of the pre-treatment data seems critical in this assessment of the fit. Ideally, we would use a larger validation sample to compare the methods on validation RSMPE and MAE.
The synthetic control method is an effective method in comparative case studies in which relatively more time periods than potential control units are available. The main advantage is the data-driven approach to control unit selection. Since the estimation of the synthetic controls is performed to maximize the pre-treatment fit to the treated unit, however, the fit may not carry over into the post-treatment period. One can argue that synthetic controls are not designed to balance bias for variance, which may lead to overfitting to the pre-treatment period despite the importance of high predictive performance in the post-treatment period.
The elastic net estimator is an extension that regularizes the weights on the control units to improve the post-treatment fit. Both methods, however, impose a linear model that may not be guided theoretically. In addition, if interactions and higher-order terms of the control units are important to approximate the treated unit but difficult to anticipate, the estimators may suffer from bias. We recast the problem of estimating a counterfactual state as a prediction problem. Specifically, we provide a data-driven method that balances bias and variance to achieve post-treatment accuracy and is able to capture nonlinearities without the need for a researcher specifying them.
Our method can be applied in domains without theoretical guidelines and is also able to recover linear models. We achieve predictive accuracy because we replace the linear component of the synthetic controls with a powerful model inspired by machine learning, namely the random forests model. The ability to capture nonlinearities in a data-driven way is a special feature of this model. This makes the tree-based synthetic control method powerful, yet simple. We provide that the random forests regression model is asymptotically unbiased as well as consistent, which we use to establish consistency of the tree-based synthetic control method.
To demonstrate the applicability of the tree-based synthetic control method, we evaluate the relocation of the US embassy from Tel Aviv to Jerusalem. Specifically, we estimate the weekly number of conflicts in Israel and Palestine in the counterfactual state of the world absent of the embassy move. The estimates cover the period from the announcement of the move on December 6, 2017, until November 3, 2018. Comparing the estimates to the observed numbers, we find that the average number of weekly conflicts in Israel and Palestine has increased by more than 26 incidents since the move was announced. By placebo tests, we show that the estimated effect of the embassy relocation is very unlikely to be replicated if one were to arbitrarily relabel the treated unit in the data given that the pre-treatment fit is reasonable. To formally justify our results, we apply exact and robust conformal inference tests and find statistical significance at the 1% level.
We further compare the tree-based controls to state-of-the-art methods and conclude that our method is data-driven and needs no linearity assumptions, while it is not dominated even by the best of the comparison methods. All comparison methods agree on the magnitude of the effect.
\phantomsection \addcontentsline{toc}{section}{References}