Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
98,250 characters · 29 sections · 107 citation commands
Synthetic learner: Model-Free Inference on Treatments over Time
\if00 \fi
\if10 { \ \\
\ \\
} \fi
{\it Keywords:} Synthetic Control, Difference In Differences, Causal Inference, Random Forests. \\ {\it JEL Code:} C10, C14, C20, C30.
\spacingset{1.5}
This paper discusses estimation and inference on the effect of a policy intervention on a single unit observed over multiple periods and exposed to treatment from one point in time onwards. We consider an aggregate time-series set-up where researchers observe the outcome of the unit of interest $T_0$ periods before treatment and $T - T_0$ after the treatment. Researchers' main goal is to conduct inference on the effect of a trajectory of treatment effects over the post-treatment period. Namely, by denoting $\tau_t = Y_{0t}^1 - Y_{0t}^0$ the difference between the two potential outcomes at time $t$, researchers want to test whether $\{\tau_t\}_{t > T_0} = \tau^o$, for some null trajectory of interest $\tau^o$. Their second goal is to precisely estimate the average effect over time.
In the same spirit of synthetic controls abadie2003economic, abadie2010synthetic, and aggregate panel data models hsiao2012panel, we construct counterfactuals using information from $n$ (possibly finitely many) control units and covariates observed over $T$ periods. We exploit information over the time series for conducting asymptotic inference. The applications of interest are those where we observe individuals at a relatively frequent level, e.g., quarterly or monthly, over multiple years. Examples are studying the effects of taxation bai2014property, changes in welfare programs maclean2019losing or (geo-localized) marketing experiments brodersen2015inferring,varian2016causal. We discuss additional applications at the end of Section (ref).
The first contribution of this paper is to derive an inferential procedure that is valid for general parametric and non-parametric (machine-learning) estimators. We propose a resampling mechanism to guarantee exact asymptotic size control without imposing restrictions on the set of estimators. The key idea for inference is to combine the sample splitting procedure with the block-bootstrap politis1992circular and exploit the differentiability properties of the proposed procedure. Our approach does not require correct model specifications.
As a second contribution, we introduce an ensemble procedure that combines many such estimators, e.g., Synthetic controls, factor models estimators, and fully non-parametric estimators such as Random Forest or Kernel Smoothing, into a single prediction. We penalize the methods with the worst out-of-sample performance over the pre-treatment period. The method's goal is to increase precision. Also, it permits replacing heuristic model selection criteria whose properties are unknown with time-dependent observations with a theoretically grounded procedure. This has important implications for applied economists, who often face the difficult model selection choice from a dictionary that includes many methods (e.g., factor models, synthetic control, difference-in-differences).\footnote{See also the discussion in athey2019ensemble.} The ensemble method that we propose builds on the literature on exponential aggregation rigollet2012sparse, cesa2006prediction, time-series and forecasts' combinations timmermann2006forecast, elliott2004optimal, which we study here for causal inference. We show that the procedure consistently recovers the average treatment effect through a bias adjustment, and it inherits strong oracle properties for its prediction performance.
Throughout the text, we assume exogeneity of the treatment time and stationarity for inference. These conditions are common when conducting inference with Synthetic Controls in the presence of a long time series\footnote{For inference via Synthetic Control chernozhukov2018exact impose stationarity of the residuals under correct model specification and similar stationarity assumptions as to the one discussed above under misspecification to show the validity of permutation tests. Stationarity, strong mixing conditions on the joint distribution of the residual errors and covariates, are also imposed for valid inference in synthetic control settings in carvalho2018arco, while covariance stationarity conditions are imposed in li2017estimation. Further discussion is in Section (ref). }, since they permit to conduct inference without relying on symmetry assumptions of placebo testing firpo2018synthetic, ben2018augmented. They imply that a pre-post treatment comparison of the means returns a consistent (but possibly inefficient) estimate of the average effect on the treated. In separate sections, we relax stationarity in two directions: (i) allowing for time-varying fixed effects, encompassing popular two-way fixed-effect models; (ii) deriving prediction guarantees for arbitrary non-stationary settings.\footnote{These are in Section (ref) and Section (ref) respectively.}
We conclude our discussion with a simulation study and an empirical application. We show that our procedure leads to larger power than existing methods while controlling the size of the test. In an application, we study the effect of Tennessee's health-insurance dis-enrollment program on health-related outcomes. Using survey data from Behavioral Risk Factor Surveillance System Data, we show that the program decreased health insurance coverage and the likelihood of visiting a doctor.
The paper is organized as follows. In Section (ref) we introduce the set-up, and identification strategy. In Section (ref), we introduce the method for estimating counterfactuals and inference on sharp nulls. In Section (ref), we discuss estimation of the average effect on the treated. In Section (ref) we discuss prediction guarantees under non-stationarity. Section (ref) discusses numerical experiments. Section (ref) discusses applications in health economics. Finally, in Section (ref) we provide extensions in the presence of carry-over effects.
While non-parametric estimators have found widespread use in microeconomic applications athey2019machine, their analysis (and consequently their applicability) in the presence of aggregate data has received much less attention. However, with aggregate data, such estimators can improve precision and better disentangle the effect of the policy from idiosyncratic shocks. This paper proposes a method that enables counterfactual prediction and hypothesis testing in the context of Synthetic Controls (SC) using predictions arising from many parametric and non-parametric estimators.
Recent literature has proposed a wide variety of methods for predicting counterfactuals in the setting under consideration, including factor and panel data models bai2009panel, hsiao2012panel, synthetic controls abadie2010synthetic, xu2017generalized,doudchenko2016balancing, arkhangelsky2019synthetic, ferman2016revisiting, two-ways fixed effects models imai2021use, ridge regression methods ben2018augmented, kernel balancing hazlett2018trajectory, functional methods gunsilius2020distributional among others. Additional references include athey2018matrix who proposes a matrix completion methods for SC; athey2019ensemble shows in a simulation exercise that ensemble methods outperform individual SC predictions on many economic data sets; amjad2018robust proposes singular value thresholding, whereas matching has been discussed by imai2018matching. Difference-in-difference methods were recently discussed in athey2018design and arkhangelsky2019synthetic in the context of staggered adoption.
However, selection among these methods remains an open research question hsiao2018panel. Combining different predictions offers a simple data-adaptive procedure to exploit information from all these models while improving the prediction performance timmermann2006forecast. As a result, our contribution can be viewed as complementary to this literature. To the best of our knowledge, we provide the first set of conditions under which predictions made by any or many machine learning methods, including Random Forests, can be used to develop valid tests for SC.
A closely related method to our inference procedure is the permutation-based inference method, discussed in chernozhukov2017exact. Their method accommodates a single (linear) model specification only and imposes stability conditions of the estimator. Also, while the permutation-based methods estimate counterfactuals using the entire sample (pre and post-treatment periods) as a training set, imposing the sharp null hypothesis of no treatment effects, here we estimate counterfactuals using the pre-treatment period only. This approach permits that estimation of the counterfactual does not depend on the outcomes observed over the post-treatment period. Our approach is particularly suited whenever the post-treatment period is proportional to the pre-treatment period as in the context of our empirical application.\footnote{In our empirical application, the post-treatment period is approximately twenty, and the pre-treatment period is approximately sixty. The reader may refer to Example (ref) for an illustration of the benefits.}
Our paper relates more broadly to inference using penalized methods, including chernozhukov2018t and carvalho2018arco, which propose penalized linear regressors for asymptotic inference on treatment effects in an SC setup. However, carvalho2018arco requires a consistent estimation of treatment effects, which is not required in our setting and focuses on a penalized model only. Differently, chernozhukov2018t proposes a bias-adjustment procedure for asymptotic inference, which in our framework is not required for hypothesis testing but used for average treatment effect on the treated (ATT) estimation. Our resampling mechanism allows for more generality than the tests in chernozhukov2018t since we do not require the use of penalized linear regressor and the related conditions for estimation of counterfactuals, but we allow for general non-parametric estimators. Related literature also includes li2017estimation, and hsiao2012panel, which discuss properties of constrained least-squares methods under stationarity and correct model specification only. li2019statistical discusses a sub-sampling procedure for inference with constrained least-squares estimators only.
Finally, we relate to kunzel2017meta who discuss model averaging with $i.i.d.$ data and classifies method into three classes, denoted as S, T, and X learners. This paper pioneers the idea of T-learning in Synthetic control setting, offering an alternative and simple weighting scheme which is inspired by the literature on boosting schapire2012boosting and online learning cesa1997use, cesa1999prediction, cesa2006prediction.
To conclude, we can list several applications of interest. The first set of relevant applications includes changes in policy at a regional or state level. For instance, studying the effects of (i) exogenous variations such as environmental disasters on policy-changes potrafke2020green, or economic outcomes cavallo2013catastrophic; (ii) of Medicaid expenditure expansions or contractions tello2016effects; (iii) of tax-reform on prices bai2014property; or (iv) of policy reforms on economic growth billmeier2013assessing. The second avenue of interesting applications includes experiments on online platforms, for which synthetic controls are becoming increasingly popular, especially in the presence of geo-localized experiments varian2016causal, li2017estimating. The third set of applications includes studying the effect of new algorithms or financial instruments xie2014impact, bojinov2019time.
Throughout this article for each unit $i$ we observe outcome variable $Y_{it}$. We denote with $Y_{0t}$ the unit treated if $D_t = 1$ and under control otherwise; the remaining $i=1,\cdots, n$ units, $Y_{1t}, \dots, Y_{nt}$ are units always observed in the control state. Additional covariate information for each unit are denoted in compact form as $Z_{it}$. $Z_{it}$ may also contain past covariates and past outcomes.
Following the literature on panel data and synthetic control models hsiao2018panel, abadie2010synthetic, we define the treatment assignment and the outcome of interest, respectively as
where $Y_{0t}^1, Y_{0t}^0$ denote the potential outcomes under treatment and control for the unit of interest $i = 0$, and $Y_{jt}^0$ denotes the potential outcome under control of unit $j$. Our definition of potential outcomes implicitely imposes that SUTVA holds rubin1990formal and no carry-over effects imai2013estimating. Extensions in the presence of carry-overs are discussed in Section (ref) .
In applications, researchers may want to test whether the difference between two potential outcomes $Y_{0t}^1 - Y_{0t}^0$ equals zero for all post-treatment periods. Namely, they may be interested in conducting inference on the time-specific treatment effects defined below.
Here, $\tau_t$ defines the difference in potential outcomes at time $t$. We begin our discussion by introducing the null hypothesis of interest.
Equation (ref) imposes that the potential outcome over the post-treatment period equals the potential outcome under control plus a known (possibly time-varying) constant. For example, we may consider $a_t^o = 0$ or we may consider also testing a linear trend of the form $a_t^o = \delta (t - T_0)$ for an arbitrary $\delta \in \mathbb{R}$. Our results also extend when we test the average of $\tau_t$.\footnote{Namely, we may also test $\mathbb{E}[\tau_t] = \alpha_t^o$, hence allowing potential outcomes under treatment and control having different idiosyncratic shocks. This is omitted for the sake of brevity only and discussed in the Appendix (Algorithm (ref)) and Remark (ref).} Finally, note that more generally, we can incorporate a more general class of hypothesis $H_0: Y_{0t}^1 = f(Y_{0t}^0,a_{t}^o), \ a_t^o \in \mathbb{R}, \ t > T_0, $ for a function $f$ being invertible in its first argument, omitted for the sake of brevity. We define $$ Y_{0t}^o =
$$ the (observed) potential outcome under control under the null hypothesis $H_0$.
Testing for treatment effects may not be satisfactory to researchers, who may also be interested in reporting estimates of average treatment effects. This is defined below.
Here, $\tau$ denotes the average effect on the treated, averaged also over the post-treatment period. The expectation is taken over the idiosyncratic shocks.\footnote{Note that whenever $\tau_t$ is non-random $\tau = \frac{1}{T - T_0} \sum_{t > T_0} \tau_t$.}
The first condition that we impose is stationarity. This is common in the literature on Synthetic Control, see carvalho2018arco, li2017estimation, chernozhukov2018exact.\footnote{Stationarity and beta-mixing conditions as stated above cover a large class of {\sc arma} processes pham1985some, {\sc {\sc ar-arch}} processes lange2011estimation, Markov Switching Processes lee2005probabilistic, {\sc garch} carrasco2002mixing, to cite some. } We relax it in Section (ref), where we allow for non-stationary observations.
Assumption (ref) imposes stationarity. In Section (ref) we show how our results for inference extend under non-stationarity, in the presence of an (unknown) time-varying fixed-effects.
The second condition we impose is an identification assumption.
Assumption (ref) states that the timing of the treatment is exogenous. The same conditions can be found in previous literature on Synthetic Controls. For instance, recent literature on Synthetic Control abadie2010synthetic, chernozhukov2018double, arkhangelsky2019synthetic, chernozhukov2018t, li2017estimation treated $T_0$ as deterministic, in which case exogeneity of $T_0$ implicitely holds. See for example, the discussion in ferman2016revisiting and bottmer2021design. carvalho2018arco also explicitely imposes exogeneity similarly to the above condition.
In the context of our empirical application, where the treatment consists of a dis-enrollment from Medicaid occurring in the early 2000s in Tennessee, and the outcomes are health-related outcomes, as argued in argys2017losing, if the policy can be attributed to budget deficit interpretable as an exogenous variation, the assumption directly holds. We warn the reader, however, that failure of the assumption may invalidate the inferential strategy.
Motivated by Assumption (ref) we will implicitely condition on $T_0$ throughout the rest of our discussion unless otherwise specified, since, conditional on $T_0$ the distribution of observables and unobserved potential outcomes remains invariant. We return instead to non-stationary conditions in Section (ref) where we relax Assumption (ref) and (ref).
We conclude this discussion with two lemmas. The first lemma provides a testable implication for the sharp null hypothesis in Definition (ref). This is stated below.
The above condition implies that the distribution before and after the intervention period $T_0$ must remain invariant under the null hypothesis. This assumption is testable since we observe the empirical distribution of the above vector before and after the intervention time $T_0$. Lemma (ref) is at the basis of our approach for testing the null hypothesis, which we discuss in Section (ref).
The second goal is to estimate the average effect. Identification of $\tau$ is discussed in the following lemma.
Lemma (ref) is an identification result. It states that the post-treatment difference in expectation equals the target estimand $\tau$. While Lemma (ref) is not invoked for constructing our test, it is used for estimating the average effect discussed in Section (ref).
The proofs of the lemmas are contained in Appendix (ref).
In this section, we discuss the problem of estimating the counterfactual prediction $\hat{Y}_{0t}^0, t > T_0$ and conducting inference on the sharp null hypothesis in Definition (ref).
Throughout our discussion, for expositional convenience, we denote in the compact form $$ X_t =(Y_{1t}, \dots, Y_{nt}, Z_{0t}^\top, Z_{1t}^\top, \dots,Z_{nt}^\top) \in \mathcal{X}. $$
To test the null hypothesis of interest, we first estimate the true potential outcome $Y_{0t}^0, t > T_0$, unobserved over the post-treatment period. We define $\widehat{Y}_{0t}^0$ its estimate constructed as follows
Here $w(F_0)$ denotes a generic functional of the empirical distribution $F_0$ of $(Y_{0t}, X_t)$ over the pre-treatment period, where $Y_{0t}$ serves as the main outcome of interest. The choice of $w(F_0)$ can be arbitrary, with the only condition required that $w(\cdot)$ is a Hadamard differentiable functional (see Assumption (ref)).\footnote{Definition of Hadamard differentiability is provided in Appendix (ref).} In Section (ref) we provide explicit expressions for $w(\cdot)$.\footnote{Since the functions $g(\cdot)$ can contain an intercept, the component $w(F_0)^\top g(X_t)$ also estimates the (time-invariant) shift in mean after subctracting $\bar{Y}_t$. See, for instance, Example (ref).}
It is important to note that the functions $g(X_t)$ can be data-dependent. Such functions are estimated as described in Algorithm (ref). Namely, first, we divide the pre-treatment period into two blocks $t \in \{T_-, \cdots, 0\}$ and $t \in \{1, \cdots, T_0\}$. We define $ F_- $ the empirical distribution for $t < 1$ of $(Y_{0t}, X_t)$. We construct the predictors as follows
with $F_-$ denoting the training set for such predictors, and $g_j$ denoting some pre-specified regressor. For example $g_1(X_t; F_-)$ can denote the prediction of a Random Forest, trained over the sample $t < 0$, with empirical distribution $F_-$. Whenever clear from the context, we will omit the second argument $F_-$ from the function $g_j(\cdot)$. Throughout the rest of our discussion, we will fix the size of $T_-$ and consider asymptotics as $T \rightarrow \infty, T_0 \propto T$.
We conclude our discussion with two cases of interest.
\paragraph{Case 1: Ensemble} To gain further intuition on Equation (ref), observe that we can interpret $w(F_0)$ as some data-dependent weights, while the functions $g(X_t)$ are interpreted as predictors or “experts" that, based on information $X_t$, predict the counterfactual outcome $Y_{jt}$ (net of time-fixed effects) at time $t$. This follows in the same spirit of forecasts combinations timmermann2006forecast, with $g_i$ denoting some “experts" or learners. Section (ref) discusses examples and properties of weighting schemes. The construction of the weights is based on out-of-sample performance: it uses information $F_0$ which has not been used for the training of the algorithms $g(\cdot)$ (that instead use information before time $t = 0$).
\paragraph{Case 2: Single Regressor} Equation (ref) is, however, more general than only ensemble methods, since it also allows for estimation with a single regressor $g_1(X_t)$ (e.g., the Synthetic Control).
Given $\widehat{Y}_{0t}$, we construct a test statistics and test with coverage $1 - \alpha$ having the following form:
where $q_{1- \alpha}^*$ is the estimated $1 - \alpha$ quantile of $\mathcal{T}$. The test statistic depends on the prediction error between the estimated counterfactual outcome and the potential outcome under the null hypothesis. Observe that under Equation (ref), we can represent in compact form the test statistic as a functional of the empirical distributions
where $F_1$ denotes the empirical distribution over the post-treatment period of $(Y_{0t}^o, X_t)$.
We obtain the critical quantile $q_{1 - \alpha}^*$ using the block bootstrap politis1992circular, politis1994stationary. Formally, we resample the entire vector $(Y_{0t}, X_t)_{t=1}^{T}$, over the pre-treatment period, constructing first $T_0$ units serving as pre-treatment period's observations and $T - T_0$ units serving as post-treatment period's observations. We then construct the empirical measure of the bootstrapped sample $(F_0^*, F_1^*)$ and compute $\mathcal{T}^* = \mathcal{T}(F_0^*, F_1^*)$. A formal description is included in Algorithm (ref).
The main intuition behind the inferential procedure is the following. We use a portion of the data to train predictors, while the remaining observations are used for the bootstrap estimate of the critical value when conducting hypothesis testing. We only estimate learners once and not on each bootstrapped sample. Figure (ref) provides a graphical illustration.
Below, we formalize the validity of the bootstrap.\footnote{Lemma (ref) provides the main intuition. The lemma implies that the distribution before and after the treatment must remain the same under the null hypothesis. Therefore, intuitively, we may expect that $(F_0^*, F_1^*)$ centered around the true empirical distribution converges to the same empirical process (after appropriate rescaling) of the limiting process of $(F_0 - \mathcal{D}_0, F_1 - \mathcal{D}_0)$, under the null hypothesis. We can then invoke Hadamard differentiability properties to show the bootstrap's validity. We use such properties in the derivation of the validity of the bootstrap. } We impose the following condition.
In Appendix (ref) we show that Assumption (ref) holds for exponential weights considered in the following sections. We now introduce the first theorem.
The above corollary follows from the validity of the bootstrap fang2014inference and it guarantees exact asymptotic coverage.
The proof of Theorem (ref) is contained in Appendix (ref). Theorem (ref) does not impose any restriction on the predictor other than Hadamard differentiability. Conditions on Hadamard differentiability are often imposed in the literature belloni2017program, and require that weights are smooth functionals of the data. These are satisfied under mild conditions, and simple examples of weights that satisfy such conditions include least squares lunde2017bootstrapping or exponential weights we discuss in Section (ref) (see Appendix (ref) for a formal discussion). Note that the theorem is valid under misspecification.
It is interesting to observe that if we were doing classical statistical inference, we would need to account for the estimation error generated by each individual prediction. Instead, the sample splitting procedure in Algorithm (ref) combined with the resampling method in Algorithm (ref) guarantees valid inference, and it overcomes the complicated generated regressor problem. The key intuition is that the empirical distributions corresponding to the group of observations used for the training and those for the resampling are asymptotically independent under standard mixing conditions. As a result, sample splitting guarantees that we can condition on the initial training period $F_-$, without affecting the asymptotic properties of the bootstrap. In this sense, our analysis extends the standard sample-splitting procedures employed in the $i.i.d.$ setting rinaldo2019bootstrapping to dependent observations.
In applications, we may also be interested in estimating the average treatment effect consistently in Definition (ref). A direct corollary of Lemma (ref) is that a simple treated and controls difference, under standard mixing conditions, provides a consistent estimate of the treatment effect. However, such an estimate can present significant variance since we do not control the variation captured by covariates $X_t$, which can be non-vanishing with high-dimensional controls.
An alternative estimator for the average treatment effect takes the following form instead: $$ \frac{1}{T - T_0} \sum_{t > T_0 } \Big(Y_{0t} - \widehat{Y}_{0t}^0\Big). $$
Unfortunately, such an estimator may be inconsistent for the true $\tau$, due to misspecification of the functions $g(x)$. We consider a bias adjustment instead.
The bias adjustment works as follows. Define $F_{0, -1/2}$ the empirical distribution of $(Y_t, X_t)$ for $t \in \{1, \cdots, T_0/2\}$, and $F_{0, 1/2}$ the empirical distribution of $(Y_t, X_t)$ for $t \in \{T_0/2 + 1, \cdots, T_0\}$. Then we construct an estimator of the form
where $\widehat{Y}_{0t, -1}^0$ is the estimator whose weights are constructed using only the first half of the treatment period, and $g$ is constructed as in Algorithm (ref). The first difference is taken over the post-treatment period, with weights computed over the entire pre-treatment period. The second difference is taken over the second half of the pre-treatment period, but with weights computed over the first half. The second difference defines the bias adjustment.
To gain further intuition on the bias adjustment, note that we can write $$ \hat{\tau} = \tau + o_p(1) - \underbrace{\int \Big(w(F_0)g(x) - w(F_{0, -1/2})g(x)\Big) d(F_1(y, x) - F_{0, 1/2}(y, x))}_{ = o_p(1)} $$ where $F_1$ denotes the empirical distribution over the treated period, $F_{0, -1/2}$ denotes the empirical distribution over the first half of the pre-treatment period, and $F_{0,1/2}$ denotes the empirical distribution over the second half. The above estimator is a difference in difference, where the first component converges to $\tau$ while the second component is of order $o_p(1)$ as shown in the following theorem.
We first impose the following condition.
Assumption (ref) states that the potential outcome under control can be decomposed into two components: a (possibly non-stationary) expectation and an additive stationary idiosyncratic shock. We can now introduce the following theorem.
Theorem (ref) shows consistency of the estimated effect. It imposes standard mixing conditions. Its proof is in Appendix (ref).
In this section, we turn to the case where time-fixed effects occur and illustrate how our results extend to this case without necessitating estimating the fixed effect.
We consider the following data generating process.
The above assumption allows for the failure of stationary as long as the time fixed effects are the same between the control and treated unit. The following lemma illustrates identification for this case.
The above lemma shows that we can identify the treatment effect by taking a difference in difference. The following lemma discusses testable implications for the sharp null hypothesis.
The proofs of the above lemmas are in Appendix (ref). The counterfactuals' estimation follows similarly as before, with a minor modification: we subtract the controls' average outcome from the outcome of the treated unit and from the control unit to guarantee stationarity. This is illustrated below. $$ \hat{Y}_{0t}^0 - \bar{Y}_t = w(\tilde{F}_0)g(\tilde{X}_t), \quad \bar{Y}_t = \frac{1}{n} \sum_{j = 1}^n Y_{jt}, \quad \tilde{X}_t = \Big(Y_{1t} - \bar{Y}_t, \cdots, Y_{nt} - \bar{Y}_t, Z_{0t}, \cdots, Z_{nt}\Big), $$ where $\tilde{F}_0$ denotes the empirical distribution of $(Y_{0t} - \bar{Y}_t, \tilde{X}_t)$ over the pre-treatment period. Here, $\bar{Y}_t$ denotes the time-specific average of the control units, but not of the treated. Theorem (ref) and Theorem (ref) directly follows once we subctract from the outcomes $(Y_{0t}, \cdots, Y_{nt})$ the controls' average $\bar{Y}_t$ to guarantee stationarity.
This section discusses a particular weighting scheme, exponential weights, and derives its properties in terms of prediction guarantees.
Although least-squares have been considered kunzel2017meta,polley2010super, elliott2004optimal, these can perform poorly when the number of learners is large compared to the sample size.\footnote{Observe that weights computed via least-squares are Hadamard differentiable lunde2017bootstrapping, hence satisfying the conditions in Theorem (ref).} Equal weighting, on the other hand, can perform poorly in the presence of many uninformative learners. To equip the method to have a better performance in the presence of a large number of algorithms, some of which may potentially be ineffective, we discuss an alternative weighting scheme.
With a slight abuse of notation, we index weights by the time period as $w(t)$, with $t \in \{1, \cdots, T_0\}$. For example $w(\tilde{t})$ denotes the weight estimated using the empirical distribution from time $t = 1$ to time $t= \tilde{t}$.
The weigthing scheme we focus are exponential weights of the following form: we write
Such weights have been widely discussed in literature on exponential aggregation, see e.g., cesa1999prediction, rigollet2012sparse among others.
By choosing the tuning parameter $\eta \propto 1/T_0$ exponential weights can be written as differentiable functionals of $F_0$. This is discussed in Appendix (ref).
We can motivate Equation (ref) as the solution to a penalized surrogate risk minimization, discussed below.
We now derive prediction guarantees of the exponential weights without imposing stationarity conditions. Our result illustrates oracle guarantees of the exponential aggregation method cesa2006prediction, here applied to the different contexts of counterfactual prediction.
We study the behavior of our algorithm trained only on $t-1$ observation and evaluated at the $th$ observation, i.e., we are willing to provide theoretical guarantees on the following cumulative loss.
where $\hat{Y}_t^0(\mathcal{F}_{t-1}) = w(t- 1) g(X_t)$. Here, $\hat Y_{0t}^0( \mathcal{F}_{t-1})$ denotes the prediction at time $t$ using only information at time $t-1$. Since $\hat Y_{0t}^0( \mathcal{F}_{t-1})$ is estimated only on the previous data and evaluated at $X_t$, this notion of performance is rooted in out-of-sample performance metric.
We first study the cumulative loss in (ref) compared to the smallest cumulative loss incurred by any of the algorithms under consideration, defined as
In the following theorem, we consider the case where $T_0 = \lambda T$ where $\lambda \in (\gamma,1 - \gamma)$ is potentially a random variable for some constant $\gamma > 0$.
The proof is presented in Appendix (ref).
Theorem (ref) provides an error bound for the empirical one step ahead prediction error. Remarkably, it does not require any stationarity assumption. The bound scales logarithmically with the number of learners, and it scales at square-root $T$ with the length of the sequence.
In the following lines, we provide stronger guarantees with respect to the conditional expectation of $Y_{0t}^0$. For the next theorem to hold, we need to introduce an additional condition, which replaces Assumptions (ref) and (ref) that we imposed in previous sections.
Assumption (ref) states that the potential outcome can be decomposed into two main components, a conditional expectation function $\mu_t(\cdot)$ and idiosyncratic shocks. Note that such a condition is only required to derive guarantees with respect to $\mu_t(\cdot)$. It is natural to compare the cumulative loss in (ref) with the smallest cumulative loss incurred by any of the algorithms under consideration. We define such a metric of comparison as the Conditional Mean Proxy Regret (CMPR).
The above definition incorporates notions of performance with respect to the conditional mean function (as opposed to the outcome itself). Our definitions above combine definitions in the literature on prediction of individual sequences cesa1999prediction with the literature on causal inference. The main difference with standard notions of regret is that CMPR is based on the unobserved deviation of the predicted counterfactual from the conditional mean evaluated at $X_t$, and not just on the cumulative loss of the predictor
The proof is presented in Appendix (ref).
Theorem (ref) provides an error bound for the empirical one step ahead prediction error with respect to the conditional mean.
If we are willing to assume more, in that our class of algorithms contains one learner that consistently estimates the unknown model, then previous results imply that our synthetic learner will preserve that consistency regardless of the number of learners in the entire class. We consider below asymptotics for $T \rightarrow \infty$ and $T_0 = \lambda T$ where $\lambda \in (0,1)$ is potentially a random variable.
In this section, we study the performance of the method in the presence of linear and non-linear outcome models, allowing for the presence of many non-informative methods. We compare the methodology to existing testing procedures, including permutation tests of Synthetic Control as well as the Difference-in-Difference method, and showcase a significant improvement.
We describe our experiments in terms of the outcome model as well as the model of the design of the covariates and the error terms. In our first experiment, DGP1, we considered a simple Linear Outcome Model $$ Y_{0t} = X_t \beta + a_t D_t + \epsilon_t $$ and tested the ability of our method to detect changes in the treatment effect $a_t$. This example is intended to model a setting where classical Synthetic Control method is optimal. Here we set $\beta_j = 1/(1+j)^2$, $j=1,\dots,p$, with the last beta chosen such that $ \sum_{j } \beta_j =1$, where we consider $p \in \{10, 50\}$. The parameter $\beta$ will be kept as above for all our experiments. We considered a simple {\sc ar} model for the errors $\epsilon_t$ with $\epsilon_t = 0.6\epsilon_{t-1} + v_t$ and $v_t \sim \mathcal{N}(0,1 - 0.6^2)$. Control units are generated according to a factor model as $$ X_{j,t} = \mu_j + \theta_t + \lambda_j F_t + u_t $$ with unit specific term $\lambda_j = \mu_j = (1+j)/j$ a time random effect $\theta_t \sim \mathcal{N}(0,1)$ and an unobserved factors $F_t \sim \mathcal{N}(0,1)$. Errors $u_t$ follow an {\sc ar} model $u_t = 0.6u_{t-1} + h_t$ with $h_t \sim \mathcal{N}(0, 1 - 0.6^2)$. In our second experiment, DGP2, we considered a Logistic-like Outcome Model $$ Y_{0t} =a_t D_t + \exp(X_t \beta + \epsilon_t)/(1+\exp(X_t \beta + \epsilon_t)) $$ with $\epsilon_t = 0.5 \epsilon_{t-1} + 0.3 v_{t-1} + v_t$. This experiment has three settings: (a), (b) and (c). Setting (a) and (b) assume $v_t \sim \mathcal{N}(0,\sigma^2)$ with $(a)\sigma = 0.1$ and $(b) \sigma = 1$, respectively. Setting (c) assumes $ \epsilon_t = 0.8 \epsilon_{t-1} + v_t$, with $ v_t = \sqrt{h_t} z_t, \quad h_t = 0.001 + 0.99 v_{t-1}^2 $ with $z_t \sim \mathcal{N}(0,1)$({\sc ar-arch} process). We report here (a) and (b). In addition we let $ X_t = h_t + u_t $ with $h_t$ being i.i.d over time with $ \mathcal{N}(0, \Sigma)$ distribution with $\Sigma_{i,j} = 0.5^{|i-j|}$ and $u_t = 0.8 u_{t-1} + k_t$ with $k_t \sim \mathcal{N}(0,1 - 0.8^2)$. This setting is designed to test the ability of the proposed Synthetic Learner to adapt to nonlinear outcome model. We consider our third setting, DGP3, that follows a factor model $$ Y_{0t} = 0.5 + a_t D_t + \theta_t + 0.5 F_t + \epsilon_t . $$ Error and design structures are the same as that of DGP1. DGP4 considers an interaction outcome model that is polynomial in structure $$ Y_{0t} = a_t D_t +(X_{1,t} + X_{2,t} + \dots + X_{10,t})^2 + \epsilon_t $$ with $\epsilon_t$ being the same as in DGP2(a) design $X_t$ is the same as throughout \textbf{DGP2}; \textbf{DGP5} postulates a cosine, hence periodic, type of outcome model $$ Y_{0t} = \cos(X_t\beta + \epsilon_t) + a_t D_t . $$ Error and design setting have three components: (a), (b) and (c) that are following the setup of \textbf{DGP2} (a), (b) and (c), respectively. Finally, \textbf{DGP6} is a simple non-stationary model, which follows similarly to DGP3, but with $F_t \sim \mathcal{N}(\mathrm{cos}(t), 1)$, with $\mathrm{cos}(t)$ capturing a non-stationary component.\footnote{Note that the component cannot be removed from simple transformations such as differentiating.} As discussed in the following subsections, we choose $p = 10$ for smaller $T (T \in \{60, 80, 100\})$ and $p = 50$ for larger $T \ge 300$.
We consider testing the following hypothesis $H_0$: $$ H_0: Y_{0t}^1 - Y_{0t}^0 = 0, \quad t > T_0. $$ We consider the Synthetic Learner with experts, including a naive XGboost (which uses the default tuning parameter of the package XGboost in R), Support Vector Regression, and {\sc arima}(0,1,1) with external regressors together with $50$ non-informative learners. Non-informative experts are randomly drawn from a multivariate gaussian with a full covariance matrix.
First, we compare Synthetic Learner's performance to existing procedures whose theoretical properties are well studied. In particular, we compare the Synthetic Control (SC) with weights being constrained to sum to one and an intercept according to Equation (7) and (8) in chernozhukov2017exact, as well as the Difference in Difference (DiD) estimator, namely $$ \hat{Y}_t^{DiD} = \hat{\alpha} + (\hat{\beta} + \hat{\Delta}) \mathbbm{1}_{t > T_0} $$ with coefficient computed as in a standard DiD with the two periods corresponding to pre and post-treatment periods. We consider the test statistics for Synthetic Control
where $\hat{w}_{SC}^0$ are computed via constrained Least Squares, with coefficients summing to one for Synthetic Control. Finally, we consider
for Difference-in-Differences. In Figures (ref), (ref) and (ref) we compare the performance of our method to permutation tests where $\hat{w}_{SC}$ and $\hat{Y}_t^{DiD}$ must be computed on the entire sample, as described in chernozhukov2017exact. We run the Synthetic Learner after training on the period running from $1$ to $T_- = T_0/2, T \in\{60, 100\}, p = 10, T - T_0 = 10$, and we use the remaining observations for computing weights and bootstrap. We consider different treatment effects, denoted by $\alpha$, and report on the x-axis the effect of the policy $\alpha$ divided by the (unconditional) standard deviation of the outcome.
We present power plots across different $T$ in Figures (ref), and (ref) for $T = 60$ and $T = 100$ respectively, while Figure (ref) collects results for the non-stationary DGP. Across all figures, we observe an improvement over permutation tests with both SC and DiD methods, with more significant improvements for the non-linear DGPs.
Improvements can be due to two factors: bootstrap outperforming permutation as well as Synthetic Learner's better performance in comparison to SC and DiD. Table (ref) (see the discussion below) and Appendix (ref) provide suggestive evidence that improvements are due to both factors.
In particular, in Appendix (ref) we also consider the oracle case where the critical value is known. This case permits to compute $\hat{w}_{SC}$ and $\hat{Y}_t^{DiD}$ only using information until time $T_0$, as discussed in doudchenko2016balancing. We show improvements also in this setting. In Appendix (ref), we report a more extensive study with $T_0$ and $T - T_0$ vary and show the robustness of our results to different settings.
Next, we study the size of our procedure for $T \in \{60, 80\}$ and $T^* = T - T_0 \in \{5, 10, 20\}$, as we vary the post-treatment period from small to longer post-treatment period. These are reported in Table (ref). We observe that in a finite sample, for $T$ relatively small, our test controls size across all DGP, except DGP3 and DGP6, where we observe a small size distortion of five percentage points for a short post-treatment period ($T^* = 5$). These results provide suggestive evidence of the correct size of the proposed test also in a finite sample.
In Table (ref) we study the performance of the algorithm and each base algorithm for $T = 60, T - T_0 = 10$. Table (ref) reports the power of each base algorithm and its corresponding weight assigned by the Synthetic Learner. We observe two striking facts: first, the largest weight is assigned to the best performing algorithm; second, the Synthetic Learner always performs approximately the same or better than any base algorithm under consideration. This result suggests the benefits of the ensemble procedure: the procedure combines predictions of different methods to maximize prediction (and ultimately power) optimally. For the sake of brevity, we report results for two linear DGPs (DGP1 and DGP2) and two non-linear ones.
Next, we study the problem in the presence of an endogenous time of the treatment. In Figure (ref) we report a representative set of results under the endogenous time of the treatment, where we simulate $$ T_0 = 280 + \min\{50, 1 + (\exp(1/\lambda) - 1/\lambda) \vee 1\} $$ where we choose $\lambda = |\sum_{j,t} X_{j,t}|$. The model follows a proportional hazard type model, similar to what is discussed in shaikh2019randomization, centered on $T_0 = 281$, with the time of treatment depending on other units' outcomes and constrained between $281$ and $320$. Figure (ref) collects results with an endogenous time of treatment when critical quantiles are estimated via resampling, where the confidence intervals for the Synthetic Control and DiD are constructed using the permutation-based method in chernozhukov2017exact. Results are consistent with the case of an exogenous treatment timing.
Next, we study the variability of the proposed method concerning the number and quality of learners included in learners' classes. We consider four different variations of the Synthetic Learner: Exponential and Least Squares weighting with $10$ and $100$ new non-informative learners. To guarantee the feasibility of the optimization problem given a large number of learners, we consider a large $T = 300$. Figure (ref) contains the results. There we observe that many non-informative learners do little to nothing to the proposed Synthetic Learner. In sharp contrast, Least Squares' weighting suffers a substantial loss in power when the number of non-informative learners is increased.
We compare the performance of the circular bootstrap against permutations proposed in chernozhukov2017exact. We consider only one learner: OLS. We compute the OLS coefficient for the bootstrap method using only the first $T_0/2$ observations, and we bootstrap the remaining ones. We estimate the full sample's coefficient for the permutation method after imposing the null hypothesis of no effect. We consider the true effect is either $\alpha_t = 0.2$ or $\alpha_t = 0.3$. We vary $T \in \{60, 80\}$ and we consider $T - T_0 = 10$. Results are collected in Table (ref). For the non-linear design, we mostly observe significant improvements in power, up to approximately fifty percentage points. Only for a few designs we observe comparable performances or slightly inferior by one to three percentage points.
Understanding the effect of public health insurance coverage on health care access is a major concern in health economics kolstad2012impact, long2009another, baicker2013oregon, anderson2012effect, garthwaite2014public.
The TennCare dis-enrollment program represents the largest reduction in public health insurance coverage ever experienced in the US. Between 2005 and 2006, approximately 170,000 individuals lost public health insurance coverage. Most of these individuals were childless adults who gained public health insurance coverage approximately ten years before, in 1994, during the Medicaid program expansion in Tennessee. In this section, we study the effect of the reform over childless adults on delayed medical care access due to medical costs. This population is of particular interest since most of the Affordable Care Act expansions target childless adults. tello2016effects estimates that the TennCare dis-enrollment significantly decreased the likelihood of having health insurance between 2 and 5 percent. The author estimates an increase of between two and three percentage points in the probability of not going to a medical center when sick.\footnote{The reader might refer to Panel C, Table 5 in tello2016effects.} Our analysis provides supportive evidence for the claim, with positive effects ranging between one and five percentage points but with higher uncertainty in the absence of stationary. Details are discussed in the following lines.
We use BFRSS data\footnote{ Behavioral Risk Factor Surveillance System Data: \href{url}{https://www.cdc.gov/brfss/annual_data/annual_data.htm}.} to investigate the effect of the reform on the percentage of people who cannot afford healthcare expenses for medical costs. BFRSS is a national survey that has been continuously run over the years since 1984. The survey contains individual-specific information, including residence, state of health, access to health coverage, and others. The survey is run on a rolling basis, and the dataset can be organized as a long sequence of monthly and quarterly observations since we can cluster observations by the date of the interview. On average, we observe $150$ childless adults between $18$ and $64$ years old in Tennessee per month from $2017$ to $1993$. To overcome survey variability, we aggregate data at a quarter level. The outcome variable is the percentage of childless adults who answered yes to the following survey question:“Was there a time in the past 12 months when you needed to see a doctor but could not because of the cost?”.
In Figure (ref), we report the distribution of respondents who were not able to afford medical costs in the past $12$ months(left panel) and who are covered by health insurance\footnote{For the latter questions, we count the number of individuals who answer yes to the question: “Do you have any kind of health care coverage, including health insurance, prepaid plans such as HMOs, or government plans such as Medicare or Indian Health Service?” We consider observations who answer “I do not know” as not having a plan.}(right panel), after clustering over the period $1993$-$2005$, $2006$-$2012$ and $2013$-$2017$ for Tennessee, other Southern States, and the United States. We observe a shift in the mean of Tennessee's outcome over these three periods, with a larger shift in the period just after the policy, between $2006$ and $2012$, while the variance remains approximately stable.
To check for stationarity of observed time-series, we test for unit roots at $95\%$ confidence level. We reject the null hypothesis of a unit root in the time series of interest displayed in Figure (ref).\footnote{We use an Augmented Dickey-Fuller test, with constant and without time trend, and include one, two, or three lags. P-values are respectively $<0.01$ for the first two tests and $0.05$ for the latter.} However, we warn the reader that a lack of stationarity or confounders may invalidate the analysis. To accommodate failures of stationarity in the presence of time-varying fixed effects, we consider two alternative estimators with and without fixed effects adjustments, as discussed in the following subsection.
As proposed in the Synthetic Control literature abadie2010synthetic, we impute the potential outcome under no dis-enrollment using a set of control variables in the other states. The starting date of the treatment corresponds to the second half of $2005$.\footnote{The overall dis-enrollment started in July 2005, and it lasted until June 2006. Most childless adults who dis-enrolled during this period were not able to requalify for Medicare garthwaite2014public.}
While the dis-enrollment program may be mostly attributable to an exogenous budget deficit argys2017losing, the series may still be affected by confounding sources over the period study, one of which is Obamacare's launch in $2014$.\footnote{The Affordable Health Care Act, also known as Obama Care, was officially approved in $2010$, but the major change entered into force in $2014$.} To control for potential confounders related to Obamacare, we consider as the post-treatment period the series until $t = 2014$, while we replicate the analysis also including periods until $2017$ in Appendix (ref). The selection into Obamacare from the states may reflect structural differences among different states. Motivated by this observation, we use a pool of control units only those Southern States that, similarly to Tennessee, did not expand Medicaid between $2010$ and $2014$, namely South and North Carolina, Mississippi, Alabama, Florida, and Georgia. In the Appendix, we replicate the analysis with all the states.
We construct the “Synthetic Control” using the Synthetic Learner described in the current paper. We consider the share of individuals in other countries who could not afford necessary health care expenses as control variables. To allow for time-varying fixed effects, we consider two variations of the Synthetic Learner, with and without fixed effects adjustments (see Section (ref)). We refer to the Synthetic Learner with fixed effect adjustment as “Demeaned Synthetic Learner" and SL otherwise.
We train Random Forest, Lasso, and a Factor model\footnote{To guarantee the validity of the algorithm through the sample splitting procedure, the factor model consists of estimating the principal component over the training period, regressing the principal component on the control states over the training period and making counterfactual predictions using the predicted factor on the remaining periods.}, and Diff-in-Diff mean proxy as discussed in doudchenko2016balancing as base predictors.
Random Forest also contains additional covariates, such as the employment level in each state. Hyperparameters for the base predictors are chosen via cross-validation within the sample used to train such predictors. We construct weights with a tuning parameter $\eta = \frac{1}{\sqrt{T} \mathrm{Var}(Y_t)}$ where the rescaling by the variance guarantees that the estimated weights are scale-invariant.\footnote{Since the loss is the squared loss, by rescaling by the variance we have losses of the form $(Y_t - \hat{Y}_t)^2/\mathrm{Var}(Y_t) = (Y_t/\mathrm{SD}(Y_t) - \hat{Y}_t/\mathrm{SD}(Y_t))^2 $ which are unit free in the outcome's unit. We rescale by $1/\sqrt{T}$ following the theoretical results of predictions. In the Appendix, we report results also after choosing different rescaling. } In Appendix (ref), we show the robustness of our results for several other choices of $\eta$. We consider two alternative sample splitting rules. First, observations between $1998$ and $2006$ are used to train the algorithms; observations between $1993$ and $1997$ are used to compute the weights and bootstrap. Second, the reverse is considered, where the training occurs over the earliest pre-treatment period. Observations from January $2006$ onwards are used to compute the test statistic.
In Table (ref) we report the estimated test statistic for testing the null hypothesis of no effect, namely $H_0: Y_{0t}^1 - Y_{0t}^0 = 0, \ t > T_0$, over the post-treatment period 2010-2014. The table reports the critical quantiles, the test statistic, and the estimated ATT when predictors are trained on the period closest to the post-treatment period (Period 1) or an earlier period (Period 2). The ATT oscillates between one and four percentage points, and its sign remains robust throughout all the setups considered. Significant effects are detected when training predictors on Period 1 for the test with a size of ten percent for Synthetic Learner. When estimating treatment effects by training the predictors more distant from the treatment timing (Period 2), results become non-significant, possibly reflecting higher uncertainty. Similarly, when we consider the adjustment for fixed effects, we observe p-values close to or larger than twenty percent, suggesting higher uncertainty for this case.
The Synthetic Learner predicts an effect larger than Lasso's one but smaller than a factor model. The reader may refer to the Appendix for further details. In Appendix (ref), we report results over the period 2010-2017, showing attenuated results over the time window 2010-2017.
In Table (ref) we collect the weights assigned to each base-algorithm. We observe that the Synthetic Learner assigns a larger weight to Lasso in the absence of fixed effects, and a larger weight to Random Forest in the presence of fixed effects.
In Figure (ref), we report the test statistics and the acceptance region for Tennessee and for placebo tests performed on the other Southern States that did not adopt Medicaid expansion. A placebo test consists of testing a policy's effect from $2006$ to $2014$ in a state different from Tennessee. Since none of the other Southern States had significant changes in the Medicaid system, we would expect no rejections for all Southern States except Tennessee. This is shown in Figure (ref). We observe that we do not reject the null hypothesis when using only the simple DiD method, potentially due to the underpowered test. This result is consistent with what we observed in simulations, where the synthetic learner outperformed other methods in terms of power.
We conclude this section with a discussion on the assumptions and possible sources of confounding. The Tennessee dis-enrollment program was a result of a budget deficit. Whenever the budget deficit was due to an exogenous variation argys2017losing, the variation in state-level Medicaid expenses can be interpreted as exogenous to childless adults' financial status validating the exogeneity of the treatment timing. In this scenario, the ATT estimator is a consistent estimator of the underlying treatment effect. However, this condition may not necessarily hold. For instance, the budget deficit may be attributed to the increase in Medicaid expenses in previous years garthwaite2014public, which may itself depend on individuals' past average income. In such a case, exogeneity may be replaced by conditional exogeneity given the past filtration without violating the prediction guarantees of the proposed algorithm.
The results of our testing procedure should instead be interpreted as conditional on the treatment assignment mechanism, similar to what is discussed in ferman2016revisiting. Here, the assumption of stationarity may fail if, for example, spillover effects of the dis-enrollment program occur over adjacent states, therefore changing the distribution of control units (see the discussion in Appendix (ref)). While the study of Synthetic controls under spillovers goes beyond this paper's scope, we observe that in this scenario, it may be necessary to estimate counterfactuals also on the other states, which may be affected by the policy intervention.
Confounding may also result from other events, such as the 1996 Clinton welfare reform and the great depression. In the former case, although the reform in Tennessee was targeted at families with children (so-called Family First policies\footnote{The reader may refer to \url{https://haslam.utk.edu/sites/default/files/ffoct00.pdf}.}), which are excluded from our analysis, it may act as a confounder in our analysis in the presence of general equilibrium effects. Time-fixed effects can (partially) accommodate for these cases if the confounder affects the mean only, and the effect is homogenous across the states considered in our analysis. These restrictions motivate studies that focus on sets of control units most similar to the treated, which, in our case, correspond to a subset of Southern States.
In many applications, treatment effects may carry over in time imai2021use. Here we extend the proposed framework to carry-over effects as follows. We consider binary treatment and denote the treatment path up to time $t$ as a vector $\mathbf{d} _{1:t}\in \{0,1\}^{t}$. Following the potential outcomes framework, we then posit the existence of potential outcomes $\tilde{Y}_{0t}(\mathbf{d}_{1:t})$, corresponding respectively to the response the treated subject would have experienced at time $t$ while being exposed to the treatment assignment contained in the treatment path $\mathbf{d}_{1:t}$. Formulating treatments and potential outcomes as paths were introduced initially by robins1986new.
Notation implicitly assumes no lead effects athey2018design. Also, we require that the realizations of potential outcomes do not depend on past $m$ lags or more. Using the same notation as in rambachan2019nonparametric, for $t \in \{T_-, \dots ,0, 1, \dots , T_0, \dots, T\}$ we assume throughout the rest of this paper that the following holds.
The assumption explicitely defines carry-over effects of size $m$. The no-anticipation assumption has been previously discussed in abbring2007econometric, athey2018design, while the restricted carryover effect is analogous to the identification assumption stated in imai2018matching, bojinov2018time, blackwell2018make among others. The estimand of interest is now defined as follows $$ \mathbb{E}\Big[Y_{0t}(\mathbf{1}) - Y_{0t}(\mathbf{0})\Big], \quad t > T_0 + m $$ which denotes the (long-run) ATT, comparing two policies always and never implemented.
The key idea in this setting consists in estimating treatment effects after removing the $m$ lag components. Formally, we construct an ATT estimator of the form
where the estimator averages after $m$ periods that the policy has been implemented. Our testing procedure remains invariant (and valid) after removing the periods $t \in \{T_0, \cdots, T_0 + m\}$.
In this paper, we have introduced a novel strategy for estimating treatment effects and testing the null hypothesis of interest in the presence of time-dependent observations. We developed a novel algorithm, denoted as Synthetic Learner, that predicts the counterfactual building on multiple regression methods. Our framework provides a starting point for performing estimation and inference, which is valid regardless of the class of models under consideration.
The presence of one single treated unit at a given time of adopting the policy brings substantial challenges from an identification perspective. We considered three scenarios of interest. First, (i) the adoption date is deterministic, $T_0$ and fixed treatment effects similarly to chernozhukov2017exact, chernozhukov2018t, arkhangelsky2019synthetic among others. We show that, under stationarity and mixing conditions, our algorithm controls the nominal size regardless of the class of base algorithm under consideration, even in the presence of misspecification bias. Extending this result to non-deterministic $T_0$ is conceptually feasible in the presence of multiple units treated at different points in time. (ii) We consider a random and exogenous time of treatment, and under stationarity assumptions, we show that the estimator for the average treatment effect is consistent under weak assumptions, letting $T_0$ be non-deterministic. Finally, (iii) we let the treatment time be sequentially exogenous, without assuming any stationarity condition. We provide bounds on the predictive performance under this complex scenario. Our paper also opens new questions on constructing valid machine learning methods for causal inference when units exhibit dependence. We leave it to future research its study for inference on conditional average treatment effects under heterogeneous effects and endogenous treatment time.