Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data

Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data

Introduction

In a seminal paper, abadie1 set out to investigate the economic impact of terrorism in Basque Country. Prior to the outset of terrorist activity in the early 1970's, Basque Country was considered to be one of the wealthiest regions in Spain. After thirty years of turmoil, however, its economic activity dropped substantially relative to its neighboring regions. Although intuition affirms that Basque Country's economic downturn can be attributed, at least partially, to its political and civil unrest, it is difficult to quantitatively isolate the economic costs of conflict. In response to this challenge, abadie1 introduced the synthetic controls framework. At its core, synthetic controls constructs a synthetic Basque Country from a weighted composition of control regions that are largely unaffected by the instability to estimate Basque Country's economic evolution in the absence of terrorism. This novel concept has inspired an entire subliterature within econometrics that is “arguably the most important innovation in the policy evaluation literature in the last 15 years” athey_imbens.

Researchers have historically tackled problems of this flavor using repeated observations of units across time, i.e., panel data, where a subset of units are exposed to a treatment during some time periods while the other units are unaffected. In the study above, the per capita gross domestic product (GDP) of $17$ Spanish regions are measured from 1955--1998. Basque Country is the sole treated unit and the remaining regions are the control units; the pre- and post-treatment periods are defined as the time horizons before and after the first wave of terrorist activity, respectively.

Synthetic controls has become a cornerstone for panel studies in recent years and across numerous fields. Beforehand, the unconfoundedness approach rubin_rosenbaum, imbens_wooldridge served as a common workhorse. Whereas synthetic controls posits a relation between treated and control units that is stable across time, unconfoundedness posits a relation between treated and pretreatment periods that is stable across units. Accordingly, synthetic controls exploits cross-sectional correlation patterns while unconfoundedness exploits time series correlation patterns. Considering the panel data format, unconfoundedness and synthetic controls based methods are commonly referred to as horizontal (HZ) and vertical (VT) regressions, respectively. Given their conceptual and computational distinctions, the two approaches are considered to be fundamentally different mc_panel.

Yet, contrary to conventional wisdom, it turns out that HZ and VT regressions can yield identical point estimates. As Figure (ref) shows, when the regression models are learned via ordinary least squares (OLS) or principal component regression (PCR), then the two approaches produce the same economic evolution for Basque Country in the absence of terrorism. Figure (ref), by contrast, shows that when the regression models are learned via lasso or lie within the simplex---as proposed by abadie1 for VT regression---then the two approaches output contrasting economic trajectories.

Curiously, Figure (ref) indicates that even when the two regressions arrive at the same point estimate, the confidence intervals can be markedly different under different sources of randomness.

The juxtaposition of these figures beg two questions:

{\bf Contribution.} This article tackles Q1--Q2 from first principles. In this endeavor, we begin by classifying several widely studied regression formulations into (i) a symmetric class that yields identical point estimates and (ii) an asymmetric class that yields contrasting point estimates. Within the symmetric class, we study properties of the estimator with randomness stemming from (i) time series patterns, (ii) cross-sectional patterns, and (iii) both patterns. We conduct our analysis from a (i) model-based perspective, which attributes randomness to the potential outcomes, and a (ii) design-based perspective, which attributes randomness to the treatment assignment mechanism. In both frameworks, we find that the source of randomness has large implications for the estimand and inference. Under the model-based framework, we construct confidence intervals for each source of randomness. Through data-inspired simulations and empirical applications, we demonstrate that the confidence interval developed for one estimand often has incorrect coverage for another estimand. Taken together, our results emphasize that the source of randomness that researchers assume has direct implications for the accuracy of the inference that can be conducted.

{\bf Organization.} Section (ref) overviews the panel data framework. Sections (ref)--(ref) provide one set of answers for Q1--Q2. Section (ref) illustrates concepts developed in this article. Section (ref) summarizes our findings. Details of simulations and empirical applications, select discussions, and mathematical proofs are relegated to the appendix.

{\bf Notation.} Let $\boldsymbol{I}$ be the identity matrix. Let $\boldsymbol{1}$ and $\boldsymbol{0}$ be the vectors of ones and zeros, respectively. The curled inequality denotes $\succeq$ the generalized inequality, i.e., componentwise inequality between vectors and matrix inequality between symmetric matrices. Let $\circ$ denote the componentwise product. For vectors $\boldsymbol{a}$ and $\boldsymbol{b}$, let $\langle a, b \rangle = a' b$ denote the inner product. Let $\operatorname{tr}(\bA)$ denote the trace of $\bA$. We define $0/0 = 0$ when applicable.

The Panel Data Framework

We anchor on the Basque study to introduce the panel data framework. Panel data contains observations of $N$ units over $T$ time periods. The Basque study, for instance, consists of per capita GDP across $N=17$ Spanish regions over $T=43$ years. In each time period $t$, each unit $i$ is characterized by two potential outcomes, $Y_{it}(0)$ and $Y_{it}(1)$, which correspond to its outcome in the absence and presence of a binary treatment, respectively. The potential outcomes framework posits that each region possesses two possible levels of economic activity each year, one that is immune to terrorism and another that is affected by terrorism. In reality, however, we can only observe one economic state, $Y_{it}(0)$ or $Y_{it}(1)$---this is the fundamental challenge of causal inference.

Let $Y_{it}$ be the observed outcome. Often, we observe all $N$ units without treatment (control) for $T_0$ time periods, i.e., $Y_{it} = Y_{it}(0)$ for all $i \le N$ and $t \le T_0$. For the remaining $T_1 = T-T_0$ time periods, $N_1$ units receive treatment while the remaining $N_0 = N-N_1$ units remain under control, i.e., if we arbitrarily label the first $N_0$ units as the control group, then $Y_{it} = Y_{it}(1)$ for all $i > N_0$ and $t > T_0$, and $Y_{it} = Y_{it}(0)$ for all $i \le N_0$ and $t > T_0$. In our study, Basque Country is the single treated unit, thus $N_1 = 1$ and $N_0=16$. The first wave of terrorist activity partitions the time horizon into pre- and post-treatment periods of lengths $T_0=15$ and $T_1=28$ years, respectively.

For ease of exposition, this article considers a single treated unit and single treated period indexed by the $N$th unit and $T$th time period, respectively. However, our results hold for any $(i,t)$ pair where $i > N_0$ is a treated unit and $t > T_0$ is a treated period. We organize our observed control data into an $N \times T$ matrix, $\boldsymbol{Y} = [Y_{it}]$, as shown in Figure (ref). In our example, $\boldsymbol{y}_N = [Y_{Nt}: t \le T_0] \in \Rb^{T_0}$ represents Basque Country's economic evolution prior to the outset of terrorism; $\boldsymbol{Y}_0 = [Y_{it}: i \le N_0, t \le T_0] \in \Rb^{N_0 \times T_0}$ represents the control regions' economic evolution prior to the outset of terrorism; and $\boldsymbol{y}_T = [Y_{iT}: i \le N_0] \in \Rb^{N_0}$ represents the control regions' economic evolution after the outset of terrorism. Our object of interest is Basque Country's counterfactual GDP in the absence of terrorism, $Y_{NT}(0)$.

Time Series Versus Cross-Sectional Based Regressions

The information across time and space motivates two natural ways to impute the missing $(N,T)$th entry. These perspectives are explored in two large and mostly separate bodies of work mc_panel.

Horizontal Regression and Unconfoundedness

The unconfoundedness literature operates on the concept that “history is a guide to the future”. As such, unconfoundedness methods express outcomes in the treated period as a weighted composition of outcomes in the pretreatment periods. This is carried out by regressing the control units' treated period outcomes $\boldsymbol{y}_T$ on its lagged outcomes $\boldsymbol{Y}_0$ and applying the learned regression coefficients to the treated unit's lagged outcomes $\boldsymbol{y}_N$ to predict the missing $(N,T)$th outcome. Following mc_panel, we refer to such methods as horizontal (HZ) regression.

Vertical Regression and Synthetic Controls

The synthetic controls literature is built on the concept that “similar units behave similarly”. Therefore, synthetic controls methods express the treated unit's outcomes as a weighted composition of control units' outcomes. This is carried out by regressing the treated unit's lagged outcomes $\boldsymbol{y}_N$ on the control units' lagged outcomes $\boldsymbol{Y}'_0$ and applying the learned regression coefficients to the control units' treated period outcomes $\boldsymbol{y}_T$ to predict the missing $(N,T)$th outcome. Following mc_panel, we refer to such methods as vertical (VT) regression.

Conventional Wisdom

The asymmetry between HZ and VT regressions has created the conception that they are fundamentally different approaches mc_panel. In fact, the unregularized forms of HZ and VT regressions are cautioned against when $T > N$ and $N > T$, respectively abadie4, sc_enet, li_bell, mc_panel. With regularization, however, mc_panel argues the two approaches can be applied to the same setting. In turn, this allows the two approaches to be systematically compared through methods such as cross-validation.

In parallel, the growth rates of the two literatures have also exhibited asymmetry. While the development of the unconfoundedness literature has seemingly plateaued, the synthetic controls literature continues to rapidly expand. Across many domains, synthetic controls based methods are arguably the de facto approach for panel studies.

Point Estimation

We tackle Q1 by studying the finite-sample estimation properties of HZ and VT regressions. We denote the singular value decomposition of $\boldsymbol{Y}_0$ as $\boldsymbol{Y}_0 = \sum_{\ell =1}^{R} s_\ell \boldsymbol{u}_\ell \boldsymbol{v}'_\ell = \bU \bS \bV'$, where $\boldsymbol{u}_\ell \in \Rb^{N_0}$ and $\boldsymbol{v}_\ell \in \Rb^{T_0}$ are the left and right singular vectors, respectively, $s_\ell \in \Rb$ are the ordered singular values, and $R = \text{rank}(\boldsymbol{Y}_0) \le \min\{N_0, T_0\}$. $\bU \in \Rb^{N_0 \times R}$ and $\bV \in \Rb^{T_0 \times R}$ denote the matrices formed by the left and right singular vectors, respectively, and $\bS \in \Rb^{R \times R}$ is the diagonal matrix of singular values. The Moore-Penrose pseudoinverse of $\boldsymbol{Y}_0$ is $\boldsymbol{Y}_0^\dagger = \sum_{\ell=1}^R (1/s_\ell) \boldsymbol{v}_\ell \boldsymbol{u}'_\ell = \bV \bS^{-1} \bU'$. Critically, we do not place any assumptions on the relative magnitudes of $N$ and $T$.

Classifying Notable Regression Formulations

We present several of the most widely studied regression formulations in the HZ and VT literatures. This list is far from exhaustive given the vastness of these literatures.

Description of Estimation Strategies

{\bf Penalized regression.} A large class of penalized regressions are expressed as follows:

We overview common choices for $(\lambda_1, \lambda_2)$ and describe the corresponding strategy.

{\em I: Ordinary least squares (OLS).} Arguably, the mother of all regressions is OLS, where $\lambda_1 = \lambda_2 = 0$. OLS is an unconstrained problem with possibly infinitely many solutions. OLS has been analyzed in numerous works on panel studies, including hcw, li_bell, and li2020.

{\em II: Principal component regression (PCR).} To formalize PCR, let

denote the rank $k < R$ approximation of $\boldsymbol{Y}_0$ that retains the top $k$ principal components. HZ and VT PCR corresponds to replacing $\boldsymbol{Y}_0$ with $\boldsymbol{Y}_0^{(k)}$ within (ref) and (ref), respectively, with $\lambda_1 = \lambda_2 = 0$. In words, PCR first finds a $k$ dimensional representation of the covariate matrix via principal component analysis; then, PCR performs OLS with the compressed $k$ dimensional covariates. Within the synthetic controls literature, rsc, mrsc and si utilize PCR.

{\em III: Ridge regression.} Consider ridge regression, where $\lambda_1=0$ and $\lambda_2 > 0$. When $\boldsymbol{Y}_0$ is rank deficient, the gram matrix, i.e., $\boldsymbol{Y}'_0 \boldsymbol{Y}_0$ for HZ regression and $\boldsymbol{Y}_0 \boldsymbol{Y}'_0$ for VT regression, is ill-conditioned. This often discourages the usage of OLS. In these settings, ridge provides a remedy by adding a {\em ridge} on the diagonal of the gram matrix, which increases all eigenvalues by $\lambda_2$, thus removing the singularity problem. sc_aug explores the properties of a doubly robust estimator that utilizes HZ ridge regression.

{\em IV: Lasso regression.} Consider lasso regression, where $\lambda_1 > 0$ and $\lambda_2 = 0$. Lasso has become a popular tool for estimating sparse linear coefficients in high-dimensional regimes. Because the lasso criterion not strictly convex, there are possibly infinitely many solutions. Thus, for our analysis of lasso only, we make the mild assumption that the entries of $\boldsymbol{Y}_0$ are drawn from a continuous distribution. As established in lasso_ryan, this guarantees the lasso solution to be unique. Several notable works in the synthetic controls literature, e.g., li_bell, arco, and sc_lasso1, analyze the lasso.

{\em V: Elastic net regression.} Mixing both $\ell_1$ and $\ell_2$-penalties, i.e., $\lambda_1, \lambda_2 > 0$, is known as elastic net. At a high level, elastic net selects variables similar to the lasso, but deals with correlated variables more gracefully as with ridge. When $\lambda_2 > 0$, the criterion is strictly convex so the solution is unique. sc_enet propose an elastic net synthetic controls variant.

{\bf Constrained regression.}

{\em VI: Simplex regression.} The next formulation constrains the regression weights to lie within the simplex, i.e., the weights are nonnegative and sum to one:

We consider a vanishing $\ell_2$ penalty since $\lambda = 0$ (standard formulation) can induce multiple minima abadie_hour. When $\lambda > 0$, the criterion becomes strictly convex and the solution is unique. Simplex regression is the original formulation set forth in the pioneering works of abadie1, abadie2, abadie4, and its properties continue to be actively studied today. Attractive aspects of simplex regression include interpretability, sparsity, and transparency abadie_survey.

Classification Results

To answer Q1, we classify the regression formulations into (i) a symmetric class, where HZ and VT point estimates agree, and (ii) an asymmetric class, where HZ and VT point estimates disagree. We use the shorthand $\textsf{HZ} = \textsf{VT}$ if the two approaches produce identical point estimates and $\textsf{HZ} \neq \textsf{VT}$ otherwise.

{\bf I: Symmetric class.} We first state the symmetric formulations.

Theorem (ref) might seem familiar at first glance. As observed in abadie4 and sc_aug, the point estimates associated with HZ OLS and HZ ridge can be written as linear combinations of the elements in $\boldsymbol{y}_T$, which take the same {\em linear forms} as the corresponding VT point estimates. However, their results stop short of establishing numerical equivalence as in Theorem (ref). From this perspective, Theorem (ref) is perhaps surprising as it takes the next step forward in contradicting the notion that the two regressions are fundamentally different. In particular, Theorem (ref) proves the same point estimate is derived from both approaches whenever the regression model belongs to the symmetric class, namely (i) OLS with minimum $\ell_2$-norm, (ii) PCR for the same choice of $k$, and (iii) ridge for the same choice of $\lambda_2$. In this view, HZ and VT regressions are not perspectives at duel---they are dual perspectives.

We emphasize that Theorem (ref) holds for any data configuration. As such, it clarifies that HZ and VT OLS are {\em not} invalid when $T > N$ and $N > T$, respectively, as previously believed. In fact, the OLS estimate can even be written as $\langle \widehat{\boldsymbol{\alpha}}, \boldsymbol{Y}'_0 \widehat{\boldsymbol{\beta}} \rangle$, which incorporates both regression models. The origin of the prior misconception may have come from the fact that infinitely many solutions exist when $\boldsymbol{Y}_0$ is rank deficient. Among these solutions, however, is the unique minimum $\ell_2$-norm model, which is arguably sufficient for inference shao_deng. This is also the solution when the problem is optimized via gradient descent, a ubiquitous optimizer in practice. Phenomena of this form are known as “implicit regularization”, where the optimization algorithm is biased towards a particular solution even though the bias is not explicit in the objective function ireg2, ireg4.

Through its connection to the $\ell_2$-penalty, the minimum $\ell_2$-norm also offers a high-level intuition for the root of symmetry. More specifically, observe that the ridge model converges to the OLS model with minimum $\ell_2$-norm as $\lambda_2 \rightarrow 0$. Since the PCR model is precisely the OLS minimum $\ell_2$-norm model that is restricted to the space spanned by the top $k$ principal components, we conjecture that the geometry of the $\ell_2$-ball is a likely source for HZ and VT estimation symmetry.

{\bf II: Asymmetric class.} Next, we state the class of formulations that fracture symmetry.

To examine the implications of Theorem (ref), we observe that the common thread between the objective functions in the asymmetric class is a penalty or constraint that promotes sparse models. Such regularizers are noticeably absent in the symmetric formulations. This highlights an interesting trade-off---while sparsity is widely considered to be a salient feature, the geometries of the $\ell_1$-ball and simplex that encourage sparsity are also the likely sources of HZ and VT estimation asymmetry.

Doubly Robust Estimators

In recent years, there has been a surge of interest in doubly robust estimators. Within panel data, we discuss two prominent works that are rising in popularity.

Synthetic Difference-in-Differences

An important approach that continues to dominate empirical work in panel data is the difference-in-differences (DID) estimator did1. At a high level, DID posits an additive outcome model with unit- and time-specific fixed effects, known more colloquially as the “parallel trends” assumption. The recent work of sdid anchors on the DID principle and brings in concepts from the unconfoundedness and synthetic controls literatures to derive a doubly robust estimator called synthetic difference-in-differences (SDID). In our setting, the SDID prediction for the missing $(N,T)$th potential outcome can be written as

where $\widehat{\boldsymbol{\alpha}}$ and $\widehat{\boldsymbol{\beta}}$ represent general HZ and VT models, respectively. The authors note that $\widehat{\boldsymbol{\alpha}} = (1/T_0) \boldsymbol{1}$ and $\widehat{\boldsymbol{\beta}} = (1/N_0) \boldsymbol{1}$ recovers DID. Moving beyond simple DID to performing a weighted two-way bias removal, the authors propose to learn $\widehat{\boldsymbol{\alpha}}$ via simplex regression and $\widehat{\boldsymbol{\beta}}$ via simplex regression with an $\ell_2$-penalty.

Augmented Synthetic Controls

Another notable work is that of sc_aug. The authors introduce the augmented synthetic control (ASC) estimator, which uses an outcome model to correct the bias induced by the classical synthetic controls estimator.\footnote{See abadie_hour for a bias correction of synthetic controls through matching.} Concretely, the ASC estimator predicts the missing $(N,T)$th potential outcome as

where $\widehat{M}_{iT}(0)$ is the estimator for the $(i,T)$th entry. The authors instantiate

Plugging the HZ outcome model in (ref) into (ref) then gives

We consider this particular variant of ASC since it takes the same form as SDID, as seen in (ref). In contrast to sdid, sc_aug learns $\widehat{\boldsymbol{\alpha}}$ via ridge regression and $\widehat{\boldsymbol{\beta}}$ via simplex regression.

When Doubly Robust Estimators are No Longer Doubly Robust

We leverage Theorem (ref) to study the estimation properties of SDID and ASC when $(\widehat{\boldsymbol{\alpha}}, \widehat{\boldsymbol{\beta}})$ are learned via OLS and PCR.

Corollary (ref) states that a researcher who uses the implicitly regularized (but explicitly unconstrained) variants of SDID and ASC inevitably arrive at the same point estimate as their colleague who simply uses HZ or VT OLS. The same phenomena occurs for PCR. From this, we observe that SDID and ASC can lose their weighted double-differencing effects under certain formulations. However, Corollary (ref) is not to discredit either approach. The fact remains that both methods allow researchers to naturally and simultaneously encode their knowledge of time and unit specific structures into the estimator. \textsf{SDID} and \textsf{ASC} warrant further studies and careful consideration.

Intercepts

Intercepts can be included in the HZ regression model by modifying the $\ell_2$-errors in (ref) and (ref) as $\| \boldsymbol{y}_T - \boldsymbol{Y}_0 \boldsymbol{\alpha} - \alpha_0 \boldsymbol{1} \|_2^2$; similarly, they can be included in the VT regression model by modifying $\ell_2$-errors in (ref) and (ref) as $\| \boldsymbol{y}_N - \boldsymbol{Y}'_0 \boldsymbol{\beta} - \beta_0 \boldsymbol{1} \|_2^2$. To date, there is still a lack of general consensus on the inclusion of intercepts within the synthetic controls literature. We attempt to shed light on the role of intercepts.

We develop an intuition for Proposition (ref) by interpreting intercepts in panel studies. A nonzero time intercept, $\alpha_0$, imposes a permanent constant difference between the treated and pretreatment periods; a nonzero unit intercept, $\beta_0$, imposes a permanent constant difference between the treated and control units. These systematic structures then create an asymmetry between the two regressions. Below, we propose a methodology based on centering the data that allows for intercepts yet retains symmetry.

Including Intercepts and Retaining Symmetry through Data Centering

Let $\boldsymbol{Y}_0$ be twice centered, i.e., the rows and columns of $\boldsymbol{Y}_0$ are mean zero. This can be satisfied by applying $\boldsymbol{I} - (1/N_0) \boldsymbol{1} \boldsymbol{1}'$ and $\boldsymbol{I} - (1/T_0) \boldsymbol{1} \boldsymbol{1}'$ to the left and right, respectively, of $\boldsymbol{Y}_0$. Consider the following modified formulations.

Similar to before, OLS corresponds to $\lambda = 0$, PCR corresponds to OLS with $\boldsymbol{Y}_0^{(k)}$ for $k < R$ in place of $\boldsymbol{Y}_0$, and ridge regression corresponds to any $\lambda > 0$.

We inspect (ref) and (ref) to understand the implications of Corollary (ref). First, we recall Theorem (ref), which establishes that the HZ and VT estimates share the same “base” estimate, i.e., $\langle \boldsymbol{y}_N, \widehat{\boldsymbol{\alpha}} \rangle = \langle \boldsymbol{y}_T, \widehat{\boldsymbol{\beta}} \rangle$. Next, we note that $\widehat{\alpha}_0 = \widehat{\beta}_1 = (1/N_0) \boldsymbol{1}' \boldsymbol{y}_T$ and $\widehat{\beta}_0 = \widehat{\alpha}_1 = (1/T_0) \boldsymbol{1}' \boldsymbol{y}_N$, which correspond to the time and unit fixed effects, respectively. Intuitively, the modified point estimates in (ref) and (ref) include both fixed effect models to compensate for $\boldsymbol{Y}_0$ being twice centered. Putting everything together, the modified HZ and VT point estimates are identical.

Inference

To answer Q2, we study the inferential properties of the counterfactual prediction. Formal discussions for classical inference require an explicit postulation of where the randomness arises from. This article takes both a (i) model-based approach, which makes assumptions about the distribution of the potential outcomes, and a (ii) design-based approach, which makes assumptions about the assignment mechanism of treatment. We start by taking a model-based route that is similar in parts to the recent works of li2020, cattaneo, and sc_lasso1, sc_t_test from the synthetic controls literature. We then transition to a design-based route that is paved by the ideas introduced in guido_inf.

For ease of exposition, we focus on OLS and its minimum $\ell_2$-norm solutions, i.e., $\widehat{\boldsymbol{\alpha}} = \boldsymbol{Y}_0^\dagger \boldsymbol{y}_T$ and $\widehat{\boldsymbol{\beta}} = (\boldsymbol{Y}'_0)^\dagger \boldsymbol{y}_N$. As such, we define $\widehat{Y}_{NT}(0) = \widehat{Y}^\text{hz}_{NT}(0) = \widehat{Y}^\text{vt}_{NT}(0)$, which is justified under Theorem (ref). To improve readability, we present several results informally and provide their precise versions in Appendix (ref).

Model-Based Inference

Within the model-based framework, we consider a classical regression model. This postulation is not always plausible but it is useful to dissect its implications for the role of randomness in conducting inference.

Generative Models

We now study properties of $\widehat{Y}_{NT}(0)$ from three different sources of randomness.

{\bf I: Horizontal model.} The HZ model focuses on time series correlation patterns.

Assumption (ref) motivates the HZ approach, which models time series patterns as the source of randomness. Under the HZ model, the statistical uncertainty of $\widehat{Y}_{NT}(0)$ is governed by the construction of $\widehat{\boldsymbol{\alpha}}$ from $(\boldsymbol{y}_T, \boldsymbol{Y}_0)$, i.e., the in-sample uncertainty.

Assumption (ref) states that the errors have zero mean and thus the regressors, i.e., lagged outcomes, are uncorrelated with the errors; this is known in the literature as strict exogeneity. The errors are also conditionally independent across space.

{\bf II: Vertical model.} The VT model focuses on cross-sectional correlation patterns.

Assumption (ref) is analogous to Assumption (ref) with the source of randomness here emanating from cross-sectional patterns. Hence, the statistical uncertainty of $\widehat{Y}_{NT}(0)$ under the VT model is governed by the construction of $\widehat{\boldsymbol{\beta}}$ from $(\boldsymbol{y}_N, \boldsymbol{Y}_0)$.

Assumption (ref) is analogous to Assumption (ref) with the errors here being conditionally independent across time.

{\bf III: Mixed model.} We introduce a new model that mixes aspects of the HZ and VT models. At a high level, the mixed model accounts for randomness along both dimensions of the data. This comes at the price of placing additional constraints on the stochastic properties of the errors.

(ref) and (ref) correspond to Assumptions (ref) and (ref), respectively. Collectively, they model time series and cross-sectional patterns as two distinct sources of randomness. Thus, the statistical uncertainty of $\widehat{Y}_{NT}(0)$ under the mixed model is governed by the constructions of both $\widehat{\boldsymbol{\alpha}}$ and $\widehat{\boldsymbol{\beta}}$.

Assumption (ref) combines Assumptions (ref) and (ref). In words, it states that $\boldsymbol{Y}_0$ contains all measured confounders and the errors are independent across both time and space. We leave a proper analysis under dependent errors as future work.

Inferential Properties

Equipped with our three models, we are ready to provide one set of answers to Q2. In what follows, we define the error covariance matrices as (i) $\bSigma^\text{hz}_T = \operatorname{Cov}(\boldsymbol{\varepsilon}_T | \boldsymbol{y}_N, \boldsymbol{Y}_0)$, (ii) $\bSigma^\text{vt}_N = \operatorname{Cov}(\boldsymbol{\varepsilon}_N | \boldsymbol{y}_T, \boldsymbol{Y}_0)$, (iii) $\bSigma^\text{mix}_T = \operatorname{Cov}(\boldsymbol{\varepsilon}_T | \boldsymbol{Y}_0)$, and (iv) $\bSigma^\text{mix}_N = \operatorname{Cov}(\boldsymbol{\varepsilon}_N | \boldsymbol{Y}_0)$, where $\boldsymbol{\varepsilon}_T = [\varepsilon_{iT}: i \le N_0]$ and $\boldsymbol{\varepsilon}_N = [\varepsilon_{Nt}: t \le T_0]$.

Theorem (ref) highlights that each model measures uncertainty with respect to a different estimand. In particular, Theorem (ref) states that the asymptotic variance is controlled by time series patterns under the HZ model, cross-sectional patterns under the VT model, and both correlation patterns under the mixed model. This clarifies that the source of randomness has substantive implications for the estimand and inference.

Confidence Intervals

Theorem (ref) motivates separate HZ, VT, and mixed confidence intervals: for $\theta \in (0,1)$,

where $z_{\frac \theta 2}$ is the upper $\theta/2$ quantile of $\mathcal{N}(0,1)$, and $(\widehat{v}^\text{hz}_0, \widehat{v}^\text{vt}_0, \widehat{v}^\text{mix}_0)$ are the estimators of $(v^\text{hz}_0, v^\text{vt}_0, v^\text{mix}_0)$. We construct

where $\widehat{\bSigma}_T$ and $\widehat{\bSigma}_N$ are the estimators of $(\bSigma^\text{hz}_T, \bSigma^\text{mix}_T)$ and $(\bSigma^\text{vt}_N, \bSigma^\text{mix}_N)$, respectively. We precisely define them under homoskedastic and heteroskedastic errors below. To reduce ambiguity, we index $(\widehat{v}^\text{hz}_0, \widehat{v}^\text{vt}_0, \widehat{v}^\text{mix}_0)$ by the covariance estimator. We also denote $\bH^u = \bU \bU'$ and $\bH^u_\perp = \boldsymbol{I} - \bH^u$. With this notation, the HZ and VT in-sample errors can be written as $\bH^u_\perp \boldsymbol{y}_T = \boldsymbol{y}_T - \boldsymbol{Y}_0 \widehat{\boldsymbol{\alpha}}$ and $\bH^v_\perp \boldsymbol{y}_N = \boldsymbol{y}_N - \boldsymbol{Y}'_0 \widehat{\boldsymbol{\beta}}$, respectively.

It is clear from (ref) that $(\widehat{v}^\text{hz}_0, \widehat{v}^\text{vt}_0)$ are plug-in estimators for $(v^\text{hz}_0, v^\text{vt}_0)$. As such, we discuss $\widehat{v}^\text{mix}_0$ with respect to $v^\text{mix}_0$. Recall $\widehat{\boldsymbol{\alpha}} = \bH^v \widehat{\boldsymbol{\alpha}}$ and $\widehat{\boldsymbol{\beta}} = \bH^u \widehat{\boldsymbol{\beta}}$ by construction. To justify the negative trace in $\widehat{v}^\text{mix}_0$, note that $\widehat{v}_0^\text{hz}$ is a quadratic involving $(\boldsymbol{y}_N, \boldsymbol{y}_T)$. Since both quantities are random, the expectation of $\widehat{v}_0^\text{hz}$ induces an additional term that precisely corresponds to the trace term in $v_0^\text{mix}$. The same property holds for $\widehat{v}_0^\text{vt}$. Thus, $\widehat{v}^\text{mix}_0$ corrects for this bias via the negative trace.

{\bf Homoskedastic errors.} Consider $\bSigma^\text{hz}_T$ with identical diagonal elements, i.e., $\bSigma^\text{hz}_T = (\sigma^\text{hz}_T)^2 \boldsymbol{I}$, where $(\sigma^\text{hz}_T)^2 = \operatorname{Var}(\varepsilon_{iT} | \boldsymbol{y}_N, \boldsymbol{Y}_0)$ for $i = 1, \dots, N_0$. Let $(\bSigma^\text{vt}_N, \bSigma^\text{mix}_T, \bSigma^\text{mix}_N)$ be defined analogously. We use the standard variance estimators

where $R = \text{rank}(\boldsymbol{Y}_0)$, which can be computed as $R = \operatorname{tr}(\bH^u) = \operatorname{tr}(\bH^v)$.

Lemma (ref) is a well known result within the OLS literature, albeit it is typically formalized under the stricter full column rank assumption.

{\bf Heteroskedastic errors.} We adopt two strategies for the heteroskedastic setting.

{\em I: Jackknife.} The first estimator is based on the jackknife. Traditionally, the jackknife estimates the covariance of the regression coefficients $(\widehat{\boldsymbol{\alpha}}, \widehat{\boldsymbol{\beta}})$. By analyzing said estimates, we derive the following:

Lemma (ref) establishes that the jackknife is conservative, provided $(\bH^u_\perp \circ \bH^u_\perp \circ \boldsymbol{I})$ and $(\bH^v_\perp \circ \bH^v_\perp \circ \boldsymbol{I})$ are nonsingular. Strictly speaking, the jackknife is well defined if these quantities are singular, as seen through the pseudoinverse in (ref) and (ref). Lemma (ref) considers the nonsingular case for simplicity. We remark that $\max_\ell H^u_{\ell \ell} < 1$ and $\max_\ell H^v_{\ell \ell} < 1$ are sufficient conditions for invertibility.

{\em II: HRK-estimator.} Next, we consider the covariance estimator proposed by variance. We index this estimator by the authors, Hartley-Rao-Kiefer:

Lemma (ref) establishes that the HRK estimator is unbiased, provided $(\bH^u_\perp \circ \bH^u_\perp)$ and $(\bH^v_\perp \circ \bH^v_\perp)$ are invertible. For the former quantity, $\max_\ell H^u_{\ell \ell} < 1/2$ is a sufficient condition for invertibility. Since $\operatorname{tr}(\bH^u) = R$, this restricts $R < N_0/2$. A similar conclusion is drawn for VT regression.

Design-Based Inference

This section studies the counterfactual prediction from a design-based perspective, whereby the potential outcomes are considered fixed and the treatment assignments are considered stochastic.

Assumptions

In congruence with the article thus far, we focus on a single treated unit and treated period. We will find it useful to separate the assignment mechanism into the selection of each component. Accordingly, let $\bA \in \{0,1\}^T$ with $\boldsymbol{1}' \bA = 1$ and $\bB \in \{0,1\}^N$ with $\boldsymbol{1}' \bB = 1$ be the indicator vectors for the treated time period and treated unit, respectively, e.g., $B_N = 1$ and $A_T = 1$ if unit $N$ is treated at time $T$. With this notation, we denote the realized outcome as $Y_{it} = B_i A_t Y_{it}(1) + (1 - B_i A_t) Y_{it}(0)$. Following guido_inf, we consider the following assignment mechanisms:

Assumption (ref) considers the treated period to be randomly selected while Assumption (ref) considers the treated unit to be randomly selected. As guido_inf notes, these assumptions are not always plausible, but they underlie the placebo tests that are commonly used in synthetic controls applications.

Estimator

To conduct design-based analysis, we consider all possible treatment assignments, not only the realized assignment. Let $Y^*_{it}(0)$ be the OLS fit of $Y_{it}(0)$ based on $[Y_{j\tau} : j \neq i, \tau < t]$, $[Y_{i \tau} : \tau < t]$, and $[Y_{j t}: j \neq i]$. We define the design-based estimator as

In words, (ref) predicts the mean counterfactual outcome under control for unit $i$ at time $t$ if unit $i$ is treated at time period $t$, i.e., $B_i = 1$ and $A_t = 1$. We reemphasize that the stochasticity of $\widehat{Y}(0)$ stems from the treatment assignment mechanism since $Y^*_{it}(0)$ is a fixed quantity. As such, while the model-and design-based estimators share the same point estimate for the realized assignment, they differ in their formulations and attributions of randomness.

Inferential Properties

In Table (ref), we summarize the estimands associated with the model-based and design-based estimators under three sources of randomness: (i) time, (ii) unit, and (iii) time and unit. Within the model-based framework, mechanisms (i)--(iii) correspond to the HZ model (Assumptions (ref)--(ref)), VT model (Assumptions (ref)--(ref)), and mixed model (Assumptions (ref)--(ref)), respectively. Within the design-based framework, mechanisms (i)--(iii) correspond to Assumption (ref), Assumption (ref), and both Assumptions (ref) and (ref), respectively. Accordingly, the model-based and design-based expectations are taken over different probability measures.

Let us compare the estimands in Table (ref). In words, $\mathbb{E}[\widehat{Y}_{NT}(0) | \boldsymbol{y}_N, \boldsymbol{Y}_0]$ is a weighted combination of outcomes under control for the treated unit across all $T_0$ pretreatment periods; $\mathbb{E}[\widehat{Y}(0) | \bB]$ is a simple average of fitted outcomes under control for the treated unit across all $T$ periods, which guido_inf calls the “HZ” effect. Similarly, $\mathbb{E}[\widehat{Y}_{NT}(0) | \boldsymbol{y}_T, \boldsymbol{Y}_0]$ is a weighted combination of outcomes under control for the treated period across all $N_0$ control units; $\mathbb{E}[\widehat{Y}(0) | \bA]$ is a simple average of fitted outcomes under control for the treated period across all $N$ units, also called the “VT” effect. Finally, $\mathbb{E}[\widehat{Y}_{NT}(0) | \boldsymbol{Y}_0]$ is a weighted combination of outcomes under control across all $N_0$ control units and $T_0$ pretreatment periods; $\mathbb{E}[\widehat{Y}(0)]$ is a simple average of fitted outcomes under control across all unit and time period pairs, which we coin the “mixed” effect.

Though our design-based analysis is brief, there two important takeaways: (i) the model-based and design-based estimators recover similar estimands for each source of randomness; and (ii) different sources of randomness lead to different estimands, which is consistent with our model-based insights. The connection between assumptions of randomness and resulting estimand has been previously noticed in related contexts, e.g., guido_inf0, guido_inf and jas_inf.

Discussion

Several remarks on this section's results and extensions are in order.

Illustrations

This section illustrates key concepts developed in this article. Our report is based on three canonical synthetic controls studies: (i) terrorism in Basque Country, (ii) California's Proposition 99 abadie2, and (iii) the reunification of West Germany abadie4. In particular, we will conduct a model-based analysis using the confidence intervals developed in Section (ref). We provide an overview of the results and relegate details (e.g., implementation) to Appendix (ref).

Background on Case Studies

{\bf Basque study.} See Sections (ref)--(ref) for details.

{\bf California study.} This study examines the effect of California's Proposition 99, an anti-tobacco legislation, on its tobacco consumption. The panel data contains per capita cigarette sales of $N = 39$ U.S. states over $T=31$ years. There are $T_0=18$ pretreatment observations and $N_0 = 38$ control units. Our interest is to estimate California's cigarette sales in the absence of Proposition 99.

{\bf West Germany study.} This study examines the economic impact of the 1990 reunification in West Germany. The panel data contains per capita GDP of $N=17$ countries over $T=44$ years. There are $T_0 = 30$ pretreatment observations and $N_0=16$ control units. Our interest is to estimate West Germany's GDP in the absence of reunification.

Data-Inspired Simulation Studies

We look to better understand the trade-offs in conducting inference under different sources of randomness. In an attempt to document our analysis in a realistic environment, we calibrate our simulations to our three studies.

Data Generating Process

We consider the single treated unit and time period setting. Specifically, we consider the actual treated unit, e.g., Basque Country, and focus on the first post-treatment period, e.g., one year after the outset of terrorism; hence, $T \leftarrow T_0 + 1$. Using the actual data, we generate the underlying regression models as

where $\boldsymbol{y}^*_N = [Y_{Nt}: t \le T_0]$, $\boldsymbol{y}^*_T = [Y_{iT}: i \le N_0]$, and $\boldsymbol{Y}^*_0 = [Y_{it}: i \le N_0, t \le T_0]$.

Observationally, we have access to the following quantities. Let $\boldsymbol{Y}_0$ be the rank $r$ approximation of $\boldsymbol{Y}^*_0$, where $r$ is chosen as the minimum number of singular values needed to capture at least $99.9\%$ of $\boldsymbol{Y}^*_0$'s spectral energy. Next, we sample $\boldsymbol{y}_T \sim \mathcal{N}(\boldsymbol{Y}_0 \boldsymbol{\alpha}^*, (N_0 - r)^{-1} \| \boldsymbol{y}^*_T - \boldsymbol{Y}^*_0 \boldsymbol{\alpha}^* \|_2^2 \boldsymbol{I})$ and $\boldsymbol{y}_N \sim \mathcal{N}(\boldsymbol{Y}'_0 \boldsymbol{\beta}^*, (T_0 - r)^{-1} \| \boldsymbol{y}^*_N - (\boldsymbol{Y}^*_0)' \boldsymbol{\beta}^* \|_2^2 \boldsymbol{I})$. We then define three estimands: (i) $\mu_0^\text{hz} = \langle \boldsymbol{y}_N, \bH^v \boldsymbol{\alpha}^* \rangle$, (ii) $\mu_0^\text{vt} = \langle \boldsymbol{y}_T, \bH^u \boldsymbol{\beta}^* \rangle$, and (iii) $\mu_0^\text{mix} = \langle \boldsymbol{\alpha}^*, \boldsymbol{Y}'_0 \boldsymbol{\beta}^* \rangle$, where $(\bH^u, \bH^v)$ are computed from $\boldsymbol{Y}_0$.

Simulation Results

For the purposes of stability, we conduct 500 replications of the above DGP for each study. In the $\ell$th simulation repeat, we learn the regression coefficients as

The corresponding point estimate is defined as $\widehat{Y}^{(\ell)}_{NT}(0) = \langle \boldsymbol{y}^{(\ell)}_N, \widehat{\boldsymbol{\alpha}}^{(\ell)} \rangle = \langle \boldsymbol{y}_T^{(\ell)}, \widehat{\boldsymbol{\beta}}^{(\ell)} \rangle$. We then construct separate HZ, VT, and mixed homoskedastic confidence intervals based on (ref) and (ref) around the point estimate.

In Table (ref), we report the coverage probabilities (CP) and average lengths (AL) for each confidence interval with respect to each estimand at the $95\%$ nominal mark. With respect to $\mu_0^\text{hz}$, the coverage of the HZ confidence interval is closer to the nominal coverage than that of the VT and mixed intervals as the latter two can substantially under- or over-cover. This storyline is consistent for the VT interval with respect to $\mu_0^\text{vt}$ and the mixed interval with respect to $\mu_0^\text{mix}$.

Collectively, our formal results and simulations demonstrate that (i) the choice of estimand directly affects the accuracy of the inference; and (ii) the variance formulas developed for one estimand may not have the correct coverage for another estimand. Accordingly, researchers should carefully consider the source of randomness in their data as it can have a significant influence over their ability to conduct valid inference. We comment that these conclusions are in line with those drawn in jas_inf, which analyzes the classical difference-in-means estimator with respect to standard estimands for randomized control trials.

Empirical Applications

Next, we analyze our three case studies of interest. All regression models are built on pretreatment data only, and the point and variance estimation formulas are separately applied for the treated unit at each post-treatment period $t > T_0$.

Point Estimation

Figure (ref) visualizes the counterfactual trajectories generated by the estimators in Section (ref). Our findings reinforce Theorems (ref) and (ref). On a separate note, we observe that, within the Basque study, the OLS estimates are wildly different from the other estimates and HZ simplex regression reduces to the last observation carried forward (LOCF) estimator. In the California and West Germany studies, the estimates are all qualitatively similar with the exception of the HZ simplex regression, which again reduces to LOCF. In fact, the OLS and ridge estimates appear to overlap, as well as the lasso and elastic net estimates.

Inference

To reduce visual redudancy, we only present the figures associated with the jackknife-based intervals for OLS and PCR in Figures (ref) and (ref), respectively. Consider the Basque study. The top row of plots in both figures demonstrates that $\mu_0^\text{vt}$ is more accurately estimated than both $\mu_0^\text{hz}$ and $\mu_0^\text{mix}$. Put differently, there is less uncertainty about conducting inference on $\mu_0^\text{vt}$ relative to the other estimands. At the same time, these plots indicate that if $\mu_0^\text{hz}$ or $\mu_0^\text{mix}$ are the estimands of interest, then the VT confidence interval will undercover in both settings. Analogous statements can be made for the remaining subfigures. As with our simulations, the large potential differences in coverage reinforce the importance of properly reasoning through the source of randomness in the data.

Conclusion

This article rewrites the conventional wisdom on HZ and VT regressions for panel data analysis. Contrary to standard notions, we show that the two regressions yield identical point estimates under several standard settings. At the same time, we articulate that the source of randomness directly affects the accuracy of the inference that can be conducted. From a practical standpoint, this stresses that researchers should carefully consider where the randomness in their data stems from as this decision will then guide their choice of estimand and inferential procedure.