Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
33,944 characters · 14 sections · 12 citation commands
A Way to Synthetic Triple Difference
In recent empirical research, the difference-in-differences (DID) and triple-difference (i.e., difference-in-difference-in-differences, or DDD hereinafter) estimators have been widely adopted for estimating causal effects due to their intuitive design and ability to control for unobserved time-invariant heterogeneity across units. However, one key challenge that researchers face is ensuring that the parallel trends assumption holds. Several strategies have been developed to facilitate or approximate parallel trends in DID frameworks, including combining propensity score matching (PSM), entropy balancing, and synthetic control with DID hainmueller2012, arkhangelsky2021. These methods improve the comparability of treatment and control groups by reweighting or constructing synthetic controls that more closely mimic the characteristics of the treated group. However, despite the attention paid to improving parallel trends in DID, there has been little work devoted to addressing this issue within the DDD framework.
In this paper, I aim to initiate a conversation with the literature by proposing an approach that combines synthetic control (SC) with DDD. The SC method, originally designed to improve the counterfactual for DID settings abadie2010synthetic, offers a powerful tool for constructing better control groups for DDD as well. However, its application to DDD settings, where treatment effects may differ across subgroups, has been underexplored. By extending the SC approach to a transformed version of DDD, I show how it can improve the validity of estimates when parallel trends are violated across multiple dimensions (including treatment status, time, and subgroup characteristics) due to the exponential growth of the outcome value.\footnote{In such situation, a synthetic DDD estimator may be needed when taking the logarithm is not a good idea, e.g., if the outcome takes the value of zero for some units.} A synthetic DDD estimator opens an avenue for causal inference when synthetic DID still performs poorly.
The contribution of this paper is two-fold. First, I contribute to the discussion of the SC methodology in DDD settings, showing how transforming DDD into DID can help facilitate parallel trends in more complex empirical applications. Second, I provide an empirical example to provide a practical guide for this approach in a real-world policy evaluation; moreover, in this empirical exercise, I compare the performance of this synthetic DDD method to standard DDD and DID estimators, illustrating the effectiveness and highlighting the potential benefits of this combined approach in improving the robustness of causal inference.
The remainder of this paper is structured as follows. In Section 2, I present the theoretical foundations of the transformed DDD model. Section 3 explains how SC can be adapted for use in a DDD context based on an empirical example, including a brief explanation of the dataset and the policy under study; the results of the empirical analysis are discussed, and the performances of standard DDD, DID, and the proposed synthetic DDD estimators are compared. In Section 4, I lay out additional notes and cautions for readers who intend to apply this procedure. Finally, Section 5 concludes with a discussion of the broader implications for empirical research and policy evaluation.
Let us start with a simple DID model without covariates. I would like to demonstrate how a DID estimator can be obtained by transforming the outcome variable through subtracting the control group's mean outcome at each time period. This approach allows us to estimate treatment effects using simplified regressions without covariates or fixed effects.
Consider the standard DID model with fixed effects:
where $Y_{it}$ is the outcome variable for unit $i$ at time $t$, $\text{Treat}_i$ is the indicator variable equal to 1 if unit $i$ is in the treatment group and 0 otherwise, $\text{Post}_t$ is the indicator variable equal to 1 if time $t$ is in the post-treatment period and 0 otherwise, $\mu_i$ refers to a unit (individual) fixed effect, $\lambda_t$ refers to a time fixed effect, $\alpha_0$ is the common intercept term, $\alpha_1$ is the average treatment effect on the treated (ATT), which is the parameter of interest, and $\epsilon_{it}$ is the error term.
Suppose that Assumptions (ref) and (ref) are satisfied.
Then, the ATT can be written as
As shown in Appendix (ref), this is equivalent to $\beta_1$ in the following model for individuals in the treatment group ($\text{Treat}_i = 1$):
where we define the transformed outcome variable as
and $\mathbb{E}[Y_{it} \mid \text{Treat}_i = 0, t = t] = \bar{Y}_{\text{Control}, t}$ as the expected outcome for the control group at time $t$. In practice, we can estimate $\bar{Y}_{\text{Control}, t}$ using sample averages:
Thus, the sample counterpart of $Z_{it}$ is:
We estimate the following regression for the treated group ($\text{Treat}_i = 1$):
where $\hat{\beta}_1$ is the estimator for the treatment effect $\alpha_1$ and $\hat{u}_{it}$ is the residual term.
Under Assumptions (ref) and (ref), $\hat{\beta}_1$ is a consistent estimator of $\alpha_1$:
where $\hat{\bar{Y}}_{\text{Group}, \text{Time}}$ denotes sample averages for Group (either treatment or control groups) and Time (either pre-treatment or post-treatment years), and “$\xrightarrow{p}$” denotes convergence in probability.
Note that, this method does not involve partialling out variables using regression residuals as in the Frisch-Waugh-Lovell (FWL) theorem. Instead, it simply adjusts the outcome variable by subtracting sample estimates of control group means at each time period. Despite not following the requirements of the FWL theorem, this method aims to control for confounding factors (e.g., time effects, covariates) by adjusting the outcome variable. The validity of the estimator relies on the assumptions stated earlier (e.g., parallel trends) and correct specification of the model.
Let us first consider a standard triple-difference model with fixed effects but without covariates as follows:\footnote{Note that, this is a fully saturated model as in olden2022, since $Treat_i \times G_i$, $Treat_i$, and $G_i$ are absorbed by $\mu_i$, while $Post_t$ is absorbed by $\lambda_i$. One may also consider heterogeneous treatment effect by replacing $\delta$ by $\delta_i$}
where $D_{it} = \text{Treat}_i \times \text{Post}_t$ is the DID interaction term, $G_i$ is the indicator variable equal to 1 if individual $i$ is in the subgroup of interest (such as a policy target) and 0 otherwise, $\delta$ is the DDD effect, and $\gamma_1$ and $\gamma_2$ are coefficients capturing interaction effects.
For individuals in the subgroup of interest ($G_i = 1$), we can define the transformed outcome variable in $\text{Treat}_i = j$ ($j = 0,1$) as
where $\mathbb{E}[Y_{it} \mid G_i = 0, \text{Treat}_i = j, t = t] = \bar{Y}_{\text{Non-target}, j, t}$ is the expected outcome for the subgroup of interest in $\text{Treat}_i = j$ ($j = 0,1$) at time $t$. In practice, we can estimate $\bar{Y}_{\text{Non-target}, j, t}$ using sample averages:
Thus, the sample counterpart of $W_{itj}$ is:
We estimate the following DID regression for individuals with $G_i = 1$:
where $\hat{\theta}_1$ is the estimator for $\delta$. As shown in Appendix (ref), under Assumptions (ref) and (ref), $\hat{\theta}_1$ is a consistent estimator of $\delta$.
With the above derivations, it is not hard to consider the case when there are covariates $X_{it}$. For individuals with $G_i = 1$, we adjust the outcome variable as
In practice, since $X_{it}$ usually involves continuous variables, we may not be able to estimate $\mathbb{E}[Y_{it} \mid G_i = 0, \text{Treat}_i = j, t = t, X_{it} = x]$ using unconditional means in subgroups but regression models fitted on the non-target subgroup ($G_i = 0$). The linear case is given by
Thus, the sample counterpart of the adjusted outcome variable is:
where $x$ is a specific value of $X_{it}$. We then estimate the following regression for individuals with $G_i = 1$:
with $\hat{\theta}_1$ being a consistent estimator for the DDD effect with covariates, under the assumption that the relationship between $X_{it}$ and $Y_{it}$ is correctly specified and stable across groups.
After transforming the DDD estimator to a DID estimator by demeaning, we are ready to discuss combining DDD with SC in practice, when the extended parallel trends assumption with covariates is violated.
As suggested by olden2022, all procedures for DID can be applied to a transformed DDD. This opens an avenue for us to combine SC with triple difference based on arkhangelsky2021. When should we apply this? How exactly can we do so in empirical research and how does it perform? In the following subsections, I answer these questions based on an empirical example.
Suppose that we are interested in studying the impact of the “4+7 Cities” Centralized Drug Procurement pilot program initiated at the end of 2018 (see Appendix (ref) for more details) on medical and drug-related innovation in China, and we have a dataset that consists of the number of patents by the first four digits of International Patent Classification symbol (i.e., four-digit IPC code), patent type, and inventor type in each city every four year from 2001 (i.e., 2001, 2005, 2009, 2013, 2017, and 2021). A natural idea is to estimate the following equation:
where $i$ denotes a “city, four-digit IPC code, patent type, and inventor type” pair (i.e., a track), $t$ denotes a year, and $\text{NumPatents}_{it}$ is the number of patent applications in a specific track and year. $\text{Treat}_i$ takes the value of 1 if a track is in one of the “4+7” pilot cities, while $\text{Post}_t$ takes the value of 1 after year 2018. Since the policy targets the medical and drug-related fields, we may limit our analysis to patents whose IPC symbol starts with “A61”. Based on the data I compiled from China National Intellectual Property Administration (see more details from Appendix (ref)), there are 2,058 observations for A61 tracks and 11,478 observations for non-A61 tracks. I only include observations with a positive number of patent applications to investigate the intensive margin.
However, as shown by (ref), neither A61 nor non-A61 tracks exhibit parallel trends in the outcome variable due to its exponential growth in value in the treated “4+7” cities, and the parallel trends are likely violated not only along the treatment dimension but also along the dimension of patent track. While synthetic DID can help create a more accurate counterfactual when parallel trends are violated along the treatment dimension, it might not capture complexities across different subgroups in this situation. As shown in (ref), pre-treatment trends are not as parallel as we may want based on synthetic DID. Thus, my derivations above open an avenue for synthetic DDD, which ensures that the counterfactuals for both the treatment and the control groups are robust and account for these multi-dimensional violations of parallel trends. There can also be complex interactions between treatment, time, and subgroups that cannot be fully captured by synthetic DID alone. Thus, situations like these motivate us to combine SC with DDD.\footnote{A caution about applying the synthetic DDD procedure to count data is given in Section (ref).}
We can write out the empirical equations for regular DDD and transformed DDD first:
where $\text{Med}_i$ takes the value of 1 if a track is related to A61 and 0 otherwise, and $\text{DiffNumPatents}_{it}$ is the difference in the number of A61 patent applications and the predicted average number of non-A61 patent applications in a specific year. To see how the transformed DDD performs, we compare results of standard and transformed DDD estimations without and with covariates in (ref).
As shown in the table, when we do not include any covariates, the transformed DDD estimation gives exactly the same point estimate of the treatment effect as the standard DDD does, although standard errors are different. They differ due to several factors related to how each model handles residual variance and degrees of freedom. For instance, regular DDD involves a larger number of interaction terms, such as $\text{Treat}_i \times \text{Post}_t \times G_i$, which increases the complexity of the residual variance. This complexity can lead to a different calculation of the error variance, particularly if there is variation in the subgroup interaction terms. The transformed DDD simplifies the model by transforming the outcome into first differences. This transformation can alter the distribution of the residuals, potentially leading to a more efficient estimator with lower standard errors. The simplification reduces the number of parameters to estimate, as fewer interaction terms are involved, which can affect how the standard errors are computed.
Since DID tends to over-reject the null hypothesis by having a smaller standard error bertrand2004, one may use clustered robust standard errors to address this issue. In Stata, the regular DDD can be estimated using command “reghdfe”, while the transformed DDD can be estimated using command “xtreg” or “reghdfe”.
When including covariates, however, point estimates can differ. This can be due to finite sample sizes, especially if the number of observations in certain groups is small. Nevertheless, my empirical exercise shows that strictly following Equations ((ref)) to ((ref)) yields the closest point estimates.
Then, we can follow arkhangelsky2021 to combine the transformed DDD with SC using the Stata command “sdid”. From (ref), we can tell that parallel trends are much more apparent than those in (ref) after the SC weighting. The point etimate suggests a treatment effect of 33.6 more patent applications in A61 track relative to non-A61 track annually. The standard error based on arkhangelsky2021's placebo procedure is only about 5.9, leading to a P-value smaller than 0.001. We may conclude that centralized drug procurement can promote innovation activities in terms of the number of patent applications.
This section presents a more formal description of the synthetic DDD estimator. I describe the estimator, the underlying assumptions, and derive its large sample distribution, emphasizing the use of the transformed outcome variable $W_{itjx}$.
I extend the synthetic DID framework to accommodate the triple-difference setting, capturing heterogeneous treatment effects across subgroups. The estimator relies on constructing a counterfactual outcome for the treated units that uses a weighted combination of control units and pre-treatment periods. The transformation we apply to the outcome is described in Equations ((ref)) and ((ref)). Let $\omega_i$ denote the unit-specific weights for the control group and $\lambda_t$ represent the time-specific weights for pre-treatment periods. The synthetic DDD (SDDD) estimator is then defined as:
where $\hat{\omega}_i^{sdid}$ and $\hat{\lambda}_t^{sdid}$ are chosen based on arkhangelsky2021's procedure.
To ensure that the proposed estimator provides consistent and unbiased estimates of the treatment effect, two important assumptions should be noted:
Following the methodology of arkhangelsky2021, one can derive the large sample distribution of the synthetic DDD estimator. Under the assumption that both \( N \to \infty \) and \( T \to \infty \), the synthetic DDD estimator converges to a normal distribution. Specifically, we have:
where \( \delta \) is the true treatment effect and \( \sigma^2 \) is the asymptotic variance of the estimator.
The variance \( \sigma^2 \) is determined by the residual variance and the structure of the weights. The SC approach minimizes the differences between treated and control units in the pre-treatment period using weights.
In the empirical application above, I have only six time periods, which challenges the large sample assumption. In practice, finite sample sizes can affect the precision of the estimates. To account for this, one may use a block bootstrap procedure to estimate the variance in cases where the number of time periods is limited.
Another challenge the empirical exercise faces is the assumption that the error terms are normally distributed. However, one may argue that even though the number of patent applications is count data, when aggregated over time or across regions, the Central Limit Theorem implies that the distribution of these aggregated counts may approximate normality, making the assumption of normally distributed error terms reasonable in large samples.
Nevertheless, the empirical example provided earlier may only serve as a working illustration of when and how to apply the synthetic DDD strategy. Researchers should always be cautious about the underlying assumptions for an appropriate use of SC in a DDD setting.
One might ask: why don't we create a SDDD estimator by directly combining SC with triple difference in the following form:\footnote{Here I do not include covariates since we can simply incorporate adjustment for them by applying SDDD to the residuals $Y_{it}^{Res} = Y_{it} - X_{it}^{\prime}\hat{\beta}$ of the regression of $Y_{it}$ on $X_{it}$.}
where $\hat{\omega}_i^{sddd}$ and $\hat{\lambda}_t^{sddd}$ should be chosen based on a new procedure.
At first glance, this approach seems intuitive. The SC method could be used to construct weights $\hat{\omega}_i^{sddd}$ and $\hat{\lambda}_t^{sddd}$ for both the cross-sectional units and the time periods, helping to create a more accurate counterfactual for treated units. The interaction terms $(\text{Treat}_i \times \text{Post}_t \times G_i)$, $(\text{Treat}_i \times \text{Post}_t)$, and $(\text{Post}_t \times G_i)$ could account for the heterogeneous treatment effects across subgroups in the DDD setup. However, there are several challenges and considerations that may prevent this approach from being ideal, or even necessary, in practice:
This paper introduces a practical way to address the parallel trends assumption in DDD models by transforming them into a DID framework and then combining them with SC, in order to improve counterfactual estimation when parallel trends are likely violated across multiple dimensions. This approach is especially relevant in evaluating complex policies such as China’s “4+7 Cities” Centralized Drug Procurement pilot program, where treatment effects may differ significantly between subgroups.
The introduction of SC in DDD settings addresses a key gap in the literature. While balancing methods have been widely applied in DID contexts to account for violations of parallel trends, their application to DDD has been largely unexplored. For empirical researchers, this method expands the use of SC to capture subgroup-specific variations in treatment effects, which are often missed by traditional DDD estimators, and allows for more precise estimation of treatment effects in complex empirical settings where interactions between subgroups and treatment are likely to influence the outcome. By offering a more flexible and robust framework, this approach encourages researchers to move beyond the standard DDD assumptions when subgroup-specific violations of parallel trends are suspected.
From a policy evaluation perspective, the application of synthetic DDD to the “4+7” program highlights a more credible consequence on high-quality development. The program, designed to reduce drug prices and promote the use of generics, affects different cities and industries in varying ways. Standard DDD may overstate its effect on innovation, particularly when pre-trends differ across subgroups. Synthetic DDD, by creating more accurate counterfactuals, offers a clearer understanding of how the program has impacted pharmaceutical innovation and competition. As shown by the figures, the effect becomes more modest if we account for the differential pre-trends.
While this paper demonstrates the efficacy of combining SC with DDD, there are opportunities for future research. Further work could explore its application in other policy contexts or investigate alternative weighting methods to enhance the performance of SC in DDD settings with complex data structures. This combination of SC and DDD offers a powerful tool for empirical research, expanding the scope of more reliable policy evaluation.