Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
110,574 characters · 26 sections · 63 citation commands
Difference-in-Differences with Unpoolable Data
Difference-in-differences (DID) is commonly used to estimate treatment effects but is infeasible in settings where data are unpoolable due to privacy concerns or legal restrictions on data sharing, particularly across jurisdictions. Unpoolable datasets are datasets which are stored on a secure server (sometimes called silos), and researchers have signed an agreement not to export or import data. In this paper, we identify and relax the assumption of data poolability in DID estimation. We propose an innovative approach to estimate DID with unpoolable data (UN-DID) which can accommodate covariates, multiple groups, and staggered adoption. We show that the conventional and the UN-DID estimators can provide equivalent estimates of the ATT and standard errors without covariates with a simulated poolable dataset. We also show that the UN-DID can recover unbiased estimates of the ATT whereas the conventional estimator can be biased when the covariates' effect is not homogeneous across silos. Two empirical examples with real-world data further underscore UN-DID’s utility. The UN-DID method allows the estimation of cross-jurisdictional treatment effects with unpoolable data, enabling new research questions to be answered.
Difference-in-differences (DID) is a widely used method to evaluate the impacts of policies and interventions. It is used to estimate both intended treatment effects and unintended consequences. The source of identifying variation in DID analyses is often a policy change that occurs in some political jurisdictions (e.g., states, countries) but not in others. Treated observations come from jurisdictions or units in which the intervention occurs, while control or untreated observations come from jurisdictions that do not experience that change (at least not at the same time). DID can be used to estimate both the intent-to-treat (ITT) effect and the average treatment effect on the treated (ATT). In the absence of randomized treatment assignment, a widely documented set of assumptions support the unbiased estimation of these effects using conventional DID methods such as the Two-way Fixed Effects (TWFE). These assumptions include parallel pre-intervention trends, no anticipation and no staggered adoption, and new DID estimators have been developed in settings where these assumptions do not hold roth2022s, abadie2005semiparametric, de2020twott, callaway2021difference, de2020twoseveral. Since treatment assignment occurs at the jurisdictional level, using conventional DiD estimation relies on an additional assumption that has not yet been made explicit: that data from the treatment group and the control group can be combined, or pooled, for analyses.
This paper develops the DID for unpoolable data (UN-DID) method, enabling the estimation of treatment effects using DID when the poolable data assumption is violated. This occurs when data come from the same jurisdictional level as the policy variation, and may not be combined on the same computer or server, usually due to privacy concerns or legal restrictions on data sharing. Usually researchers accessing data stored in secured servers, often called silos, sign an agreement not to import data to or export data from these servers. The data are therefore siloed at the level of treatment, and traditional DID methods cannot be used to estimate the ATT. Examples include health care data from different insurers (e.g., Canadian provinces HDRN, private and public insurers, all-payer claims databases across states aspe2023), restricted-use micro-data held in secure environments (e.g., vital statistics, income tax, business data held in Statistics Canada's or the US Census Bureau's Research Data Centres CanRDC, USRDC), and electronic health records federated across countries (e.g., the European Health Data Space EHDS). Sometimes data are considered unpoolable because the datasets are not sufficiently harmonized. We do not address that issue here - UN-DID requires that data that are physically unpoolable are sufficiently harmonized (either de facto or after cross-jurisdictional collaboration). That said, UN-DID is able to accommodate datasets from different jurisdictions that measure the same underlying concepts, but code or categorize variables slightly differently (e.g., income in different bins). UN-DID enables the estimation of treatment effects when policies and interventions vary at the level of data silos (county, state, country, etc.), opening opportunities to estimate less-biased treatment effects using better controls and to answer new research questions.
When the ability to pool data across treatment and control jurisdictions is limited, the choice of appropriate counterfactuals is necessarily constrained. Researchers may resort to estimating first-differences Fischer2021 or using interrupted time series Tamblyn2001, bernal2017interrupted, Farin2024, which only provide unbiased treatment effect estimates if no other changes coincide with the intervention of interest. More convincing treatment effect estimates are obtained leveraging within-jurisdiction controls Rudoler2023, Strumpf2017, LiHurley2014, GIRMA2007 or individual fixed effects Dumont2008, both of which may still suffer from selection on unobservables. The impacts of national policies that take effect everywhere at the same time can sometimes be identified using pre-policy variation Finkelstein2007, Duflo2001, but in other cases, including nationwide cannabis legalization in Canada in 2018, rigorous policy evaluations using external controls remain extremely limited or non-existent.
If policy designs permit the use of Regression Discontinuity (RD) or Instrumental Variables (IVs) to estimate local average treatment effects almond2010, Hutcheon2020, the identifying variation may be convincing, but the generalizability of the estimates are narrow by definition. Meta-analysis can be used with unpoolable data, and is an effective way to leverage sample size by combining silo-specific treatment effect estimates into a single parameter xiong2023federated. For example, the Canadian Network for Observational Drug Effect Studies (CNODES) routinely employs meta-analytical techniques to more comprehensively examine drug safety using siloed provincial or national data Suissa2012, Filion2014, Durand2021. However, meta-analysis cannot be used to estimate treatment effects of an intervention in one jurisdiction using another unexposed jurisdiction as a control. UN-DID will allow researchers to estimate treatment effects using better counterfactuals, leveraging data from non-treated jurisdictions and limiting selection bias while maintaining generalizability.
Data poolability limitations can also constrain which research questions get asked, specifically which outcomes are examined in evaluations of policy interventions. National and international surveys include multiple states/provinces or countries, so DID models are easily estimated with treated and control units in the same dataset Baker2008, Charters2013, Dube2019. The range of outcomes available in a survey, however, is necessarily limited with respect to content, detail, and method of reporting (self-report vs. administrative data). In the absence of a method like UN-DID to allow treatment effect estimation across separate datasets, the questions that have been asked and answered have been a function of the (pooled) datasets that are available. This constraint has become even more problematic in recent years, precisely because large, detailed, and high-quality data are increasingly available for research. All-payer health insurance claims, electronic medical records, detailed individual- and firm-level tax filings, genetic data, and immigration records offer comprehensiveness, accuracy and detail that allow researchers to ask and answer new questions. However, these data are generally accessible only in secure research data centers and are not shareable or poolable across jurisdictions. By enabling treatment effect estimation using siloed data, UN-DID will open up opportunities to answer novel research questions and exploit new sources of identifying variation using separate datasets across jurisdictions.
The UN-DID method allows the estimation of treatment effects with unpoolable data. This has important implications for research across the fields of applied economics, and across a range of disciplines including public policy, epidemiology, health research, environmental science, political science, sociology, and others. With UN-DID, new research questions can be answered and better counterfactuals can be used to reduce bias when estimating treatment effects. While researchers with treatment and control observations in separate datasets could already calculate the double-difference “by hand", our formal results justify this approach and extend beyond the 2$\times$2 case, accomodating staggered adoption, more than two groups, and covariates. Furthermore, UN-DID enables the estimation of appropriate standard errors and therefore proper inference. We have developed software packages in R, STATA and Julia, which are described in Section (ref). In a separate publication, we will guide users of the packages through an empirical example using real-world siloed datasets. These will enable UN-DID to be implemented across secure data silos, even when local data analysts are not skilled econometricians.
The intuition behind UN-DID is based on non-parametric DID estimation where conditional mean outcomes are estimated for each of four treatment$\times$timing groups: pre-intervention treatment, pre-intervention control, post-intervention treatment and post-intervention control. The post-pre first difference can then be calculated for each treatment and control group, and then the DID estimate is simply the difference between those first differences. UN-DID estimates first-differences and standard errors within jurisdictional silos using a regression based tool and provides a method to combine these estimates into a treatment effect estimate and to test hypotheses. We provide formal results which show that this method accurately estimates the ATT. UN-DID also estimates the parameters necessary to evaluate parallel pre-intervention trends.
In this paper, we begin by reviewing the DID identifying assumptions and formally introduce the poolability assumption, that data can be combined across treatment and control groups. We propose a method for estimating the ATT when the poolability assumption is violated, which we call UN-DID (DID for unpoolable data). Because researchers commonly rely on regression-based tools for estimating treatment effects, we introduce the UN-DID method as a regression-based tool for this purpose. Through analytical proofs, we first demonstrate the equivalence of estimates between conventional DID and UN-DID without covariates. We then relax the strong parallel trends to the conditional parallel trends assumption and add the assumption of common causal covariates to extend the analysis using covariates. This assumption was introduced to the literature in karim2024good and assumes that each covariate has a common coefficient across silos. In this setting, we show the equivalence of conventional DID and UN-DID with time-invariant covariates. In contrast, the conventional and the UN-DID estimators converge to two distinct population parameters when time varying covariates are used. In addition, we show that the UN-DID is unbiased, while the conventional estimator can be biased when the CCC assumption is violated. We also turn to the many groups and many periods (G$\times$T) setting and relax the no staggered adoption assumption to demonstrate how the UN-DID method works in this setting, in the spirit of callaway2021difference. Cluster robust inference is a challenge, but possible, as described in Section (ref) and a companion project \citep*{karim2025slides}.
With analytical results in hand, we conduct Monte Carlo experiments with simulated panel data designed to mimic populations in Canada's two largest provinces over 10 years (2$\times$10). We consider six data generating processes (DGPs): either with or without treatment effects and with no covariates, with a time-invariant covariate, and with a time-varying covariate. We consider equal and unequal sample sizes between the two provinces, and vary the sample size from 500 to 50,000 observations. To assess unbiasedness, we plot kernel densities of UN-DID and conventional DID estimates of the ATTs against the known ATT value from the DGP. To assess the asymptotic performance of UN-DID and the conventional DID hetero-robust standard errors, we compare estimates with the true values of the standard errors against to the known true values as the sample size increases. Specifically, we calculate the mean squared error between SEs for each pair: UN-DID vs. truth, conventional DID vs. truth, and UN-DID vs. conventional DID. Finally, to assess UN-DID's performance and to highlight its utility, we include two empirical examples: one with common treatment timing and one with staggered adoption. Here, real poolable data are treated as if they were unpoolable across treated and control groups, allowing us to compare the estimated ATTs and SEs from UN-DID and conventional DID with real-world data.
In the simulation study, we demonstrate that UN-DID and conventional DID are both unbiased without covariates. With a time-invariant covariate, the two ATTs are unbiased; the SEs are equivalent but not numerically equal and converge to the true value. With a time-varying covariate, the UN-DID and conventional DID ATTs are unbiased but are not numerically equivalent as they converge to two different population parameters. When the state-varying CCC assumption is violated, we demonstrate that the UN-DID is unbiased and the conventional DID is biased. The above results hold with both equal and unequal sample sizes across silos. These results hold in our two empirical examples using real-world data. Without covariates, UN-DID's ATT and SE estimates are exactly the same as conventional DID. With covariates, the estimates differ slightly but the statistical inference and substantive conclusions remain the same. As the number of covariates and/or data silos increases, the more flexible UN-DID model may differ more from conventional DID which constrains covariates to have the same slopes across silos. These results reinforce the theoretical and simulation study findings, that the UN-DID procedure estimates either identical or very similar treatment effects compared to the conventional method.
This paper contributes to the dynamic literature on DID methods. In the past five years, an important series of papers have looked under the hood of the conventional DID estimator, exploring what is actually being estimated and what can go wrong when each of the identifying assumptions is violated roth2022s, abadie2005semiparametric, de2020twott, callaway2021difference, de2020twoseveral, goodman2021difference. They have developed methods to identify the ATT in each case, changing the way that DID analyses are conducted. We highlight the heretofore implicit assumption of data poolabilty and provide a solution to estimate the ATT when this assumption does not hold. We assess UN-DID's performance and demonstrate its equivalence to conventional DID.
We evidently aim to contribute to the large literature that uses DID to estimate treatment effects of policy changes and other interventions, both in economics and across a wide range of disciplines. In this paper, we apply UN-DID to two published policy evaluations that used conventional DID methods sabia2017effect, conley_2011. Our primary goal is to compare the estimates from the two methods, with a secondary goal of highlighting how UN-DID could be used in empirical analyses. We have produced UN-DID software packages in several languages, which are explained in detail in Section (ref). A forthcoming user guide will use UN-DID to evaluate a policy change using real-world siloed data, where a conventional DID analysis is impossible and no ATT estimates exist for comparison.
Throughout the paper, longer proofs are relegated to appendices. The paper proceeds as follows: Section (ref) presents the theoretical framework and the identifying assumptions underlying DID in the canonical 2x2 setting. It also formally introduces the poolability and the no entry assumptions. Section (ref) introduces the UN-DID in detail, and how it is utilized in the canonical 2x2 setting. Section (ref) discusses why methods such as IPW, RA and DR-DID are infeasible with unpoolable datasets. Section (ref) highlights two key advantages of the UN-DID over the conventional estimator. Section (ref) extends UN-DID to settings with multiple silos and/or staggered adoption and Section (ref) further extends UN-DID in the staggered adoption framework where there are multiple silos per treatment time. Section (ref) describes the basic Monte Carlo design, and Section (ref) presents simulation results for the cases with and without time-fixed and time-varying covariates. Section (ref) compares UN-DID and conventional DID ATT estimates for two empirical examples using real-world data and Section (ref) concludes.
In this section, we introduce the core assumptions of DID, most of which also apply to UN-DID. Suppose an intervention was introduced at different times across different jurisdictions (e.g., states, provinces). Individual-level data are held by jurisdictions and are not poolable across them - in other words, the data cannot be easily combined for analysis. There are several possible variations of this common scenario: for example, implementation timing may be common or staggered, there may be multiple treatment and control groups (e.g., several states may have enforced a policy, while several others did not), and so on. UN-DID offers a novel approach to handle these analytical contexts.
We begin by describing the canonical case (two groups and two periods, with common adoption and covariates) before expanding to staggered adoption scenarios in Section (ref). The basic idea of DID estimation is to compare the difference in outcomes before and after the treatment between groups that received the treatment and groups that did not bertrand2004much. The change in outcomes for the control group is used as the unobserved counterfactual of the treated group. card1993minimum first used the conventional DID to estimate the effect of an increase in minimum wage on unemployment in the American state of Pennsylvania using changes in outcomes over the same period in New Jersey as the control. The conventional DID estimate of the ATT uses the following regression:
Here, $D^{Pooled}_{s}$ is a dummy variable that takes on a value of 1 if group $s$ is in the treatment group, and 0 otherwise. $P^{Pooled}_{t}$ is a dummy variable that takes on a value of 1 if the observation is in the post-intervention period, and 0 otherwise. $X^{Pooled}_{i,s,t}$ are the covariates that researchers include to improve the plausibility of parallel trends. The superscript $Pooled$ implies that we are combining the dataset from both the treated group ($Treat$) and the control group ($Control$) into a common dataset ($Pooled = Treat + Control$). Under very strong assumptions, $\hat{\beta_3}$ (the coefficient of the interaction term between $D^{Pooled}_{s}$ and $P^{Pooled}_{t}$) identifies the ATT roth2022s. To simplify notations, we will not use the $Pooled$ superscripts to denote a combined dataset for the rest of the paper.
In Equation (ref), $Y_{i,s,t}(0)$ is the untreated potential outcome for the relevant group and period. Assumption (ref) ensures that the selection bias is 0. A plausibility test for parallel trends is to analyze the evolution of outcomes conditional on covariates for both treated and control groups in the pre-intervention period and check if the trends in outcomes are parallel.
Under Assumption (ref), the treated potential outcome is equal to the untreated potential outcome for all units in the treated group in the pre-intervention period abadie2005semiparametric, de2020twott. Here, $t'$ is the period which the treated group is first treated. Violation of no anticipation can also lead to deviations in parallel trends before treatment. Here, $Y_{i,s,t}(t)$ is the treated potential outcome of individual $i$ in state $s$ at calendar year $t$. Under Assumptions (ref) and (ref), the conditional ATT for a given value of $X_{i,s,t}$ is shown in Equation (ref). Refer to callaway2021difference for a simple proof.
In Equation (ref), $E[Y_{i,s,t}|D_s = 1, P_t = 1,X_{i,s,t} = x]$ is the expected outcome for the population in the treated group in the post-intervention period with a covariate value $x$ and $E[Y_{i,s,t}|D_s = 1, P_t = 0,X_{i,s,t} = x]$ is the expected outcome for the population in the treated group in the pre-intervention period with a covariate value $x$. Similarly, $E[Y_{i,s,t}|D_i = 0, P_t = 1,X_{i,s,t} = x]$ is the expected outcome for the population in the control group in the post-intervention period with a covariate value $x$ and $E[Y_{i,s,t}|D_s = 1, P_t = 0,X_{i,s,t} = x]$ is the expected outcome for the population in the control group in the pre-intervention period with a covariate value $x$. It is important to distinguish the unconditional ATT from the conditional ATT shown below, which is just the ATT for a single realization of $X_{i,s,t} = x$.
Assumption (ref) implies treatment occurs only once callaway2021difference,de2020twott. For now, we introduce Assumption (ref) to develop an intuitive understanding of the “simple” two group, two period building blocks of the UN-DID with staggered adoption. This assumption will be relaxed later when we explore the more complex setup where multiple groups adopt treatment at different times.
The papers cited above have contributed to a rich recent literature investigating what occurs when each of the aforementioned assumptions are violated. In this paper, we introduce a new assumption that has been implied in previous literature but has not been explicitly explored or discussed in the context of DID.
Here, $Y^{Treat}$ and $Y^{Control}$ are two matrices containing the outcome variable for group $Treat$ and $Control$, respectively, and $N_{Treat}$ and $N_{Control}$ the total number of observations in the matrices, respectively. $X^{Treat}$ and $X^{Control}$ are the matrices of covariates for the treated and the control groups, respectively, which can also be stacked together into a common matrix $X$.
\[ Y^{Treat} =
, Y^{Control} =
, \]
\[ X^{Treat} =
, X^{Control} =
\]
Under Assumption (ref), the two matrices can be stacked together into a common matrix $Y^{pooled}$, which is not feasible when there are legal restrictions preventing them from being stacked together (data are siloed or unpoolable). For settings where Assumption (ref) is violated and the data are not poolable, we introduce the UN-DID method to estimate the ATT. In such cases, when we access the data from the treated silo, we have no knowledge of the data from the control silo. Similarly, once we have access to the data from the untreated silo, we have no information about the data from the treated silo. The conventional DID method becomes impractical if siloed datasets cannot be combined for regression analysis.
Assumption (ref) implies that when data are unpoolable, researchers are unable to exchange any information between silos, including coefficients and standard errors from regressions conducted in a different silo. The extracted coefficients and standard errors can only be combined in a server outside the respective data silos.
In this section we introduce the UN-DID estimator in the simplest setting, where there are only two silos and two time periods. The estimator is extended to multiple time periods in Section (ref) and multiple silos per time period in Section (ref).
When Assumptions (ref) and (ref) are violated, we can visit each silo (physically or virtually) and run regressions shown in equations (ref) and (ref) for the treated and untreated silos, respectively. In the simplest case, we assume that there are only two silos: the treated silo ($Treat$) and the untreated silo ($Control$).
Here, $Y^{Treat}_{i,t}$ is the outcome for an individual from the silo that is treated at time $t$ and $Y^{Control}_{i,t}$ is the outcome of an individual from the silo that is untreated at time $t$. $post_t^{Treat}$ is a dummy variable that takes on a value of 1 when the treated observation is in the post-intervention period, and 0 otherwise. Similarly, $post_t^{Control}$ is a dummy variable that takes on a value of 1 when the untreated observation is in the post-intervention period, and 0 otherwise. $pre_t^{Treat}$ is a dummy variable that takes on a value of 1 if the treated observation is in the pre-intervention period, hence $pre_t^{Treat} = 1 - post_t^{Treat}$. Similarly, $pre_t^{Control}$ is a dummy variable that takes on a value of 1 if the untreated observation is in the pre-intervention period, hence $pre_t^{Control} = 1 - post_t^{Control}$. $X^{Treat}_{i,t}$ and $X^{Control}_{i,t}$ are the covariates for the treated and the control groups respectively. Note that the regressions in equations (ref) and (ref) do not include a constant. Therefore, none of the variables are dropped because of multicollinearity.
These regressions are silo-specific. Because data are siloed, the treated regression does not contain any data from the untreated silo and the untreated regression does not contain any data from the treated silo. Here, $(\widehat{\lambda}_2^{Treat} - \widehat{\lambda}_1^{Treat}) - (\widehat{\lambda}_2^{Control} - \widehat{\lambda}_1^{Control})$ is the estimate of the ATT, as shown in Equation (ref). The associated hetero-robust standard error of the ATT is estimated as the square root of the sum of the variances used to estimate the ATT estimated from the above UN-DID regressions minus twice the covariances between the relevant coefficients. This is shown in Equation (ref).
In this paper, we assume that the error terms are independent. Accordingly, we can estimate standard errors that are robust to heteroskedasticity but not robust to clustering. It is straightforward to extend the procedures that we discuss to clustering at the sub-silo level. For instance, one could cluster at the person-level when using panel data, or at the city level if the silos are state level. However, researchers will typically cluster at the level of the policy change bertrand2004much, MACKINNON2023, which we have assumed is by silo. Silo level clustering is challenging when models like those in equations (ref) and (ref) are also estimated at the silo level. Conventional methods of cluster-robust inference will not work with silo-specific data because there is only one cluster in each dataset. With only one cluster, the variance matrix is rank deficient and cannot be computed. In a companion project karim2025slides, we explore two methods for cluster robust inference. The first is a cluster jackknife, similar to MNW-bootknife, hansen2025standard. The second is a Randomization Inference procedure similar to mackinnon2020randomization. However, the finite sample properties of cluster-robust inference under silo level clustering are not addressed in this paper. The UNDID software packages, see Section (ref), produce cluster robust standard errors and p-values using these two methods.
One interesting feature of siloed data is that the information may be collected and coded differently according to the processes in place in each jurisdiction. Each silo is primarily interested in analyzing the data in isolation for their own purposes, not in comparisons with other jurisdictions. It is therefore likely that not all variables will be standardized or harmonized across silos. UN-DID is designed for settings where data are unpoolable in the sense that they cannot be combined on one server or in one regression, but where the information they capture is similar enough that fundamental issues of data harmonization are not a problem. It is convenient, therefore, that UN-DID can accomodate settings where data are sufficiently harmonized so that variables capture the same concepts across silos, even if they are coded somewhat differently.
Consider, for instance, a variable like family income. Researchers must confirm that this variable is comparable across silos. For example, they should check whether this variable includes the same types of income (wages, investment earnings, government transfers, etc.) and is continuous and not-censored in both silos. However, it is plausible that both silos record family income as categorical: one using bins in increments of \$10,000, while the other uses increments of \$25,000. This would cause issues for conventional DID, as the categories are non-overlapping across silos without a problematic degree of aggregation. With UN-DID, the silo-specific coefficients can handle the different categories easily, each first-difference can be estimated conditional on the covariate, and the conditional ATT and SEs can also be estimated. While data must be sufficiently harmonized to capture the same underlying concepts and the effects of covariates must be homogeneous across silos (formally stated in section (ref) as the common causal covariates assumption), the UN-DID can more flexibly accommodate differences in datasets across data silos than conventional DID.
In this section, we will discuss three alternatives to the conventional regression which are used to estimate the ATT with covariates: Inverse Probability Weighting (IPW), Regression Adjustment (RA) and the Double Robust Difference-in-differences (DR-DID) estimators. In particular, we show that these methods are infeasible when Assumption (ref) and (ref) are violated. The IPW method is a semi-parametric approach, which involves re-weighting the first difference of outcomes for the control group which are more similar to the treated group based on observable characteristics abadie2005semiparametric. In other words, the units in the control group are re-weighted based on the propensity scores, which can be estimated using any econometric tool of our choice. However, the IPW method relies on Assumption (ref) in order to estimate the propensity scores. As a result, this is not feasible when Assumption (ref) is violated. The DR-DID, which is a combination of both the IPW and the OR method sant2020doubly, is also infeasible when Assumption (ref) is violated.
Now, let us explore the RA with covariates in more details. In RA with the canonical two-group and two-period setup with repeated cross-sectional data, the dataset is sub-divided into four parts based on treatment status and treatment timing heckman1997matching. For each of the sub-groups, we regress the outcome variable on the covariates, and store the fitted values from these regressions for the treated group. In other words, we obtain four fitted values $\widehat{Y_{i,t}}$ from the following regressions:
The fitted values can be estimated using any econometric model of our choice. However, to keep things simple, we use the linear specification shown in Equation (ref). Since the above regressions run separate regressions using each sub-division, RA can provide unbiased estimates of the ATT under violations of the state-invariant CCC assumption formally stated in Section (ref). The next step involves estimating the ATT using the following formula:
In the above equation, $\widehat{Y^{j,p}_{i,t}}$ are the fitted values for group $j$ in period $p$. When Assumption (ref) is violated, it may seem feasible to transfer $\varrho^{j,p}_0$ and $\varrho^{j,p}_1$ from the control silo to the treated silo, and calculate the fitted values $\widehat{Y^{{Control},post}_{i,t}}$ and $\widehat{Y^{{Control},pre}_{i,t}}$ for each treated unit. However, this is not allowed, as under Assumption ((ref)) information cannot be brought into silos.
When researchers have access to a balanced panel, RA no longer involves estimating the four fitted values shown in Equation (ref). Instead, we use the individual level data for the first difference in outcomes for the treated group. We only estimate $\widehat{Y^{{Control},post}_{i,t}}$ and $\widehat{Y^{{Control},pre}_{i,t}}$ for each treated unit using the coefficients from the control silo, and then estimate the ATT using the following formula shown in Equation (ref). However, this approach is also not feasible with unpoolable data, as information cannot be transferred into the treated silo from the control silo (Assumption (ref)).
In this section, we highlight two key advantages of UN-DID over the conventional estimator. The first advantage of UN-DID is that it provides researchers with a simple yet innovative econometric tool to circumvent the problem of “unpoolable" data and conduct difference-in-differences analysis. The second advantage is that it allows for a greater variety of covariates, specially time-varying covariates. We then present theoretical results of the UN-DID estimator relative to the conventional estimator. We begin with the simplest case with no covariates and with poolable data. Here, running separate first difference regressions for the treated and control groups and taking the difference of the two first differences can easily recover an estimate of the ATT which is numerically equivalent the estimate from a conventional DID regression.
We begin by analyzing the simplest case with no covariates. Theorem (ref) proves the equivalence of the conventional and UN-DID estimators without covariates. A proof of Theorem (ref) is shown in the Appendix (ref).
The above proof relies on a stronger version of Assumption (ref), called the strong parallel trends assumption, introduced in Equation (ref). The proof of Theorem (ref) also requires the data to be poolable between groups. It is important to note that, the UN-DID can be run on both poolable and unpoolable datasets, allowing for a comparison of results between the conventional and UN-DID estimators. However, the conventional regression is no longer feasible when the dataset is unpoolable.
In the more realistic case, researchers typically want to incorporate covariates in conventional DID to ensure the plausibility of parallel trends given covariates, particularly when strong parallel trends do not hold heckman1997matching, abadie2005semiparametric. Likewise, we are including only covariates that are required for conditional parallel trends to be plausible. For valid estimation of the ATT with covariates, we require that the covariates are correlated with the outcome but not with the treatment on the basis of the conditional independence assumption.
The second main advantage of the UN-DID estimator is that it allows researchers to include a greater variety of covariates compared to the conventional DID estimator. The existing DID literature emphasizes on the importance of the thoughtful selection of covariates in DiD analyses. In particular, it suggests using covariates that are time-invariant abadie2005semiparametric, callaway2021difference, caetano2022timevarying, roth2022s, sant2020doubly. When dealing with time-varying covariates, researchers frequently select a specific value of these covariates during a pre-treatment period, as if it were a time-invariant covariate caetano2022timevarying. The UN-DID enables researchers to use time-varying covariates with differing effects on outcomes, which is discussed in more details in the following subsection. In particular, we will show that the UN-DID can provide unbiased estimates of the ATT when time varying covariate with differing effects on outcome are used, while the conventional estimator is biased.
Time-varying covariates are known to complicate DID analysis caetano2022timevarying. Additionally, karim2024good demonstrates that the conventional TWFE estimator can be biased when the common causal covariates (CCC) assumption is violated. The CCC assumption has been implied in previous DiD literature, but has not been explicitly addressed. Their paper introduces three types of CCC assumptions, but this study focuses on the state-invariant CCC assumption, as stated below.
One important distinction between the conventional DID regression using poolable data in Equation (ref) and the UN-DID regressions using siloed data in Equations (ref) and (ref) is how the covariates are handled. In the conventional DID regression there is a single coefficient, $\beta_4$, whereas in the UN-DID regressions there are two $\lambda_3^{Treat}$ and $\lambda_3^{Control}$. Whether or not this matters depends on the relationship between the covariate(s) and the outcome variable and if that varies across silos. When Assumption (ref) is violated, the conventional DID estimator and other estimators such as callaway2021difference can be biased karim2024good. Conversely, the UN-DID estimator, is unbiased in these settings. So is the closely related estimator developed for pooled data in karim2024good. This is called the state-varying Intersection Difference-in-Differences (DID-INT).
In this sub-section, we show that the estimate of the ATT from the conventional and the UN-DID are not numerically equivalent with time varying covariates. For now, we do not impose Assumption (ref) on the covariates. To prove equivalence, it is sufficient to show the equivalence of the conditional ATTs shown in equation (ref) between the two methods with a common poolable dataset.
To prove Theorem (ref), we first establish and prove Lemma (ref) and Lemma (ref). The key difference between Lemma (ref) and Lemma (ref) lies in how the four expectations are conditioned. In Lemma (ref), the four expectations are conditional on the stacked matrix X, which contains the covariates for both the treated and the control silo. This follows from the structure of the conventional regression, where there is a single coefficient of X, $\beta_4$. If Assumption (ref) does not hold, $\beta_4$ is a weighted average of the true coefficients of Xs across all silos. However, in Lemma (ref), the expectations of the treated group are conditioned on $X^{Treat}$, which is a matrix of covariates for the treated silo. Similarly, the expectations of the control group are conditioned on $X^{Control}$, which is a matrix of covariates for the control silo. This follows from the structure of the UN-DID regressions, where separate first difference regressions are run for each silo.
A proof of Lemma (ref) is found in Appendix (ref).
Here, the unconditional ATT, $ATT^{UNDID}$, is defined as:
Here, the conditional ATTs, $ATT^{UNDID}(x)$, is defined as:
The main difference between the two conditional estimands shown in Equations (ref) and Equation (ref) lies in how they incorporate covariates. The estimand for the conventional regression is derived using the full set of covariates across both the treated and control groups. In contrast, the UN-DID estimates two separate regressions for the treated and control groups. Therefore, the conditional expectations for each group are conditioned only on the covariates for its own group. A proof of this Lemma is found in Appendix (ref).
The above problem may be circumvented by interacting the covariates with an indicator for each silo in the conventional regression shown in Equation (ref). This approach allows researchers to capture the differing effect of the covariates between silos, and mitigate the bias which arises from CCC violations as highlighted in karim2024good. A recent alternative is the FLEX estimator proposed in deb2024flexible, however, this only considers treatment cohort group level variation in coefficients rather than silo level variation. However, researchers have typically not done this in practice.
karim2024good show that the Two-Way Fixed Effects (TWFE) DID estimator is biased when any of the three types of CCC assumptions stated in their paper are violated. They also demonstrate that the three types of CCC violations are plausible in real datasets. In light of this issue, karim2024good propose a new estimator called the Intersection Difference-in-differences (DID-INT) which can provide an unbiased estimate of the ATT under the three types of CCC violations. In a simple two group and two time period setup, the TWFE estimator is equivalent to the the conventional regression shown in Equation (ref), and is therefore biased.
In this paper, we show that the UN-DID estimator is equivalent to the state-varying DID-INT without staggered adoption, which is shown in Equation (ref). The state-varying DID-INT is DID-INT variant designed to handle violations of Assumption (ref). It is important to note that, the UN-DID is not able to capture violations of the time-invariant CCC violations. However, a modified version of UN-DID could handle two-way violations.
In the above regression, $pre_t^{Treat} = pre_t * D_i$; $post_t^{Treat} = post_t * D_s$; $pre_t^{Control} = pre_t * (1 - D_s)$; $post_t^{Control} * (1 - D_s)$; $X^{Treat}_{i,t} = X_{i,t} * D_s$ and $X^{Control}_{i,t} = X_{i,t} *(1 - D_s)$. In other words, $pre_t$, $post_t$ and $X_{i,t}$ are interacted with each silo. In the above regression, $\left[ (\hat{\psi_2} - \hat{\psi_1}) - (\hat{\psi_4} - \hat{\psi_3})\right]$ is the estimate of the ATT and is unbiased karim2024good.
A proof of Theorem (ref) can be found in Appendix (ref). Theorem (ref) imples that the UN-DID regression yields the same estimate of the ATT as the DID-INT regression in the absence of staggered adoption. Therefore, the UN-DID regression can provide an unbiased estimate of the ATT even under violations of Assumption (ref). In contrast, the estimate of the ATT from the conventional regression is biased.
To summarize, this section demonstrates two key advantages of the UN-DID over the conventional estimator. First, we show that the UN-DID method can estimate the ATT with unpoolable data, while the conventional DID regression cannot. Second, we show that UN-DID allows time-varying covariates with differing slopes accross silos to be used. We also establish the equivalence of the UN-DID and the conventional estimators in settings without covariates, or with time-invariant covariates. However, with time varying covariates, the two estimators converge in probability to two different ATT estimands, but remain unbiased. In addition, we have introduced the state-invariant CCC assumption, and showed that the conventional estimator becomes biased when the assumption is violated and time-varying covariates are used. In contrast, the UN-DID remains unbiased.
UN-DID is easily extended to staggered adoption contexts. Suppose we have data for multiple silos, labeled as $s = 1, 2, ..., S$. The silos which received the intervention or treatment are called the treated groups, and those that did not are called the control group. For expositional clarity, consider that only one silo is included in each timing group, and only one silo is a control. We will relax this requirement in Section (ref). We also assume data for several calendar years, $t = 1,2,.....,T$. In this setting, different silos adopt treatment at different points in time, creating a staggered adoption framework. The period before a group $s$ is treated is called the pre-intervention period. We impose an additional restriction called absorptive state, which implies that, once a unit is treated, it remains treated for the rest of the period of the study. Several papers in the existing DID literature, such as callaway2021difference, de2020twott and borusyak2017revisiting, have already proposed methods to estimate the ATT when data are poolable.
Similar to callaway2021difference, UN-DID with staggered adoption decomposes the dataset into several $2\times2$ group--time blocks, which are the building blocks for estimating the ATT under staggered treatment timing. Following callaway2021difference, we group together each individual into $s \in S$ cohorts based on the treatment start-time dummies. Let $t^s$ be the year group $s$ is first treated. For a particular group $s$, any period where $t<t^s$ are called pre-treatment periods. However, following callaway2021difference, we define the pre-treatment period for group $s$ as the year before the group is treated ($t^s-1$). Each of these group--time blocks contains a group that is currently treated and a group that has not yet been treated. For each $2\times2$ group--time block, we will run the following two regressions:
In the above regressions, $Treat$ is an index for the treated group and $Control$ is an index for the control group. $X^j_{i,t}$'s are the covariates researchers may be interested in controlling for, where $j = \{Treat,Control\}$. $\nu^j_{i,t}$ are the identically and independently distributed (i.i.d) error terms. After these two regressions are estimated, the estimate of the ATT(s,t) is:
Note, the ATT${(s,t)}$ defined here is identical to the ATT${(g,t)}$ from callaway2021difference when there is only one silo per treatment timing group. However, that is not the case when there are additional silos, as in Section (ref). There are many cases in which ATT${(s,t)}$ are the more relevant objects as discussed in detail in our companion project karim2025slides.
The above process is repeated for all relevant $2\times2$ group--time blocks. These ATT(s,t) cells are then aggregated together to get an overall estimate of the ATT based on Equation (ref). $w_{s,t}$ are the weights associated with each of the ATT(s,t) cells. Common weighting schemes include equal weighting for each cell and population-based weighting. Refer to callaway2021difference and xiong2023federated for details.
Up until this point, we have concentrated on settings in which there is only one silo per treatment time. However, even with common timing it is typically the case that many treated and many comparison silos will be used. In that case, it is easier to estimate the ATT by means of a second stage regression. The details differ slightly depending on whether there is common or staggered adoption. Our software packages always use this approach, in part because it makes it computationally much faster to conduct cluster robust inference.
Imagine that there are two treated silos, A and B, and one control silo C. Under Assumption (ref), calculation of the treatment effect must occur on a fourth computer, call this a server. Let us also assume that we are not interested in estimating dynamic treatment effects, so that aggregating time points into pre- and post-intervention will suffice. In that case, we need three pieces of information from each silo: the difference between the pre-period and the post-period, as estimated as $\lambda_2^{j} - \lambda_2^{j}$ for the $j$th silo in Equation (ref), the standard error for this difference, and possibly the weight for the silo. We typically recommend the weight for silos $s$ in period $t$ is $w_{s,t} = N_{s,t} / \sum_{s,t}(N_{s,t}) \; \forall s \in S^T$. Here, $S^T$ is the set of all silos which are treated.
The first stage of the analysis creates a dataset which contains this information, along with treatment status for each of the three silos. In this case, the dataset would have three observations, and four variables: $\textrm{diff}_s$ is the within silo pre-post difference, potentially after controlling for covariates; $se_s$ is the standard error for $\textrm{diff}_s$; $w_s$ is the silo weight; and $d_s$ is whether that silo is treated $d_s=1$ or not $d_s=0$. We can calculate the ATT by estimating the regression:
In this regression, $\beta$ is the estimate of the $ATT^{UNDID}$, which will give the same estimate that one would get from a regression using poolable data when the groups are of equal size. When the groups are of unequal size, a weighted least squares version of the regression is needed to match the regression with pooled data. This second stage regression approach is particularly useful for cluster robust inference. This is being explored in a companion paper in development by the authors of this paper.
Staggered adoption imposes an additional computational burden with multiple silos. Each individual $ATT(s,t)$ can be calculated as before, and then an overall ATT can be estimated as a weighted average (see Section (ref)). To fix ideas, consider the simplest staggered adoption setting, where there are three silos (A,B,C) and three time periods (1,2,3). In this section we use the treatment timing notation from callaway2021difference. Silo A is first treated in period 2 ($g=2$). Silo B is first treated in period 3 ($g=3$). Silo C is never treated ($g=\infty$). In this 3$\times$3 example, without covariates, and using the never-treated cells as controls, the three $ATT{s,t}$s are:
In this example, we need to calculate differences in mean outcomes within each silo for different sets of years for each silo. Specifically, we need to estimate the difference between year 2 and 1, and 3 and 1 for silo A. For silo B, we only need the difference between year 3 and 2. Finally, for silo C, we need all three of these differences. Similar to the multiple silo with common timing example, we propose storing these differences, and their standard errors, in a separate matrix. The staggered treatment timing requires us to also keep track of which “block” we are estimating. To do this, we introduce a new index $h$. This index is equal to $g$ for the treated silos. For the control silos, $h$ takes on the value of $h$ for the relevant comparisons. For example,for the control silo in the equation above, $h=2$ for the first two lines, and $h=3$ for the last line. We also need to keep track of $t$ which is analogous to the $t$ from callaway2021difference. Again, for the control silo in the equation above, this $t=2$, $t=3$, and $t=3$. We also need to record whether a given silo is a treatment or control silo. To fill this matrix, each of the within-silo, two-year differences can be calculated using a separate regression. Table (ref) illustrates this matrix for a simple 3$\times$3 example. With multiple silos per group, each silo would have its own row for each difference across years.
The various $ATT(g,t)$ terms, can then be estimated using this matrix estimating this regression:
In this regression $\beta$ estimates the ATT${s,t}$ given by the sample restriction. For the 3$\times$3 example, the three restrictions needed would be h=2, T=2; h=2, T=3; and h=3, T=3. Our software packages implement this algorithm.
We have developed a set of software packages to aid in the implementation of the UN-DID estimator. There are versions available in R, Stata, Python and Julia. At present, the R version is the most polished and has been accepted on CRAN. The documentation for the R-package can be found at undidr. The Stata package is available here: \url{https://github.com/ebjamieson97/undid}. The \texttt{Julia} program can be found here: \url{https://github.com/ebjamieson97/Undid.jl}. The \texttt{Python} version works as a wrapper to call the \texttt{Julia} program, and can be found here: \url{https://github.com/ebjamieson97/undidPyjl}. Work is in progress to fully document using the software with worked examples. As mentioned, these software packages estimate the treatment effects using the two stage procedure. They also estimate cluster $P$-values using both a cluster jackknife, and a Randomization Inference routine. Which one of these procedures is preferred depends on many things, such as how many silos there are, and how many are treated karim2025slides.
We conduct a set of Monte Carlo experiments with synthetic data to demonstrate the unbiasedness of the UN-DID and its relative performance compared to the conventional DID estimator. The overall experimental design is depicted in Figure (ref), where we apply both conventional and UN-DID regression methods to the same simulated dataset to estimate ATTs and assess their properties. Note that, with poolable data, we can apply both the conventional and UN-DID methods to estimate the ATT. With unpoolable data, we are not able to run the conventional regression. Because these are synthetic data, they are poolable. Therefore, we can run both methods on the dataset.
Our generated dataset is intended to replicate samples from census data for Ontario and Quebec. To be more precise, we aligned the age and gender distributions to closely resemble those observed in the populations of these provinces in 2001 for individuals aged 65 years and older. We keep the gender constant for all individuals and age them by one year for the next nine years, resulting in our final dataset. Our outcome is a continuous variable. In our initial simulation, we created panels of 50 individuals each in both Ontario and Quebec, covering the period from 2000 to 2009. This results in a total of 500 observations evenly distributed over 10 years. We chose a small sample size to ensure the robustness of the results even when dealing with small datasets. For the simulation, we assume that Quebec is treated by some arbitrary policy in the year 2005. This approach ensures a balanced and symmetrical dataset, maintaining an equal number of observations in both pre- and post-treatment periods and in the treatment and control groups. We then repeat the simulations with roughly two thirds of the total observations in Ontario and one third of the total observations in Quebec.
We also consider four types of data-generating processes (DGPs) to assess whether the estimates of the ATT align between conventional and UN-DID regressions: one without covariates, one with time-invariant covariates, one with time-varying covariates when Assumption (ref) holds and one with time-varying covariates where Assumption (ref) does not hold. Within each DGP, we analyze two cases. In the first case, we set the true treatment effect for the treated group at 0. In the second case, the true treatment effect for the treated group is set at 0.1. After generating the data, we estimate ATTs using both conventional and UN-DID regressions and assess their unbiasedness properties. This is then repeated 1,000 times for each case. The data-generating process for the two cases without covariates are as follows:
In all the simulations in the paper, the error term is idiosyncratic and follows a normal distribution, denoted as $e \sim \mathcal{N}(0, 1)$. The $DID^s_{it}$ represents the interaction term between $P_t$ and $D_s$, corresponding to the post and treat dummy variables. The coefficient associated with this interaction term signifies the “true" effect of the treatment in both cases.
With time-invariant covariates, we introduce a modification to our DGP, where the outcome is now influenced by a time-invariant covariate, denoted by $X_i$. In our simulations, the time-invariant covariate is represented by a binary variable indicating gender. Specifically, it assumes a value of 1 if the individual is female, and 0 otherwise. Additionally, we assign a common slope parameter of 0.5 to the covariate in both the treated and the control silos. As proven in Theorem (ref), the ATTs from both methods are equivalent regardless of whether Assumption (ref) holds. Therefore, we just explore one scenario.
With time varying covariates, we replace the time-invariant covariate with a time-varying covariate, denoted by $X_{it}$. In our simulations, the time-varying covariate is age, which is unaffected by treatment even though it changes over time. We consider two types of time-varying covariates: one where Assumption (ref) holds, and one where Assumption (ref) is violated.
When Assumption (ref) holds, $\varsigma$ is 1.5 for both silos, and when Assumption (ref) is violated, $\varsigma^T = 0.5$ and $\varsigma^C = 2$.
In this simulation study, the generated dataset is a balanced panel. The results displayed in the following sections also apply to repeated cross sectional data. Furthermore, the results for time invariant covariate with repeated cross sectional data are comparable to the results with time varying covariates with panel data, as the distribution of the covariate varies over time due to different individuals being present in the data in each year. After generating the datasets and estimating both the conventional and the UN-DID ATTs, we create kernel density plots to evaluate the bias in both estimators.
The proofs shown in the preceding section are limited to the estimation of the ATT and do not show the equivalence of the Standard Errors between the two methods. We employ scatter plots, where the conventional DID standard errors are plotted on the y-axis and the UN-DID standard errors are plotted on the x-axis, to assess the equivalence of the standard errors between the two methods. If all the points lie along the $45^{\circ}$ line, it demonstrates equivalence in standard errors among the two methods. However, any deviations from the $45^{\circ}$ line indicates differences between them, which may arise due to small sample size.
In order to assess whether the conventional and the UN-DID standard errors approach the true standard errors as sample size increases, we repeat the Monte Carlo simulation by increasing the sample size to 1,000, 2,000, 4,000, 8,000, 10,000, and 50,000 while maintaining unequal sample sizes between both silos. Subsequently, we compute the mean squared error (MSE) between the estimated SE using UN-DID and the true SE from the underlying DGP according to Equation (ref):
The true standard error of the ATT is determined using the following formula:
Note that the $\epsilon_{it}$ are the actual disturbances used in the DGP in the Monte Carlo. Here, $\overline{DID}$ is the mean of the DID term from the generated dataset and $f$ is the number of coefficients. In the case with no covariates, the number of coefficients is set at 4, following the conventional regression shown in Equation (ref). We then check whether the MSE decreases as sample size increases. This will allow us to verify that any variations in the scatter plots are simply a result of small sample sizes. The following sub-sections present the simulation study results for the DGPs with unequal sample size, as it is more complicated than the DGPs with equal sample size. However, the results for both DGPs are similar.
We now present the results on a case by case basis.
\FloatBarrier
In Theorem (ref), we have shown that the conventional and the UN-DID regressions are numerically equivalent and unbiased. To assess unbiasedness, we generate kernel density plots for both conventional DID and UN-DID estimates of the ATT on a shared axis. The kernel densities for unequal sample sizes are shown in Figure (ref). The DGP's are generated such that Assumptions (ref), (ref) and (ref) holds. Since we can run both the UN-DID and the conventional method on poolable data, Assumption (ref) is not necessary. In Panel (a), the kernel density estimates for Case 1 are presented, and in Panel (b), the kernel density estimates for Case 2 are presented. Notably, for Case 1, both kernels are centered around 0, aligning with the true value of the ATT. Similarly, for Case 2, both kernels are centered around 0.1, which is the true value of the ATT for Case 2. The blue line representing the conventional ATT lies exactly beneath the red line representing the UN-DID ATT (except at the tails) and is therefore not visible. This demonstrates that the ATTs of the two methods are exactly numerically equivalent. This finding demonstrates that both conventional and UN-DID estimation methods exhibit unbiasedness, as evidenced by the distributions of estimates being centered around the true values of the ATT.
\FloatBarrier
In Theorem (ref), we have shown that both the conventional DID and the UN-DID methods can recover numerically equivalent values of the ATT when implemented on a poolable dataset and that both are unbiased. The DGP's for this section are generated such that Assumptions (ref), (ref), (ref) and (ref) holds. Similar to the preceding sub-section, we assess the unbiasedness of the UN-DID estimator with unequal sample sizes and time-invariant covariates using kernel densities, illustrated in Figure (ref). The kernel densities show that the distribution of ATTs is centered on the true ATT value, implying unbiasedness with time-invariant covariates. As in Figure (ref), the blue line is not visible as it is perfectly underneath the red line (except at the tails), demonstrating numerical equivalence.
Figure (ref) presents the scatter plots assessing the equivalence of the standard errors between the conventional and the UN-DID estimators. Panels (a) and (b) depict the relationships between the conventional DID standard error (y-axis) and the UN-DID standard error (x-axis) for Case 1 and Case 2 with unequal sample sizes and time-invariant covariates, respectively. Reviewing the scatter plots, we note that the standard errors lie along the $45^{\circ}$ line but do not align perfectly with it. This indicates that the standard errors are equivalent but not numerically equal.
\FloatBarrier
We hypothesize that the standard errors are not numerically equal due to small sample sizes. To verify that the two methods converge to the true standard error as sample size increases, we increase the sample sizes and check if the MSE's converge to the true value. The results are shown in Figure (ref). The MSEs on the y-axis are plotted against the sample sizes on the x-axis. The MSEs between the UN-DID and the true SE are illustrated by the red lines in Figure (ref). On the same axis, we plot the MSEs comparing the conventional DID to the true SE (the blue lines in Figure (ref)) and the MSEs between the conventional DID and UN-DID methods (the green lines in Figure (ref)). Panels (a) and (b) showcase the MSE of standard errors for Case 1 and Case 2, respectively with time-invariant covariates. We observe that standard error MSEs decrease with larger sample sizes, confirming that both the conventional and UN-DID methods converge to the true values of the standard errors as sample size increases. The blue lines (which illustrate the MSEs for the conventional DID vs. true values of SE) are not visible, as they lie underneath the red line (MSE for the UN-DID vs. true values of SE). This also shows the close equivalence between the conventional DID and UN-DID SEs with time-invariant covariates. Figure (ref) also illustrates a near zero MSE between the conventional DID and UN-DID methods, indicating the equivalence between the two estimators.
\FloatBarrier
In this section, we examine two DGPs. Assumptions (ref), (ref) and (ref) are satisfied for both DGPs, whereas Assumption (ref) holds in the first DGP but is violated for the second. In Theorem (ref), we have shown that the conventional DID estimator and the UN-DID converge in probability to two distinct population parameters $ATT^{Conditional}$ and $ATT^{UNDID}$ respectively. However, Theorem (ref) demonstrates that when Assumption (ref) holds, the two population parameters are equivalent (not numerically equal). Therefore, both estimators are unbiased when Assumption (ref) holds. In contrast, Theorem (ref) shows that the two population parameters are not equivalent when Assumption (ref) is violated. In this section, we show that the conventional regression is biased when Assumption (ref) is violated, whereas the UN-DID estimator is unbiased using a kernel density plot. The results are shown in Figure (ref). In Panel (a) and Panel (c), both the UN-DID and the conventional estimators are centered around the true value of 0 when Assumption (ref) holds. However, when Assumption (ref) is violated, we observe that the kernel density for the UN-DID (Panel (b)) is centered around the true value of 0, but the kernel density of the conventional estimator (Panel (d)) is centered roughly around 16. This shows that the UN-DID estimator is unbiased when Assumption (ref) is violated, but the conventional estimator is biased. We observe a similar pattern when the true effect is not 0 (figure not shown).
\FloatBarrier
From the simulation study, we observe that both the conventional and UN-DID estimators are unbiased in DGPs without covariates, with time invariant covariates or when time-varying covariates satisfy Assumption (ref). However, when time varying covariates are used and Assumption (ref) is violated, the conventional estimator is biased, but the UN-DID remains unbiased. Additionally, the standard errors of both methods converge to the true standard errors as sample size increases.
We now consider two empirical examples, one with common adoption timing and one with staggered adoption. We include these examples to highlight the properties of the UN-DID and conventional DID methods. In contrast to our Monte Carlo simulations, we do not know the true effects of the policies we analyze here. Using survey data that is pooled across US states, we can estimate the ATT using both conventional DID and UN-DID (treating the data as if it were siloed by state), and then compare the ATTs and SEs estimated using these two approaches. Our purpose here is to compare the two methods with real-world data - not to answer the substantive questions underlying the specific regressions. Accordingly, we diverge somewhat from the designs in the original analyses, and our analysis falls well short of complete empirical analyses of the substantive treatment effects.
We revisit the question asked in sabia2017effect of how access to medical marijuana impacts body mass index. The original study investigates the role of medical marijuana laws in 21 US states and finds that access to medical marijuana decreases the likelihood of being obsese. To assess the performance of UN-DID in a 2$\times$T setting with common treatment timing, we explore the impact of the District of Columbia's (DC) medical marijuana law, using New York State as the control.
Data come from the Behavioral Risk Factor Surveillance System (BRFSS). This repeated cross-sectional survey is pooled across US states but can also be analyzed as if the data were siloed by state. We use data from 2006--2013 from New York and DC, which enacted and enforced their medical marijuana law in 2010. We therefore have a $2 \times 8$ design with an equal number of pre-treatment and post-treatment years. We restrict our attention to respondents aged 25 or older, resulting in a final sample of 84,783 observations.
With these data, we estimate treatment effects using four different models: a conventional DID regression and UN-DID, each with and without controls. The control variables are a binary variable female and categorical variables for age, race, and education, all of which are time-varying covariates in a repeated cross-sectional dataset. The UN-DID estimates are calculated by artificially siloing the data by state. The conventional DID model is shown in Equation (ref).
We also calculate the UN-DID ATT by first estimating the following model separately for New York and DC.
Table (ref) presents the ATT and standard error estimates. As expected, without covariates the conventional DID and UN-DID yield the same estimates: an estimated ATT of 0.2560 and a standard error of 0.0853. When we add covariates, the estimated coefficient drops to 0.1322 for the conventional DID and 0.1204 for UN-DID. These estimates are neither substantively nor statistically different from one another. The small difference between them is expected given that the models include 25 covariates which are restricted to common slopes with conventional DID and allowed to vary as silo-specific slopes with UN-DID. The standard errors are also slightly different, but substantively equivalent: 0.0807 for the conventional DID and 0.0811 for UN-DID.
This empirical example demonstrates the strong performance of UN-DID relative to conventional DID in a 2$\times$T setting with common or single treatment timing. We reiterate that our effect estimates in this simplified example should not be interpreted as responding to the substantive question. However, this analysis using real-world, artificially siloed data demonstrates that UN-DID can accurately estimate the ATT and SE as well as conventional DID, both with and without covariates.
To highlight the use of UN-DID in a staggered adoption setting, we revisit the analysis of merit scholarships from conley_2011. These authors estimate the average impact of 10 state-level merit scholarship programs with different adoption dates. These scholarships vary in their design, generally awarding scholarships to high-achieving high school students to enroll in a college within their home state (see deming_2010 and the references therein for details).
Data come from the Current Population Survey (CPS), another repeated cross-sectional survey that is pooled across US states. There are 51 “states" (including DC) and 12 years (1989-2000) in the dataset, for a total of 42,161 individual-level observations. We estimate the ATT of a state's adoption of a merit scholarship program on the likelihood of a student in that state graduating college. We first estimate this effect individually for each of the 10 treated states, and then combine those for an aggregate ATT. This allows us to compare 11 distinct treatment effect estimates using both UN-DID and conventional DID.
Ten states adopt a merit scholarship across varying adoption dates. Four states adopt the scholarship in 1991, 1993, 1996, and 1999, respectively, while two states each adopt in 1997, 1998, and 2000. For each treated state, one control state was chosen from the set of 41 untreated states to roughly match the number of treated and untreated observations in each model. We use only one treated state to highlight the properties of the UN-DID estimator rather than to estimate the best possible counterfactuals. We estimate the treatment effects both with and without covariates. Specifically, using only data from treated state $T$ and control state $C$, we estimate the model:
The coefficients $\delta_1, \delta_2$, and $\delta_3$ are estimated only when the covariates are included. The remaining variables are defined the standard way: $treat_s$ is set equal to 1 for observations in state $T$ and is set equal to 0 for observations in state $U$. Similarly, $post_t$ is set equal to 1 for years greater than and equal to the year when state $T$ first offered a merit scholarship. Finally, $did_{st}$ is the product of $treat_s$ and $post_t$. We estimate treatment effects, or $\widehat{did}_{st}$, and standard errors using the pooled dataset.
We then artificially silo the data to estimate treatment effects and standard errors using UN-DID. Specifically, for each treated silo we estimate
Here, $\text{pre}_{t}$ is equal to one in the years before the merit program was adopted, and equal to zero afterwards. Conversely, $\text{post}_{t}$ is equal to zero in the years before the merit program was adopted, and equal to one afterwards. We estimate the same regression for the control state used for each treated state. We then calculate the treatment effect using Equation (ref) and the standard error using Equation (ref).
The results are found in Table (ref). Panel A shows the estimates for the models without covariates. The treatment effect and standard error estimates are identical to the fourth decimal place using conventional DID with the pooled data or UN-DID with the unpooled data. These results match the simulation results presented in Section (ref).
Panel B of Table (ref) shows the treatment effect estimates conditional on covariates. Here, some of the individual state treatment effect estimates are very similar using conventional DID and UN-DID. For instance, for state 88, the conventional DID effect estimate is 0.1811 and the UN-DID estimate is 0.1819. However, some of the estimates are not as close. The estimated treatment effect for state 61 is 0.0370 using conventional DID and 0.0314 with UN-DID. Even though the difference between the estimates is larger, they are not statistically different and may not be substantively different from each other. The estimates of the standard errors differ more between conventional DID and UN-DID. Again, these differences are not large enough to impact inference in this example. These findings are consistent with the results in Section (ref).
Lastly, we estimate average treatment effects using data from all groups and all necessary years per group. We do this using two methods, both including covariates. We estimate the ATT using the CSDID command, which implements the callaway2021difference procedure. We again pretend that each state's data are siloed, and estimate the ATT using UN-DID. We do this using our UNDID command, which excludes forbidden comparisons. Simplifying the procedure, we estimate an ATT for each post treatment period for each treated state. This is done by estimating a model like in Equation (ref) for each treated state, but using only one pre-treatment and one post-treatment year. Using those same years, we estimate the same model for each of the control states. The state-specific first differences $\beta_2 - \beta_1$ are then calculated, and added to a table, like Table (ref) as described in Section (ref). We can then estimate each ATT$({s,t})$ by running a properly specified regression like the one in Equation (ref). For both CSDID and UN-DID we estimate one ATT by taking a simple average of the ATT$({s,t})$ (or ATT$({g,t})$) and one by averaging over the $s$ (or $g$) groups. Additionally, we calculate cluster-robust standard errors, assuming state level clustering, for all ATT estimates. The UN-DID standard errors use the jackknife discussed in Section (ref).
The aggregate ATT estimates are shown in Panel C of Table (ref). We find the simple average ATT estimates from the two procedures to be quite similar. Using simple weights, the UN-DID ATT is 0.0485 and the CSDID ATT is 0.0464. The two standard errors are also very close to one another.
The group-averaged ATTs are not as similar to one another, but still close. The UN-DID ATT is 0.0459, remaining close to the simple average ATT. The standard errors are also similar at 0.0110 for UN-DID and 0.0133 for CSDID. The CSDID standard errors are cluster-jackknife standard errors using the Stata package csdidjack.\footnote{This is based on a soon to be released paper by MacKinnon, Nielsen, Webb, and Karim. The package can be downloaded here \url{https://github.com/liu-yunhan/csdidjack}.} Using group weights, the UNDID ATT is 0.0459 while the CSDID ATT drops to 0.0339. The standard errors are also similar, with the UNDID standard error equal to 0.0188 and the CSDID one equal to 0.0211. Note that the CSDID ATT is no longer significant at the 10% level.
Finally, we use the DID-INT estimator to estimate ATTs. These results are very comparable to the the UN-DID results. With group aggregation, the UN-DID estimate is 0.0459 and the DID-INT estimate (allowing for state level CCC violations) is 0.0458. The standard errors are more different, where the DID-INT standard error is 0.0084 compared to the UN-DID 0.0188. Results are very similar with simple aggregation. We also investigate the sensitivity to the remaining CCC assumptions. These results can be found in Appendix (ref). Generally, the ATTs broadly agree with one another, with the smallest being 0.041 when allowing for time varying parameters, and the largest is the 0.0511 when allowing for two-way varying parameters.
Difference-in-differences is commonly used to estimate treatment effects, particularly for policies or other interventions that vary at the jurisdictional (e.g., state, country) level. However, this approach was previously infeasible in settings where the data are siloed by jurisdiction. In this paper, we propose the unpoolable DID (UN-DID) method, assess its performance, and demonstrate its utility in settings with siloed data. We also formalize the assumption of data poolability in the DID context, which was previously implied but not directly articulated. Specifically, we show that UN-DID is an unbiased and reliable extension to the conventional regression-based DID method in settings with unpoolable data.
We began by presenting analytical proofs to demonstrate the equivalence of UN-DID and conventional DID with no covariates and time invariant covariates. With time varying covariates, we show that the two estimators converge in probability to two different population parameters. When the state-invariant CCC assumption holds and time varying covariates are used, both estimators remain unbiased, but are no longer equivalent. However, when the state-invariant CCC is violated, the conventional estimator becomes biased, while the UN-DID remains unbiased.
We then extended our assessment to simulated panel data using a series of Monte Carlo simulations (incorporating a range of DGPs and sample sizes). These experiments permitted a direct assessment of UN-DID’s performance and unbiasedness compared to both conventional DID and the true treatment effects. Across all assessments, we find that the estimated ATT from UN-DID remains unbiased. However, the conventional estimator is biased when time-varying covariates are used and the state-invariant CCC is violated. The estimated standard errors are also not numerically equal, but converge to the true value as the sample size increases. These findings suggest that UN-DID is a valid approach to DID when data are unpoolable. Finally, we compared the performance of UN-DID and conventional DID using pooled data from existing empirical examples, treating these data as unpoolable.
Our work contributes to a growing body of literature developing DID methods for use in increasingly complex policy and data environments. UN-DID offers an important methodological tool for researchers assessing multi-jurisdictional (or cross-jurisdictional) policy effects. By enabling the use of DID estimation in siloed data settings, UN-DID effectively opens the door to new research questions and new evidence on policy impacts.
The UN-DID software packages in Stata, R, Python, and Julia will facilitate the practical application of UN-DID by a range of different end-users; the accompanying user guide will further support this aim while additionally showcasing the use of UN-DID in a truly siloed setting. Taken together, we expect our development of UN-DID (both as an analytical method and associated software packages) to catalyze innovative cross-jurisdictional policy evaluation in siloed data settings, thereby eliminating a common barrier for an otherwise widely used and powerful method.