Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
46,728 characters · 26 sections · 54 citation commands
Testing the Significance of the Difference-in-Differences Coefficient via Doubly Randomised Inference
\def\spacingset#1{ {#1}} \spacingset{1}
\if00 \fi
\if10 {
} \fi
{\it Keywords:} Difference-in-Differences, \and econometrics, \and randomisation, \and causal inference, \and randomised inference
{\it JEL Codes:} C12, \and C15, \and C21, \and C23, \and C31
\spacingset{1.75}
The Difference-in-Differences (DiD) framework, also referred to as Difference-in-Differences (DD) or Diff-in-Diff, has become one of the most influential quasi-experimental tools in applied econometrics and the social sciences Halkiewicz2024. Its intuitive structure and relative simplicity make it a preferred method for evaluating causal effects of discrete interventions—such as legal reforms, natural disasters, or policy programs—when randomized experiments are infeasible. The fundamental logic of DiD is to compare the evolution of outcomes between a treatment group, affected by an intervention, and a control group, unaffected by it. By differencing both across time and across groups, the method aims to eliminate time-invariant unobserved heterogeneity and isolate the average treatment effect.
Since its formal introduction in applied economics by Card1994 and its systematic exposition in Angrist2008 and Cameron2005, DiD has become a cornerstone of empirical policy evaluation. Applications span health sciences—where early uses included epidemiological analyses of disease propagation Caniglia2020—education and social programs Domina2015, Furquim2020, Asadi2020, Halkiewicz2023, as well as labor and public economics Callaway2022. More recent studies have extended DiD to domains such as marketing Deng2019, human resources Chen2016, and production management Distelhorst2017. Methodological advances have further expanded its scope, introducing heterogeneous treatment effects Fredriksson2019, synthetic control and interactive fixed-effects extensions Arkhangelsky2021, and refinements in statistical inference MacKinnon2020, Conley2011, Ferman2019.
Despite its ubiquity, challenges remain in assessing the statistical significance and reliability of the estimated DiD parameter, especially in studies with limited group sizes or complex treatment assignment structures. Classical inference methods—such as $t$-tests with robust or clustered standard errors—often rely on asymptotic approximations that may be inaccurate in small samples or when key assumptions (parallel trends, homoskedasticity) are only weakly satisfied. These limitations raise important questions about how to properly evaluate the precision and robustness of DiD estimates in non-ideal empirical settings.
To address these issues, this paper introduces a simulation-based framework for significance testing of the DiD estimator. Our approach builds on the logic of randomization-based inference Fisher1935, Neyman1923, ImbensRubin2015, but adapts it to the quasi-experimental context of Difference-in-Differences. Specifically, we propose a dual randomization scheme in which both the treatment and time vectors are permuted to generate a Monte Carlo distribution of the DiD coefficient. This distribution allows for non-parametric inference on treatment effects and provides more reliable significance measures in small or irregular panels.
The remainder of the paper is structured as follows. Section (ref) introduces the Difference-in-Differences framework, clarifies identification and interpretation, and highlights why seemingly similar setups may yield different estimates. Section (ref) reviews the theoretical foundations of randomization-based inference and its adaptation to econometric applications. Section (ref) formalizes the proposed simulation procedure and presents its statistical properties. Section (ref) validates the proposed significance test and its relevancy through numerical experiments. Finally, section (ref) discusses implications for applied research and outlines directions for future methodological development.
The Difference-in-Differences (DiD) estimator measures the average effect of a treatment or intervention by comparing outcome changes over time between treated and untreated groups. Its central appeal lies in its simplicity: by differencing twice—across time and across groups—it removes time-invariant unobservables and isolates the component of change attributable to treatment. The standard two-period, two-group setup can be summarized by the following model:
where $Y_{it}$ is the outcome variable for unit $i$ at time $t$, $\text{TIME}_t$ indicates the post-treatment period, $\text{AFFECTED}_i$ marks the treated group, and $\delta$ represents the DiD estimator—the estimated average treatment effect on the treated (ATT).
Under the parallel trends assumption, i.e. \[ \mathbb{E}[Y_{it}(0) - Y_{i,t-1}(0)\,|\,\text{AFFECTED}_i=1] = \mathbb{E}[Y_{it}(0) - Y_{i,t-1}(0)\,|\,\text{AFFECTED}_i=0], \] the parameter $\delta$ consistently identifies the causal effect of treatment. Violations of this assumption—such as pre-existing differential trends, heterogeneous shocks, or group-specific dynamics—can lead to biased estimates.
Equation (ref) can be estimated using Ordinary Least Squares (OLS), or equivalently derived from a set of mean comparisons. In its simplest form, the DiD estimator is:
where $\bar{Y}_{g,t}$ denotes the sample mean for group $g \in \{0,1\}$ at time $t \in \{0,1\}$. The first difference $(\bar{Y}_{1,1} - \bar{Y}_{1,0})$ captures the temporal change in the treated group, and the second $(\bar{Y}_{0,1} - \bar{Y}_{0,0})$ measures the analogous change for the control group. The DiD estimate, being the difference between these two quantities, reflects the excess change attributable to the treatment.
This simple expression hides several subtleties. When treatment assignment is not perfectly exogenous or when timing varies across units, the estimate may no longer represent a single, homogeneous effect. For example, if treated and untreated groups experience unequal exposure to macroeconomic shocks or policy spillovers, $\widehat{\delta}_{DiD}$ can differ markedly depending on the grouping or time horizon considered.
To highlight this sensitivity, consider two stylized empirical setups with identical baseline means but different underlying dynamics:
Although both cases share similar raw means, the implied DiD estimates differ five-fold. Moreover, in small samples such differences can easily arise by chance, especially when only a few units define the group averages. Classical significance tests based on asymptotic normality may then yield unreliable $p$-values or conflicting conclusions about whether $\delta$ is statistically meaningful.
Determining the statistical significance of $\widehat{\delta}_{DiD}$ is nontrivial. In large samples, robust or clustered standard errors often suffice. However, when treatment assignment is limited to a small number of groups, the conventional variance estimator tends to understate uncertainty Cameron2005, MacKinnon2020. Furthermore, DiD inference depends on the specific composition of the treatment and control groups: re-defining them, or even altering time coding, can yield different standard errors and significance levels Conley2011, Ferman2019.
These issues motivate the simulation-based framework proposed in this paper. By repeatedly randomizing both the AFFECTED and TIME vectors, we construct an empirical distribution of $\widehat{\delta}_{DiD}$ under the null hypothesis of no treatment effect. This randomization provides a reference distribution that accounts for the actual data structure and sample size, offering more reliable inference than asymptotic formulas—particularly in small-$N$ and unbalanced panels.
In the next section, we formalize this approach within the broader context of randomization-based inference, showing how it generalizes classical permutation logic to Difference-in-Differences estimation.
The method proposed in this paper can be inherently seen as an extension of randomised inference ImbensRubin2015, Puchkin2023, MacKinnon2020.
Randomization-based inference (often called randomization inference or Fisherian inference Ahmad2024, Efron2016) is an approach to statistical inference that leverages the random assignment of treatments as the basis for testing hypotheses and estimating causal effects. The intellectual foundation dates back to the pioneering work of Fisher1935, who introduced the idea of the randomization test in the context of designed experiments. Fisher’s famous “lady tasting tea” experiment exemplified how one could test a null hypothesis by considering all possible reassignments of treatment labels and computing the probability of obtaining results as extreme as the observed outcome under random assignment alone.
Around the same time, Neyman1923 (translated in 1990) developed the potential outcomes framework for causal effects in experiments, emphasizing that each unit has a potential outcome under treatment and control and that random assignment allows for unbiased estimation of average effects Keller2024. Building on Neyman’s framework, the Rubin1974 Causal Model (RCM) formalized causal inference using potential outcomes in both randomized and observational studies Imbens2010, ImbensRubin2015. Together, these contributions established the logic that inference about treatment effects can (and should) be grounded in the known randomization mechanism, rather than requiring strong modeling assumptions about outcomes Rosenbaum2002, ImbensRubin2015.
The core principle of randomization inference is that under the null hypothesis of no treatment effect, the distribution of any test statistic can be derived entirely from the random assignment process. Consider $N$ units, of which $N_T$ receive treatment and $N_C$ receive control, with $N = N_T + N_C$. Let $Y_i(1)$ and $Y_i(0)$ denote unit $i$'s potential outcomes with and without treatment, respectively Neyman1923, Rubin1974. The sharp null hypothesis is:
\[ H_0: Y_i(1) = Y_i(0) \quad \text{for all } i. \]
Under $H_0$, the observed outcome is $Y_i^{\text{obs}} = Y_i(0)$ regardless of treatment. Hence, the vector $\mathbf{Y}^{\text{obs}} = (Y_1^{\text{obs}}, \dots, Y_N^{\text{obs}})$ is fixed, and any variation in a test statistic arises solely from the randomness in the treatment assignment vector $\mathbf{W}$.
Let $T(\mathbf{W}, \mathbf{Y})$ be a test statistic of interest. The randomization distribution of $T$ is the set $\{ T(\omega, \mathbf{Y}^{\text{obs}}) : \omega \in \Omega \}$ where $\Omega$ is the set of all allowable treatment assignments consistent with the experimental design Bonnini2024. For a completely randomized design, $|\Omega| = \binom{N}{N_T}$. The $p$-value for a two-sided test is then:
\[ p = \frac{1}{|\Omega|} \sum_{\omega \in \Omega} \mathbb{I}\left( |T(\omega, \mathbf{Y}^{\text{obs}})| \ge |T_{\text{obs}}| \right). \]
This provides an exact significance level for the sharp null Caughey2023. When $|\Omega|$ is large, a Monte Carlo approximation can be used to sample a subset of $\Omega$.
Randomization inference constructs the null distribution by permuting treatment labels and recalculating the test statistic on each permutation. For each simulated assignment $\omega \in \Omega$, the test statistic $T(\omega, \mathbf{Y}^{\text{obs}})$ is calculated. The empirical distribution of these values represents the permutation distribution under the null Caughey2023, Bonnini2024.
In classic applications, such as Fisher’s exact test, the randomization distribution corresponds to a hypergeometric distribution Fisher1935. In general, this approach extends to any test statistic and design, offering flexibility beyond conventional parametric frameworks Bonnini2024.
Randomization tests are exact in finite samples under the null hypothesis. They do not rely on asymptotic approximations and are therefore especially useful in small samples or non-standard settings Rosenbaum2002. They also allow researchers to construct confidence intervals by inverting a series of hypothesis tests (e.g., using Hodges–Lehmann-style estimation Hershberger2011).
Randomization inference offers several advantages:
Randomization inference is increasingly used in empirical economics Caughey2023, Canay2017, Abadie2022, especially in development and labor economics, where randomized controlled trials (RCTs) are common. Researchers often supplement regression-based inference with permutation-based $p$-values to ensure robustness Young2019, Basse2024. Another rapidly growing field of application is network analysis, where interference and non-independence of units violate the assumptions underlying classical asymptotic inference. Randomization inference provides exact tests under complex exposure mappings and graph-structured dependence, and has become a standard tool in experimental and observational network settings Aronow2017, Athey2018, Ugander2023, Li2019.
It is also used in quasi-experimental settings. For example, in regression discontinuity designs, permutation tests are applied locally around the cutoff to assess robustness of estimated effects Cattaneo2015. In observational studies with matched samples, permutation inference allows testing under assumptions of “as-if” randomization Rosenbaum2002.
Outside of economics, randomization inference has long-standing applications in biomedical research, particularly in clinical trials and epidemiology - e.g. Khim2020 developed a permutation test for edge structure of an infection graph.
Randomization inference provides a principled and flexible framework for testing hypotheses about treatment effects. By leveraging the known assignment mechanism, it avoids strong assumptions, delivers exact inference in finite samples, and accommodates complex designs ImbensRubin2015. As computing power increases and interest in robust causal inference grows, the use of randomization-based methods in econometrics and related fields continues to expand Athey2017.
The proposed procedure provides a simulation-based significance test for the Difference-in-Differences (DiD) estimator that does not rely on asymptotic approximations. Instead, the test constructs an empirical reference distribution of the DiD coefficient under the null hypothesis of no treatment effect through repeated randomization of the treatment and time indicators.
The test is based on the following hypotheses:
The rejection region is determined by the empirical $\frac{\alpha}{2}$ and $1-\frac{\alpha}{2}$ quantiles of the simulated distribution of DiD estimates, denoted $P_{\frac{\alpha}{2}}$ and $P_{1-\frac{\alpha}{2}}$, respectively. The null hypothesis $H_0$ is rejected if the observed estimate $DD_e$ lies outside the interval $(P_{\frac{\alpha}{2}},\, P_{1-\frac{\alpha}{2}})$, i.e. \[ DD_e \notin (P_{\frac{\alpha}{2}},\, P_{1-\frac{\alpha}{2}}). \] This logic mirrors the two-sided test based on asymptotic critical values, but replaces the theoretical reference distribution with a simulation-based one generated from the data. The quantiles can be directly obtained using standard statistical software such as R or Stata Team2021, Villa2016.
The algorithm proceeds as follows:
This procedure yields a data-dependent reference distribution that reflects the sample’s structure and accounts for limited group sizes, unequal variances, or potential departures from model assumptions. We can write the procedure above into a pseudocode as shown in Appendix (ref).
The function to perform this procedure is included in the sigDD R package, available on GitHub at \href{https://github.com/profsms/sigDD}{https://github.com/profsms/sigDD} Halkiewicz2024thesis.
Let $\mathcal{X} = \{(Y_i, A_i, T_i)\}_{i=1}^n$ denote the sample, where $A_i \in \{0,1\}$ and $T_i \in \{0,1\}$ are treatment and period indicators, respectively. Define $\Omega$ as the set of admissible label permutations, and $\mathcal{P}$ as the uniform probability measure on $\Omega$: \[ \mathcal{P}(\omega) = \frac{1}{|\Omega|}, \quad \forall\, \omega \in \Omega. \] The test statistic $D(\mathcal{X})$ is a function $D:\mathcal{X}\!\to\!\mathbb{R}$, and its randomization distribution is given by \[ F_D(t) = \mathcal{P}\big( D(\omega\mathcal{X}) \le t \big), \] where $\omega\mathcal{X}$ denotes the relabeled data. The empirical $p$-value is thus \[ p = \frac{1}{|\Omega|}\sum_{\omega\in\Omega} \mathbf{1}\!\left\{\,|D(\omega\mathcal{X})|\ge |D(\mathcal{X})|\,\right\}. \] In this setting, the following theorem provides a precious property of our procedure.
We now establish two simple but useful theoretical results that characterize the size and convergence properties of the permutation space introduced above.
The central design choice concerns how to randomize the Time and Affected indicators. Both are binary variables and can naturally be modeled as Bernoulli random variables. However, care must be taken in specifying the probability parameter $p$ governing the fraction of treated or post-treatment observations.
Several randomization strategies were initially considered:
Empirical testing indicated that randomizing $p$ or the interaction term alone substantially increases computational burden without providing theoretical or practical benefits. Moreover, such approaches can distort the joint structure of the data, leading to reference distributions that depend more on random sampling design than on the observed dataset.
Following conventions from experimental design, especially clinical trial methodology Rosenbaum1985, Kang2008, the most consistent approach is to preserve balanced group sizes. Hence, both Time and Affected variables are randomized under a fixed Bernoulli parameter $p = \frac{1}{2}$, ensuring equal proportions of 0s and 1s in each vector in the long run. This choice stabilizes the test distribution and maintains comparability across different simulations and datasets. Additionally, as further discussed in Section (ref), such choice maximizes the information gain.
The proposed test inherits several desirable features from randomization-based inference:
These properties make the procedure especially useful when the number of groups is small (which will be further investigated in Sections (ref) and (ref)), standard errors are unreliable, or model assumptions are questionable. The algorithm can be implemented in any environment capable of generating random permutations and linear regressions, such as R, Stata, or MATLAB.
This section presents a set of numerical experiments designed to validate the proposed doubly randomized significance test for the Difference-in-Differences coefficient. We apply the procedure to four empirical datasets commonly used in the DiD literature. For each dataset, we provide: (i) a brief description and summary statistics, (ii) permutation-based inference results under single and dual randomization, and (iii) visualizations of the resulting permutation distributions. All tables and figures are referenced in the text.
The first dataset, Inpress_data, is derived from the large-scale primary school construction program conducted in Indonesia between 1973 and 1978 Duflo2001, also discussed in Halkiewicz2023. Each observation contains an educational outcome and two binary indicators: affected (treated cohort) and time (high-intensity school-building regions). A summary of pre- and post-program averages across treated and control groups is reported in Table (ref).
Permutation-based inference results appear in Table (ref), showing that neither randomization scheme rejects the null hypothesis for this dataset. The corresponding permutation distributions are visualized in Figure (ref).
The second dataset contains daily search-intensity data (0–100 scale) for Ben & Jerry’s (treated) and Häagen-Dazs (control), collected from March–September 2020 and normalized within each keyword Halkiewicz2024thesis. Ben & Jerry’s issued a public Black Lives Matter statement in June 2020, providing a treatment event. Summary statistics are shown in Table (ref).
Table (ref) reports the permutation-based DiD results, which show strong evidence against the null of zero effect. The associated permutation distributions appear in Figure (ref).
The third dataset reproduces the fast-food restaurant survey from CardKrueger1994. Each restaurant reports pre- and post-treatment values for employment, starting wage, and meal price. Summary statistics are given in Tables (ref)--(ref).
The permutation-based DiD inference results are shown in Table (ref). The corresponding permutation distributions for all three outcomes are visualized in Figure (ref).
The final dataset comes from Dinas2019 and covers 96 municipalities over four Greek parliamentary elections. The treatment corresponds to substantial refugee-inflow exposure. Table (ref) presents summary statistics.
Table (ref) reports permutation-based inference, and Figure (ref) presents the two permutation distributions.
To this day, most researchers have approached the problem of testing the significance of Difference-in-Differences (DD) estimations by employing tools borrowed from Ordinary Least Squares (OLS) models, such as $t$-tests. However, as discussed in Sec. (ref), even though a DD estimate can be obtained from an OLS regression, its meaning differs fundamentally from that of a structural parameter. Therefore, there remains a need for a single, well-defined significance test tailored specifically to the DD estimator. Such a procedure would allow researchers to assess whether the observed DD effect carries real empirical weight, without relying on subjective or model-dependent approaches. Moreover, by introducing an interpretable scale, the value of the DD parameter could be understood not only in binary significance terms, but also with respect to its effect magnitude and strength.
The results presented in this study demonstrate that the proposed permutation test, which randomizes both the TIME and AFFECTED dimensions, performs comparably to the conventional approach that randomizes only AFFECTED. Across all examined datasets, both methods led to consistent inference decisions: strong effects were detected by both tests, while weak or null effects were not. Importantly, the proposed full randomization procedure provides a substantially larger set of admissible permutations, leading to a denser empirical null distribution. This feature stabilizes empirical $p$-values and critical thresholds, which is particularly advantageous in small-sample applications. Hence, the method constitutes a promising, more objective alternative to conventional significance testing in Difference-in-Differences designs.
Theoretical support for these empirical findings is provided in Sec. (ref), where the properties of the permutation space are analyzed formally. In particular, Proposition (ref) quantifies the combinatorial expansion induced by dual randomization, and Lemma (ref) establishes that a larger permutation space yields a finer and faster-converging approximation of the null distribution. Together, these results explain the observed stability of $p$-values and the robustness of inference under full randomization.
As established in Proposition (ref), dual randomization expands the admissible permutation space by a factor of $\binom{n}{n_T}$ relative to single-margin permutation.
Let $p_T = n_T/n$. By Stirling’s approximation Morris2024, the binomial coefficient admits the asymptotic expression \[ \binom{n}{n_T} \approx \frac{1}{\sqrt{2\pi n p_T(1-p_T)}}\, \exp\!\big(n\,H(p_T)\big), \qquad H(p)=-p\log p-(1-p)\log(1-p), \] where $H(p)$ is the binary Shannon entropy Saraiva2023. Therefore, the logarithmic growth rate of the permutation space is approximately $nH(p_T)$, reaching its maximum at $p_T=\tfrac{1}{2}$. This shows that the information content of the permutation universe, measured by entropy, is maximised when the randomisation is balanced, further justifying the choice of $p=\frac{1}{2}$, posed in Section (ref).
Entropy interpretation. Let $p_A = n_A/n$ and $p_T = n_T/n$ denote the empirical probabilities of treatment and post-treatment status, respectively. The combined permutation space then satisfies \[ \frac{1}{n}\log \#\Omega(\text{AFFECTED \& TIME}) \approx H(p_A) + H(p_T). \] Hence, adding the second randomized margin contributes an independent entropy term $H(p_T)$, quantifying the increase in information complexity of the test. When both margins are balanced ($p_A=p_T=1/2$), the entropy of the randomization space is maximized, \[ \#\Omega(\text{AFFECTED \& TIME}) \asymp \frac{2^{2n}}{2\pi n}\,. \] Consequently, the number of possible relabelings grows exponentially with $n$, and the full randomization scheme explores an exponentially richer configuration space. This combinatorial and entropic expansion formally explains the empirically observed stability of the test: by enlarging the support of the null distribution, the method achieves greater precision in finite samples and reduces discretization bias.
There is an expanding number of researches conducted using Difference-in-Differences models and new areas for possible applications are described every year. Given this fact, there is a growing need for efficient, strict and unbiased tests to measure the effects presented by the DD model. Inquires may be posed whether there exists a way to standardize the test distribution, such that it would not be dependent on the sample size. Further research endeavors may be also conducted to generalize the test distribution to one of well-described distributions and finding the parameters of such, therefore eliminating the need for, often time-consuming, simulations. Debate about whether this testing method is truly unbiased is also welcomed.