Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
82,711 characters · 15 sections · 48 citation commands
Is Inference Conditional on Not Rejecting a Pre-test Less Reliable than Unconditional Inference?
Specification tests are ubiquitous in applied research. To estimate the effect of a treatment, one has to rely on an identifying assumption to recover treated units' outcome without treatment, and researchers typically conduct pre-tests to assess the credibility of that assumption. For example, difference-in-difference (DID) estimation relies on a parallel-trends assumption. Then, researchers usually start by computing a pre-trend estimator, and they may report a DID estimator of, and a confidence interval (CI) for, the average treatment effect on the treated (ATT) only if treated and control units do not experience significantly different outcome trends before treatment. Similarly, in randomized controlled trials (RCTs), if there is attrition or one fears that the randomization may have been compromised, researchers may first conduct balancing tests comparing the average of some pre-determined covariates in the treated and control samples, and they may report the usual difference-in-means estimator and its CI only if covariates are balanced. Balancing tests are also common in instrumental variable (IV) and regression discontinuity design (RDD) studies. Beyond treatment-effect estimation, pre-tests also arise in generalized method of moments (GMM) estimation. There, researchers may only report the GMM estimator and its CI if a J-test is not rejected.
While pre-tests can help researchers assess if their identifying assumption is valid, they may also undermine inference. Let us explain this issue and our results in a particular case of our more general setup. Assume that a researcher uses an estimator $\widehat{\beta}$ (e.g. a DID) to learn a scalar parameter $\beta_0$ (e.g. the ATT). Under an assumption referred to as the null hypothesis of valid specification (e.g. parallel trends), $\widehat{\beta}$ is $\sqrt{n}-$consistent and asymptotically normal for $\beta_0$. Letting $q_{x}(\widehat{\beta})$ denote the $x$-th quantile of the asymptotic distribution of $\sqrt{n}(\widehat{\beta}-\beta_0)$, $$\text{CI}_{1-\alpha}:=\left[\widehat{\beta}-q_{1-\alpha/2}(\widehat{\beta})/\sqrt{n},\widehat{\beta}+q_{1-\alpha/2}(\widehat{\beta})/\sqrt{n}\right]$$ is the usual $1-\alpha$-level confidence interval for $\beta_0$. By construction, under the null $$\underset{n\to \infty}{\lim}P(\beta_0\in \text{CI}_{1-\alpha})=1-\alpha:$$ the unconditional coverage rate (UC) of $\text{CI}_{1-\alpha}$ is equal to its nominal coverage rate (NC).
Then, let us further assume that the null has a testable implication: if it holds, an estimator $\widehat{\theta}$ (e.g. a pre-trends estimator) should not be significantly different from zero. Then, the researcher only reports $\text{CI}_{1-\alpha}$ if $\widehat{\theta}$ is not significantly different from zero, namely if $||\sqrt{n}\widehat{\theta}||\leq q_{1-\alpha}(\widehat{\theta})$, where $q_{x}(\widehat{\theta})$ denotes the $x$-th quantile of the asymptotic distribution of $||\sqrt{n}\widehat{\theta}||$ under the null and $\|.\|$ is an appropriate norm. The issue is that while $\text{CI}_{1-\alpha}$ is only computed when the null is not rejected, we do not have any guarantee on its coverage conditional on not rejecting the null. Specifically, $$\underset{n\to \infty}{\lim}P(\beta_0\in \text{CI}_{1-\alpha}|~||\widehat{\theta}||\leq q_{1-\alpha}(\widehat{\theta})/\sqrt{n}),$$ the conditional coverage rate (CC) of $\text{CI}_{1-\alpha}$, could be strictly larger or strictly lower than $1-\alpha$, and the direction of the inequality could depend on the correlation between $\widehat{\theta}$ and $\widehat{\beta}$. Of course, one could modify $\text{CI}_{1-\alpha}$ to account for this selection; but our aim is precisely to study the CC of this naive confidence interval, following standard practice.
We first show that irrespective of the correlation between $\widehat{\beta}$ and $\widehat{\theta}$, under the null of valid specification the CC of $\text{CI}_{1-\alpha}$ is always larger than its NC:
Thus, if we are under the null (e.g. if we have parallel trends), pre-testing can lead $\text{CI}_{1-\alpha}$ to over-cover but cannot lead it to under-cover. (ref) readily follows from the Gaussian correlation inequality royen2014, which implies that for a centered normal vector $(Y,X')\sim \mathcal{N}(0,\Sigma)$,
We also show that the inequality in (ref) is strict only if the covariance between $Y$ and $X$ is equal to zero, a result that appears to be new. This allows us to show that under the null, $\text{CI}_{1-\alpha}$'s CC is equal to its NC if and only if $\widehat{\beta}$ and $\widehat{\theta}$ are asymptotically independent. We also extend (ref) by considering infinite-dimensional (e.g., Kolmogorov-Smirnov) tests, as well as one-sided inference for $\beta_0$.
Then, we consider results under the alternative, namely when the null hypothesis fails (e.g. we have differential trends). In that case, even without pre-testing, $\text{CI}_{1-\alpha}$ does not cover $\beta_0$ $(1-\alpha)$% of the time, because $\widehat{\beta}$ is biased (e.g. the DID estimator is biased for the ATT). Then, one may want to compare $\text{CI}_{1-\alpha}$'s CC both to its NC and to its UC. One could for instance have that $\text{CI}_{1-\alpha}$'s CC is below its NC but still larger than its UC. Then, pre-testing does not decrease the coverage rate of $\text{CI}_{1-\alpha}$.
We start by noting that if $\widehat{\beta}$ and $\widehat{\theta}$ are not asymptotically independent, then the inequality in (ref) is strict under the null, so that by a continuity argument, $\text{CI}_{1-\alpha}$'s CC is still larger than its NC and UC “locally”, namely for small departures from the null. Next, we show that if $\mu_1$, the standardized bias of $\widehat{\beta}$, is equal to $\mu_2$, the standardized bias of $\widehat{\theta}$, multiplied by their correlation $\Sigma_{12}$, then the CC of $\text{CI}_{1-\alpha}$ is larger than its UC “globally”, namely for any value of $\mu_2$. In RCT (resp. IV) studies, if $\widehat{\theta}$ is a vector of differences in means of covariates, then $\mu_1=\mu_2 \Sigma_{12}$ holds if the treatment (resp. instrument) is exogenous once the covariates in the balancing tests are controlled for, and if the treatment (resp. instrument) effect is independent of the covariates. On the other hand, in DID studies $\mu_1=\mu_2 \Sigma_{12}$ fails for instance if there is a differential linear trend and the error in the untreated outcome equation follows a stationary AR(1). Finally, we show numerically that our global result is robust to violations of the condition $\mu_1=\mu_2 \Sigma_{12}$: when $\widehat{\theta}$ is of dimension one, $\text{CI}_{1-\alpha}$'s CC is still larger than its UC if $\mu_1=\mu_2 \Sigma_{12}(1+R)$, for a wide range of values of $(\mu_2,\Sigma_{12},R)$. In RCT, IV, and RDD studies, $R$ represents the bias of $\widehat{\beta}$ coming from variables not considered in the balancing tests, divided by the bias coming from the variables considered in those tests. Then, our numerical result shows that in this type of studies, the CC of $\text{CI}_{1-\alpha}$ can be larger than its UC even with a substantial bias coming from other variables than those in the balancing tests.
Finally, as the condition under which the CC of $\text{CI}_{1-\alpha}$ is larger than its UC is unlikely to hold in DID studies, we compare the CC and UC of $\text{CI}_{0.95}$ in data generating processes (DGPs) calibrated to the twelve DID studies in the meta-analysis of roth2022pretest, assuming differential linear trends. On average, the CC and UC are respectively equal to 78.4 and 80.3%: while both are well below 95%, the CC is very close to the UC so pre-testing only slightly reduces coverage. Note that this comparison is not entirely fair, because it does not account for the power of the pre-test, which can lead the analyst to correctly reject the null of parallel trends.
Inference conditional on not rejecting a pre-test has received some attention in the DID literature. Propositions 1 and 4 in roth2022pretest show that under parallel trends, the DID estimator remains conditionally unbiased, and that its conditional variance is smaller than its unconditional variance. We complement these results by showing that under parallel trends, the CC of $\text{CI}_{1-\alpha}$ is weakly larger than its NC.
Our paper is related to, but different from, the vast literature on inference after model selection leeb2005model,leeb2006can,berk2013valid, lee2016exact,heller2019post,kuchibhotla2020valid, andrews2024inference. A simple example of the settings considered by that literature goes as follows. A researcher seeks to estimate $\beta_0$, the coefficient on $X$ in the regression of $Y$ on $X$, but they do not know whether they should control for another variable $W$. First, the researcher estimates the regression of $Y$ on $(X,W)$, and tests if $\beta_W$, the coefficient on $W$, is different from zero. If the test is not rejected, the researcher's CI for $\beta_0$ is based on $\widehat{\beta}^s$, the coefficient from the regression of $Y$ on $X$, while if the test is rejected their CI is based on $\widehat{\beta}^\ell$, the coefficient from the regression of $Y$ on $(X,W)$. The resulting sequential CI is defined irrespective of the outcome of the pre-test. This is an important difference with our setting, where we do not take a stance as to what happens if the pre-test is rejected. Another important difference is that most papers in this literature consider inference that accounts for selection, whereas we study “naive” inference that ignores such selection. For instance, while lee2016exact and andrews2024inference exploit properties of the normal distribution to characterize estimators' distributions conditional on selection, we exploit the Gaussian Correlation Inequality to show that naive inference can remain conditionally valid, albeit conservative.
Our paper still relates to some results in this literature. In particular, in the example of linear regressions above, Proposition 3.1 of leeb2003finite shows that under the null that $\beta_W=0$, $\widehat{\beta}^s$ is asymptotically independent of the pre-test, so that the CC of $\widehat{\beta}^s$'s naive CI is equal to its NC. angelini2024identification obtain a similar result in the context of proxy-SVARs.
Finally, this literature has also shown that under local alternatives (e.g., when $\beta_W$ differs from but converges to zero), the UC of sequential CIs can be well below their NC. We also consider results under local alternatives, but they apply to the CC of the CI based on $\widehat{\beta}^s$, which we mainly compare to the UC of the same CI. This is a relevant comparison to make in our setting, as there may be no estimation taking place when the pre-test is rejected, and we do not assume that we have access to an estimator like $\widehat{\beta}^\ell$ that is consistent even under the alternative.
The remainder of the paper is organized as follows. Section (ref) presents our results under the null of valid specification. Section (ref) presents our results under the alternative. Section (ref) concludes by highlighting what researchers should and should not take away from our paper.
We are interested in $\beta_0\in \mathbb R^p$. We use an estimator $\widehat{\beta}$, which is consistent and asymptotically normal if $(\theta_0,\eta_0)=0$, where $\theta_0\in \mathbb R^q$ and $\eta_0\in \mathbb R^r$. We assume for now that $\theta_0$ is finite-dimensional, but we consider the infinite-dimensional case in Section (ref). Similarly, in all our examples $\eta_0$ is finite-dimensional, but our results would still hold with an infinite-dimensional $\eta_0$. We refer to $(\theta_0,\eta_0)=0$ as the null hypothesis of valid specification. We also have a consistent and asymptotically normal estimator of $\theta_0$, $\widehat{\theta}$. Therefore, $\theta_0=0$ is testable. Accordingly, the null has a testable implication, though it may not be fully testable, unless $\eta_0$ is by definition equal to zero. We review four examples where this set-up applies.
\newcounter{forreset} \setcounter{forreset}{\arabic{example}}
We now introduce the conditions underlying our main result below.
Assumption (ref).(ref) requires that $(\theta_0,\eta_0)=0$ and $\widehat{\beta}$ is consistent and asymptotically normal for $\beta_0$. Note that $\widehat{\beta}$ could still be consistent for $\beta_0$ under a weaker condition. For instance, in the DID, AE, and RDD examples, it is sufficient to have $\eta_0=0$. Importantly, Assumption (ref).(ref) does not restrict the asymptotic dependence between $\widehat{\beta}$ and $\widehat{\theta}$. It also allows for estimators converging at rates lower than $n^{1/2}$, as is the case in the RDD example. The $J$ different tests in Assumption (ref).(ref) reflect the fact that one often considers different pre-tests. The functions $T_1,...,T_J$ in these pre-tests have to be convex and symmetric around zero: this condition is key to obtaining our main result. On the other hand, we do not require the pre-tests to be consistent or to reach their nominal level asymptotically. The condition $q_j>0$ in Assumption (ref).(ref) ensures that under the null, the probability to reject does not tend to one.
In our four leading examples, Assumption (ref) holds under regularity conditions.
Our first result provides a lower bound on the asymptotic probability that $\widehat{\Sigma}_\beta^{-1/2}\left(\widehat{\beta}-\beta_0\right)$ belongs to a convex set, conditional on not rejecting the pre-tests.
Heuristically, the proof goes as follows. First, by convexity and symmetry of the $(T_j)_{j=1,...,J}$, the event $\{T_{1,n} \le q_{1,n},...,T_{J,n}\le q_{J,n}\}$ is asymptotically equivalent to $\widehat{\Sigma}_{\theta}^{-1/2}\widehat{\theta}\in C'$, for some centro-symmetric convex set $C'$. Second, under the null of valid specification, $\left(\widehat{\Sigma}_\beta^{-1/2}\left(\widehat{\beta}-\beta_0\right)\right.$, $\left.\widehat{\Sigma}_\theta^{-1/2}\widehat{\theta}\right)\stackrel{d}{\longrightarrow} (Z_1,Z_2)$, where $Z_2$ is such that $(Z_1, Z_2)\sim \mathcal{N}(0,\Sigma)$. Then,
where the inequality follows from the Gaussian correlation inequality royen2014. Finally, the result follows from dividing both sides of the previous display by $P\left(\widehat{\Sigma}_\theta^{-1/2}\widehat{\theta}\in C'\right)\approx P(Z_2 \in C')$, the probability that the pre-test is not rejected.
We now turn to the implications of Theorem (ref) for (conditional) inference on $\beta_0$. We consider the usual F-test of $\beta_0=b_0$ with test statistic $F_n(b_0):=(\widehat{\beta}-b_0)'\widehat{\Sigma}_{\beta}^{-1}(\widehat{\beta}-b_0)$ and critical region $\{F_n(b_0)>q_{1-\alpha}(p)\}$, with $q_{1-\alpha}(p)$ the quantile of order $1-\alpha$ of a chi-squared distribution with $p$ degrees of freedom. We also consider the standard confidence region (CR)
Hence, under Assumption (ref), conditional on not rejecting the pre-tests, the usual F-tests and CRs are asymptotically conservative. When $\beta_0$ is a scalar ($p=1$), this implies that two-sided tests and CIs on $\beta_0$ are asymptotically conservative. We have focused for simplicity on F-tests and standard CRs but given that Theorem (ref) applies to any centro-symmetric convex set, the same result as Corollary (ref) also holds for sup-tests on $\beta_0$ and cartesian products of symmetric two-sided confidence intervals.
One may wonder when conditional inference is actually not conservative. An obvious sufficient condition is asymptotic independence between the estimator and the pre-test ($\Sigma_{12}=0$), but is this condition also necessary? The answer is affirmative when we focus on two important types of pre-tests, namely F-tests and sup-tests.
The result under (i) readily follows from royen2014. In case (ii), the result is new, to the best of our knowledge. To prove it, we show in particular that asymptotically, $$f\mapsto P\left[\widehat{\Sigma}_\beta^{-1/2}\left(\widehat{\beta}-\beta_0\right) \in C \big|T_{1,n}=f\right]$$ is non-increasing and non-constant when $\Sigma_{12}\ne 0$. This follows from a convenient representation of $(Z_1,T_1(Z_2))$, with $(Z_1,Z_2)\sim\mathcal{N}(0,\Sigma)$ (see Equation (ref) in the proof) and Anderson's theorem anderson1955integral. Note that we impose $\Sigma_{22}=I_q$ in (ii). This is a weak condition: it holds whenever a consistent estimator of the asymptotic variance of $\widehat{\theta}$ is available.
Assume that $p=1$. Then, one-sided tests and CIs are not covered by Corollary (ref). Yet it follows from Lemma (ref) below, a consequence of the Gaussian correlation inequality,\footnote{We thank Fedor Petrov for giving us the proof of this result.} that a similar result holds for such tests and CIs.
We consider the test statistic $T_n(b_0):=(\widehat{\beta}-b_0)/\widehat{\Sigma}_{\beta}^{1/2}$. To test the null that $\beta_0\leq b_0$, we consider the critical region $\{T_n(b_0)>z_{1-\alpha}\}$, with $z_{1-\alpha}$ the quantile of order $1-\alpha$ of a standard normal distribution. Accordingly, we consider the unilateral CI $(-\infty, \widehat{\beta} + \widehat{\Sigma}_{\beta}^{1/2} z_{1-\alpha}]$. The result below still holds if one considers instead the critical region $\{T_n(b_0)<z_{\alpha}\}$, to test the null that $\beta_0\geq b_0$, or if we consider the unilateral CI $[\widehat{\beta}+\widehat{\Sigma}_{\beta}^{1/2} z_{\alpha},+\infty)$.
Similarly, if $q=1$, meaning that the pre-test estimator $\widehat{\theta}$ is of dimension one, one can apply Lemma (ref) again, but now swapping the roles of $\widehat{\beta}$ and $\widehat{\theta}$, to show that conditional on not rejecting a unilateral pre-test, bilateral tests or CRs on $\beta_0$ are asymptotically conservative. Hence, while Assumption (ref) does not apply to moment inequality models, the main take-away from Theorem (ref) still applies to moment inequality models with one inequality. On the other hand, unilateral tests or CRs on $\beta_0$ are not guaranteed to be conservative conditional on not rejecting a unilateral pre-test.\footnote{When $p=q=1$, the distortion does not seem large. For instance, if we consider a 5% level test of $\theta_0\le 0$, we find by numerical evaluation that across all possible values of $\Sigma_{12}$, the conditional coverage of the confidence interval $(-\infty,\widehat{\beta}+1.64\widehat{\sigma}_\beta]$ is (asymptotically) at least equal to 94.7%.}
The pre-tests in Assumption (ref) are based on a finite-dimensional estimand $\theta_0$. However, as the next example shows, some pre-tests are in fact based on infinite-dimensional estimands.
To cover pre-tests based on infinite-dimensional estimands, we introduce the following assumption.
Contrary to Assumption (ref), Assumption (ref) allows for pre-tests that rely on infinite-dimensional estimands: the class $\mathcal{F}$ in Assumption (ref).(ref) may be infinite and even uncountable. Then, the continuity condition in Assumption (ref).(ref) is important, to apply the continuous mapping theorem. Note also that Assumption (ref) does not allow for estimators converging at rates lower than $n^{1/2}$ or for non-independent variables. However, we conjecture that Theorem (ref) below still holds if we assume, instead of Assumption (ref).(ref) and (ref).(ref), the weak convergence of $(\widehat{\Sigma}_\beta^{-1/2}(\widehat{\beta} -\beta_0),\widehat{\Sigma}_\theta^{-1/2}(\widehat{\theta} -\theta_0))$ to a Gaussian process.
Beyond Example AME, Assumption (ref) applies more generally to tests of functional form, where one wants to test if $E(Y|D)$ is equal to, say, a polynomial of known degree $K$ in $D$, or more generally to tests of conditional moment restrictions such as $E[g(U,\nu_0)|X]=0$.
Theorem (ref) follows from the same steps as Theorem (ref), but also from the approximation of the limit distribution of $T_j\left(n^{1/2}\widehat{\theta}\right)$ by a convex function of a finite-dimensional Gaussian measure. Though we do not state them, Theorem (ref) readily implies results equivalent to those in Corollary (ref) if Assumption (ref) holds.
In this section, we consider results when $(\theta_0,\eta_0)\ne 0$. This requires modifying our modeling framework. With a fixed distribution such that $\theta_0\ne 0$, the power of the pre-tests converges to one when $n\to \infty$. Then, pre-testing can never lead to wrongly accept the null. In practice however, pre-tests may have limited power and may thus fail to reject under the alternative. To allow for this possibility, we consider local alternatives, where the probability distribution of the data can change with the sample size. Accordingly, we now index $(\beta_0, \theta_0,\eta_0)$ by $n$ and assume that $(\theta_{0n},\eta_{0n})\ne 0$ but
The term $r_n$ corresponds to the rate at which the estimators converge under the null. In our examples above, $r_n=n^{1/2}$ except in RDDs for which $r_n=n^{2/5}$. $\delta_n$ and $\delta$ can be interpreted as (properly normalized) distances between the distribution of the data and the null. Then, (ref) states that the distance between the distribution of the data and the null shrinks to zero at the same rate at which the estimators converge. In this framework, the power of the pre-tests does not converge to one, because as the sample size increases, deviations from the null become closer to zero and harder to detect.
Assumption (ref) below is the analogue of Assumption (ref) under local alternatives.
Assumption (ref).(ref) requires that if the distance between the distribution of the data and the null of valid specification converges to zero at the same rate at which the estimators converge, then $\widehat{\beta}-\beta_{0n}$ and $\widehat{\theta}$ are asymptotically normal, potentially with first-order biases. If $\delta=0$, then the asymptotic biases are equal to zero, as would happen under the null. On the other hand, $\widehat{\beta}$ may have an asymptotic bias even if the testable part of the null hypothesis holds (i.e. $\delta_\theta=0$). Allowing for this generality is important: for instance, in the DID example, parallel trends before $t_0$ may not necessarily imply parallel trends after.\footnote{Similarly, under Assumption (ref), having $\delta_\theta=0$ is not sufficient to have $\mu_2(\delta)=0$. This generality may not be as important: in our four leading examples, $\delta_\theta=0$ does imply $\mu_2(\delta)=0$.} We now give conditions under which Assumption (ref).(ref) holds in three of our four examples.\footnote{We conjecture that it should be possible to exhibit conditions under which Assumption (ref).(ref) also holds in the RDD example.} Hereafter, we let $\Sigma_\beta$ and $\Sigma_\theta$ denote the probability limit of $r_n^2\widehat{\Sigma}_\beta$ and $r_n^2\widehat{\Sigma}_\theta$ (we show that these limits exist in all our examples).
Theorem (ref) below shows that when the estimators used in the pre-tests $\widehat{\theta}$ and the estimators of interest $\widehat{\beta}$ are not asymptotically independent, and the pre-tests are either sup-tests or F-tests, then there exists neighborhoods around the null ($\delta=0$) such that if $\delta$ belongs to that neighborhood, CRs' conditional coverage (CC) is at least as large as their nominal coverage (NC) and their unconditional coverage (UC).
The intuition of the proof of Theorem (ref) is as follows. First, by Proposition (ref), for sup-tests and F-tests, CRs' CC is greater than their NC under the null if $\widehat{\theta}$ and $\widehat{\beta}$ are not asymptotically independent. Then, by continuity, there must be a neighborhood where CRs' CC is still greater than their NC. By the same reasoning, there must also be a neighborhood around the null where CR's CC is greater than their UC, as the UC is equal to the NC under the null. Theorem (ref) focuses on CRs, but a similar result holds for tests.
Theorem (ref) raises the following question: in practice, how large are the neighborhoods around the null where CRs' CC is not lower than their NC (resp. UC)? The answer to that question depends heavily on $(\mu_1(\delta),\mu_2(\delta),\Sigma_{11}(\delta),\Sigma_{22}(\delta),\Sigma_{12}(\delta))$. Figure (ref) below considers the case where $p=q=1$: the pre-test is based on a scalar estimator, in which case the sup-test and the F-test are equivalent, and the parameter of interest is also scalar. Then, the figure shows the asymptotic CC and UC of $\text{CI}_{0.95}$, when $\mu_1(\delta)=\lambda \mu_2(\delta)$, $\Sigma_{11}(\delta)=\Sigma_{22}(\delta)=1$, and $\Sigma_{12}(\delta)=\Sigma_{12}$, as a function of $\mu_2:=\mu_2(\delta)$ and for four different values of $(\lambda,\Sigma_{12})$. $\mu_2$, the bias of the pre-test estimator divided by its standard deviation, can directly be mapped to the power of the pre-test. For instance, if $|\mu_2|=1.96$ (resp. $2.8$), the pre-test has a 50% (resp. 80%) chance of being rejected.
In the top-left panel, $\Sigma_{12}=0.5$ and $\lambda=-1$: $\widehat{\beta}$ and $\widehat{\theta}$ are positively correlated, and the standardized bias of $\widehat{\beta}$ is the negative of that of $\widehat{\theta}$. In that DGP, under the null (i.e. $\mu_2=0$), the CC of $\text{CI}_{0.95}$ is equal to 95.7%, barely above 95%. Accordingly, the neighborhood around $\mu_2=0$ where the CC is above 95% is small ($[-0.22,0.22]$), and the neighborhood where the CC is above the UC is also small ($[-0.51,0.51]$). At $|\mu_2|=1.96$, the CC is equal to $33\%$, which is much lower than 95%, and quite a bit lower than the UC (50%). At $|\mu_2|=2.8$, the CC is equal to $4.2\%$, which is orders of magnitude lower than 95%, and much below the UC (20%). In the top-right panel, $\Sigma_{12}=0.9$ and $\lambda=-1$. With very strongly positively correlated estimators, the neighborhoods where the CC is above 95% and above the UC become larger ($[-0.36,0.36]$ and $[-0.63,0.63]$, respectively). On the other hand, the CC declines more steeply when $|\mu_2|$ increases. For instance, at $|\mu_2|=1.96$, the CC is equal to $14.4\%$. In the bottom-left panel, $\Sigma_{12}=0.5$ and $\lambda=1$: the standardized bias of $\widehat{\beta}$ is equal to that of $\widehat{\theta}$. Then, while the neighborhood where the CC is above 95% remains small ($[-0.29,0.29]$), the CC is larger than the UC for all $\mu_2$. The bottom-right panel shows that results are similar when $\Sigma_{12}=0.9$, though the neighborhood where the CC is above 95% becomes larger ($[-0.91,0.91]$).\footnote{Considering only positive values of $\Sigma_{12}$ is without loss of generality: if $\Sigma_{12}<0$ one can let $\widetilde{\theta}=-\widehat{\theta}$.}
The bottom panels of Figure (ref) suggest that under some conditions, the CC is larger than the UC for all $\delta$. The following theorem confirms this idea.
Our result relies on the same assumptions as Theorem (ref), and three additional conditions. First, we need to have $p=1$: the target parameter should be a scalar. Second, we should have $\Sigma_{11}(\delta)=1$ and $\Sigma_{22}(\delta)=I_q$. This essentially requires that once divided by the estimators' convergence rate squared, $\widehat{\Sigma}_\beta$ and $\widehat{\Sigma}_\theta$ are consistent for the estimators' asymptotic variances under the null and under local alternatives, a condition that often holds in practice. Third, we need to have $\mu_1(\delta)=\Sigma_{12}(\delta)\mu_2(\delta)$: the standardized bias of $\widehat{\beta}$ should be equal to the standardized bias of $\widehat{\theta}$ multiplied by their correlation. Under those additional conditions, (ref) holds uniformly: CRs' CC is always greater than their UC.
Theorem (ref) only applies to F-tests. Its proof shares some similarities with that of Proposition (ref), but with important differences. Let $(Z_1(\delta),Z_2(\delta))$ be such that $$\left[\widehat{\Sigma}_\beta^{-1/2}(\widehat{\beta}-\beta_0), \widehat{\Sigma}_\theta^{-1/2}\widehat{\theta}\right]\stackrel{d}{\longrightarrow} (Z_1(\delta),Z_2(\delta)).$$ We cannot apply Anderson's theorem to $Z_1(\delta)|Z_2(\delta)'Z_2(\delta)$, as in Proposition (ref), because its distribution is not centered. Instead, we apply it to $N(\delta):=Z_1(\delta) - \Sigma_{12}(\delta)Z_2(\delta)$: when $\mu_1(\delta)=\Sigma_{12}(\delta)\mu_2(\delta)$, $N(\delta)$ has mean zero. We also exploit $\mu_1(\delta)=\Sigma_{12}(\delta)\mu_2(\delta)$ to establish asymptotic independence between $Z_1(\delta)^2$ and $Z_2(\delta)'Z_2(\delta)$ conditional on $V:=[\Sigma_{12} Z_2(\delta)]^2$.
We now exhibit assumptions under which the additional conditions underlying Theorem (ref) hold in the AE example, before exhibiting assumptions under which $\mu_1(\delta)=\Sigma_{12}(\delta)\mu_2(\delta)$ cannot hold in the DID example.
Theorem (ref) above shows that under weak conditions, the CC of $\text{CI}_{1-\alpha}$ is larger than its UC in a neighborhood of the null, but the result is not informative on the neighborhood itself. Theorem (ref) shows that under stronger conditions, notably $\mu_1(\delta)=\Sigma_{12}(\delta)\mu_2(\delta)$, the neighborhood is actually the full space. Point 1 of the following numerical result gives quantitative guarantees on the neighborhood when $p=q=1$, under a weaker restriction than $\mu_1(\delta)=\Sigma_{12}(\delta)\mu_2(\delta)$. Point 2 provides guarantees on the CC itself, when $\mu_1(\delta)=\Sigma_{12}(\delta)\mu_2(\delta)$.
To establish, say, the first result in Point 1, we minimize $P(\beta_0\in\text{CI}_{0.95}| T_{1,n} \le q_{1,n})/P(\beta_0\in$ $\text{CI}_{0.95})$ across all $(R(\delta),\mu_2(\delta),\Sigma_{12})\in [0,1]\times [-3.9,3.9]\times [-1+\zeta,1-\zeta]$, using Matlab's fminunc function. For 500 randomly chosen starting values, the objective function is always greater than or equal to one at the minimum, so the result seems robust to the fact that the objective function exhibits local minima. We proceed similarly to establish all other numerical results.
Point 1.a) means that for essentially any value of $(\mu_2(\delta),\Sigma_{12})$,\footnote{The result is probably true for any value of $(\mu_2(\delta),\Sigma_{12})$, though up to Matlab's numerical precision we can only establish it numerically for $(\mu_2(\delta),\Sigma_{12})\in [-3.9,3.9]\times [-1+\zeta,1-\zeta]$.} the CC is larger than the UC if the standardized bias of $\widehat{\beta}$ belongs to $\Sigma_{12}\mu_2(\delta)\times[1,2]$.\footnote{If $\Sigma_{12}\mu_2(\delta)<0$, this interval should be understood as $[2\Sigma_{12}\mu_2(\delta),\Sigma_{12}\mu_2(\delta)]$.} Point 1.b) means that for essentially any value of $\Sigma_{12}$ and for small to moderately large values of $\mu_2(\delta)$, the CC is still larger than the UC if the standardized bias of $\widehat{\beta}$ belongs to $\Sigma_{12}\mu_2(\delta)\times[0,2]$. Finally, Point 1.c) means that for essentially any value of $\Sigma_{12}$ and any value of $\mu_2(\delta)$ for which the pre-test has less than 80% chances of being rejected, the CC is still larger than the UC if the standardized bias of $\widehat{\beta}$ belongs to $\Sigma_{12}\mu_2(\delta)\times[0.15,2]$. Thus, (ref) can still hold under fairly large departures from $\mu_1(\delta)=\Sigma_{12}\mu_2(\delta)$.
While (ref) holds uniformly when $\mu_1(\delta)=\Sigma_{12}\mu_2(\delta)$, (ref) does not: $\mu_1(\delta)=\Sigma_{12}\mu_2(\delta)$ is not sufficient to ensure that the CI's CC is least as large as its NC. Still, Point 2 of Numerical Result (ref) shows that if $p=q=1$ and $\alpha=0.05$, (ref) holds for all $|\mu_2(\delta)| \le 0.84$. Moreover, for all $|\mu_2(\delta)| \le 2.8$, the coverage rate of the 95%-level CI for $\beta_0$ cannot be lower than 88%. By contrast, the UC can be as low as 20% in such cases.
Point 1 of Numerical Result (ref) can be fruitfully applied to the AE example. There, rationalizing $\mu_1(\delta)=\Sigma_{12}\mu_2(\delta)$ requires in particular assuming that the treatment is uncorrelated to $Y_n(0)$ once the covariates in the balancing tests are controlled for. This is restrictive: controlling for $X_{b,n}$ may not be enough to ensure that the treatment is exogenous. We now allow for effects of omitted variables.
\paragraph{Motivation and set-up.} In this section, we compare the CC and UC of $\text{CI}_{0.95}$, in DGPs calibrated to the 12 DID studies in the meta-analysis conducted by roth2022pretest.\footnote{Those are all the event-study DID papers published in the AER, AEJ:Policy, and AEJ:Applied between 2014 and 2018, with publicly available datasets.} We choose DID rather than cross-sectional examples (e.g. RCT, IV, matching, or RDD), because in the DID case $\Sigma_{12}$ is likely to be positive (this holds for ten of the 12 studies we consider, see Column (2) of Table (ref) below), while $\mu_1(\delta)$ and $\mu_2(\delta)$ are likely to be of opposite signs (this for instance holds with differential monotone trends). Then, Theorem (ref) and Numerical Result (ref) are unlikely to apply, thus making DID more adversarial than these other examples for the comparison of the CC and UC. For each paper, we consider the event-study graph revisited by roth2022pretest, and restrict attention to the first pre-trend estimate and to the first event-study estimate, so that $p=q=1$. Then, we calibrate $\mu_2(\delta)$ to the t-statistic of the pre-trend and $\Sigma_{12}$ to the correlation between the pre-trend and the event-study estimator. Finally, we let $\mu_1(\delta)=-(\Sigma_{\theta}/\Sigma_{\beta})^{1/2}\mu_2(\delta)$, which holds if we have differential linear trends. We compute the CC and UC for these 12 values of $(\mu_1(\delta),\mu_2(\delta),\Sigma_{12})$, for $\alpha=0.05$. Note that setting $\mu_2(\delta)$ at the realized t-stat of the pre-trend is an adversarial choice: even if trends were perfectly parallel in the twelve papers, the twelve $\mu_2(\delta)$ would correspond to realizations of twelve standard normals, with some of them being mechanically large, thus leading to a low CC and UC.
\paragraph{Results.} Results are shown in Columns (1) to (5) of Table (ref) below. The pre-trend is significant at the 5% level in only one paper. Yet, violations of parallel trends calibrated to mostly insignificant pre-trend estimates can lead to significant size distortions: on average, UC and CC are respectively equal to 81.0% and 79.2%, far below the 95% NC of $\text{CI}_{0.95}$. At the same time, there is only one value of $(\mu_1(\delta),\mu_2(\delta),\Sigma_{12})$ for which CC is more than 10 percentage points smaller than UC. Among the remaining values, there are four for which CC is larger than UC and three for which CC is less than 1 percentage point smaller than UC. Thus, for those 12 values of $(\mu_1(\delta),\mu_2(\delta),\Sigma_{12})$, pre-testing rarely decreases much the coverage of $\text{CI}_{0.95}$. In Table (ref) in the appendix, we keep the same values of $\Sigma_{12}$, but we calibrate $\mu_2(\delta)$ to the lower and upper bounds of the 95% CI of the first pre-trend estimate, while still letting $\mu_1(\delta)=-(\Sigma_{\theta}/\Sigma_{\beta})^{1/2}\mu_2(\delta)$. Unsurprisingly, larger differential trends worsen the coverage of $\text{CI}_{0.95}$: across these 24 values of $(\mu_1(\delta),\mu_2(\delta),\Sigma_{12})$, the UC and CC are respectively equal to 53.8% and 49.7% on average. Now, there are two values of $(\mu_1(\delta),\mu_2(\delta),\Sigma_{12})$ for which CC is more than 10 percentage points smaller than UC, but for those the UC is lower than 70%, thus implying that $\text{CI}_{0.95}$ is unreliable even without pre-testing.
The conventional rationale for specification tests is that they enable the rejection of misspecified models, thereby increasing the likelihood that researchers estimate correctly specified ones. Such tests have nonetheless been criticized on at least two grounds. First, they may lack power in practice, and second, they may distort post-test inference. While the first concern remains valid and important, this paper offers a more optimistic view of the second: not only does conditional inference remain valid under the null of correct specification, it may even be less distorted than unconditional inference under local alternatives. Overall, the “cost” of running specification tests —such as conservative inference under the null, the possibility of wrongly rejecting some true specifications, and potentially more distorted inference under certain local alternatives — may be relatively limited compared to their benefits, namely a higher likelihood of selecting correct specifications and, in some cases, less distorted inference under local alternatives.
This tentative conclusion should however be taken with a grain of salt, given a notable limitation of our current work. Specifically, we assume that researchers consider a single identifying assumption, and we do not take a stance as to what happens when that assumption is rejected. In such cases, researchers may pre-test another identifying assumption until they find one that is not rejected. For instance, if a pre-trend test is rejected in a DID study, one could add control variables and assess if those are sufficient to remove the pre-trend. The results in our paper do not apply to such sequential pre-testing procedures. Investigating the consequences of sequential pre-tests for inference is an important avenue for future research.
Finally, our results also have perhaps unexpected implications for practitioners. For instance, inasmuch as one favors some form of protection against misspecification over valid inference under the null, one may prefer using a non-optimal rather than optimal GMM estimator. Also, our “global” and numerical results under local alternatives highlight that pre-testing may be more contentious in some contexts (e.g., DIDs) than in others (e.g., randomized experiments, IV studies). As these results require specific relationships between the two asymptotic biases and the asymptotic correlation of the test statistic and the estimator, they also imply that some specification tests (e.g., a single test on a summary index of the potential bias generated by covariate imbalances in Example AE) may be more desirable than others (e.g., a joint test that all covariates are balanced). Again, we leave for future research the question of how to best choose the specification test from this perspective.