Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
98,409 characters · 15 sections · 117 citation commands
Specification Tests for the Propensity Score
The propensity score, which is defined as the conditional probability of receiving treatment given covariates, is one of the most widely used tools for causal inference. Part of its popularity can be credited to the seminal result of Rosenbaum1983: if the treatment assignment is independent of the potential outcomes conditional on a vector of covariates, then one can obtain unbiased and consistent estimators of different treatment effect measures by adjusting for the propensity score alone, greatly reducing the dimensionality of the underlying problem. Several methods that exploit this important insight are now an essential part of the applied researcher's toolkit. Examples include matching, see e.g. Rosenbaum1985, Heckman1997 and Abadie2016; inverse probability weighting (IPW), see e.g. Rosenbaum1987a, Hirano2003 and Donald2013; regression methods, see e.g. Hahn1998 and Firpo2007; and many others. For literature reviews, see Heckman2007 and Imbens2009.
Despite their popularity, a main concern of these methods is that the propensity score is usually unknown, and therefore has to be estimated. Given the high dimensionality of available covariates, researchers are usually coerced to adopt a parametric model for the propensity score since nonparametric estimation methods suffer from the \textquotedblleft curse of dimensionality\textquotedblright , implying that the resulting treatment effect estimators can have considerably poor properties, even for large sample sizes. Such a common practice raises the important issue of model misspecification. Indeed, as shown by Frolich2004, Millimet2009 , Huber2013 and Busso2014, propensity score misspecifications can lead to misleading treatment effect estimates.
In this paper we propose new specification tests for parametric propensity score models. Our proposal builds on the common practice of comparing the density of the propensity score between treated and control groups to determine the covariate overlap region, see e.g. Heckman1998. However, instead of comparing conditional densities, we focus on comparing conditional cumulative distribution functions (CDFs). In particular, we derive a restriction between the propensity score CDFs among treated and control groups that gives information on overlapping\footnote{ We thank an anonymous referee for making this suggestion.}, show that such a restriction is equivalent to a particular infinite number of unconditional moment conditions, and develop tests based upon it.
In contrast to existing proposals, our tests are fully data-driven, do not require user-chosen tuning parameters such as bandwidths, and are able to detect a broad class of local alternatives converging to the null at the parametric rate $n^{-1/2}$, with $n$ the sample size. Furthermore, our tests do not suffer from the \textquotedblleft curse of dimensionality\textquotedblright\ when the vector of covariates is of high dimensionality, and have greater power than competing tests for many alternatives. Of course, such power gains do not come without a cost: there exist some classes of alternative hypotheses against which our tests have trivial power. Nonetheless, we believe that such a compromise is reasonable since, as pointed out by Janssen2000 and Escanciano2009a, achieving reasonable power over all possible directions seems hopeless.
The proposal closest to ours is Shaikh2009. Despite using a similar characterization of the null hypothesis as Shaikh2009, our proposal greatly differs from theirs. Whereas Shaikh2009 adopts the local smoothing approach, see e.g. Hardle1993, Zheng1996, Fan1996 and Li1998, we adopt the integrated conditional moment (ICM) approach, see e.g. Bierens1982, Bierens1990, Bierens1997 , Stute1997 and Escanciano2006b. As a consequence, our approach inherits some advantages of ICM when compared to Shaikh2009. First, our tests do not require delicate bandwidth choice, unlike Shaikh2009's test whose performance can be sensitive to it. Second, in contrast with Shaikh2009, our approach has power against local alternatives converging to the null at the parametric rate.
Another popular procedure to assess misspecification of the propensity score model is to use \textquotedblleft balancing\textquotedblright\ tests. Initially proposed by Rosenbaum1985, these tests consist of assessing if each covariate is independent of the treatment assignment, conditional on the propensity score. This is often implemented examining whether moments (usually just the mean) of the observable characteristics between the two \textquotedblleft matched\textquotedblright\ or \textquotedblleft weighted\textquotedblright\ groups are the same; see e.g. Dehejia2002 and Smith2005. One should bear in mind that because \textquotedblleft balancing\textquotedblright\ tests are usually based on a finite number of orthogonality conditions, there are uncountably many directions of misspecification that cannot be detected with these tests. Furthermore, as shown by Lee2013a, balancing tests may have size distortions due to the \textquotedblleft multiple testing problem\textquotedblright , the failure to account for the estimation effect of the propensity score, and poor covariate overlap. Such drawbacks put at stake the reliability of many of these procedures. Our proposal, on the other hand, does not suffer from these.
Our paper also contributes to the literature on ICM tests. What appears distinctive to our approach is that $(i)$ we exploit the dimension-reduction coming from our derived restriction between propensity score CDFs, and $ \left( ii\right) $ we acknowledge our lack of knowledge of the \textquotedblleft true\textquotedblright\ correct specification of the propensity score, by means of an orthogonal projection onto the tangent space of nuisance parameters. The result of $(i)$ and $(ii)$ is a test with improved power properties, and with a simple bootstrap implementation. The power improvement due to the dimension reduction has been noticed by Stute2002, Escanciano2006b and Shaikh2009, whereas the power improvement due to the use of orthogonal projections has been noticed in different contexts, see e.g. Neyman1959, and more recently, Bickel2006 and Escanciano2014a. To the best of our knowledge, our proposal is the first to incorporate both procedures.
As mentioned above, our paper is related to the relatively scarce literature on projection-based specification tests, see e.g. Escanciano2009simple and Escanciano2014a for two notable exceptions. Escanciano2009simple proposes a simple bootstrap testing procedure for conditional moment restrictions that acknowledges specifically the fact that the nuisance parameters are unknown and introduces the projection methodology, while Escanciano2014a propose a projection-based testing procedure for linear quantile regression using a related projection weight function. Our proposal builds on these papers, with the important difference that our test statistics also exploit the dimension-reduction property of the propensity score.
The rest of the paper is organized as follows. In Section (ref) we present the testing framework and derive the restriction upon which our tests are based. The asymptotic properties of our tests are established in Section (ref). We next examine the finite sample properties of our tests by means of a Monte Carlo study in Section (ref). We provide an empirical illustration of our procedures in Section (ref). Section (ref) concludes. Mathematical proofs are gathered in an appendix at the end of the article.
Finally, all proposed tests discussed in this article can be implemented via open-source R package pstest, which is freely available from GitHub (\url{https://github.com/pedrohcgs/pstest}).
Let $D$ be a binary random variable that indicates participation in the program, i.e. $D=1$ if the individual participates in the treatment and $D=0$ otherwise. Define $Y\left( 1\right) $ and $Y\left( 0\right) $ as the potential outcomes under treatment and control, respectively. The realized outcome of interest is $Y=DY\left( 1\right) +\left( 1-D\right) Y\left( 0\right) $, and $X$\ is an observable $d\times 1$ vector of pre-treatment covariates.\ Denote the support of $X$ by $\mathcal{X\subseteq }\mathbb{R}^{d}$ and the propensity score $p\left( x\right) =\mathbb{P} \left( D=1|X=x\right) $. We have a random sample $\left\{ \left( Y_{i},D_{i},X_{i}^{\prime }\right) ^{\prime }\right\} _{i=1}^{n}$ of size $ n\geq 1$ from $\left( Y,D,X^{\prime }\right) ^{\prime }$. Throughout the rest of this article, all random variables are defined on a common probability space $\left( \Omega ,\mathcal{A},\mathbb{P}\right) .$
The main goal in causal inference is to assess the effect of a treatment $D$ on the outcome of interest $Y$. The most popular parameters of interest include the average treatment effect, $ATE=\mathbb{E}\left[ Y\left( 1\right) -Y\left( 0\right) \right] $, and the average treatment effect on the treated, $ATT=\mathbb{E}\left[ Y\left( 1\right) -Y\left( 0\right) |D=1\right] $. Note that such parameters of interest depend on potential outcomes $ Y\left( 1\right) $ and $Y\left( 0\right) $ which cannot be jointly observed for the same individual, precluding estimating $ATE$ and $ATT$ using their sample analogues. One of the most popular identification strategies in policy evaluation that resolves such difficulty is to assume that selection into treatment is solely based on observable characteristics, the so-called unconfoundedness setup, see e.g. Rosenbaum1983. Formally, the unconfoundedness setup requires the following assumptions:
As shown by Rosenbaum1987a, under Assumptions (ref)-(ref), $ATE$ and $ATT$ are identified by
respectively. This result motivates the two-step procedure in which one first estimates the propensity score, computes its estimated values $\hat{p} \left( X_i\right) $, and then uses the analogy principle to estimate $ATE$ and $ATT$, that is,
Alternatively, one could estimate $ATE$ and $ATT$ using propensity score matching, see e.g. Rosenbaum1983, Heckman1998a and Abadie2016.
In order to ensure that such estimators are well-defined and stable, it is important to assess the overlap between the distribution of the propensity score among treatment and control groups, i.e. to check whether the propensity score is bounded away from zero and one, and if the support of the propensity score in both groups are nearly the same, see e.g. Heckman1998, Smith2005, Crump2009 and Khan2010. Following Heckman1998, it is now routine to compare kernel density estimates of the propensity score among treated and control samples to determine the common support region. In cases where there is strong overlap one proceeds as described above, otherwise, one usually considers trimmed samples, see e.g. Crump2009 and Sasaki2018.
Although kernel density estimators are popular, they involve choosing tuning parameters such as bandwidths and often suffer from boundary bias. Of course such inconveniences can be easily avoided if one focuses on CDFs instead of densities. In the following we show that propensity score overlap implies a particular set of restrictions between the CDFs of treated and control groups, and that these restrictions can form the basis for testing the correct specification of propensity score models.
Assume that the propensity score $p\left( X\right) $ has a density with respect to a dominating measure, and that the density is bounded away from zero and infinity uniformly over its support. The following lemma builds on Shaikh2009 and formalizes the above discussion.
Lemma (ref) implies that, when the propensity score is correctly specified, one can expect that the sample analogue of ((ref)) should hold. Thus, ((ref)) provides a graphical diagnostic tool for propensity score misspecification; see Lemma 3.2 of Sloczynski2018 for a result related to ((ref)). Perhaps more importantly, note that ((ref)) provides an infinite number of simple unconditional moment restrictions that can be used to formally test whether a parametric model for the propensity score is correctly specified or not.
Motivated from Lemma (ref), we seek to test whether a parametric putative model for $p\left( x\right) $ is correctly specified based on
where $\Theta \subset \mathbb{R}^{k}$, $\Pi=\left[ 0,1\right]$ is the unit interval, and $q\left( X,\theta \right) :$ $\mathcal{X\times }\Theta \mapsto \left[ 0,1\right] $ is a family of parametric functions known up to the finite dimensional parameter $\theta $. Common specifications for $q\left( X,\theta \right) $ in empirical applications are the Probit, $\Phi \left( X^{\prime }\theta \right) ,$ and the Logit, $\Lambda \left( X^{\prime }\theta \right) $, where $\Phi \left( \cdot \right) $ and $\Lambda \left( \cdot \right) $ are the normal and logistic link functions, respectively.
Note that ((ref))\ can be equivalently written as
see e.g. Stute1997\footnote{ Alternative representations of $H_{0}$ are also possible, see e.g. Bierens1997, Escanciano2006a and a previous version of this article. }. Thus, in order to assess $H_{0}$, one can either use the infinite number of unconditional moment restrictions in ((ref)), or the conditional moment restriction in ((ref)). In this article we use ((ref)) whereas Shaikh2009 exploits ((ref)). More precisely, Shaikh2009 consider a test statistic based on
where $\varepsilon _{i}\left( \hat{\theta}_{n}\right) =D_{i}-q\left( X_{i}, \hat{\theta}_{n}\right) $, $\hat{\theta}_{n}$ is a $\sqrt{n}-$consistent estimator of $\theta _{0}$ under $H_{0}$, $h_{n}$ is a positive scalar bandwidth parameter converging to zero at a suitable rate as $n\rightarrow \infty $, and $K\left( \cdot \right) $ is a kernel function. Note that, in addition to the estimation of $\theta _{0}$ under $H_{0}$, Shaikh2009 's procedure requires local smoothing of the data, implying that its finite sample properties rely on the adequate choice of the smoothing parameter $ h_{n}$, a task that is far from trivial in testing problems. Since our approach is based on ((ref)) and only involves unconditional expectations, our testing procedure is free of tuning parameters such as bandwidth sequence $h_{n}$.
Tests based on a continuum of unconditional moment restrictions such as ((ref)) fall into the ICM approach, see Gonzalez-Manteiga2013 for a review. Nonetheless, our tests have two main differences with respect to the standard ICM tests. First, ((ref)) depends on $X$ only through the propensity score model under $H_{0}$, a one-dimensional (though unknown) function. As a consequence, the ICM in ((ref)) is insensitive to the dimension $d$ of the explanatory variables $X$, avoiding the so-called \textquotedblleft curse of dimensionality\textquotedblright . Second, in contrast to the standard ICM tests, we explicitly acknowledge that $\theta _{0}$ is a nuisance parameter in testing ((ref)) by proposing to use orthogonal projections on the tangent space of nuisance parameters. As discussed in the Introduction, the use of orthogonal projections leads to important advantages. In the next subsection we describe how we construct such projection-based tests, paying particular attention to the role played by the orthogonal projection; see also Escanciano2009simple and Escanciano2014a for related results in different contexts.
Recall that $\varepsilon _{i}\left( \theta \right) =D_{i}-q\left( X_{i},\theta \right) .$ For all $u\in \Pi $, define
where $g(x,\theta )=\partial q(x,\theta )/\partial \theta$ is the score function of $q(x,\theta )$,
and
Given a random sample $\left\{ \left( D_{i},X_{i}^{\prime }\right) ^{\prime }\right\} _{i=1}^{n}$, our test statistics are based on continuous functionals of the projection-based empirical process $\hat{R}_{n}^{p}(u)$,
where $\hat{\theta}_{n}$ is a $\sqrt{n}-$consistent estimator for $\theta _{0}$ under $H_{0}$. Two popular examples of such functionals are the Cram \'{e}r-von Mises-type and Kolmogorov-Smirnov-type functionals,
respectively, where $F_{n}(u)=n^{-1}\sum_{i=1}^{n}1\left( q\left( X_{i},\hat{ \theta}_{n}\right) \leq u\right) $ is the empirical distribution function (EDF) of $q\left( X_{i},\hat{\theta}_{n}\right) $, $1\leq i\leq n.$
At this point, one may wonder why our test statistics are based on the empirical process $\hat{R}_{n}^{p}(u)$ ((ref)) instead of the usual sample analogue of ((ref)),
i.e. the \textquotedblleft unprojected\textquotedblright\ analogue of $\hat{R }_{n}^{p}(u)$. To answer such a query, note that under $H_{0}$ and some weak regularity conditions given in Section (ref), the unprojected process $\hat{R}_{n}(u)$ can be decomposed as
uniformly in $u\in \Pi $, see Lemma (ref) in the Appendix. The asymptotic representation in ((ref)) implies that the effect of replacing $\theta _{0}$ by $\hat{\theta}_{n}$ is non-negligible, and therefore the asymptotic null distributions of tests based on ((ref)) are sensitive to the estimator $\hat{\theta}_{n}$ being used. As a consequence, for a given parametric specification $p\left( x\right) =q(X,\theta _{0})$, the asymptotic null distributions of tests based on ((ref)) will depend on whether one estimates $\theta _{0}$ using maximum likelihood (ML), nonlinear least squares (NLS), or generalized method of moments (GMM), even though the underlying specification for the propensity score is the same across these estimation methods.
The projection-based process $\hat{R}_{n}^{p}(u)$, on the other hand, avoids such drawback since
almost everywhere in $u\in \Pi $, where
with
and
The intuition behind ((ref)) is simple. First, note that $\Delta ^{-1}\left( \theta \right) G\left( u,\theta \right) $ is the vector of linear projection coefficients of regressing $1\left\{ q(X,\theta )\leq u\right\} $ on $g(X,\theta )$. Thus, it follows that $g(X,\theta )^{\prime }\Delta ^{-1}\left( \theta \right) G\left( u,\theta \right) $ is the best linear predictor of $1\left\{ q(X,\theta )\leq u\right\} $ given $g(X,\theta )$, and that ((ref)) is nothing more than the associated projection error, which is by definition orthogonal to $g(X,\theta )$. As a consequence of ((ref)), it follows that under some weak regularity conditions, uniformly in $u\in \Pi $,
where
see Theorem (ref) in Section (ref). Thus, $\hat{R}_{n}^{p}(u)$ is (asymptotically) invariant to the choice of estimator $\hat{\theta}_{n}$. Furthermore, as we discuss in Section (ref), the above asymptotic representation of $\hat{R}_{n}^{p}(u)$ in terms of $R_{n0}^{p}(u)$ allows for a multiplier-type bootstrap procedure that greatly simplifies the computation of asymptotically valid critical values.
In summary, by focusing on the projection-based process $\hat{R}_{n}^{p}(u)$ instead of the more traditional process $\hat{R}_{n}(u),$ our proposed test statistics are $\left( a\right) $ invariant to the choice of estimator $\hat{ \theta}_{n}$, and $\left( b\right) $ allow for simplified bootstrap implementation. In addition, tests based on $\hat{R}_{n}^{p}(u)$ acknowledge that deviations in the direction of the score function $g(x,\theta)$ cannot be distinguished from deviations within the parametric model, and therefore do not \textquotedblleft waste\textquotedblright\ power in such directions. As a result, tests based on $\hat{R}_{n}^{p}(u$) can have higher power when compared to tests based on $\hat{R}_{n}(u)$, though in general none of them is strictly better than the other uniformly over the space of alternatives. We defer the discussion of these power properties to Section (ref).
In this section, we establish the asymptotic behavior of the projection-based empirical process $\hat{R}_{n}^{p}(u)$ under the null hypothesis $H_{0}$, under the fixed alternative hypothesis $H_{1}$, which is the negation of ((ref)), and under a sequence of local alternatives $ H_{1n}$ that converges to $H_{0}$ at the parametric rate $n^{-1/2}$, $n$ being the sample size. We also characterize classes of alternative hypotheses against which our tests have no power, and argue that such classes are rather exceptional. Finally, we show that critical values can be computed with the assistance of a multiplier-type bootstrap that is easy to implement.
The asymptotic null distributions of our tests are the limiting distributions of continuous functionals of $\hat{R}_{n}^{p}(u)$ under $H_{0}$ . To derive the asymptotic results, we adopt the following notation. For a generic set $\mathcal{G}$, let $l^{\infty }\left( \mathcal{G}\right) $ be the Banach space of all uniformly bounded real functions on $\mathcal{G}$, equipped with the uniform metric $\left\Vert f\right\Vert _{\mathcal{G} }\equiv \sup_{z\in \mathcal{G}}\left\vert f\left( z\right) \right\vert $. We study the weak convergence of $\hat{R}_{n}^{p}(u)$ and its related processes as elements of $l^{\infty }\left( \Pi \right) $, where $\Pi \equiv \left[ 0,1 \right] $. Let “$\Rightarrow $” denote weak convergence on $\left( l^{\infty }\left( \Pi \right) ,\mathcal{B}_{\infty }\right) $ in the sense of J. Hoffmann-J$\phi $rgensen, where $\mathcal{B}_{\infty }$ denotes the corresponding Borel $\sigma $-algebra - see e.g. Definition 1.3.3 in VanderVaart1996.
We assume the following regularity conditions. Let $\Theta _{0}$ be an arbitrarily small neighborhood around $\theta _{0}$ such that $\Theta _{0}\subset \Theta $. For any $d_1\times d_2$ matrix $A=(a_{ij})$, let $ ||A|| $ denote its Euclidean norm, i.e. $||A||=[\text{tr}(AA^{\prime})] ^{1/2}$.
Assumptions (ref)-(ref) are weaker than related conditions in the literature. For instance, Assumption (ref) only requires $\sqrt{n} \left( \hat{\theta}_{n}-\theta _{0}\right) =O_{p}\left( 1\right) ,$ but does not require $\sqrt{n}\left( \hat{\theta}_{n}-\theta _{0}\right) $ to admit an asymptotically linear representation. Assumption (ref) is a condition concerning the degree of smoothness of the propensity score $ q(x,\theta )$, and is satisfied for standard parametric models such as the Probit and the Logit specifications. It also only requires finite first moment of $g(X,\theta )$, instead of more than four moments as in Shaikh2009. Assumption (ref) simply imposes a Lipschitz type continuity condition on the CDF of the parametric propensity score.
Next, we derive the asymptotic behavior of the projection-based empirical process $\hat{R}_{n}^{p}(u)$ under $H_{0}$. We do this in two steps. First, we show that, under $H_{0}$, $\hat{R}_{n}^{p}(u)$ is asymptotically equivalent, with respect to the supremum norm on $\Pi $, to the process $ R_{n0}^{p}(u)$ given in ((ref)). From this result it follows that the weak convergence under $H_{0}$ of the process $\hat{R}_{n}^{p}(u)$ can be conveniently established from that of $R_{n0}^{p}(u)$. More importantly, the limiting null behavior of $\hat{R}_{n}^{p}(u)$ does not depend on $\hat{ \theta}_{n}$ nor how $\hat\theta_n$ is obtained.
Theorem (ref) and the continuous mapping theorem (CMT), see e.g. Theorem 1.3.6 in VanderVaart1996, yield the asymptotic null distributions of continuous functionals of $\hat{R}_{n}^{p}(u),$ including the test statistics $CvM_{n}$ and $KS_{n}$ given in ((ref)) and ((ref) ), respectively.
Note that the integrating measure in $CvM_{n}$ is a random measure, but Corollary (ref) shows that the asymptotic distribution is not affected by this fact. Further details can be found in the Appendix A.
Now, we investigate the power properties of tests based on continuous functionals $\Gamma (\hat{R}_{n}^{p})$, like $CvM_{n}$ and $KS_{n}$ in ((ref)) and ((ref)), respectively. We consider fixed alternatives, and a sequence of local alternatives $H_{1n}$ that converges to $H_{0}$ at the parametric rate $n^{-1/2}$.
Next theorem analyzes the asymptotic properties of our tests under fixed alternatives of the type
where $\Pi =[0,1]$ is the unit interval. Note that $H_{1}$ is simply the negation of $H_{0}$ in ((ref)).
From Theorem (ref), we see that test statistics of the form of $\Gamma ( \hat{R}_{n}^{p})$ are not consistent against all fixed alternative hypotheses in ((ref)), but only those not collinear to the score function $g(X,\theta_0)$. To see this, note that
is equal to zero under ((ref)) if $p\left( X\right) -q\left( X,\theta _{0}\right)$ and $g(X,\theta _{0})$ are collinear almost surely. We do not see this as a limitation. First, when one estimates $\theta _{0}$ using the NLS method, the population first order condition for $\theta _{0}$ sets $ \mathbb{E}\left[ \left( D-q\left( X,\theta _{0}\right) \right) g^{\prime }(X,\theta _{0})\right] =0,$ implying that, for some $u\in \Pi ,$
As a consequence, our projection-based tests would be consistent against all alternative hypotheses of the type of ((ref)), avoiding the aforementioned problem.
Second, and perhaps more importantly, even when one does not use NLS to estimate $\theta _{0},$ we argue that the lack of power against alternatives collinear to the score function $g(X,\theta _{0})$ is not a main concern. As shown by Escanciano2009a, every test based on ICM approach has trivial local power against these alternatives, and as a consequence, the global power of all ICM tests in the direction of the score function will also be low, see e.g. Strasser1990. In fact, instead of considering this as a limitation one may consider this property as a feature of our tests: by acknowledging that such alternatives cannot be powerfully detected, our projection-based test statistics do not waste power in such directions, and therefore may have higher power against other, perhaps more important, alternatives; see Section (ref) for an illustration.
The above discussion raises the question: are there other classes of fixed alternative hypotheses that our specification tests are not able to detect? As first pointed out by Shaikh2009, the answer is yes. Because our test statistics depend on $X$ only through the propensity score $q(X,\theta _{0})$, our tests will have trivial power against the class of misspecified propensity scores, where
in a set of positive probability, but
where $\theta _{0}$ now is the probability limit of $\hat{\theta}_{n}$. In other words, tests based on ((ref)) (or equivalently on ((ref))), will have trivial power against alternatives such that $p\left( X\right) $ is different from $q\left( X,\theta _{0}\right) $ but
The leading case of such a class of alternatives is when the propensity score is correctly specified for a subvector of $X$, but not for the entire vector $X$, see e.g. Shaikh2009. Given the nonlinear nature of $ p\left( X\right) $, one may consider such a class of alternatives rather exceptional. However, they can still arise in practice under some particular circumstances as we will illustrate below.
Consider the case where $X=\left( X_{1},X_{2}\right) $ and that both covariates are relevant for the propensity score, i.e., $p\left( X\right) =p\left( X_{1},X_{2}\right) $. Suppose that a researcher considers a Probit model for the $p\left( \cdot \right) $ but only included $X_{1}$ as a covariate, i.e., the researcher assume the model $q\left( X,\theta _{0}\right) =\Phi \left( \theta _{00}+\theta _{01}X_{1}\right) $, with $\Phi \left( \cdot \right) $ the normal link function. Of course, $p\left( X\right) \not=q\left( X,\theta _{0}\right) $ since $q\left( X,\theta _{0}\right) $ only includes a subset of relevant covariates. Thus, in light of ((ref)), our tests will have no power if there exists $\theta _{0}=\left( \theta _{00},\theta _{01}\right)^{\prime }\in \Theta \subset \mathbb{R}^2$ such that $\mathbb{E}\left[ p\left( X_{1},X_{2}\right) |X_{1} \right] =\Phi \left( \theta _{00}+\theta _{01}X_{1}\right) $ $a.s.$. The existence of such $\theta _{0}$ depends on the underlying conditional distribution of $X_{2}$ given $X_{1}$, and also on the form of true unknown propensity score $p\left( X_{1},X_{2}\right) $. For instance, assume that the true propensity score is a Probit, e.g., $p\left( X_{1},X_{2}\right) =\Phi \left( X_{1}+X_{2}\right) $, and that $X_{2}$ given $X_{1}$ follows a normal distribution with conditional mean $\mu \left( X_{1}\right) $ and conditional variance $\sigma ^{2}\left( X_{1}\right) $. In this case, we can show that
Thus, if $\mu \left( X_{1}\right) =a+bX_{1}$ and $\sigma^2 \left( X_{1}\right) =c$ for some constants $a,b$ and $c$ with $b\neq -1$ and $c>0$, we have that
Thus, in this particular case such $\theta _{0}$ does exist with $ \theta_0\equiv\left(a/\sqrt{1+c},(1+b)/\sqrt{1+c}\right)^{\prime }$ and our tests would have trivial power. On the other hand, if $\mu \left( X_{1}\right) $ is nonlinear in $X_{1}$ and $\sigma ^{2}\left( X_{1}\right) $ is a nontrivial function of $X_{1}$, no such $\theta _{0}$ exists and therefore ((ref)) is ruled out. Thus, it is clear that the conditional distribution of $X_{2}$ given $X_{1}$ plays an important role in the \textquotedblleft empirical relevance\textquotedblright\ of alternative hypotheses like ((ref)).
The role played by the functional form of the true propensity score in the \textquotedblleft empirical relevance\textquotedblright\ of ((ref)) can be illustrated by assuming that the true propensity score is a Logit instead of a Probit, e.g., $p\left( X_{1},X_{2}\right) =\Lambda \left( X_{1}+X_{2}\right) $, with $\Lambda(\cdot)$ the logistic link function. In this case, because of the nonlinear nature of $\Lambda $, there exists no $ \theta _{0}\in \Theta $ such that $\mathbb{E}\left[ \Lambda \left( X_{1}+X_{2}\right) |X_{1}\right] =\Phi \left( \theta _{01}+\theta _{01}X_{1}\right) $ $a.s.$, ruling out ((ref))\footnote{ In practice, however, the power against this particular alternative can be low since the Logit and Probit specifications are relatively \textquotedblleft close\textquotedblright\ to each other.}.
In summary, the above discussion shows that our projection-based tests are consistent against a broad range of alternatives, though not all. This is the main drawback of our proposal when compared to the standard ICM specification tests. However, in our simulations that follow we show that, for the alternatives considered, our projection-based tests are the best or comparable to the best tests in terms of power in finite samples. Thus, from a practical point of view, the benefits of using our procedure can outweigh the costs in many relevant situations.
Next, we study the performance of our projection-based tests under a sequence of local alternative hypotheses converging to the null at the parametric rate $n^{-1/2}$ given by
for some $\theta _{0}\in \Theta $, where $r\left(q(X,\theta _{0})\right)$ represents directions of departure from $H_0$, and $n^{-1/2}$ indicates the rate of convergence of $H_{1n}$ to $H_0$. The function $r:\left[ 0,1\right] \rightarrow \mathbb{R}$ is required to satisfy the following assumption.
Note that, in general, the deterministic shift function $\Delta _{r}\left( u\right) \not=0$ for at least some $u\in \Pi $, implying that tests based on continuous even functionals of $\hat{R}_{n}^{p}(\cdot )$ will have non-trivial power against local alternatives of the form in ((ref)). A situation in which our tests will have trivial local power against such alternatives is when directions $r(q\left(x,\theta _{0}\right) )$ are a linear combination of score function $g(x,\theta _{0})$, i.e. $r(q(x,\theta _{0}))=\beta g(x,\theta _{0})$ for some $\beta $. In such a case, the limiting distribution of $\hat R_n^p(u)$ under $H_0$ and $H_{1n}$ is the same so that $H_{1n}$ cannot be detected. On the other hand, note that tests based on the local smoothing approach such as Shaikh2009 are not able to detect alternatives of the form ((ref)).
From the above theorems, we see that the asymptotic distribution of continuous functionals $\Gamma \left( \hat{R}_{n}^{p}\right) $ depend on the underlying data generating process and of course on $\Gamma \left( \cdot \right) $ itself. Furthermore, the complicated covariance structure of $ K^{p}(\cdot ,\cdot )$ given in ((ref)) does not allow for a simple representation of $R_{\infty }^{p}$ in terms of a well-known distribution-free Gaussian process for which critical values are readily available. To overcome this problem, we propose to compute critical values with the assistance of a multiplier bootstrap. The proposed procedure has good theoretical and empirical properties, is computationally easy to implement, and does not require computing new parameter estimates at each bootstrap replication.
More precisely, in order to estimate the critical values, we propose to approximate the asymptotic behavior of $\hat{R}_{n}^{p}(u)$ by that of
where $\{V_{i}\}_{i=1}^{n}$ is a sequence of $i.i.d.$ random variables with zero mean, unit variance and bounded support, independent of the original sample $\{(D_{i},X_{i}^{\prime })^{\prime }\}_{i=1}^{n}$. A popular example involves $i.i.d.$ Bernoulli variates $\left\{ V_{i}\right\} $ with $\mathbb{P }\left( V=1-\kappa \right) =\kappa /\sqrt{5}$ and $\mathbb{P}\left( V=\kappa \right) =1-\kappa /\sqrt{5}$, where $\kappa =\left( \sqrt{5}+1\right) /2,$ as suggested by Mammen1993.
With $\hat{R}_{n}^{p\ast }(u)$ at hands, the bootstrapped version of our test statistics $\Gamma \left( \hat{R}_{n}^{p}\right) $ is simply given by $ \Gamma \left( \hat{R}_{n}^{p,\ast }\right) $. For instance, the bootstrapped versions of $CvM_{n}$ and $KS_{n}$ in ((ref)) and ((ref)), respectively, are given by
The asymptotic critical values are then estimated by
where $\mathbb{P}_{n}^{^{\ast }}$ means bootstrap probability, i.e. conditional on the original sample $\{(D_{i},X_{i}^{\prime })^{\prime }\}_{i=1}^{n}$. In practice, $c_{n,\alpha }^{^{\Gamma ~\ast }}$ is approximated as accurately as desired by $\left( \Gamma \left( \hat{R} _{n}^{p,\ast }\right) \right) _{B(1-\alpha )}$, the $B\left( 1-\alpha \right)-$th order statistic from $B$ replicates $\left\{ \left(\Gamma \left( \hat{R}_{n}^{p,\ast }\right)\right)_l \right\} _{l=1}^{B}$ of $\Gamma \left( \hat{R}_{n}^{p,\ast }\right) $.
The next theorem establishes the asymptotic validity of the multiplier bootstrap procedure proposed above.
In this section, we conduct a series of Monte Carlo experiments in order to study the finite sample properties of our proposed projection-based tests. In particular, we compare our Cram\'{e}r-von Mises and Kolmogorov-Smirnov tests $CvM_{n}$ and $KS_{n}$ given in ((ref)) and ((ref)) to $(i)$ the Shaikh2009's test,
where $\hat{V}_{n}\left( h_{n}\right) $ is given in ((ref)) and
the analogues of $CvM_{n}$ and $KS_{n}$ based on either $\left( ii\right) $ the unprojected process $\hat{R}_{n}(u)$ given in ((ref))\footnote{ For conciseness, we do not discuss the asymptotic properties of tests based on the unprojected empirical process $\hat{R}_{n}(u)$ in the main text. However, most of the asymptotic properties follow from arguments analogous to those we used to study $\hat{R}_{n}^{p}(u)$.}, or on $\left( iii\right) $ the traditional empirical process
We also compare our proposal with $\left( iv\right) $ balancing tests based on the normalized IPW estimators
where $w_{1,i}=D_{i}/q\left( X_{i},\hat{\theta}_{n}\right) $, $ w_{0,i}=\left( 1-D_{i}\right) /\left( 1-q\left( X_{i},\hat{\theta} _{n}\right) \right) $, $\bar{w}_{d,n}$ is the sample mean of $w_{d,i},$ $ d=\left\{ 0,1\right\} $, and $X^{j}$ is the $j$-th element of $d$ -dimensional vector $X=\left( X_{1},\dots ,X_{d}\right) ^{\prime }$. We consider two different test statistics for the balancing tests: the Wald test, and the maximum of the $d$ marginal two-sided $t$-tests.
Critical values (CVs) for the $CvM_{n}$ and $KS_{n}$ tests are obtained using the multiplier bootstrap procedure described in Section (ref), whereas for the ICM tests in $\left( ii\right) $ and $\left( iii\right) $ we use the bootstrap procedure described in Stute1998b. For $T_{n}\left( h_{n}\right)$ test, we use one-sided CVs from the standard normal distribution. CVs for the Wald test are from the chi-squared distribution with $d$ degrees of freedom. For the $t$-test, we consider Bonferroni corrected CVs based on the standard normal distribution. Note that the Bonferroni correction is necessary to address the multiple testing problem.
We consider sample sizes $n$ equal to $100$, $200$, $400$, $600$, $800$ and $ 1,000$. For each design, we consider $1,000$ Monte Carlo experiments. The $ \left\{ V_{i}\right\} _{i=}^{n}$ used in the bootstrap implementations are independently generated as $V$ with $\mathbb{P}\left( V=1-\kappa \right) =\kappa /\sqrt{5}$ and $\mathbb{P}\left( V=\kappa \right) =1-\kappa /\sqrt{5} $, where $\kappa =\left( \sqrt{5}+1\right) /2$, as proposed by Mammen1993. The bootstrapped critical values are approximated using $B=999$ bootstrap replications. To compute Shaikh2009's test, we use the standard normal kernel
Following Shaikh2009, the bandwidth sequence $h_{n}$ is chosen to be equal to $cn^{-1/8}$ for $c$ equal to 0.01, 0.05, 0.10 and 0.15. We choose different $c$'s to assess how sensitive Shaikh2009's test may be with respect to the bandwidth $h_{n}$.
We first consider the following data generating processes (DGPs):
For each of these five DGPs, $D=1\left\{ D^{\ast }>0\right\} ,\varepsilon \protect\mathpalette{\protect\independentT}{\perp} \left( X_{1},X_{2}\right) $, where $X_{1}=Z_{1}$, $X_{2}=\left( Z_{1}+Z_{2}\right) /\sqrt{2}$, and $Z_{1}$, $Z_{2},$ and $\varepsilon $ are independent standard normal random variables. All the DGPs considered have the propensity score bounded away from zero and one. Finally, for each of these DGPs we consider the potential outcomes
where $m_{1}\left( X\right) =1+X_{1}+X_{2}$, $u\left( 1\right) $ and $ u\left( 0\right) $ are independent normal random variables with mean zero and variance $0.1$. The observed outcome is $Y=DY\left( 1\right) +\left( 1-D\right) Y\left( 0\right) $, and the true $ATE$ is 1. Although these outcome equations are not necessary to assess the size and power properties of the tests, they can be used to assess the utility of our proposed tests to distinguish between \textquotedblleft good\textquotedblright\ and \textquotedblleft bad\textquotedblright\ estimates of the $ATE$.
Let $X=\left( 1,X_{1},X_{2}\right) ^{\prime }$. For $DGP1-DGP5$, the $H_{0}$ considered is
where $\Phi (\cdot )$ is the cumulative distribution function (CDF) of the standard normal distribution. We estimate $\theta _{0}$ using the Probit ML, i.e.
Clearly, $DGP1$ falls under $H_{0}$, whereas $DGP2-DGP5$ fall under $H_{1}$. Note that $D$ follows a heteroskedastic Probit model in $DGP4$ and $DGP5$.
The simulation results are presented in Table (ref). We report empirical rejection frequencies at the $5\%$ significance level. Results for 10% and 1% significance levels are similar and are available upon request. We also report the bias of the normalized IPW estimator
and the average length and coverage of its estimated 95% confidence interval based on its asymptotically normal approximation (assuming that the propensity score model is correctly specified).
We first analyze the size of our test. From the results of $DGP1$, we find that the actual finite sample size of both $KS_{n}$ and $CvM_{n}$ tests is close to their nominal size, even when the sample size is as small as 100. The same holds for the other ICM-type tests. On the other hand, we find that Shaikh2009's test is, in general, conservative, and sensitive to the choice of bandwidth. For instance, when $c=0.15$, the empirical size is close to zero even with $n=1,000$. On the other hand, with $c=0.01$, the empirical size of Shaikh2009's test is closer to the nominal value. In terms of traditional balancing tests, we find that Wald test and Bonferroni-corrected $t$-test do not control size. Such a drawback is due to the random denominator being relatively close to zero: when we trim observations with estimated propensity score outside the $[0.05,0.95]$ range, we find that classical balancing tests can control size. We report these results in Table (ref) in the Appendix B. Finally, note that when the propensity score is correctly specified, the bias of the $ \widehat{ATE_n}$ estimator in ((ref)) is small, the length of the 95% confidence interval reduces as sample size increases, but the coverage probability is smaller than its nominal value even when $n=1,000$. However, as we show in Table (ref) in the Appendix B, such undercoverage disappears when we trim observations with extreme estimated propensity scores.
\afterpage{
}
Note that when the propensity score is misspecified in $DGP2-DGP5$, the $ATE$ estimator ((ref)) can be severely biased, and its confidence interval can be \textquotedblleft too small\textquotedblright\ when the propensity score is misspecified, leading to severe undercoverage\footnote{ In $DGP3$, the bias, confidence interval length, and coverage of the $ATE$ estimator are good. However, it is important to have in mind that we consider only one particular DGP for the $ATE$, and that such \textquotedblleft robust\textquotedblright\ results may not translate to other DGPs.}. Thus, detecting propensity score misspecifications can prevent misleading inference about $ATE$. Our proposed $KS_{n}$ and $CvM_{n}$ tests perform admirably well in such a task, particularly in moderate sample sizes. In these scenarios, $CvM_{n}$ performs slightly better than $KS_{n}$. Looking at the results from Shaikh2009's test, we note that bandwidth choices can play an important role, and the choice of the \textquotedblleft best\textquotedblright\ bandwidth $h_n$ via $c$ varies across DGPs. Perhaps, what is more important to emphasize in terms of power is that in all alternative hypotheses and sample sizes analyzed, our projection based tests have higher power than Shaikh2009's test, regardless of the bandwidth choice. Our proposed tests also dominate the balancing tests, as balancing tests have little to no power in all DGPs considered but $DGP2$, even with $ n=1,000$. Finally, the results in Table (ref) show that projection-based tests perform either better or as well as the other ICM tests in the DGPs considered. Among the considered DGPs, $DGP3$ is the only one where some existing specification testing has higher power than our proposed procedure. For this particular DGP, ICM type tests $KS^{trad}_n$ and $CvM^{trad}_n$ based on ((ref)) have higher power than our proposed tests when sample size $n$ is large. Given the discussion in Section (ref) and the fact that none of the ICM tests are strictly better than the others uniformly over the space of alternatives, such a finding does not come with a surprise.
In this simulation, we push forward the dimensionality of the covariates to see how our proposed tests and the other alternative tests perform in scenarios with 10 continuous covariates. To investigate further this issue, we consider the following five DGPs:
where $X_{1}$, $X_{2}$ and $\varepsilon $ are defined as before, $\left\{ X_{i}\right\} _{k=3}^{10}$ are independent standard normal random variables, $D=1\left\{ D^{\ast }>0\right\} ,$ and $\varepsilon \protect\mathpalette{\protect\independentT}{\perp} X$, with $X=\left( 1,X_{1},X_{2},\dots ,X_{10}\right) ^{\prime }$. For each of these DGPs we consider the potential outcomes
where $m_{2}\left( X\right) =1+\sum_{j=1}^{10}X_{j}$, $u\left( 1\right) $ and $u\left( 0\right) $ are independent normal random variables with mean zero and variance $0.1$. The observed outcome is $Y=DY\left( 1\right) +\left( 1-D\right) Y\left( 0\right) $, and the true $ATE$ is 1.
For $DGP6-DGP10$, the $H_{0}$ considered is
We estimate $\theta _{0}$ by ML$.$ Note that $DGP6$ falls under $H_{0},$ whereas $DGP7$-$DGP10$ fall under $H_{1}$. The simulation results for $DGP6$- $DGP10$ are presented in Table (ref).
As before, we first discuss the size properties of the tests. From the results of $DGP6$, we find that $KS_{n}$ and $CvM_{n}$ tests are oversized when $n$ $=100$, but as sample size $n$ increases, the empirical size gets closer to its nominal value. The same holds true for ICM tests based on the unprojected process ((ref)). Shaikh2009's test tends to be conservative (with the exception when $c=0.01),$ and sensitive to the bandwidth choice. The traditional balancing tests, and the ICM tests based on ((ref)) are conservative, reflecting the \textquotedblleft curse of dimensionality\textquotedblright . Finally, note that when the propensity score is correctly specified, the finite sample properties of the $ATE$ estimator ((ref)) are good: the bias and the length of the 95% confidence interval get smaller when sample size increases, and the coverage probability is relatively close to its nominal value. When one trims observations with extreme estimated propensity score, these properties further improve; see Table (ref) in the Appendix B.
Note that when the propensity score is misspecified, the $ATE$ estimator ( (ref)) can be severely biased, and inference can be unreliable. Thus, tests with higher power to detect such misspecifications can prevent one to make misleading conclusions about the effectiveness of a given policy. What is clear from Table (ref) is that, regardless of the sample size and bandwidth considered, Shaikh2009's test seems to have little to no power to detect the alternatives described in $DGP7$, $DGP9$, and $DGP10$ . For $DGP8$, the maximum power for their test is approximately 55% when $ n=1,000$ and $c=0.1.$ However, with $c=0.01,$ the power of Shaikh2009 's test reduces to approximately 30%, highlighting again how important (and non-trivial) is to \textquotedblleft appropriately\textquotedblright\ choose the bandwidth. In sharp contrast with Shaikh2009's test, note that for moderately sized samples, our proposed $KS_{n}$ and $CvM_{n}$ tests have non-trivial power to detect all the alternatives. Our projection-based tests seem to dominate the other tests in the scenarios considered. Note that the traditional balancing tests have no power to detect the alternatives considered. Finally, ICM tests based on the traditional empirical process ( (ref)) have substantially less power than our proposed tests, reflecting the cost of the \textquotedblleft curse of dimensionality\textquotedblright . The power gains from using the projection-based process ((ref)) instead of the unprojected process ((ref)) can also be noted.
\afterpage{
}
Overall, the simulation results highlight that the proposed projection-based tests perform favorably compared to other alternative testing procedures in terms of size and power. Importantly, the simulations illustrate that the gains in power can be credited to each distinguished feature of our tests, that is, $\left( i\right) $ the avoidance of smoothing parameters by using the ICM approach, $\left( ii\right) $ the dimension-reduction coming from considering $1\left\{ q\left( X,\theta _{0}\right) \leq u\right\} $ instead of $1\left\{ X\leq u\right\} $, and $\left( iii\right) $ the use of orthogonal projections. Given these attractive features, we believe that our tests can be of great use in practice.
In this section, we provide an empirical illustration of our testing procedure. We revisit the analysis of Frankel2005 and Millimet2009, and study the effect of trade on the environment. More specifically, following Millimet2009, we assess the effect of a country being a member of the General Agreement on Tariffs and Trade (GATT) or World Trade Organization (WTO) on 5 different measures of environmental quality: per capita $CO_{2}$ emissions, average annual deforestation rate for 1990-1996, energy depletion, rural access to clear water and urban access to clear water.
As Millimet2009, we use three different covariates to model the probability of being a GATT/WTO member: real per capita GDP, land area per capita, and polity, which is a measure of how democratic (versus autocratic) is the structure of the government. The motivation to include these three covariates is to increase the plausibility of Assumption (ref) as discussed by Frankel2005. GDP per capita is associated with the probability of being member of the GATT/WTO and at the same time may have effects on different measures of environment quality, for instance, via the environmental Kuznetz curve. Land area per capita is another potential confounder since higher population density may lead to environmental degradation and \textquotedblleft larger\textquotedblright\ countries are more likely to trade more, affecting the probability of being member of he GATT/WTO. Finally, as noted by Frankel2005, low-democracy countries tend to have lower measures of environmental quality, and can also confound the effect of GATT/WTO membership. In what follows and in the same spirit of Millimet2009, we assume that Assumption (ref) holds after controlling for these three confounding factors\footnote{ If one finds the plausibility of this assumption rather low, all the estimates presented below should be interpreted as associations/correlations and not as causal effects. In light of Remark (ref), however, Assumption (ref) plays no role in our specification tests.}.
The unbalanced country-level panel data we use follows from Millimet2009, and includes observations from 1990 (before the WTO) and 1995 (after the WTO). However, it is important to have in mind that treatment is defined as being a GATT/WTO member, and therefore there are countries who were treated in both times, and others who were treated only in 1995. Table (ref) provides summary statistics and more detailed description of the variables. Finally, we highlight that the data we analyze is from an unbalanced country-level panel and instead of only considering the \textquotedblleft always observed\textquotedblright\ countries, we follow Millimet2009 and run a separate analysis for each outcome. For further details, see Section 4.1 of Millimet2009.
The main goal of this section is to assess the \textquotedblleft reliability\textquotedblright\ of different treatment effect measures by analyzing if different propensity score models are correctly specified or not. Given that the sample constitutes of an unbalanced panel, we estimate separate propensity score models for each outcome subsample. More specifically, for each outcome, we model the probability of a country being a GATT/WTO member ($D=1$ if member, $D=0$ otherwise) by a standard Probit model and consider two different specifications:
Spec1: $X$ includes real per capita GDP, land area per capita, and polity.
Spec2: $X$ is defined as in Spec1 but adds pairwise interaction terms between each covariate.
For each of these specifications, we test the null hypothesis
against $H_{1}$, which is simply the negation of $H_{0}$. Table (ref) reports the testing results for each specification, together with normalized IPW estimator for the $ATE$ based on ((ref)). We also consider the normalized IPW estimator for the $ATT$,
where $w_{1,i}^{treat}=D_{i}$, $w_{0,i}^{treat}=\left( 1-D_{i}\right) q\left( X_{i},\hat{\theta}_{n}\right) /\left( 1-q\left( X_{i},\hat{\theta} _{n}\right) \right) $, and $\bar{w}_{d,n}^{treat}$ is the sample mean of $ w_{d,i}^{treat},$ $d=\left\{ 0,1\right\} $. The associated standard errors and $p$-values are in parenthesis and brackets, respectively. Following\ Millimet2009, we trim observations with estimated propensity score outside the interval $[0.05,0.95]$ to avoid denominators arbitrarily close to zero. Bootstrapped $p$-values for our proposed specification tests are based on 100,000 bootstrap draws\footnote{ Note that the variables in Spec1 and\ Spec2 are all functions of the same three covariates: real per capita GDP, land area per capita, and polity. As so, Spec1 and Spec2 have the same information content with respect to the reliability of Assumption (ref).} .
At the 5% level we find that, based on the $CvM_{n}$ test statistic ((ref)), Spec1 is rejected for per capita $CO_{2},$ deforestation and energy depletion, but is not rejected for rural and urban accesses to clean water. The evidence of propensity score misspecification is weaker when using the $KS_{n}$ test statistic ((ref)). Spec2, on the other hand, is not rejected for any outcome at the usual significance levels, using either $CvM_{n}$ or $KS_{n}$ test statistic. Thus, our tests suggests that Spec2 should be preferred when analyzing per capita $ CO_{2},$ deforestation and energy depletion, whereas for urban and rural water access, our tests do not favor either specification.
Next we comment on the consequences of propensity score misspecification. For per capita $CO_{2}$, our results suggest that the overall effect of GATT/WTO membership on emissions is negative and statistically significant at the 5% level under both propensity score specifications. On the other hand, we find the effect of GATT/WTO membership on per capita $CO_{2}$ is not statistically significant among the treated sub-population using either specification. In terms of point estimates, however, there are important differences. For example, the $ATE$ point estimate under Spec1 (misspecified propensity score) is 30% higher (in absolute terms) than under Spec2. Note that the $0.3$ difference in $ATE$ represents roughly 8% of the overall per capita $CO_{2}$ emissions.
When we analyze the effect of GATT/WTO on deforestation and energy depletion, our results again highlight the consequences of propensity score misspecifications. We find that the $ATE$ point estimate for the effect of GATT/WTO membership on deforestation is 30% larger under Spec2 than under Spec1, and the $ATT$ point estimate for the effect of GATT/WTO membership on energy depletion is 25% smaller under Spec2 than under Spec1. Such large differences are economically significant, as the 0.08 difference in $ATEs$ on deforestation represents nearly 12% of the mean annual deforestation, and the 0.46 difference in $ ATTs$ on energy depletion represents nearly 15% of the mean energy depletion among countries in the sample. Interestingly, the $ATT$ on energy depletion is statistically significant at 10% level under Spec1, but not under Spec2, highlighting that propensity score misspecifications can also lead to invalid inference. Finally, we note that although our tests do not detect propensity score misspecification for the rural and urban water access, the results suggest that GATT/WTO membership is not statistically significant using either propensity score specification, perhaps because of the relatively high standard errors due to the limited sample size.
Overall, we find that propensity score misspecifications can affect both the economic and statistical conclusions about the effect of GATT/WTO membership on environmental quality. When the propensity score is correctly specified, our results suggest that GATT/WTO membership is associated with improved environmental performance in terms of $CO_{2}$ and energy depletion but not in terms of rural and urban water access. We also find that GATT/WTO membership is associated with higher deforestation, though the statistical evidence is relatively weaker. Furthermore, our results uncover interesting heterogeneity, as the aforementioned results are statistically significant for the overall population ($ATE$) but not for the treated-subpopulation ($ ATT$).
In this article, we have shown that, when propensity scores are correctly specified, a particular restriction between the propensity score CDFs of treated and control groups must hold. Based on such restriction, we propose new nonparametric projection-based tests for the correct specification of the propensity score. In contrast to other proposals, our tests are not severely affected by the \textquotedblleft curse of dimensionality\textquotedblright ', are not sensitive to the different estimation methods used to estimate the propensity score under the null, do not rely on the potentially ad hoc choice of bandwidths, and enjoy some optimal power properties against particular classes of alternative hypotheses. We have derived the asymptotic properties of the proposed tests, and have proved that they are able to detect local alternatives converging to the null at the parametric rate, and that critical values can be easily computed via a simple multiplier bootstrap procedure. Our Monte Carlo simulation study illustrates that, for a large class of alternatives, our projection-based tests perform better in finite samples than existing tests, though there are some classes of alternatives in which our tests have trivial power. All these finite sample findings are in line with our asymptotic results. Finally, our empirical application concerning the effect of trade on the environment shows the feasibility and appeal of our tests in relevant scenarios. Given that the validity of many policy evaluation procedures relies on the correct specification of the propensity score, we argue that the tests proposed in this article are important additions to the applied researchers' toolkit.
We would like to mention that, in general, our tests should be seen as a \textquotedblleft model validation\textquotedblright\ and not a \textquotedblleft model selection\textquotedblright\ procedure. Once a propensity score model is selected, our specification tests can provide evidence of its reliability or lack thereof. In case the putative propensity score model is rejected, one can consider more flexible specifications. For instance, one can add additional interaction terms into the original model, consider semiparametric single-index or partially linear models, among other possible strategies. Having said so, we emphasize that if one uses our testing procedure as a \textquotedblleft model selection\textquotedblright\ device, one must bear in mind that standard inference procedures for treatment effects may be invalid if one treats the resulting selected propensity score as the \textquotedblleft true\textquotedblright\ one, see e.g. Leeb2005. Thus, in case one uses our proposal for model selection, one must account for the model selection step in order to make valid inference about the treatment effect, see e.g. Belloni2014, Chernozhukov2016, and Belloni2017. A full discussion of this procedure is beyond the scope of this article and we leave it for future research.
Finally, we note that results in Lemma (ref) can also be used for estimating the propensity score such that, for a given specification $ q\left( X,\theta _{0}\right) $, condition $\mathbb{E}\left[ D-q\left( X,\theta _{0}\right) |q\left( X,\theta _{0}\right) \right] =0~a.s.$ is directly imposed. One could use the minimum distance method described in Dominguez2004 to estimate $\theta _{0}$, for example. A detailed discussion of this estimator is beyond the scope of this article and is deferred to future work.