Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
90,131 characters · 12 sections · 123 citation commands
Testing for Idiosyncratic Treatment Effect Heterogeneity
The literature on program evaluation interrogates whether and how the effect of a program varies across people. The literature argues that such investigation helps assess important causal mechanisms for treatment effect heterogeneity based on observed characteristics crump2008nonparametric,ding2019decomposing. Likewise, policymakers often wish to discern who benefits the most out of a policy intervention, to administer the intervention to those individuals, and to exclude other individuals, so-called targeting manski2004statistical,dehejia2005program,hirano2009asymptotics,bhattacharya2012inferring,kitagawa2018should,wager2018estimation. After conducting such an exercise, it may be useful to know if there is still some treatment effect heterogeneity left, also known as idiosyncratic heterogeneity, across individuals with the same characteristics. If so, that may indicate that there is room for further targeting, and that collecting more individual characteristics could help improve the treatment assignment even further. It is therefore critical to establish whether there is unexplained, idiosyncratic heterogeneity in treatment effects. heckman1997making,djebbari2008heterogeneous,ding2016randomization.
This paper provides asymptotically valid tests for treatment effect heterogeneity. In deriving tests to detect idiosyncratic heterogeneity, I first explore tests to detect general heterogeneity. When analyzing general heterogeneity, this paper proposes a test that circumvents the estimation of the ATE using the characteristic function, which, unlike previous tests, renders the asymptotic distribution independent from the ATE. Simulations show that the test detect true heterogeneity at higher rates than previous tests in small samples.
Additionally, a second test aims to detect differences across treatment and control groups after accounting for differences in their conditional means given covariates. Consequently, I propose a permutation procedure based on the characteristic function that is both asymptotically valid and consistent against general alternatives. Finally, I apply the methods to a microcredit experiment in Bosnia and Herzegovina aimed at marginally rejected loan applicants.
I start with the simple hypothesis of no treatment effect heterogeneity. In this case, the null hypothesis implies that the distribution of potential outcomes for the treated and control groups shifts by a constant, which is the average treatment effect. A typical solution to this inference problem requires the researcher to estimate this constant, which acts as a parameter of no direct relevance to the problem. This parameter renders the asymptotic distribution of the test statistic dependent of the average treatment effect, which is often called the nuisance parameter or Durbin problem durbin1973distribution,basu1977elimination,sen1984aligned,babu2004goodness,ding2016randomization,CHUNG2021148. To circumvent the estimation of this constant, I formulate a nonparametric test based on the symmetrized outcomes, which are the difference between two independent observations of the outcomes. Since the distribution of the symmetrized outcomes should be equivalent across the treatment and control groups under the null hypothesis, their characteristic functions should also be equivalent. Accordingly, I construct an asymptotically valid test that compares the characteristic functions of the symmetrized outcomes of the treatment and control groups.
I then introduce a procedure for the more prominent case of detecting idiosyncratic treatment effect heterogeneity or, equivalently, heterogeneity that is not explained by observed characteristics. First, I remove the variation in the observed outcomes predicted by covariates in separate regressions, by treatment and control groups, of the observed outcome on covariates. Using the residuals from those regressions, I perform a homogeneity test based on the empirical characteristic function to check for equality of the unexplained variation across treatment and control groups. I show that the test is asymptotically valid and consistent for general alternatives. Except for the use of regression residuals, the test statistic is identical to the one used in meintanis2005permutation in the case with covariates. Given that the asymptotic distribution of the test statistic depends on the potential outcomes distributions, I implement a permutation test to approximate the asymptotic distribution and obtain critical values. This permutation test offers an alternative to current approximations based on the weighted bootstrap in the empirical characteristic function literature rivas2019two.
This paper contributes to the growing literature that tests for treatment effect variation in program evaluation. In the absence of covariates, ding2016randomization use randomization inference and Kolmogorov-Smirnov statistics to test for idiosyncratic treatment effect variation, but their test requires the estimation of a confidence interval for the average treatment effect. sen1984aligned suggests a test that uses the empirical distribution function under a restrictive symmetry assumption. In the case of no covariates, one can also compare the variances of the treated and untreated outcome distributions cox1984interaction,white1980heteroskedasticity,breusch1979simple or compare other moments of these two counterfactual distributions. However, those tests can only be applied to distributions with finite moments, and still they encounter the nuisance parameter problem ding2016randomization.\footnote{Other literature has explored systematic treatment effect variation explained by covariates in the form of quantile regression koenker2002inference,chernozhukov2005subsampling, finite-population regressions ding2019decomposing, nonparametric tests for equality of conditional average treatment effects crump2008nonparametric, multiple testing for treatment effect heterogeneity across subgroups defined by discrete covariates lee2014multiple, variation in subgroups' average treatment effects explaining treatment effect heterogeneity bitler2017can,buhl2022some, censored outcomes sant2016nonparametric, and multiple testing comparison of quantiles across treatment and control groups goldman2018comparing.} Likewise, in the case without covariates, CHUNG2021148 propose a permutation test that eliminates the nuisance parameter using a martingale transformation of the empirical process; however, unlike the test based on the characteristic function, that test requires the outcomes to be continuous.
The use of the empirical characteristic function for hypothesis testing is not new. To construct such statistical tests, feuerverger1977empirical and csorgo1981limit show point-wise consistency of the empirical characteristic function to its population counterpart, which is implied by a mild moment condition. Using the characteristic function, meintanis2005permutation and chen2019some introduce tests for equality of two distributions, joint independence and univariate symmetry. In the case of no covariates available to the researcher, the test based on the characteristic function is closest to the location-shift test in henze2005checking, but that paper requires the estimation of the nuisance parameter. In contrast, I avoid the estimation of the true treatment effect using a statistic that does not depend on this parameter.\footnote{The statistic in this paper is equivalent to the the difference in weighted integrals of the characteristic function's squared modulus of the treated and control groups. Several authors have used the squared modulus of the characteristic function meintanis1996robustness. For instance, using this statistic, hu2021simple test for completeness in a class of nonparametric models, and murota1981studentized formulate a normality test based on standardization of the empirical characteristic function.}
In a simulation study, I compare the small sample properties of the tests to competing methods based on the characteristic function, the distribution function (specifically, the Kolmogorov-Smirnov tests), and the quantile regression process. In the case of general heterogeneity, the tests proposed in this paper achieve higher power compared to the methods based on the characteristic function henze2005checking, the confidence interval randomization inference test ding2016randomization and the subsampling test based on the the quantile regression process chernozhukov2005subsampling. I also compare two versions of this paper's tests: one that circumvents the estimation of the average treatment effect and one that does not. Accordingly, the simulations show that circumventing the estimation of the average treatment effect has zero or at most modest gains in small samples. This modest gains are likely due to two main reasons: the success of resampling techniques in approximating the asymptotic distribution of the test statistic without worrying about the effect of unknown parameters such as the average treatment effect or the potential outcomes distributions bickel1969distribution,romano1988bootstrap,praestgaard1995permutation,abadie2002bootstrap, and secondly, the consistency of the common difference-in-means estimator. I also apply the confidence interval randomization test in berger1994p and ding2016randomization to the tests based on the characteristic function. In the simulations, I find that the confidence interval randomization test is conservative and loses power compared to the procedure I propose and that of henze2005checking.
In the more salient case of covariates available to the researcher, the simulations show the validity and power of the homogeneity test that compares the characteristic function of the observed outcomes' residuals. Given the relative success of the resampling techniques for the case without covariates, I propose a permutation test that has power against alternatives.
I apply the characteristic function tests to 40 outcomes in the microfinance experiment that augsburg2015impacts implemented in Bosnia and Herzegovina. That experiment targeted 1,196 loan applicants who were marginally rejected by a microfinance institution, with 628 of these applicants randomly selected for a typical loan. As one would expect most of the outcomes for which the authors found significant treatment effects are the candidates to exhibit treatment effect variation. Among these outcomes, there are several outcomes for which the covariates collected by the authors do not completely explain treatment effect variation including loan take-up, ownership of inventory and hours worked by working-age teenagers in the household business. For those outcomes, researchers could potentially try to collect more covariates to inspect the remaining treatment effect heterogeneity. For four outcomes, the tests suggest that any treatment effect variation seems to be accounted for by the covariates collected by the researchers; these outcomes include the incidence of self-employment income and the incidence of wage income, main business expenses and hours worked in activities other than business and wage labor. Finally, the test based on the statistic in henze2005checking tends to reject more often than the tests with and without covariates proposed in this paper.
The paper proceeds as follows. The second section introduces the setup. The third section explains the test based on the empirical characteristic function. The fourth section shows simulations exploring the small sample properties of the tests. The fifth section presents the application. The sixth section concludes. All proofs are included in the Appendix.
Consider a sample of $n$ units indexed by $i=1,\dots,n$. Let $D_i$ be a binary variable that indicates treatment status of individual $i$: $D_i=1$ if individual $i$ was treated, and $D_i=0$ otherwise. Let $Y_i(1)$ and $Y_i(0)$ represent random potential outcomes of individual $i$ with and without treatment, respectively. The observed outcome is given by
Let $n_1$ and $n_0=n-n_1$ be the number of treated and control units, respectively. Let $\mathbf{X}_i$ be a $(q\times 1)$ vector of covariates not affected by treatment for unit $i$, which may include a constant, and let $\mathbf{X}_i'$ be its transpose.
I now make two assumptions on the treatment assignment and sampling mechanisms.
Assumption (ref) guarantees that the treatment status does not influence the potential outcomes. Assumption (ref) states that the researcher has a random sample of units from the same population, which is a commonly made assumption in the literature crump2008nonparametric,wager2018estimation,heckman1997making,djebbari2008heterogeneous.
This paper aims to test for equality in the distributions of the treated and control potential outcomes after taking into account different conditional mean responses given covariates. For instance, if gender is a covariate of interest, the paper aims to answer whether there are differences between the distributions of the treatment and control groups after allowing for different mean responses for female and non-female participants by treatment assignment. This hypothesis could be of importance as a participant would be expected to have the same uncertainty (i.e., having the same distribution) over their response in either the treatment or the control group once the mean response specific by gender and treatment assignment is removed.\footnote{The paper differentiate itself from the literature that test for constant conditional average treatment effect crump2008nonparametric. Likewise, the paper does not pursue any decomposition or inference over the treatment effect, $Y_i(1)-Y_i(0)$, which is in general an unobservable quantity ding2019decomposing. }
To make this purpose explicit, one can write the following decomposition of the potential outcomes into systematic and idiosyncratic components
The component captured by $\operatorname{\mathbb{E}}[Y(1)|\mathbf{X}]$ is the conditional expectation of the outcome when treated given covariates $\mathbf{X}$ or systematic component. The component given by $\varepsilon(1)$ is the part of the potential outcome after stripping down the systematic component. For instance, if $\mathbf{X}$ is a categorical variable that takes the values female and non-female, $\varepsilon(1)$ will be the treated potential outcome after removing the outcome means for female and non-female subpopulations.\footnote{ding2016randomization used a similar procedure in a finite population setting where, instead of using the conditional expectation, they suggest using a linear regression projection $\mathbf{X}'\boldsymbol{\beta}$.}
The null hypothesis can be written as follows,
where $\overset{d}{=}$ means equality in distribution. This hypothesis conveys that the marginal distributions of treatment and control groups are the same after subtracting different conditional means given covariates.
To give simple intuition about $H_0$, in the simplest case of no covariates $\operatorname{\mathbb{E}}[Y(1)|\mathbf{X}]=\operatorname{\mathbb{E}}[Y(1)]$ and $\operatorname{\mathbb{E}}[Y(0)|\mathbf{X}]=\operatorname{\mathbb{E}}[Y(0)]$, so $H_0$ implies the following null hypothesis
where $\tau=\operatorname{\mathbb{E}}[Y(1)-Y(0)]$ is a fixed constant.
In this case, $H_{0}^\text{nocov}$ implies that the cumulative distribution functions of the potential outcomes when treated and control are shifted by a constant, $\tau$. Previous statistical tests require eliminating this constant from the statistical problem and multiple methods have been proposed such as studentization, substitution by estimated values, invariance principles, conditioning or maximizing across the parameter space, among several others basu1977elimination,lancaster2000incidental,koenker2002inference,henze2005checking,ding2016randomization,CHUNG2021148.\footnote{Other alternatives include, as koenker2002inference state, the use of resampling of the test statistic under conditions consistent with the null hypothesis in tests based on the empirical process to obtain critical values bickel1969distribution,romano1988bootstrap,praestgaard1995permutation,abadie2002bootstrap.} In the statistics literature, $H_0^\text{nocov}$ is also known as the location-shift hypothesis sen1984aligned. cox1984interaction asserts that if the marginal distributions of the treatment group stochastically dominates the marginal distribution of the control group, or vice versa, then there is a increasing transformation of the outcomes such as there is constant treatment effect in the transformed outcomes.
In this section, I introduce a hypothesis test using the characteristic function. To formulate a test for the null of no idiosyncratic treatment effect, I exploit the fact that the characteristic function has a one-to-one correspondence with the distribution function. I start with the simplest case when the researcher does not have access to exogenous characteristics $\mathbf{X}_i$, and I then develop the case when those variables are available.
To simplify the exposition, I first present the case in which the researcher is interested in testing for idiosyncratic treatment effect variation but there are no observed characteristics available. As noted in Equation (ref), $H_0$ reduces to $H_{0}^\text{nocov}: Y(1) \overset{d}{=} Y(0) + \tau, \text{ for some } \tau \in \mathbb{R}.$
I now introduce a test based on the characteristic function to test for the null hypothesis $H_{0}^\text{nocov}$. Let $$\varphi_{d}(t)=\operatorname{\mathbb{E}}[e^{\mathrm{i}Y(d)t}]= \operatorname{\mathbb{E}}[\cos\{tY(d)\}+\mathrm{i}\sin\{tY(d)\}],$$ be the characteristic function of the potential outcome $Y(d)$ with treatment assignment $d$, for $d=0,1$, and $\mathrm{i}=\sqrt{-1}$. As noted in henze2005checking, the null hypothesis $H_{0}^\text{nocov}$ implies that
After straightforward algebra and using the properties of the characteristic function, Equation (ref) in turn implies that
for all $t \in \mathbb{R}$. Then, one can restate Equation (ref) as
where I use the fact that the modulus of the complex number $|\varphi_d(t)|^2$ is defined as $|\varphi_d(t)|^2=\varphi_d(t)\varphi_d(-t)$, $d=0,1$.\footnote{If $c$ is a complex number, then $|c|^2$ is the square modulus of $c$. Other tests using the square modulus of the characteristic function include, for example, meintanis1996robustness, murota1981studentized and hu2021simple.}
Note that Equation (ref) does not depend on the average treatment effect $\tau$. To provide more intuition about this equation, for $d=0,1$, $|\varphi_1(d)|^2$ is also known as the characteristic function of the symmetrized variable $Y(d)-Y'(d)$, where $Y'(d)$ is independent from and has the same distribution as $Y(d)$ feller2008introduction. The connection with the symmetrized outcomes is indeed distinctive; symmetrized outcomes are centered at zero which implies that their associated location parameter or expected value is, if any, zero. In a nutshell, Equation (ref) implies that the characteristic functions of the symmetrized outcomes of treatment and control groups is the same, and provides the advantage of removing the dependence on the average treatment effect, $\tau$, i.e., the difference in the counterfactual distributions' expected values.
The weighted integral difference corresponding to Equation (ref) is then
where $w(t)$ is an appropriately chosen weighting function.\footnote{Although it is clear that Equation (ref) implies Equation (ref), the converse is not generally true, so one may lose power against certain alternatives by focusing on integral differences. Alternatively, one can formulate a test based on integrals of squared differences of the modulus of characteristic functions by using a quantity such as $\mathbb{L}_w=\int_\mathbb{R}[|\varphi_1(t)|^2-|\varphi_0(t)|^2]^2w(t)dt$. In this case, since the $|\varphi_d(t)|^2$ terms in $\mathbb{L}_w$ are indeed characteristic functions, the theorems in Section (ref) simplify and follow directly from results in meintanis2005permutation. When $w(t)$ in $\mathbb{L}_w$ takes the form of a density of a stable distribution with parameter $\theta$, $\theta\in (0,2]$, that quantity becomes $\operatorname{\mathbb{E}}\left[\exp(-|Y(1)-Y'(1)-Y''(1)+Y'''(1)|^\theta)\right.$$\left.-\exp(-|Y(0)-Y'(0)-Y''(0)+Y'''(0)|^\theta) \right]$ where $Y'(d)$, $Y''(d)$, and $Y'''(d)$ are independent copies of $Y(d)$, $d=0,1$. In practice, the use of this double difference, $Y(d)-Y'(d)-Y''(d)+Y'''(d)$, implies computing a large number of differences which becomes computationally expensive in the order of $O(n^4)$. }
I summarize the previous observations in the following result.
Proposition (ref) states that whenever the null hypothesis is true, the quantity defined in Equation (ref) is zero.
Now, let $L_\theta$ be the quantity $L_w$ where one substitutes $w(t)$ with the density of a spherical stable random variable with parameter $\theta \in (0,2]$, which I denote as $w_\theta(t)$. Following known facts about the integrals of cosine functions for this family of densities as in chen2019some,zolotarev1981integral, one can write Equation (ref) in the following way
with $Y'(0)$ and $Y'(1)$ being independent variables with the same distribution as $Y(0)$ and $Y(1)$, respectively.\footnote{Two notable cases of the spherical stable family are the standard normal and Cauchy distributions corresponding to $\theta = 2$ and $\theta = 1$, respectively.} Equation (ref) shows that the test statistic $L_\theta$ is based on the difference of the symmetrized variables $Y(0) - Y'(0)$ and $Y(1)-Y'(1)$ feller2008introduction.\footnote{To obtain $Y'(1)$, a researcher can reproduce the following steps. Given a data point within a random sample, let's say $Y_i(1)$, the researcher obtains a value for $Y'(1)$ by drawing $Y_j(1)$ from the rest of the sample with $j$ different from $i$. Since one has a random sample, $Y_i(1)$ and $Y_j(1)$ are independent and have the same distribution.}
The use of $L_\theta$ leads to the following corollary.
Both Proposition (ref) and Corollary (ref) show that, when the null hypothesis $H_{0}^\text{nocov}$ is true, the statistics $L_w$ and $L_\theta$ are equal to zero. However, in certain curious cases, those statistics can be zero without $H_{0}^\text{nocov}$ being true (see lukacs1970characteristic; feller2008introduction). Despite this fact, in the simulations of Section (ref), I show that a test based on $L_\theta$ is more powerful to reject plausible alternatives than previous tests based on the characteristic function henze2005checking. Intuitively, since a test based on $L_\theta$ does not require the estimation of $\tau$, the approximation to the large sample distribution of the recentered statistic will contain less error in the order of $O(n^{-1})$, which is the condition that the test in henze2005checking assumes.
Finally, I introduce the finite sample version of the test quantity $L_\theta$ based on a sample of size $n$,
In this subsection, I study the asymptotic behavior of sample analogs of the test statistic $L_w$ under both general alternatives and the null hypothesis $H_{0}^\text{nocov}$.\footnote{I offer an alternative characterization of the asymptotic distribution of the sample analog of $L_\theta$ in Appendix (ref), which applies when the weighting scheme $w(t)$ belongs to the spherical stable family.}
Let
be the sample analog of $L_w$, where $\varphi_{n,1}(t)=\frac{1}{n_1}\sum_{j:D_j=1} e^{\mathrm{i}tY_{j}}$ and $\varphi_{n,0}(t)=\frac{1}{n_0}\sum_{j:D_j=0} e^{\mathrm{i}tY_{j}}$ denote the empirical characteristic functions computed on the treated and untreated samples, respectively.
A convenient space to describe the convergence in law of $L_{n,w}$ is the separable Hilbert space of measurable real-valued functions on $\mathbb{R}$ that are square integrable with respect to $w(t)$ and denoted by $\mathcal{L}_2$, with the inner product and norm in this space, respectively, defined by
I use similar results for $\mathcal{L}_2$-type weighted integral distances explored in, for instance, henze2005checking, meintanis2005permutation and chen2019some. $L_{w,n}$ differs from the statistics in those papers slightly so; whereas $L_w$ is a difference of weighted integrals of squared characteristic functions, the statistics in those papers are weighted integrals of squared differences of characteristic functions.
I now introduce two assumptions regarding the sample size and the weighting function. Let $\xrightarrow[]{P}$ stand for convergence in probability.
Assumption (ref) specifies that any large sample has a positive fraction of control and treatment units. The symmetry of $w(t)$ around the origin stated in Assumption (ref) is for convenience; since one can define for any non-symmetrical weight function $w(t)$ a symmetrized version $w_1(t)= 0.5\{w(t)+w(-t)\}$ that leaves the statistic $L_{w}$ unchanged jimenez2017fast.
In the following theorems, I describe the asymptotic distribution of $L_{n,w}$ under both general alternatives and the null hypothesis $H_{0}^\text{nocov}$. Let $\rightsquigarrow$ stands for convergence in distribution. Additionally, let $\operatorname{Re}(a)$ and $\operatorname{Im}(a)$ be the real and imaginary parts of a complex number $a$.
I now introduce the following corollary that describes the limit distribution of $L_{n,w}$ under $H_{0}^\text{nocov}$.
Corollary (ref) guarantees that the test based on $L_{n,w}$ is asymptotically exact in the sense that its limiting rejection probability under the null hypothesis equals the nominal level.
The asymptotic null distribution of the statistic $L_{n,w}$ is highly non-standard. Typically, the test criteria have the same distribution as an infinite linear combination of independent chi-squared distributions and it is extremely hard to find analytically. Following the literature, I will use resampling techniques to approximate the distribution of the test statistic under the null hypothesis and general alternatives henze2005checking,meintanis2005permutation,jimenez2017fast.
One can extend the test based on $L_{n,w}$ to consider constant treatment effects for multiple outcomes. Specifically, one can test the hypothesis $H_0^{MO} : \mathbf{Y}(1) \overset{d}{=} \mathbf{Y}(0) + \tau $, for some $\boldsymbol{\tau} \in \mathbb{R}^p$, where $\mathbf{Y}(d)$ is a random variable in $\mathbb{R}^p$, $p$ a positive integer and $d=0,1$. The characteristic function of $\mathbf{Y}(d)$ is defined by $\varphi_d(\mathbf{t}) = \operatorname{\mathbb{E}}[\exp\{\mathrm{i}\mathbf{t}'\mathbf{Y}(d)\}]$, for $d=0,1$ and $\mathbf{t} \in \mathbb{R}^p$. Accordingly, one can redefine $L_w$ and $L_{n,w}$ to be integrals over $\mathbb{R}^p$ instead of $\mathbb{R}$ in Equations (ref) and (ref).\footnote{Specifically, let $w(\mathbf{t})$ be a density on $\mathbb{R}^p$ and $\varphi_{n,d}(\mathbf{t}) = \sum_{i:D_i=d}^n\exp\{\mathrm{i}\mathbf{t}'\mathbf{Y}_i(d)\}$, $d=0,1$. One can redefine $L_w = \int_{\mathbb{R}} \left\{ |\varphi_{n,1}(\mathbf{t})|^2-|\varphi_{n,0}(\mathbf{t})|^2 \right\} w(\mathbf{t}) d\mathbf{t}$ and $L_{n,\theta}= \frac{1}{n_1^2} \sum_{i:D_i=1}^n\sum_{j:D_j=1}^n \exp(-||\mathbf{Y}_i-\mathbf{Y}_j||^\theta)-\frac{1}{n_0^2} \sum_{i:D_i=0}^n\sum_{j:D_j=0}^n \exp(-||\mathbf{Y}_i-\mathbf{Y}_j||^\theta), \ 0<\theta \leq 2$, with $||\cdot||$ the Euclidean norm.}
A second hypothesis of interest is whether there is remaining variation in treatment effects after removing the variation predicted by predetermined characteristics as shown in Equation (ref). Testing for this hypothesis is equivalent to testing for the equality, across treatment groups, of the distributions of the conditional expectation errors:
where $\varepsilon(1) \equiv Y(1)-\operatorname{\mathbb{E}}[Y(1)|\mathbf{X}]$ and $\varepsilon(0)\equiv Y(0)-\operatorname{\mathbb{E}}[Y(0)|\mathbf{X}]$ are the conditional expectation errors of the treated and control groups, and $\operatorname{\mathbb{E}}[Y(d)|\mathbf{X}]$ is the conditional expectation of $Y(d)$ given $\mathbf{X}$, $d=0,1$. By recentering the potential outcomes using the conditional expectations, one would remove, in a mean-squared sense, the variation in the treatment effects accounted for predetermined characteristics.\footnote{ding2016randomization used a similar procedure in a finite population setting where, instead of using the conditional expectation, they suggest using a linear regression projection $\mathbf{X}'\boldsymbol{\beta}$.}
The following regularity condition guarantees that the conditional expectation function and the best linear predictor of $Y(d)$ given $\mathbf{X}$ exists.
In Assumption (ref), $E[|Y(d)|]<\infty$ guarantees that the conditional function $\operatorname{\mathbb{E}}[Y|X]$ exists, whereas the existence and finiteness of the elements of $\operatorname{\mathbb{E}}[\mathbf{X}Y(d)]$ and $\operatorname{\mathbb{E}}[\mathbf{X}\mathbf{X}']$ guarantees that the best linear predictor of $Y(d)$ given $\mathbf{X}$ exist. \footnote{Note that the second condition in Assumption (ref) is weaker than assuming that $\operatorname{\mathbb{E}}[\mathbf{X}\mathbf{X}']$ is invertible, which guarantees that the linear projection coefficient is unique hansen2021econometrics.} The best linear predictor of $Y(d)$ given $\mathbf{X}$ is denoted by
where $\beta_d = \operatorname{\mathbb{E}}[\mathbf{X}\mathbf{X}']^{-}\operatorname{\mathbb{E}}[\mathbf{X}Y(d)]$ is the linear projection coefficient, and $A^{-}$ is the generalized inverse of matrix $A$.
We can use the conditional expectation errors, $\varepsilon(1)$ and $\varepsilon(0)$, to compute the following $\mathcal{L}_2$-type distance based on the characteristic function
where $\varphi_{\varepsilon(d)}(t)=\operatorname{\mathbb{E}}[\exp\{\mathrm{i}t\varepsilon(d)]$ is the characteristic function of the error $\varepsilon(d)$, $d=0,1$. The distance in Equation (ref) has been previously used in meintanis2005permutation, chen2019some and rivas2019two.
I formulate the following proposition that resembles Proposition (ref) for instances when the researcher has available covariates.
One possibility to estimate the functions $\varphi_{\varepsilon(0)}(t)$ and $\varphi_{\varepsilon(1)}(t)$ is to calculate the residuals from a nonparametric regression
where $\widehat{m}_d(x)$ is a nonparametric estimator of ${m}_d(x)\equiv\operatorname{\mathbb{E}}[Y(d)|\mathbf{X}=x]$ chen2007large,chen2008semiparametric,crump2008nonparametric. In particular, I use the following kernel estimators for the conditional expectation function ${m}_d(x)$
and $K(v)$ is a kernel and $h$ is the bandwidth such that $K_{h}(v)=\frac{1}{h}K(v/h)$. For simplicity, I assumed that the same kernel function, $K$, is used for both treatment and control groups.
The following regularity conditions ensure that the Nadaraya-Watson type kernel estimator $\widehat{m}_d$ will be a consistent estimator for the conditional expectation functions $m_d(x)$ hansen2008uniform.
After estimating the residuals, I can compute the sample counterpart of $\mathcal{D}_w$ as
where $\varphi_{\varepsilon(1),n}(t)= \frac{1}{n_1} \sum_{j=1}^n D_j \exp\{\mathrm{i}t\widehat{\varepsilon}_j\}$ and $\varphi_{\varepsilon(0),n}(t)= \frac{1}{n_0} \sum_{j=1}^n (1-D_j) \exp\{\mathrm{i}t\widehat{\varepsilon}_j\}$ are the empirical characteristic functions of the residuals $\widehat{\varepsilon}(1)$ and $\widehat{\varepsilon}(0)$, respectively. Large values of $\mathcal{D}_{n,w}$ indicate that there is treatment effect variation after accounting for covariates. Similar to the case without covariates, I can compute critical values for the test associated with $\mathcal{D}_{n,w}$ using resampling methods, which I describe in Section (ref).
The following two theorems are the basis for the asymptotic validity and consistency of the test based on $\mathcal{D}_{n,w}$.
Theorem (ref) shows that the sample analog $\mathcal{D}_{n,w}$ is consistent for the population quantity $\mathcal{D}_{w}$. Theorem (ref) together with Proposition (ref) make it possible to use $\mathcal{D}_{n,w}$ as the basis for a consistent test based on the empirical characteristic function. In view of Proposition (ref), the quantity $\mathcal{D}_w$ is equal to zero if and only if the null hypothesis $H_0$ holds true provided that the weight function $w$ is positive with probability 1. Consequently, large values of $\mathcal{D}_{n,w}$ should imply a rejection of the null hypothesis $H_0$.
The following theorem shows that a standardized $\mathcal{D}_{n,w}$ has a limiting distribution as the sample size increases.
As with the limiting distribution of statistic $L_{n,w}$, the asymptotic distribution of $\mathcal{D}_{n,w}$ is highly non-standard rivas2019two. Following the literature, I will show the use of resampling techniques to approximate the null distribution of this test statistic in the next section.
When the weighting scheme belongs to the spherical stable density family with parameter $\theta$, it is straightforward to define the corresponding weighted integrals $\mathcal{D}_\theta$ and $\mathcal{D}_{n,\theta}$ as in the case of $L_\theta$ based on Equations (ref) and (ref). Similar results to those in Corollary (ref) and Theorems (ref) and (ref) based on the quantity $\mathcal{D}_\theta$ and the statistic $\mathcal{D}_{n,\theta}$ follow. For the sake of brevity, I omit those results.
Finally, I introduce the finite sample version of the test quantity $\mathcal{D}_\theta$ based on a sample of size $n$,
I now describe a simulation study that confirms the validity of the tests based on $L_{n,w}$ [see Equation (ref)] and assesses their power under a range of plausible scenarios. To compare these results with previous studies, I follow the data generating processes suggested in ding2016randomization, koenker2002inference, and chernozhukov2005subsampling. \footnote{ding2016randomization kindly provide simulation code in their paper's supplemental material.}
When implementing the test based on $L_{n,\theta}$ in Equation (ref), I select the density function of the spherical stable distribution with parameter $\theta=2$, which corresponds to the normal density, and rewrite $L_{n,\theta}$ in the following way:
To compute critical values for the test based on $L_{n}$, I use the bootstrap, which does not require the estimation of the constant treatment effect, and a permutation test, which does require that estimation. Intuitively, the bootstrap and permutation tests provide good approximations to the asymptotic distribution of $L_{n,\theta}$ since the empirical characteristic function is a sample average of a smooth function of the empirical process.
Besides $L_{n,2}$, I include simulation results for the test statistic in henze2005checking:
where $\widehat{\tau}=\frac{1}{n_1}\sum_{i:D_i=1}Y_{i}-\frac{1}{n_0}\sum_{i:D_i=0}Y_{i}$ is the classic differences-in-means estimator of $\tau$. The test based on $L_{n}^{HKZ}$ requires that $\widehat{\tau}-\tau$ has a representation as the difference-in-means of an appropriate measurable function with mean zero and finite variance, which boils down to $\sqrt{n}(\widehat{\tau}-\tau)=O_P(1)$ when $E[Y(0)^2]<\infty$ and $E[Y(1)^2]<\infty$ henze2005checking. To compute critical values for the test based on $L_{n}^{HKZ}$, I follow henze2005checking and use the bootstrap and a permutation test.
The most striking difference between the two test statistics $L_{n,2}$ and $L_{n}^{HKZ}$ is that $L_{n}^{HKZ}$ depends on the estimation of the average treatment effect, whereas $L_{n,2}$ does not. Furthermore, as suggested by Equation (ref) and to possibly increase power, one can generalize $L_{n,2}$ to use other parameters of the spherical stable distribution as long as $\theta \in (0,2]$. Although, the statistic $L_{n}^{HKZ}$ can also be generalized in a similar way, the formulation in henze2005checking does not easily introduces this possibility.
First, I examine the various methods under the null hypothesis of a constant treatment effect (no covariates) as specified in the following model:
with $i = 1, \dots, n$. I assess the methods based on $L_{n,2}$ and $L_{n}^{HKZ}$ for the following distributions: $\varepsilon_i$ distributed standard normal, $t_5$, standard exponential and log-normal; each with a constant treatment effect of 1 unit ($\tau = 1$) as suggested in koenker2002inference, chernozhukov2005subsampling, and ding2016randomization.
To assess validity, I replicate the following procedure 5,000 times for each combination of a test statistic, a sample size and an underlying distribution:
As suggested in ding2016randomization, there is a nuisance parameter problem when implementing permutation tests based on the Kolmogorov-Smirnoff statistic in the form of the average treatment effect, $\tau$. The bootstrapped test based on $L_{n,2}$ does not directly depend on this parameter, so it does not require the estimation of this parameter, whereas all implementations of the test in henze2005checking does require the estimation of such parameter. On the other hand, the permutation version of the test based on $L_{n,2}$ does require the estimation of the average treatment effect to calculate an approximate science table, i.e., the table that lists all of the pairs of potential outcomes for every unit.
Table (ref) shows the rejection rates of 5%-level tests for the tests based on the characteristic function, the plug-in and confidence-interval Fisher randomization tests ding2016randomization, the martingale transformation test CHUNG2021148 and the subsampling quantile regression test chernozhukov2005subsampling, for each sample size and data-generating process. The actual level of the bootstrap and permutation procedures based on ${L}_{n,2}$ is close to the nominal level of 5% for samples sizes as small as 200 observations. The bootstrap and permutation methods based on $L_{n}^{HKZ}$ exhibit severe overrejection when the underlying distribution is log-normal. However, this overrejection is not as severe as the one exhibited by the plug-in Fisher randomization test. The confidence-interval Fisher randomization, the subsampling and the martingale transformation tests seem to control size in samples as small as 50 observations.
To overcome the overrejection of the characteristic-function based tests in samples with less than 200 observations, I implement the confidence interval correction in berger1994p and ding2016randomization. Specifically, I repeat the steps in the permutation test (see item (ref) in the simulation procedure above) for $m$ values of $\tau$ chosen from a valid $99.9\%$-level confidence interval for $\tau$. In practice, this method may require performing permutation tests for a large $m$. When applying this test to data generated under constant treatment effects, as in Equation (ref), with $m=21$, the confidence interval permutation test based on the characteristic function (either $L_{n,2}$ or $L_{n}^{HKZ}$) controls the probability of rejecting a true null in samples as small as 50 observations.
To assess the power of these methods under selected alternatives, I adopt the simulation study in koenker2002inference, chernozhukov2005subsampling and ding2016randomization. For these simulations, I repeatedly generate data with different levels of treatment effect heterogeneity, denoted by $\sigma_\tau$, and I estimate the probability that a method would reject the null hypothesis of constant treatment effect at 5% nominal level given draws of data and random treatment assignment,
for each $i=1,\dots,n$, with $\varepsilon_i$ distributed either standard normal, $t_5$, standard exponential or log-normal, each with $\sigma_\tau=0.2$ or $\sigma_\tau=0.5$.\footnote{If $\varepsilon_i \sim \mathcal{N}(0,1)$, the $F$-test is the uniformly most powerful test cox1984interaction.}
Table (ref) shows the rejection rates of $H_{0}^\text{nocov}$ for those tests that control simulated size in Table (ref), specifically, the confidence-interval permutation versions of the tests based in $L_{n,2}$ and $L_{n}^{HKZ}$, the confidence-interval Fisher randomization test, the martingale transformed test, and the subsampling quantile regression test. Particularly, the simulations show that the confidence interval permutation test based on $L_{n,2}$ has higher power than the competing tests in log-normal, normal and $t_5$ distributions. In exponential outcomes, the confidence-interval test based on $L_{n,2}$ tends to be more powerful in samples with 100 or less observations and with high heterogeneity ($\sigma_\tau=0.5$). Surprisingly, the martingale transformation test seems to be overly conservative.
To compare power across tests with different sizes, I compute size-adjusted power using the simulated percentiles of the corresponding test statistics for each underlying distribution and sample size. Table (ref) shows the size-adjusted power of the tests based on statistics $L_{n,2}$ and $L_{HKZ}$, the quantile regression test chernozhukov2005subsampling, the martingale transformed test CHUNG2021148, and the FRTCI statistic (Kolmogorov-Smirnoff statistic). These calculations presume the availability of the critical values that correspond to each of the underlying distributions and sample sizes. The test based on $L_{n,2}$ has higher power than the competing tests when the distributions are symmetric, i.e., the standard normal and the $t_5$ distributions. In log-normal outcomes, the martingale-transformed statistic has higher power than the competing tests. The tests based on $L_{n,2}$ and the FRTCI tend to have higher power for exponential outcomes. Finally, within the statistics based on the characteristic function, the test based on $L_{n,2}$ appears to have higher power than the tests based on $L_{n}^{HKZ}$.
To assess the validity and power of the test based on $\mathcal{D}_{n,w}$, I simulate a completely randomized experiment in which potential outcomes and treatment effects follow a linear model in covariates as in ding2019decomposing. \footnote{Although the simulations in that paper are to show the properties of a test for systematic treatment effect variation, those simulations encompass a framework where both systematic and idiosyncratic treatment effect variation may coexist.} Specifically, potential outcomes observe the following equations:
with covariates distributed as follows $X_{1i}\sim \mathcal{N}(0,1)$, $X_{2i}\sim \mathrm{Bernoulli}(0.5)$, $X_{3i}\sim \mathrm{Bernoulli}(0.25)$ and $X_{4i}\sim \mathcal{N}(0,1)$; $i=1,\dots,n$. The terms $\delta_i$ and $\varepsilon_i$ configure four types of variation in treatment effects: $\tau_i$ can have either no systematic ($\delta_i=0.3$) or systematic variation ($\delta_i=0.2 + 0.1X_{1i} + 0.4X_{3i}$); and $\tau_i$ can have either no idiosyncratic ($\varepsilon_i=0$) or idiosyncratic variation ($\varepsilon_i=\mathcal{N}(0,0.2^2)$). In each replication, sixty percent of the units are randomly assigned to treatment.
When implementing the test based on $\mathcal{D}_{n,\theta}$ (Equation (ref)), I use the density function of the spherical stable distribution with parameter $\theta=2$ as weight function $w(t)$, and rewrite $\mathcal{D}_{n,\theta}$ in the following way:
I select $\theta=2$ as suggested in the literature, which corresponds to a standard normal weighting function. As stated in Section (ref), $\widehat{\varepsilon}_i$ is the residual from fitting a linear model of the observed outcome $Y_{i}$ on $D_i$, $\mathbf{X}_i=(X_{1i},X_{2i},X_{3i},X_{4i})$ and $D_i\mathbf{X}_i$.
To compute critical values, I use a permutation test based on oja1987permutation. Specifically, I implement the following steps:
Table (ref) shows the rejection rates for 5%-level tests based on $\mathcal{D}_{n,2}$. In the presence of no idiosyncratic treatment effect variation (columns 3 and 5), the test rejects about 5% of the replications indicating the test's asymptotic validity. The test achieves power for large samples, which are similar to the sample sizes in the simulations in ding2019decomposing, as shown in columns 2 and 4.
I apply the tests to the microcredit RCT implemented by augsburg2015impacts in Bosnia and Herzegovina. The experiment targeted 1,196 loan applicants who were marginally rejected by an microfinance institution. The researchers offered the loan to 628 randomly selected applicants. Successful applicants received the loan within a week with similar interest rate and maturity to the regular institution's clients. The authors interviewed participants 14 months after. For those participants, the researchers collected baseline characteristics such as gender, age, marital status of the marginal applicant, and information on the household composition (number of children in the age ranges of 0--5, 6--10, 11--16, and number of household members that are: female, employed, in school, and retired). They found evidence that the treatment induced higher self-employment, increases in inventory, a reduction in the incidence of wage work, an increase in the labor supply of 16--19-year-olds in the household’s business, increases in profits and a reduction in consumption and savings.
Table (ref) shows the results for the characteristic-function based tests in 40 outcomes presented in the main tables in augsburg2015impacts.\footnote{It is also possible to evaluate the multiple hypothesis that all 40 outcomes (or subsets of these outcomes) have no idiosyncratic treatment effect variation as pointed out above at the end of section (ref).} The table shows the empirical cumulative probability, or cumulative relative frequency, which is the fraction of quantiles of the corresponding resampling distribution that are below the calculated value of the statistic, of three statistics: $L_{n,2}$, $L_{n}^{HKZ}$ and $\mathcal{D}_{n}$. The resampling distributions are computed using 2000 samples with replacement (bootstrap) and permuting the treatment without replacement (permutation; see Section (ref) for details). The test based on $L_{n,2}$ rejects at the 5% level if the empirical cumulative probability is lower than 2.5% or greater than 97.5%. The tests based on $L_{n}^{HKZ}$ and $\mathcal{D}_{n,2}$ reject at the 5% level if the empirical cumulative probability is greater than 95%.
In the case without covariates, the bootstrap and permutation tests based on the statistic $L_{n,2}$ reject the null of no treatment effect variation at 5% significance level for 13 and 22 outcomes, respectively, whereas the test based on $L_{n}^{HKZ}$ rejects the null for 31 outcomes.
As expected, accounting for covariates tends to reject less often with only 12 outcomes rejecting the null based on $\mathcal{D}_{n,2}$. Likewise, as one would anticipate, those outcomes affected by the treatment are the candidates to exhibit treatment effect variation. Indeed, some of the outcomes for which the tests based on $L_{n,2}$ and $\mathcal{D}_{n,2}$ reject the null hypothesis of no idiosyncratic treatment effect coincide with the outcomes for which the authors find significant average treatment effects; those outcomes are related to loan take-up, ownership of inventory and hours worked by 16--19-year-olds in the household business. Additionally, both tests also reject the null of no idiosyncratic treatment effect for the following outcomes: self-employment income, an indicator for having 16--19-year olds, the number of 16--19-year olds in the household, and the home durable good index. In contrast to what augsburg2015impacts state, the tests suggest that the distributions of the outcomes related to the presence of 16--19-year olds in the household do not seem to be the same across treatment and control groups.
The tests suggest that, for four outcomes, any treatment effect variation seems to be accounted for by the covariates collected by the researchers. Specifically, four (ten) outcomes are rejected by the bootstrap (permutation) test based on $L_{n,2}$ but not by the test based on $\mathcal{D}_{n,2}$, namely, the average yearly expenses of the main business, indicators for whether the household received self-employment income and wages, and the number of hours worked per household member in a week in other activities.
Furthermore, Table (ref) shows the $p$-values for the confidence-interval Fisher randomization test (FRTCI) and the plug-in Fisher randomization test (FRTPI) in ding2016randomization, the martingale transformation test in CHUNG2021148, and the subsampling quantile regression test in chernozhukov2005subsampling for each outcome in augsburg2015impacts. The FRTCI is conservative in general detecting heterogeneity for only 3 outcomes, whereas the FRTPI and the subsampling tests detect heterogeneity in 18 and 16 outcomes, respectively. In principle, the FRT and the martingale permutation tests cannot be applied to a variety of outcomes in this application since they tend to be discrete.\footnote{ding2016randomization recognize that the Fisher randomization test does not directly address the case of discrete outcomes.}
I show the usefulness of the tests based on the characteristic function to detect idiosyncratic treatment effect variation. The tests are asymptotically valid and have power against plausible alternatives. As further variation is detected, the test can help researchers in their data collection efforts to determine who benefit the most out of policy interventions, to disentangle causal mechanisms, and to identify characteristics that predict treatment effect variation. An application of the tests to a microcredit experiment in Bosnia and Herzegovina shows that outcomes related to loan take-up, self-employment, savings, expenses on durable goods, and hours worked by 16--19-year-olds in the household business exhibit idiosyncratic treatment effect variation not accounted for baseline characteristics. For those outcomes, researchers could potentially try to collect more covariates to inspect the remaining treatment effect heterogeneity, and potentially, improve treatment targeting.