Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
Rendered from LaTeX for readability, not typeset faithfully. Citation keys are highlighted; maths is left as source; figures, tables and equation environments are summarised rather than reproduced; unrecognised commands are greyed out so nothing is silently dropped. Email addresses are removed.
Inference for multi-valued heterogeneous treatment effects when the number of treated units is small
abstractWe propose a method for conducting asymptotically valid inference for treatment effects in a multi-valued treatment framework where the number of units in the treatment arms can be small and do not grow with the sample size. We accomplish this by casting the model as a semi-/non-parametric conditional quantile model and using known finite sample results about the law of the indicator function that defines the conditional quantile. Our framework allows for structural functions that are non-additively separable, with flexible functional forms and heteroskedasticy in the residuals, and it also encompasses commonly used designs like difference in difference. We study the finite sample behavior of our test in a Monte Carlo study and we also apply our results to assessing the effect of weather events on GDP growth.
Introduction
Many applications in economics and elsewhere require us to acknowledge and learn about treatment effect heterogeneity. Particularly in panel data where individuals, firms, or states can have different characteristics, belong to different groups or adopt the treatment in different instances. Moreover, in this setup, outcomes may be depend not only on the current treatment status but on the whole past history of treatment
, thereby given raise to multi-valued treatments rather than simple binary ones.
These features --- the heterogeneity and the multi-valued nature of treatments --- force the researcher to dissagregate the data in a way that the treatment groups can have a small number of units even for moderate and even large sample sizes. Moreover, this issue becomes even more salient if one disaggregates further, for instance, by conditioning on observed characteristics of the individuals to handle potential heterogeneity of the treatment affects. Thus, in these cases, standard inferential procedures that rely on asymptotic results may be untrustworthy or even invalid. The main contribution of this paper is to address this problem by proposing a valid inference method for testing the treatment effect over outcomes that is valid when the number of treated units is small even though the sample size is large. We do so in a potential outcomes framework that is sufficiently general to allow for non-/semi-parametric models with multi-valued heterogeneous treatments and dynamic treatment effects, and to to cover commonly used identification approaches such as RCTs, difference in difference and control variable.
We consider a panel data potential outcomes framework (splawa1990application) wherein the so-called outcome function relates the current outcome to the treatment profile --- i.e., the current and past treatment status --- past outcomes and characteristics of the individual (both observed and unobserved). Under our assumptions, this outcome function is the object of interest as it encodes the causal effect of the treatment profile on current outcomes. In fact, the null hypothesis boils down to whether the outcome function for user-specified treated groups --- defined by the observed characteristics and treatment profile --- coincides with the outcome function associated to a benchmark/control group.
An important first step in our method is to relate this outcome function for a given treatment group to the solution of a conditional moment restriction defined by the conditional quantile of the current outcome, given the past outcomes and other observed covariates for this group. As a consequence, we are able to construct a test statistic that is akin to a model specification test statistic for non-parametric GMM models where we input an estimator of the outcome function for the benchmark/control group on the moments of the treated groups and reject the null if these moments are far away from zero. The non-standard part of the test lies on approximating its distribution; to do this we employ two insights. First, we note that since the number of treatment groups is finite and fixed, for sufficiently large sample sizes the number of observations will be large in at least one of the groups; we normalize such a group to be the benchmark/control group. Based on this observation, we construct a consistent sieve-based estimator (cf. chen2012estimation, CvKL2003) of the outcome function for such group. Under our assumptions this estimator converges to the true outcome function, but since the effective sample size of each treated group does not grow with the number of units in the overall sample, standard approximations results like the Central Limit Theorem do not apply. To circumvent this issue our second insight is to apply the results by chernozhukov2009finite to show that the conditional quantile for the treated groups evaluated in the outcome function of the control group, under the null, has a known distribution that can be approximated by simulations. Based on these logic we are able to prove the main result of the paper stating that our test has correct asymptotic size, even if the number of units in the treatment groups stays fixed. In addition, we are also able to construct confidence regions for certain semi-parametric models wherein the treatment effect is encoded in a finite dimensional vector.
The key assumption used to link the outcome function, which is a structural object, to the conditional quantile of the respective groups, which is related to the properties of observed random variables, imposes that some quantile of the random variable representing the unobserved characteristics does not depend on the treatment profile once we condition on observed characteristics and past outcomes. Even though this assumption does restrict the scope of our framework, it still allows for commonly used designs such as RCTs and difference in difference and the control function approach (see Section (ref) for an application).
We are not the first paper to propose inference methods that are valid for finite samples. Closest to our methodology is the paper by ferman2019inference
which belongs to a strand of the literature that proposes inference methods when the treatment group is small but the control group is large (e.g. donald2007inference, conley2011inference and Cameron et al (2008).). Both, us and this literature, leverage the observations in the control group --- for which the number of observations is assumed to be large when the sample size is large --- to learn certain features of the problem and then, under the null, extrapolate this knowledge to the treated groups. The difference lies on how we accomplish this. This literature focuses on linear models within a difference in difference framework and imposes, to different degrees, restrictions on the conditional CDF of the residuals given observables, in particular in terms of its heterosckedasticity. In contrast, by casting the problem within a non-parametric conditional quantile moments framework, we are able to perform this extrapolation without imposing any functional form restrictions in the outcome function. The outcome function can be non-separatively additive, can depend on both finite and infinite dimensional parameters and allow for flexible --- albeit not completely unrestricted --- conditional distribution in the residuals. On the other hand, ferman2019inference allow for within group dependence, whereas we assume units to be IID.\footnote{If the treatment only varies at the group level --- e.g. as in ferman2019inference leading example --- then one can apply our framework taking the groups as units.}
In Section (ref) we apply our framework to a two way fixed effect (TWFE) model with heterogeneous treatment effects under staggered adoption, which has been recently studied in many papers (e.g. ChaiseMartin2020, SUN2020, callaway2018difference, athey2018designbased). This literature emphasizes identification, estimation and inference of treatment effects allowing for certain types of heterogeneity and dynamics. To the best of our knowledge all of these papers rely on standard asymptotic theory which assumes that the size of the treated groups grows with the sample size. Given the dynamic nature of treatment profiles and the group heterogeneity of effects, having a small number of treated units could be a salient feature in many applications. Hence, the results in this paper could complement this literature by offering a novel and alternative method for inference.
\paragraph{Related Literature} Beyond the aforementioned papers, our paper builds on and contributes to three main strands of the literature about treatment effects. First, our setup is similar to the one consider by han2018identification. However, han2018identification focuses on identification in an IV approach whereas we focus on estimation and testing under a control variable approach when the number of treated units is small.
There is a vast literature concerned with inferential methods that have finite sample properties such as Fisher randomization or permutation methods (see rosenbaum2010design for a review). These methods, however, are quite different to our approach as they are derived under a different set of assumptions that impose certain symmetry or invariance of the data distribution under the null (see econometrics5040052 for a discussion in the context of synthetic control models). We refer the reader to ferman2019inference for a more thorough literature review and references.
From a technical point of view, our results can be viewed as deriving the asymptotic distribution of a test statistic constructed upon a GMM criterion function in a non-parametric conditional moment model, where each of the moments in the (sample) GMM criterion function (which in our case are given by the conditional quantiles of the outcome given observable characteristics for each treatment group) may have different and even non-vanishing rates of convergence, even as the sample size goes to infinity. To our knowledge, the asymptotic results in the non-parametric conditional moment models literature (cf. ai2003efficient, CvKL2003, chen2012estimation and horowitz2007nonparametric among many others) do not apply. To bridge this gap, we combine this framework with the finite sample results by chernozhukov2009finite.
Illustrative Example
We now present a simple canonical model, which encompasses several existing applications, and which we develop and extend throughout the paper to illustrate different concepts.
example[Canonical Example]
For each unit $i$ and time $t$, let $Y_{i,t}$ be the economic outcome of interest and let $D_{i,t}$ be a binary variable that encodes the treatment status for this unit at time $t$. We postulate that, for some $L,P \in \mathbb{N} \cup \{0\}$,\footnote{$P=0$ simply denotes that no lags of $Y$ are included. Also, additional variables such as exogenous covariates and region-time fixed effects are omitted to simplify the exposition.}
\begin{align}
Y_{i,t} = \kappa + \sum_{s=0}^{L} \theta_{s} D_{i,t-s} + \sum_{s=1}^{P} \gamma_{s} Y_{i,t-s} + \epsilon_{i,t}.
\end{align}
in which $(\theta_{s})_{s=0}^{L}$ are the parameters of interest as they summarize effect of the treatment on the outcome variable.
Similar models have been used in several applications. For instance, to capture the effect that weather has on GDP or GDP growth (dell2012temperature, dell2014we and references therein); or the effect that climate shocks (miguel2004economic, burke2015climate) or economic shocks (bazzi2014economic) have on conflicts; or the effect that democratization has on GDP growth (acemoglu2019democracy); or the effect that elections on bond issuance have on economic outcomes such as spending in infrastructure or prices of houses (cellini2010value).
Our point of departure is to cast this as a multi-treatment effect problem where there are $\{0,1\}^{L+1}$ potential outcomes, each indexed by a particular treatment profile, i.e., $Y_{t}(d_{t},...,d_{t-L})$. While several of the aforementioned applications use $L$ equal to 0, 1 or 2, the treatment variables (e.g. climate variables, democratizations, economic shocks, etc) may be highly autocorrelated thus give raise to an omitted variable bias; this has already been pointed out (e.g. burke2015climate). One way to address this issue is to consider large values of $L$ or even set $L=t$ and let the data determined which lags matter. In these cases, however, the number of countries in each “treatment profile bin" may be very small. In order to illustrate this feature, we use data from dell2012temperature and first encode temperature increases as a binary variable. We define that country $i$ suffered a temperature increase in period $t$ if the difference in temperature between periods $t-1$ and $t$ is above one degree Celsius. We specify $L=4$ and tally the number of countries in each treatment profile for three different time periods, which we report in Table (ref) presents the results. For all the choices of $t$ the number of observations in the control group (i.e, the profile with all zeros) is large, however, for each of the other treatment profiles the number of observations is much smaller --- in all cases less than 20 and in many less than 10 --- even though the sample size is moderate and $L$ is not particularly big.
\begin{table}[h]
\begin{center}
\begin{tabular}{lrrr}
\hline
Profile & N 1960-1964 & N 1980-1984 & N 1990-1994 \\
\hline
00000 & 100 & 113 & 93 \\
00001 & 7 & & 17 \\
00010 & 1 & 5 & 3 \\
01000 & 4 & 1 & \\
01001 & 1 & & \\
10000 & 7 & 1 & 6 \\
10100 & 4 & & \\
00100 & & 3 & 1 \\
01010 & & 1 & \\
00011 & & & 1 \\
00101 & & & 2 \\
10001 & & & 1 \\
\hline
\end{tabular}
\caption{Number of countries for each treatment profile with $L=4$ and for different periods $t \in \{ 1964, 1984, 1994 \}$. Empty cells mean no countries for that profile. Data: Panel of 124 countries taken from dell2012temperature.}
\end{center}
\end{table}
Without further restrictions on the coefficients $(\theta_{s})_{s=1}^{L}$, this feature will render the usual asymptotic techniques untrustworthy or directly unviable. Of course, by imposing additional restrictions on the coefficients $(\theta_{s})_{s=1}^{L}$ --- e.g. $\theta_{s} = \theta$ for all $s$ --- one can use the linearity assumption to extrapolate; these further restrictions, however, may not be warranted in many applications. The goal of this paper is to propose an inference method that delivers correct results in these situations where the number treated units may be small, even asymptotically. $\triangle$
Setup
Our point of departure is a simplified version of the setup in han2018identification which is in turn an extension of the setup considered by chernozhukov2005iv. This setup corresponds to a potential outcomes framework with multiple treatments.
We consider a panel structure in which for each unit, indexed by $i \in \mathbb{N}$, we observe $T$ time periods of data; we define $\mathbb{T}:=\{1,...,T\}$ as the set of time periods. For each $t \in \mathbb{T}$, a time $t$ treatment profile is a sequence in $\{0,1\}^{t}$, $ {D}^{t} : = (D_{1},...,D_{t})$, such that the $s$-th element, $D_{s} \in \{0,1\}$ indicates the treatment status at time $s$. We use $ {D}^{t}_{i}$ to denote the time $t$ treatment profile of unit $i \in \mathbb{N}$.
For each $t \in \mathbb{T}$ and time $t$ treatment profile, $d^{t}$, we use $Y_{t}(d^{t}) \in \mathbb{Y} \subseteq \mathbb{R}$ to denote the $d^{t}$ potential outcome, that is the outcome at time $t$ had the time $t$ treatment profile been $d^{t}$. We use $Y_{i,t}(d^{t})$ to denote the $d^{t}$ potential outcome of unit $i \in \mathbb{N}$. Finally, let $ {Y}^{t}(d^{t}) : = (Y_{1}(d^{1}),...,Y_{t}(d^{t}))$. For each $t \in \mathbb{T}$, the time $t$ (observed) outcome profile is given by $ {Y}^{t} : = (Y_{1},...,Y_{t})$ where $Y_{t} : = Y_{t}( {D}^{t})$.
Finally, for each $t \in \mathbb{T}$, let $ {X}^{t} : = (X_{1},...,X_{t}) \in \mathbb{X}^{t}$ with $\mathbb{X} \subseteq \mathbb{R}^{q}$ be the vector of observed covariates up to time $t$. We assume the covariates are discrete, i.e., $|\mathbb{X}| < \infty$. While it is straightforward to extend our framework to handle both continuous and discrete covariates --- provided the former are not used to define the treatment groups (see Remark (ref) below for more details) ---, we maintain the assumption of discrete covariates to simplify some of the derivations and exposition, as well as to ease the notational burden.
We now impose restrictions on the process for the treatment and potential outcomes profiles. For each $t \in \mathbb{T}$, let
align[align omitted — 86 chars of source]
where $V^{t}: = (V_{1},...,V_{t})$ are random variables that amount to the shocks for the treatment profiles. Expression (ref) allows for dynamic treatment assignments whereby the treatment status at time $t$ can depend on past outcomes, past treatment status and a random shock.
We assume that the process for $Y_{t}(.)$ satisfies
align[align omitted — 137 chars of source]
where $U_{t}(d^{t})$ is a random variable with support in $[0,1]$ whose CDF will be specified below, and it should be viewed as unobserved characteristics of the individual. We can view the vector $(X^{t}, U_{t}(d^{t}))$ as the characteristics of the unit, and thus the outcome function, $\mu_{t}$, determines how these characteristics, the time $t$ treatment profile and the past (potential) outcomes affect the current outcome. In particular, under the assumptions below, the mapping
$$d^{t} \mapsto \mu_{t}(Y^{t-1}(d^{t}), x^{t},d^{t},\tau)$$
measures the causal effect of the treatment profile $d^{t}$ on the outcomes, for a unit with characteristics $(x^{t},\tau) \in \mathbb{X}^{t} \times (0,1)$.
We impose the following assumptions on the DGP.
assumption[IID]
For each $i \in \mathbb{N}$, $((Y_{i,t}(d^{t}),U_{i,t}(d^{t}) )_{t,d^{t}},D^{T}_{i},X^{T}_{i},V^{T}_{i})$ are IID drawn from $P$, such that expressions (ref) and (ref) hold and $D_{0}$ and $Y_{0}(.)$ are assumed to be exogenous.
assumption[Monotonicity]
$u \mapsto \mu_{t}(y^{t-1},x^{t},d^{t},u)$ is increasing.
assumption[Conditional Rank Invariance restriction]
For any $d^{t} \in \{0,1\}^{t}$ and some $\tau \in (0,1)$, the distribution of $U_{t}(d^{t})$ conditional on $Y^{t-1}(d^{t-1}),X^{t}$ and $D^{t}$ satisfies the following restriction,
\begin{align*}
P( U_{t}(d^{t}) \leq \tau \mid Y^{t-1},X^{t} ,D^{t}) = \tau.
\end{align*}
remark[Discussion of the Assumptions]
These assumptions are standard in the literature of conditional quantile estimation and are use to link the structural function to a conditional quantile restriction on the observables. Assumption (ref) is the standard IID assumption and can be somewhat relaxed; Lemmas (ref) and (ref) in Appendix (ref) illustrate what aspects of this assumption are used in the proofs. Assumption (ref) imposes a monotonicity restriction on the structural function; cf. chernozhukov2005iv.
The key assumption is Assumption (ref), which conflates two restrictions on the distribution of $U_{t}(d^{t})$ for each $d^{t}$. First, a shape restriction that requires, the $\tau$-th quantile of the CDF of $U_{t}(d^{t})$ given $Y^{t-1},X^{t}$ and $D^{t}=d^{t}$ to be equal to $\tau$. Second, an invariance restriction that requires the $\tau$-th quantile of the CDF of $U_{t}(d^{t})$ given $Y^{t-1},X^{t}$ to be the same for each treatment profile $D^{t}$ (not just $D^{t} = d^{t}$). While it is only the first restriction that is used to show our results, it is the second restriction that warrants the qualifier causal when describing the function $\mu_{t}$; this last point is discussed further in Remarks (ref) and (ref) below. In essence, this last restriction can be interpreted as a type of selection on observables, in which the observed characteristics are $Y^{t-1},X^{t}$. While this restriction is by no means innocuous and rules out many models, given the general specification of our model, it still allows for many popular approaches like
RCTs and difference in difference designs (see Section (ref)) and control variable approach in general.
To shed more light onto this assumption, we now discuss it under different laws for the treatment profiles and the assumption --- to simplify the presentation --- that $(U_{t}(d^{t}))_{t}$ are independent across time. First, consider the case where $D^{t}$ is independent of $(U_{t}(d^{t}))_{t,d^{t}}$, e.g., $D_{t} = \delta_{t}(X^{t},D^{t-1},V_{t})$ with $V_{t}$ being independent of $U_{t}(d^{t})$. In this case, conditioning on $Y^{t-1}(d^{t-1})$ is not needed and can be omitted, and thus Assumption (ref) boils down to
\begin{align*}
P( U_{t}(d^{t}) \leq \tau \mid X^{t}) = \tau.
\end{align*}
This expression is a shape-restriction in the distribution of $U_{t}(d^{t})$ given $X^{t}$, which essentially amounts to a re-normalization of $\mu_{t}$ under the assumption that this distribution is increasing. Moreover, it is straightforward to check that an analogous conclusion can be reached when $D^{t}$ is independent of $U_{t}(d^{t})$ conditional on $Y^{t-1}$; in this case the shape-restriction is over the distribution of $U_{t}(d^{t})$ given $(Y^{t-1}, X^{t})$.
While these cases cover many applications, there are many other applications where $V_{t}$ is not independent of $U_{t}(d^{t})$. In this case, $V^{t}$ can be viewed as a control variable, and Assumption (ref) is analogous to the control function framework for quantile regression (e.g. lee2007endogeneity). All three cases are further illustrated in Example (ref) below. $\triangle$
We now use example (ref) to illustrate the outcome function and verify the assumptions.
example[Canonical Example: Verification of Assumptions (ref) and (ref)]
We now verify Assumptions (ref) and (ref) in the setup of Example (ref) and by doing so illustrate how the structural function looks for this example. To do this we need to impose some restrictions. First, we assume that, for each $t \in \mathbb{T}$, given $Y_{t-1}$, $\epsilon_{t}$ is independent of $D^{t-1}$, $\epsilon^{t-1}$ and $Y^{t-2}$. This assumption can be somewhat relaxed to allow for a richer dependence on past outcomes, but, due to the non-linear nature of our framework, the standard assumption of $E[\epsilon_{t} \mid D^{t}] = 0$ will typically be insufficient to verify assumption (ref). Let $F(.|.)$ be the conditional CDF of $\epsilon_{t}$ given $Y_{t-1}$, which we assume is an increasing function of $\epsilon$.
Let $U_{t} : = U_{t}(d^{t}) : = F(\epsilon_{t} \mid Y_{t-1})$. It is straightforward to verify the following relationship: For any $u\in (0,1)$,
\begin{align*}
P(U_{t} \leq u \mid Y^{t-1}(d^{t-1}),D^{t}=d^{t}) = P( \epsilon_{t} \leq F^{-1}( u \mid Y_{t-1}(d^{t-1})) \mid Y^{t-1}(d^{t-1}),D^{t}=d^{t}) = u.
\end{align*}
Thus, Assumption (ref) holds. Let $(y^{t-1},d^{t},u) \mapsto \mu_{t}(y^{t-1},d^{t},u) : = \kappa + \sum_{s=0}^{L+1} \theta_{s} d_{t-s} + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + F^{-1}(u \mid y_{t-1})$. It is straightforward to check that this choice satisfies Assumption (ref) and that $Y_{t} = \mu_{t}(Y^{t-1},D^{t},U_{t})$ coincides with (ref). Note that $\mu_{t}$ is indexed by $(\kappa,(\theta_{s})_{s=1}^{L},(\gamma_{s})_{s=1}^{P})$ and the function $F$.
We observe that this specification allows the residual $\epsilon_{t}$ to be heterosckedastic and correlated with lags of the outcome variable. The cost of such flexibility is the estimation of the function $F^{-1}$. Indeed, if $\epsilon_{t}$ is assumed to be IID, the structural function reduces to $(y^{t-1},d^{t}) \mapsto \mu(y^{t-1},d^{t},\tau) = \kappa + \sum_{s=0}^{L} \theta_{s} d_{t-s} + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + F^{-1}(\tau)$, which is entirely parameterized by finite dimensional parameters.
Finally, we note that $F^{-1}(u \mid Y_{t-1})$ is the $u$-th conditional quantile of $\epsilon_{t}$ given $Y_{t-1}$. Thus, the structural function $\mu_{t}(\cdot,\cdot,u)$ coincides with the conditional quantile function in lee2007endogeneity (see equation 4). This fact is present in many of the examples below and it illustrates the link between our framework and a quantile regression model under the control function approach. $\triangle$
The next example illustrates different examples of the process for $(D_{t})_{t=1}^{T}$.
example[Canonical example: The process for $(D_{t})_{t=1}^{T}$]
Expression (ref) allows for many specifications for the treatment process. One simple case is where $D^{T}$ is exogenous but autocorrelated, e.g., for all $t \in \mathbb{T}$, $D_{t} = \delta_{t}(D^{t-1},V_{t})$ where $V^{T}$ is IID. In some applications, however, this assumption might be too strong, as $D_{t}$ could depend not only on past realizations of itself but also on past outcomes. For these applications we can set $D_{t} = \delta_{t}(Y^{t-1},D^{t-1},V_{t})$ where $V^{T}$ is assumed to be IID. Observe that, conditional on $Y^{t-1}$, $D_{t}$ is independent of $U_{t}(d^{t})$. Finally, our framework allows for more general situations where $V^{t}$ is correlated with the error in the outcome equation.
We illustrate this case in a standard election setup, i.e., $D_{t} = 1\{X_{t} \geq 0.5 \}$ with $X_{t}$ being the shares of votes, but with the additional feature that past election outcomes can influence the voting behavior, i.e., $X_{t} := m(D_{t-1},V_{t})$ for some function $m$.\footnote{In this case $X$ is continuous. In Remark (ref) we discuss how to include continuous covariates in our setup.} The “election shock" $V_{t}$ is independent across time but can be correlated with $\epsilon_{t}$ in the outcome equation. Under the condition that the conditional CDF of $\epsilon$ given $X$, $F(.|.)$, is increasing on $\epsilon$, assumptions (ref) and (ref) hold in this case with $U_{t} : = F (\epsilon_{t}|X_{t}) $ and
\begin{align*}
(y^{t-1},x^{t},d^{t},u) \mapsto \mu_{t}(y^{t-1},x^{t},d^{t},u) : = \kappa + \sum_{s=0}^{L} \theta_{s} d_{t-s} + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + F^{-1}(u \mid x_{t}),
\end{align*}
because, for any $u \in (0,1)$ any $Y^{t-1}$, and any $X^{t}$ such that $ 1\{ X_{s} \geq 0.5 \} = D_{t}~\forall s\leq t$,
\begin{align*}
P(U_{t}(d^{t} ) \leq u \mid Y^{t-1}, D^{t} ,X^{t} ) = E [ 1\{ U_{t}(d^{t}) \leq u \} \mid X_{t} ] = E [ 1\{ \epsilon_{t} \leq F^{-1}(u|X_{t}) \} \mid X_{t} ] = u.
\end{align*}
$\triangle$
Finally, we provide some extensions of example (ref) to illustrate the scope of our framework. The first of these extensions shows that it allows for specifications that go beyond additive-separable models.
example[Canonical example: Quantile Regression]
In Example (ref) we study the conditional mean of the economic outcome given past realizations of the treatment (and other variables). However, it could be of interest to also assess the effect of the treatment on other aspects of the outcome distribution. To do this, we consider the following quantile regression model.
\begin{align*}
Y_{i,t} = \kappa(U_{i,t}) + \sum_{s=0}^{L} \theta_{s}(U_{i,t}) D_{i,t-s} + \sum_{s=1}^{P} \gamma_{s}(U_{i,t}) Y_{i,t-s} + U_{i,t}
\end{align*}
where $U_{i,t} \sim U(0,1)$. It is easy to show that the $\tau$-quantile of $Y_{t}$ given $(D_{s})_{s=t-L}^{t}$ and $(Y_{s})_{s=t-P}^{t-1}$ is given by $\kappa(\tau) + \sum_{s=0}^{L} \rho_{s}(\tau) D_{t-s} + \sum_{s=1}^{P} \gamma_{s}(\tau) Y_{t-s}$, that Assumption (ref) holds and that $(y^{t-1},d^{t}) \mapsto \mu_{t}(y^{t-1},d^{t},\tau) = \kappa(\tau) + \sum_{s=0}^{L} \theta_{s}(\tau) d_{t-s} + \sum_{s=1}^{P} \gamma_{s}(\tau) y_{t-s} + \tau$. Moreover, under the assumption that $\kappa(\cdot)$, $\theta_{s}(\cdot)$ and $\gamma_{l}(\cdot)$ are increasing for all $s \in \{0,...,L\}$ and $l \in \{1,...,P\}$, Assumption (ref) holds. Observe that as opposed to the linear case studied in Example (ref), the structural function, $\mu_{t}(\cdot,\cdot,\tau)$ is only indexed by finite dimensional parameters. $\triangle$
The second of these extensions shows how our framework can handle richer specifications with less functional form restrictions.
example[Canonical example: More general functional form]
In many applications the correct order of lagged outcomes is unknown, and it is not even clear that the lagged outcomes affect current outcomes linearly. Thus, a possible extension for the specifications in Examples (ref) and (ref) is given by
\begin{align*}
Y_{t}(d^{t}) = \sum_{s=0}^{t} \theta_{s} d_{t-s} + \varphi( Y^{t-1}(d^{t-1}), X^{t} ,\epsilon_{t}(d^{t}) )
\end{align*}
where $\epsilon \mapsto \varphi(y^{t-1},x^{t},\epsilon)$ is an unknown increasing function, and $\epsilon$ is such that its $\tau$-th conditional quantile given $(Y^{t-1}, X^{t} , D^{t} )$ equals $q(Y^{t-1},X^{t})$. This specification can be handled by our framework by defining $U_{t}(d^{t}): = \varphi( Y^{t-1}(d^{t-1}), X^{t} ,\epsilon_{t}(d^{t}) ) - \varphi( Y^{t-1}(d^{t-1}), X^{t} , q(Y^{t-1}(d^{t-1}),X^{t}) ) + \tau$ and $\mu_{t}(y^{t-1},x^{t},d^{t},u) : = \sum_{s=0}^{t} \theta_{s} d_{t-s} + \kappa( y^{t-1}, x^{t} ) + u $ where $\kappa(y^{t-1},x^{t} ) : = \varphi( y^{t-1}, x^{t} , q(y^{t-1},x^{t}) ) - \tau $; it is to see that Assumptions (ref)-(ref) hold. Under these assumptions $\sum_{s=0}^{t} \theta_{s} d_{t-s} $ is the $\tau$-th quantile treatment effect (cf. chernozhukov2005iv) and would be the object of interest and $\kappa$ would be a nuisance (infinite dimensional) parameter.
Implicit in this specification --- as well as those in Examples (ref) and (ref) --- lies the assumption that the treatment effect function, $d^{t} \mapsto \sum_{s=0}^{t} \theta_{s} d_{t-s} $, is the same for each observed characteristic and linear on the treatment profile; e.g., the treatment effect for $d^{t}=(1,1,0,...,0)$ is the same as the sum of the effects for $(1,0,0,...,0)$ and $(0,1,0,...,0)$. In some applications these restrictions may be too strong as they restrict potential sources of heterogeneity of treatment effects. For instance --- in the context of the applications mentioned in Example (ref) --- the effect of having two (or more) consecutive weather/climate shocks may have a larger effect on GDP growth than the sum of the effects of each individual shock. To account for this heterogeneity, we can consider the following “saturated" model,
\begin{align*}
Y_{t}(d^{t}) = \sum_{\tilde{d}^{t} \in \{0,1\}^{t} } \theta( \tilde{d}^{t} , X^{t} ) 1\{ \tilde{d}^{t} = d^{t} \} + \varphi( Y^{t-1}(d^{t-1}), X^{t} ,\epsilon_{t}(d^{t}) ) ,
\end{align*}
where $ \theta( d^{t} , x^{t} ) $ is the $\tau$-th quantile treatment effect for $x^{t}$. In our empirical application in Section (ref), we employ a (simplified) version of this saturated model to study the effect on weather shocks on GDP growth. $\triangle$
Henceforth, we will typically omit the $t$ index for the functions $\mu_{t}$ and $\delta_{t}$.
The object of interest and null hypothesis
In this paper, we want to test the hypothesis whether the effect of an specific set of treatment profiles, $\mathcal{D}^{t} \subseteq \{0,1\}^{t}$, is the same as effect of the control treatment profile, $0^{t}$, for all units that share the same characteristics $(x^{t},\tau)$.
Formally,
align[align omitted — 135 chars of source]
for some $(x^{t},\tau) \in \mathbb{X}^{t} \cup \{\emptyset\} \times (0,1)$. The “$\emptyset$" in the definition is used to denote cases wherein no conditioning on the observed characteristics takes place; in this case, $\mu(\cdot,\emptyset,\cdot,\tau) = : \mu(\cdot,\cdot,\tau)$.
To shed some light on this hypothesis, consider an outcome function that has certain separability properties regarding $Y^{t-1}$ and $D^{t}$, i.e., $ \mu(y^{t-1},x^{t},0^{t},\tau) = \sum_{\tilde{d}^{t} \in \{0,1\}^{t} } \theta(\tilde{d}^{t},x^{t},\tau) 1\{ \tilde{d}^{t} = 0^{t} \} + g( y^{t-1}, x^{t} , \tau ) $ for some unknown functions $\theta$ and $g$. The causal effect of $d^{t}$ on the outcome $Y_{t}$ is given by $\theta(d^{t},x^{t},\tau)$ and is heterogeneous in the characteristics $(x^{t},\tau)$ (but not $y^{t-1}$). The null hypothesis is thus equivalent to inquiring whether the treatment effect is zero, $\theta(d^{t},x^{t},\tau) = 0 $, for treated groups given by $(d^{t},x^{t},\tau)$ with $d^{t} \in \mathcal{D}^{t}$.
remark[Some further remarks about expression (ref)]
(1) While we use the group with $(x^{t},0^{t})$ as control group, our method allows to use any other group as benchmark, provided the cardinality of the group diverges with the sample size; see Section (ref) for a more thorough discussion.
(2) Our approach allows us to condition on observed and unobserved characteristics, making our null “sharper" than the typical null in Fisher randomization methods (e.g., rosenbaum2010design). We, however, impose stronger conditions that those typically used in that literature and we also do not condition on potential outcomes.
(3) While our null is written for a particular calendar time $t$, in Appendix (ref) we extend this setup to allow for multiple time periods.
$\triangle$
We now illustrate the null hypothesis in the context of example (ref).
example[Canonical example: Null hypothesis]
For example (ref) and $\mathcal{D}^{t} = \{d^{t}\}$, it is easy to see that our null hypothesis translates to $H_{0}~:~\sum_{s=0}^{L} \theta_{s} d_{t-s} = 0$. Hence, by choosing different treatment profiles we can replicate the common null hypothesis studied in the literature, for instance to test for cumulative effects $H_{0}~:~\sum_{s=0}^{L} \theta_{s}=0$. We can also test joint hypothesis, e.g., $H_{0}~:~\theta_{0} = ... = \theta_{L} = 0$ by setting $\mathcal{D}^{t}$ as the set of profiles of the type $(0,...,0,1,0,...,0)$. $\triangle$.
The next example illustrates our null hypothesis for non-additively separable models.
example[Canonical example: Quantile Regression (cont.)]
For the model in example (ref) our null hypothesis with $\mathcal{D}^{t} = \{d^{t}\}$ translates into $H_{0}~:~\sum_{s=0}^{L} \theta_{s}(\tau) d_{t-s} = 0$.
That is, our framework allow us to test the null hypothesis of no treatment effect for units with unobserved characteristics equal to $\tau$; it can also allow us to test for joint hypothesis by specifying a non-singleton value of $\mathcal{D}^{t}$. By shifting the value of $\tau$ we can test the direct treatment at different percentiles of the cross-unit outcome distribution.
$\triangle$.
Main Result
In what follows, $t \in \mathbb{T}$ and $(x^{t},\tau) \in \mathbb{X}^{t} \cup \{\emptyset \} \times (0,1)$ and $\mathcal{D}^{t} \subseteq \{0,1\}^{t}$ are taken to be fixed. The goal is to construct a test for the aforementioned null hypothesis,
align*[align* omitted — 110 chars of source]
The test is akin to an specification test as it based on the behavior of conditional moments that identify the structural mapping $\mu$. The non-standard feature is that the number of units in each group --- a group is all units with the same $(X^{t},D^{t})$ --- does not necessarily grow with $n$ or it does so at a very slow rate for all the groups. However, since the number of groups are fixed, as $n$ diverges, necessarily at least one group's size must be proportional to $n$. We normalize this group to be $(x^{t},0^{t})$. We use this insight to construct a consistent estimator of $\mu_{t}$, and, under the null, we plug-in this quantity in the (sample) conditional quantile moments for other groups in $\mathbb{X}^{t} \times \mathcal{D}^{t}$. Finally, in order to derive the distribution of the test statistic, we apply the results in chernozhukov2009finite who derive the finite-sample distribution of sample quantile moments. It is worth to point out that the estimator presented here is standard and is presented for completeness. Any other estimator of $\mu_{t}$ that satisfies the conditions listed in sub-section (ref) can be used to construct our test.
In what follows, let $O^{t} : = (Y^{t-1},X^{t},D^{t})$. Also, it is useful to keep track of the units that belong to different treatment groups. A group is indexed by $g \in \mathbb{X}^{t} \cup \{ \emptyset \} \times \{0,1\}^{t} $, and let $\mathcal{G}_{n}(g) : = \{ 1 \leq i \leq n \colon (X^{t}_{i},D^{t}_{i}) = g \}$ be all units in a sample of $n$ units that belong to group $g$. As mentioned above, we “normalize" the group $(x^{t},0^{t})$ to grow with the sample size $n$; i.e., $P( (X^{t},D^{t}) = (x^{t},0^{t}) ) > 0$. Importantly, we allow for the size of other groups to grow slower than $n$ or not grow at all. Finally, let $M : = |\mathcal{D}^{t}|$ and sometimes we index the elements in $\mathcal{D}^{t}$ as $d^{t}[m]$ for $m \in \{1,...,M\}$, and for convenience $d^{t}[0] : = 0^{t}$.
We conclude this introduction with a remark about continuous covariates.
remark[Continuous covariates]
Our methodology can accommodate situations in which the outcome function $\mu$ also depends on continuous covariates, denoted as $X_{c}$, i.e., $\mu(Y^{t-1},X^{t}_{c},X^{t},D^{t},U_{t})$, provided we cast our null hypothesis as
\begin{align*}
H_{0} \colon \mu(\cdot,\cdot,x^{t},d^{t},\tau) = \mu(\cdot,\cdot,x^{t},0^{t},\tau), \forall d^{t} \in \mathcal{D}^{t}.
\end{align*}
Importantly, the treatment groups are still defined in terms of the treatment profiles and the discrete covariates. In fact, we can define $\tilde{Y}^{t-1} : = (Y^{t-1},X^{t}_{c})$ and all the estimation and inferential results go through with $\tilde{Y}$ instead of $Y$.
As such, this extension is useful for situations where $X^{t}_{c}$ acts as a control variable. In other situations, where we want to learn the treatment effect for particular values of $X^{t}_{c}$, e.g., whether $\mu(\cdot,x^{t}_{c},x^{t},d^{t},\tau) = \mu(\cdot,x^{t}_{c},x^{t},0^{t},\tau)$, a more involved modification of our estimation and inference method is needed. While this could be an interesting extension, it is outside the scope of this paper. $\triangle$
Estimation Technique
For each $\delta^{t} \in \mathcal{D}^{t} \cup \{ 0^{t} \} $, let $\mathbb{H}(\delta^{t})$ denote the class of functions which $\mu(\cdot,x^{t},\delta^{t},\tau)$ belongs to. The exact structure of this set will depend on the particular application, but it could be a class of functions indexed by both finite and infinite dimensional parameters. For each $h \in \mathbb{H}(\delta^{t})$, let $y^{t} \mapsto \rho(y_{t},h(y^{t-1})) : = 1\{ y \leq h(y^{t-1}) \} $.
Under Assumptions (ref) and (ref), Lemma (ref) in Appendix (ref) and LIE imply that, for any $\delta^{t} \in \mathcal{D}^{t} \cup \{ 0^{t} \} $, $\mu(\cdot,\delta^{t},x^{t},\tau)$ satisfies the following equation
align*[align* omitted — 196 chars of source]
where $h \mapsto Q(x^{t},\delta^{t},h,P)$ is defined as the population criterion function. The test exploits this relationship, but since the conditional mean inside the population criterion function is unknown, we need to estimate this conditional mean in order to construct our test statistic. To do this, we follow a sieve-based approach (see chen2007large for a review) and approximate the conditional mean (as a function of $Y^{t-1}$) with $K$ unconditional moments and allowing $K$ to grow with the sample size. Formally, for any $\delta^{t} \in \{ 0^{t} , d^{t} \}$, let
align*[align* omitted — 113 chars of source]
be the regularized criterion function, where
align*[align* omitted — 154 chars of source]
and $\Phi_{K} : \mathbb{Y}^{t-1} \rightarrow \mathbb{R}^{K}$ is some integrable vector-valued function that defines the unconditional moment to be taken into account; for instance $\Phi_{K} : = (\phi_{1},...,\phi_{K})$ where the $(\phi_{k})_{k}$ is a basis functions such as polynomials, wavelets, etc. This construction is standard (e.g. chen2012estimation, CvKL2003 among others) with the minor caveat that in our case we condition on discrete random variables, $(X^{t},D^{t})$ as well as continuous, $Y^{t-1}$, so the regularization is only done over $Y^{t-1}$ not $(X^{t},D^{t})$.
Our sample criterion function takes the regularized one and replaces $P$ --- the unknown probability over the data --- by its empirical counterpart $$P_{n}(A) : = n^{-1} \sum_{i=1}^{n} 1\{ (Y^{t}_{i},X^{t}_{i},D^{t}_{i}) \in A \}$$ for any measurable set $A \subseteq \mathbb{Y}^{t} \times \mathbb{X}^{t} \times \{0,1\}^{t}$. For any $\delta^{t} \in \mathcal{D}^{t} \cup \{ 0^{t} \} $, the estimator of $\mu(\cdot,\delta^{t},x^{t},\tau)$ is analogous to a sieve-based minimum distance one (ai2003efficient) and it's defined as follows: For any $(n,K) \in \mathbb{N}^{2}$,\footnote{Implicit is the assumption that there exists one and only one solution. In lieu of this assumption one can use approximate minimizers as the estimators. In addition, we assume that there is at least one unit with $(x^{t},0^{t})$ and $(x^{t},d^{t})$.}
align[align omitted — 158 chars of source]
where and $(\mathbb{H}_{K}(\delta^{t}) )_{K}$ are a sequence of sieves, i.e., sets such that $\overline{\cup_{K} \mathbb{H}_{K}(\delta^{t}) } = \mathbb{H}(\delta^{t})$. Observe that $m_{K}(x^{t},\delta^{t},h,P_{n})$ can be cast as
align*[align* omitted — 200 chars of source]
Thus, $m_{K}(x^{t},\delta^{t},h,P_{n})$ --- and consequently $Q_{K}(x^{t},\delta^{t},h,P_{n}) $ --- only depend on the data corresponding to the group with treatment profiles equal to $\delta^{t}$ and discrete observed characteristics $x^{t}$.
Before presenting our test statistic and the main result, we illustrate the aforementioned quantities in the context of Example (ref).
example[Canonical Example (cont.): Illustration of the Estimator]
For any $d^{t}$, $y^{t-1} \mapsto \mu(y^{t-1},d^{t},\tau) = \kappa + \sum_{s=0}^{L} \theta_{s} d_{t-s} + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + F^{-1}(\tau \mid y_{t-1})$. Hence, $\mathbb{H}(d^{t})$ is a class of functions indexed by finite dimensional vectors --- $(\kappa, (\theta_{s})_{s=0}^{L} , (\gamma_{s})_{s=1}^{P})$ --- and a function $ y \mapsto F^{-1}(\tau \mid y)$. In particular, $\mathbb{H}(0^{t})$ is the class of functions of the form $y^{t-1} \mapsto b + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + F^{-1}(\tau \mid y_{t-1}) $ and we assume that $(b, (\gamma_{s})_{s=1}^{P}) \in \mathcal{B}$ where $\mathcal{B}$ is a compact subset of $\mathbb{R}^{1+P}$ and $F^{-1}(\tau \mid \cdot ) \in \mathcal{H}$ where $\mathcal{H}$ is a class of real-valued functions to be determined later.
We now discuss the estimation of $\mu(\cdot,0^{t},\tau)$ in detail. For any $K \in \mathbb{N}$, the sieve space $\mathbb{H}_{K}(0^{t})$ is constructed analogously to $\mathbb{H}(0^{t})$ but we restrict the index parameters to $(b,(\gamma_{s})_{s=1}^{L}, h) \in \mathcal{B} \times \mathcal{H}_{K}$ where $\mathcal{H}_{k} : = \{ \sum_{l=1}^{K} \pi_{l} \psi_{k} \colon (\pi_{1},...,\pi_{k}) \in \mathbb{R}^{k} \} \cap \mathcal{H}$ and $(\psi_{k})_{k}$ is some basis functions such polynomials, P-splines, etc; non-linear sieves such as neural networks or trees and shape restrictions (e.g. freyberger2015identification, horowitz2017nonparametric) can also be employed. For any $(n,K)$, the estimator, denoted by $ (\hat{\kappa}_{n,K}, (\hat{\gamma}_{n,K,s})_{s=1}^{P} , \hat{h}_{n,K} ) $ is the solution to the optimization problem \footnote{ If the “min" does not exist, we can replace by an approximate minimizer; similarly, if there are many solutions, we just select one.}
\begin{align*}
\min_{ (b,(\gamma_{s})_{s=1}^{P}, h) \in \mathcal{B} \times \mathcal{H}_{K} } \left \Vert \frac{1}{|\mathcal{G}_{n}(0^{t})|} \sum_{i \in \mathcal{G}_{n}(0^{t})} (1\{ Y_{t,i} \leq b + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + h( y_{t-1}) \}- \tau)\Phi_{K}(Y^{t-1}_{i}) \right \Vert^{2}.
\end{align*}
This estimator corresponds to a sieve-based estimator of a quantile IV semi-parametric problem where the effective sample size is not $n$ but $|\mathcal{G}_{n}(0^{t})|$; which by our assumptions is proportional to $n$. This estimation problem has been extensively studied in the literature, e.g. CvKL2003, chen2009efficient, chen2012estimation and its asymptotic properties --- rate of convergence in particular --- are well understood. $\triangle$
Test Statistic
We now construct our test statistic and state our main result about it's asymptotic properties. First, we present one of the building block for our result is the following lemma, which uses the insights by chernozhukov2009finite and gives us the distribution of the moment function, under the null, in finite samples.
lemmaSuppose Assumptions (ref) - (ref) hold. Then, for any $(n,K) \in \mathbb{N}^{2}$ and any $\delta^{t} \in \mathcal{D}^{t} \cup \{ 0^{t} \} $, under the null hypothesis, conditionally on $(Y^{t-1}_{i},X^{t}_{i},D^{t}_{i})_{i=1}^{n}$ such that $\mathcal{G}_{n}(x^{t},\delta^{t}) \ne \{\emptyset \} $, the distribution of $m_{K}(x^{t},\delta^{t},\mu(0^{t},x^{t},\tau),P_{n})$ is the same as the distribution of
\begin{align*}
\ell_{n,K}(\delta^{t}) : = \frac{1}{| \mathcal{G}_{n}(x^{t},\delta^{t})| } \sum_{i \in \mathcal{G}_{n}(x^{t},\delta^{t}) } (B_{i} - \tau)\Phi_{K}(Y^{t-1}_{i})
\end{align*}
where $(B_{i})_{i=1}^{n}$ are IID drawn from $Ber(\tau)$.
proofSee Appendix (ref).
The qualifier $\mathcal{G}_{n}(x^{t},\delta^{t}) \ne \{\emptyset \} $ ensures we do not have division by zero. Henceforth, we will re-define $m_{K}(x^{t},\delta^{t},.,P_{n})$ to be naught if $\mathcal{G}_{n}(x^{t},\delta^{t}) = \{\emptyset \} $ and avoid this qualification; the same is done for $\ell_{n,K}(\delta^{t})$.
We now construct our test statistic. We have $M+1$ moments $(m_{K_{l}}(x^{t},d^{t}[l],\cdot,P_{n}))_{l=0}^{M}$ and each of these is a vector of dimension $K_{l} \times 1$. Thus, there are several ways of constructing our test statistic, which is basically a notion of distance between these moments and zero. To model this, let $H : \mathbb{R}^{K_{0}} \times .... \times \mathbb{R}^{K_{M}} \rightarrow \mathbb{R}_{+}$ be function such that $H(0) = 0$ and there exists a finite constant $C$ such that $| H(a_{0},...,a_{M}) - H(b_{0},...,b_{M}) |\leq C \max_{l \in \{0,...,M\}} | ||a_{l}||^{2} - ||b_{l}||^{2} | $ for any $(a_{0},...,a_{M})$ and $(b_{0},...,b_{M})$. Examples of this function are $H(x_{0},...,x_{M}) = \sum_{l=0}^{M} ||x_{l}||^{2}$ or a weighted version of it; $H(x_{0},...,x_{M}) = \max_{l \in \{0,...,M\}} ||x_{l}||^{2}$; or, when all the vectors have the same dimension, $H(x_{0},...,x_{M}) = (\sum_{l=0}^{M} x_{l})^{2}$ or a weighted version of it.
Our test statistic is thus given by $Q_{\mathbf{K}}(x^{t},\hat{\mu}_{n,K_{0}}(\cdot,x^{t},0^{t},\tau),P_{n}) $, where, abusing notation
align*[align* omitted — 126 chars of source]
for any $h \in \mathbb{H}$. In addition, let $\mathcal{B}_{n,\mathbf{K}}$ be the analogous quantity but using the moments introduced in Lemma (ref), i.e.,
align*[align* omitted — 104 chars of source]
The other building block for the main result of the paper is to show that, as the size of the control group grows with the sample size, $Q_{\mathbf{K}(n)}(x^{t},\hat{\mu}_{n,K_{0}(n)}(\cdot,x^{t},0^{t},\tau),P_{n})$ approximates $Q_{\mathbf{K}(n)}(x^{t},\mu(\cdot,x^{t},0^{t},\tau),P_{n})$ for some sequence $(\mathbf{K}(n))_{n}$. The following high-level assumptions are used to show this result.
assumptionLet $F(.|.)$ be the conditional CDF of $Y_{t}(d^{t})$ given $Y^{t-1} ,D^{t}$. There exists a $\delta >0$ and a finite $y^{t-1} \mapsto A(y^{t-1})$ such that for any $(y_{t-1},d^{t}) \in \mathbb{Y}^{t-1} \times \{0,1\}^{t}$,
\begin{align*}
| F( a \mid y^{t-1} , d^{t} ) - F( b \mid y^{t-1} , d^{t}) | \leq A(y^{t-1}) |a-b|^{\delta}
\end{align*}
for all $a$ and $b$ in $\mathbb{R}$
assumptionThere exists a $q \in[1,\infty]$, a sequence $(\mathbf{K}(n): = K_{0}(n),K_{1}(n),...,K_{M}(n))_{n \in \mathbb{N}}$ and diverging sequences $(\eta^{-1}_{n},\varpi_{n})_{n \in \mathbb{N}}$ such that: (i) \footnote{The $\delta>0$ in the expression corresponds to the one in Assumption (ref).}
\begin{align*}
P \left( \max_{d^{t} \in \mathcal{D}^{t} \cup \{0^{t}\}} ||\hat{\mu}_{n,K_{0}(n)}(\cdot,x^{t},0^{t},\tau) - \mu(\cdot,x^{t},0^{t},\tau)||_{L^{\delta q}(P(\cdot| x^{t},d^{t} )) } \geq \eta_{n}/\varpi_{n} \right) \leq \varpi^{-1}_{n},
\end{align*}
and (ii)
\begin{align*}
(\eta_{n})^{\delta} \max_{l \in \{0,...,M\}} || \sqrt{A} \Phi_{K_{l}(n)} ||^{2}_{L^{q'}(P(\cdot|x^{t},d^{t}[l]))} = o(1) and \max_{l \in \{1,...,M\}} \frac{ || \Phi_{K_{l}(n)} ||^{2}_{L^{2}(P(\cdot|x^{t},d^{t}[l]))} } { \sqrt{\varpi_{n}} }=o(1)
\end{align*}
with $q'$ such that $1/q'+1/q = 1$.
assumptionThere exists a positive real-valued sequences, $(\varrho_{n})_{n}$ and $(K_{0}(n))_{n}$, such that $\varrho_{n} = o(1)$ and
\begin{align*}
\Vert \sup_{f \in \mathcal{M}_{K_{0}(n)}(0^{t}) } n^{-1} \sum_{i=1}^{n} 1\{ G_{i} = (x^{t},0^{t}) \} \{ f(Y_{t,i},Y^{t-1}_{i} ) - E_{P}[ f(Y_{t},Y^{t-1} ) \mid G = (x^{t},0^{t}) ] \} \Vert^{2} = o_{P} ( \varrho_{n} ),
\end{align*}
where
\begin{align*}
\mathcal{M}_{K_{0}(n)}(0^{t}) : = \left\{ (y_{t},y^{t-1}) \mapsto (1\{ y_{t} \leq h(y^{t-1}) \} - 1\{ y_{t} \leq \mu (y^{t-1},x^{t},0^{t},\tau) \}) \Phi_{K_{0}(n)}(y^{t-1}) \colon h \in \mathbb{H}_{K_{0}(n)}(0^{t}) \right\}.
\end{align*}
remark[Comments about the assumptions]
Assumption (ref) imposes uniform H\"{o}lder continuity of $y \mapsto F( y \mid y^{t-1} , d^{t} ) $, where the constant $A(Y^{t-1})$ needs to satisfy certain integrability conditions described in Asssumption (ref)(ii). Under assumptions (ref) and a strengthening of Assumption (ref) that imposes a uniform conditional CDF for $U$, it follows that
\begin{align*}
y \mapsto F(y \mid y^{t-1} , d^{t} ) = \mu^{-1}(y^{t-1} ,d^{t} , y )
\end{align*}
where $y \mapsto \mu^{-1}(y^{t-1} ,d^{t} , y )$ denotes the inverse function of $\mu_{t}$ with respect its last component. Thus, Assumption (ref)(i) actually imposes uniform H\"{o}lder continuity on this function.
Asssumption (ref)(i) is a high level condition. As Lemma (ref) in Appendix (ref) shows, it is implied by the standard condition that $\hat{\mu}_{n,K_{0}(n)}$ consistently estimates $\mu$ at a rate faster than $\eta_{n}$; the sequence $(\varpi^{-1}_{n})_{n}$ simply quantifies the “faster than". The fact that we split the rate into $(\eta_{n})_{n}$ and $(\varpi_{n})_{n}$ is merely for exposition, as it facilitates the presentation of part (ii) of the assumption. Note that the assumption is silent about $\hat{\mu}_{n,k}$ for the “treated groups", it only restricts the behavior of $\hat{\mu}_{n,K_{0}(n)}$ for the “control group", which under our assumptions has cardinality diverging with the sample size. Hence, while high level, this assumption can be established invoking the asymptotic results for estimation of semi-/non-parametric conditional moment models such as CvKL2003, chen2012estimation, horowitz2007nonparametric, gagliardini2012nonparametric with one caveat which we now discuss. The notion of distance used in this assumption is non-standard and is given by $|| \cdot ||_{L^{q\delta}(P(\cdot| x^{t},d^{t} )) }$. The non-standard aspects are the fact that we are using an $L^{p}$ norm with $p=q \delta$ and the measure is $P(.|x^{t},d^{t})$, however, this norm can be linked to several standard norms. For instance, it is easy to see that this norm is bounded by the $L^{\infty}$ norm. Moreover, if $P((X^{t},D^{t}) = (x^{t},d^{t}) ) > C^{-1} > 0$ for all $(x^{t},d^{t})$, it follows that $\max_{d^{t} \in \{0,1\}^{t}} || \cdot ||_{L^{\delta q}(P(\cdot| x^{t},d^{t} )) } \leq C^{-1/(\delta q)} ||.||_{L^{\delta q}(P)}$.
Assumption (ref)(ii) imposes growth restrictions on $ k \mapsto || \sqrt{A} \Phi_{k} ||^{2}_{L^{q'}(P(\cdot|x^{t},d^{t}))} $ and $k \mapsto || \Phi_{k} ||^{2}_{L^{2}(P(\cdot|x^{t},d^{t}))}$ based on $(\eta_{n})_{n} $ and $(\varpi_{n})_{n}$. We observe that, given the nature of our problem wherein the number of treated units may not grow, $K_{1}(n),...,K_{M}(n)$ can be chosen not to grow with $n$, in this case the second display of part (ii) becomes vacuous.
Assumption (ref) is used in Lemma (ref) which handles the asymptotic behavior of the moment functions for the control group. If this group is not included in the test statistic --- i.e., we only include treatment groups --- then this assumption is not needed. This assumption is commonly used in semi-/non-parametric conditional moment models (see andrews1994empirical and van2000asymptotic for a review and CvKL2003 for usage in the context of quantile IV); it essentially is a uniform LLN over $\mathcal{M}_{K_{0}(n)}$, and requires controlling the growth of the complexity of the set $\mathbb{H}_{K_{0}(n)}(0^{t})$. $\triangle$
For a cutoff $t \geq 0$, we define our test as rejecting the null if $\phi_{\mathbf{K}}(t,P_{n}) = 1$ with
align*[align* omitted — 151 chars of source]
That is, our test statistic is akin to specification tests for GMM models in which one rejects the null hypothesis if the empirical criterion function is “too large" under the null. “Too large" is measured by the asymptotic distribution of the criterion function under the null. The difference with the classical results is that our asymptotic distribution, derived in Lemma (ref), is non-standard.
We now proceed to show that our test statistic controls size asymptotically for a suitably chosen cutoff.
theoremSuppose Assumptions (ref) - (ref) hold and there exists a sequence $(\mathbf{K}(n))_{n}$ that satisfies Assumptions (ref)-(ref). Then, under the null, for any $\alpha \in (0,1)$ and any $\delta>0$,
\begin{align*}
\limsup_{n \rightarrow \infty} E_{P}[\phi_{\mathbf{K}(n)}(t_{ \mathbf{K}(n) }(\alpha,(O^{t}_{i})_{i=1}^{n} ) + \delta ,P_{n})] \leq \alpha,
\end{align*}
where
\begin{align*}
t_{\mathbf{K}(n)}(\alpha,(O^{t}_{i})_{i=1}^{n} ) : = \inf \{ t \colon \Pr( \mathcal{B}_{n,\mathbf{K}(n)} \leq t \mid (O^{t}_{i})_{i=1}^{n} ) \geq 1-\alpha \}.
\end{align*}
proofSee Appendix (ref).
The $\delta>0$ in the theorem can be considered as an “infinitesimal" correction due to the fact that the distribution of $\mathcal{B}_{n,\mathbf{K}(n)}$ is discontinuous; its role is depicted in Lemma (ref) in the Appendix (ref).
remark[Heuristics behind the Proof]
To understand the idea behind the proof of the theorem, let's define an intermediate quantity
\begin{align*}
\tilde{\phi}_{\mathbf{K}}(t,P_{n}) : = 1\left\{ Q_{\mathbf{K}}(x^{t},\mu(\cdot,x^{t},0^{t},\tau),P_{n}) \geq t \right\}.
\end{align*}
This quantity is analogous to our test but with $\mu(\cdot,x^{t},0^{t},\tau)$ instead of the estimated version of it.
Lemma (ref) in the Appendix (ref) shows that $Q_{\mathbf{K}}(x^{t},\mu(\cdot,x^{t},0^{t},\tau),P_{n})$ and\\ $Q_{\mathbf{K}}(x^{t},\hat{\mu}_{n,K_{0}(n)}(\cdot,x^{t},0^{t},\tau),P_{n})$ are asymptotically equal. Using this result and Lemma (ref) in Appendix (ref), we then show that the law of our statistic is, asymptotically, well approximated by that of $\tilde{\phi}_{\mathbf{K}(n)}(t,P_{n})$, uniformly over $t$. In view of this result, it suffices to show that, under the null, the distribution of $\tilde{\phi}_{\mathbf{K}(n)}(t,P_{n})$ is well approximated by that of $ 1\left\{ \mathcal{B}_{n,\mathbf{K}(n)} \geq t \right\}$. This result follows immediately from Lemma (ref), and it is not asymptotic in nature, but holds for all sample sizes.
We conclude by pointing out that Assumption (ref) is used only for this last part of the result and, indeed, a weaker version of this assumption is used, which is:
\begin{align*}
P( U_{t}(d^{t}) \leq \tau \mid Y^{t-1}(d^{t-1}),X^{t} ,D^{t} = d^{t}) = \tau.
\end{align*}
The key difference is that in this version we only require the restriction to hold for $D^{t} = d^{t}$ (the same $d^{t}$ as in $U_{t}(d^{t})$) not for $D^{t} \ne d^{t}$. In fact, under certain mononotonicity restrictions on the conditional CDF of $U_{t}(d^{t})$, this condition amounts just to a re-normalization of the residuals. Hence, our main result still holds when Assumption (ref) is replaced by this weaker condition. However, by relaxing this assumption, the outcome function $d^{t} \mapsto \mu_{t}(y^{t-1},x^{t},d^{t},U_{t}(d^{t}))$ may no longer have an structural interpretation as the CDF of $U_{t}(d^{t})$ may also vary with $d^{t}$. $\triangle$
We now show that our technical assumptions hold --- under standard conditions used in the literature --- in the context of examples (ref) and (ref). We also use these examples to illustrate the test statistic and to show how our main result could be used to obtain confidence regions.
example[Canonical Example (cont.): Test Statistic and Confidence Regions ]
Recall that in this case, $(y^{t-1},d^{t}) \mapsto \mu(y^{t-1},d^{t},\tau) = \kappa + \sum_{s=0}^{L} \theta_{s} d_{t-s} + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + F^{-1}(\tau \mid y_{t-1})$ and thus $\mu^{-1}_{t}(y^{t-1},d^{t},y) : = F(y_{t} - \kappa - \sum_{s=0}^{L} \theta_{s} d_{t-s} - \sum_{s=1}^{P} \gamma_{s} y_{t-s} \mid y_{t-1})$. Hence, given Remark (ref), Assumption (ref) holds provided $y \mapsto F(y \mid y_{t-1})$ is H\"{o}lder continuous, i.e., $|F(a|y^{t-1}) - F(b|y^{t-1}) | \leq A ||a - b||^{\delta}$ with $\delta > 0$ and $A < \infty$.
As discussed in Example (ref), the estimation of $\mu(\cdot,0^{t},\tau)$ for the control group is akin to a sieve-based estimator for semi-parametric conditional quantiles, where the effective sample size is not $n$ but $|\mathcal{G}_{n}(0^{t})|$; which by our assumptions is proportional to $n$. Hence, the rate of convergence in Assumption (ref)(i) can be obtained from the literature, e.g. chen2012estimation. Based on this results and the choice of $\Phi_{K}$ one can choose $K_{I}(n)$ to ensure the validity of Assumption (ref)(ii).
Regarding Assumption (ref), recall that the set $\mathbb{H}_{K}(0^{t})$ is completely characterize by $\mathcal{B}$ and $\mathcal{H}_{K}$. Thus, assumption (ref) boilds down to
\begin{align*}
| \sup_{ f \in \mathcal{M}_{K_{0}(n)}(0^{t}) } n^{-1} \sum_{i=1}^{n} 1\{ D^{t}_{i} = 0^{t} \} f(Y_{i,t},Y^{t-1}_{i} ) - E[ 1\{D^{t} = 0^{t} \} f(Y_{t},Y^{t-1} ) ] |= o_{P}(\varrho_{n}),
\end{align*}
where \begin{align*}
\mathcal{M}_{K_{0}(n)}(0^{t}) : = \left\{ y^{t-1} \mapsto 1\{ y_{t} \leq b + \sum_{s=1}^{P} \gamma_{s} y_{t-s} + g(y_{t-1}) \} \colon (b,(\gamma_{s})_{s=1}^{P},g) \in \mathcal{B} \times \mathcal{H}_{K_{0}(n)} \right\}
\end{align*}
In Proposition (ref) in Appendix (ref) we show that if $\mathcal{H}$ (and thus $\mathcal{H}_{K}$ for any $K$) belongs to a certain class of sufficiently smooth functions, then Assumption (ref) holds with $\varrho_{n} = n^{-1/2} \log \log n $ and any sequence $(K_{0}(n) )_{n}$.
We have thus verified all the assumptions for Theorem (ref). We now show how to use this theorem to obtain confidence regions and joint hypothesis testing. For illustration, we test whether there is no treatment effect neither contemporaneously nor lagged, i.e., $\theta_{m} = 0$ for all $m \in \{0,...,L\}$. To test this hypothesis we choose $H(x_{0},...,x_{L}) = \sum_{m=0}^{L} ||x_{l}||^{2}$. Under this choice, $ Q_{\mathbf{K}}(x^{t},\hat{\mu}_{n,K}(\cdot,x^{t},0^{t},\tau),P_{n}) $ equals $\hat{Q}_{n,\mathbf{K}}(0)$ where, for any $\boldsymbol{\theta} \in \mathbb{R}^{L+1}$ (in this context, the "0" in the function is understood as the vector 0),
{ {
\begin{align*}
& \hat{Q}_{n,\mathbf{K}}(\boldsymbol{\theta}) \\
: = & \sum_{m=0}^{L} \left \Vert \sum_{i \in \mathcal{G}_{n}( 1_{t-m} ) } \Phi_{K_{l}}(Y^{t-1}_{i}) \left( \frac{ 1\{ Y_{t,i} \leq \hat{\kappa}_{n,K_{0}} + \theta_{m} + \sum_{s=1}^{P} \hat{\gamma}_{n,K_{0},s} Y_{t-s,i} + \hat{h}_{n,K_{0}}(Y_{i,t-1}) \} - \tau }{ |\mathcal{G}_{n}( 1_{t-m} ) | } \right) \right \Vert^{2}
\end{align*}
}}
and $1_{t-m}$ for $m \in \{ 0 , 1,..., L \} $ be a $t \times 1$ vector with $1$ in the $m$-th coordinate and 0 everywhere else. Thus, our test will reject the null hypothesis that $ \theta_{m} = 0$ for all $m \in \{0,...,M\}$, if $\hat{Q}_{n,,\mathbf{K}(n)}(0) > t(\alpha) + \delta$ where $$t(\alpha) = \inf \{ t \colon \Pr( \sum_{m=0}^{L} \left \Vert \sum_{i \in \mathcal{G}_{n}(1_{t-m} ) } \Phi_{K_{l}(n)}(Y^{t-1}_{i}) \frac{ B_{i} - \tau }{ |\mathcal{G}_{n}( 1_{t-m} )| } \right \Vert^{2} \geq t \mid (Y^{t-1}_{i},D^{t}_{i})_{i=1}^{n} ) \geq 1-\alpha \}$$ and $\delta$ is an “infinitisimal small" correction defined in Theorem (ref).\footnote{The dependence of $t(\alpha)$ on $(Y^{t-1}_{i},D^{t}_{i})_{i=1}^{n}$ is left implicit for the sake of the exposition.}
We define the $1-\alpha$ confidence region as
\begin{align*}
CR_{n}(\alpha): = \left\{ \boldsymbol{\theta} \in\mathbb{R}^{L+1} \colon \hat{Q}_{n,\mathbf{K}(n)}(\boldsymbol{\theta}) \leq t(\alpha) + \delta \right\}.
\end{align*}
The next proposition shows that these confidence regions have correct asymptotic confidence level.
\begin{proposition}
Suppose all the assumptions in Theorem (ref) hold. Let $(\theta_{s})_{s=0}^{L}$ be the true coefficients. Then, under the null hypothesis $ \theta_{s} = 0$ for all $s \in \{0,...,L\}$ and $ \liminf_{n \rightarrow \infty} P(CR_{n}(\alpha) \ni (\theta_{s})_{s=0}^{L} ) \geq 1-\alpha $.
\end{proposition}
\begin{proof}
See Appendix (ref).
\end{proof}
$\triangle$
example[Canonical Example: Quantile regression (cont.)]
For the quantile regression case (example (ref)), $(y^{t-1},d^{t}) \mapsto \mu(y^{t-1},d^{t},\tau) = \kappa(\tau) + \sum_{s=0}^{L} \theta_{s}(\tau) d_{t-s} + \sum_{s=1}^{P} \gamma_{s}(\tau) y_{t-s} + \tau$ and thus the CDF of $Y_{t}$ given $Y^{t-1},D^{t}$ is such that $F(y \mid y^{t-1},d^{t}) = y - \kappa(\tau) - \sum_{s=0}^{L} \theta_{s}(\tau) d_{t-s} - \sum_{s=1}^{P} \gamma_{s}(\tau) y_{t-s} $ for any $(y,y^{t-1},d^{t})$ such that the RHS is above 0 and less than 1 (otherwise is 0 or 1, resp). It is easy to see that such $F$ is H\"{o}lder continuous with $\delta=1$ and $A(.)=1$, so Assumption (ref) holds.
For this case we set $\Phi_{K} = 1$ and thus the problem of estimating $\mu(\cdot,0^{t},\tau)$ is given by
\begin{align*}
(\hat{\kappa}_{n},(\hat{\gamma}_{n,s})_{s=1}^{P}) : = \arg\min_{\kappa , (\gamma_{s})_{s=1}^{P} } \left( |\mathcal{G}_{n}(0^{t}) |^{-1} \sum_{i \in \mathcal{G}_{n}(0^{t})} 1\{ Y_{t,i} \leq \kappa + \sum_{s=1}^{P} \gamma_{s} Y_{t-s,i} \} - \tau \right)^{2}
\end{align*}
which coincides with estimation problem for a linear quantile regression (e.g. koenker1978regression). Since $ |\mathcal{G}_{n}(0^{t})| $ is proportional to $n$, Assumption (ref)(i) holds under mild conditions with $\eta_{n} = n^{-1/2} \log \log n $ and $\varpi_{n} = \sqrt{ \log \log n }$.\footnote{ The $\log \log n$ factor can be replaced by any arbitrarily slowly diverging sequence.} Part (ii) of this assumption is vacuous.
Regarding Assumption (ref), observe that $\mu( \cdot , 0^{t} , \tau)$ is of the form $b_{\tau} + \sum_{s=1}^{P} \gamma_{s}(\tau) y_{t-s} $ where $b_{\tau}$ is a number that can depend on $\tau$. Hence, $\mathbb{H}(0^{t}) : = \{ y^{t-1} \mapsto b + \sum_{s=1}^{P} \gamma_{s} y_{t-s} \colon (b,(\gamma_{s})_{s=1}^{P}) \in \mathcal{B} \} $ where $\mathcal{B}$ is some compact subset of $\mathbb{R}^{1+P}$. Moreover, we can take $\mathbb{H}_{k}(0^{t}) = \mathbb{H}(0^{t}) $ for all $k$. Hence, Assumption (ref) boils down to verifying that
\begin{align*}
\sup_{(b,(\gamma_{s})_{s=1}^{P}) \in \mathcal{B} } n^{-1} \sum_{i=1}^{n} 1\{ D^{t}_{i} = 0^{t} \}1\{ Y_{t,i} \leq b + \sum_{s=1}^{P} \gamma_{s} Y_{i,t-s} \} - E[ 1\{D^{t} = 0^{t} \} 1\{ Y_{t} \leq b + \sum_{s=1}^{P} \gamma_{s} Y_{t-s} \} ]
\end{align*}
converges in probability to 0 faster than some rate $\varrho_{n}$. Since $\mathcal{B}$, it follows that we can take $\varrho_{n} = n^{-1/2} \log \log n $.
We have thus verified all our assumptions for Theorem (ref) and we apply it to do hypothesis testing and construct confidence regions for the hypothesis that there is no total treatment effect, i.e., $\theta_{0} + \theta_{1} + ... + \theta_{L} = 0$.\footnote{In general, we can set the null to be $\sum_{s=0}^{L} \theta_{s}d_{t-s}[l] = 0$ for $l \in \{0,...,M\}$ whereby choosing $d_{t-s}[.]$ judiciously it allow us to test for many linear transforms of the vector $\boldsymbol{\theta} = (\theta_{0},..., \theta_{L}) \in \mathbb{R}^{L+1}$.} To do this, we choose $H$ to be the square of the absolute value of the moment corresponding to the treated group $1^{t} : = (1,1,...,1)$. Under this choice, $ Q_{\mathbf{K}}(x^{t},\hat{\mu}_{n,K}(\cdot,x^{t},0^{t},\tau),P_{n}) $ equals $\hat{Q}_{n}(0)$ where
\begin{align*}
\hat{Q}_{n}(\boldsymbol{\theta}) : = \left| \frac{\sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} 1\{ Y_{t,i} \leq \hat{\kappa}_{n} + \sum_{s=0}^{L} \theta_{s} + \sum_{s=1}^{P} \hat{\gamma}_{n,s} Y_{t-s,i} \} }{\sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} } - \tau \right|
\end{align*}
for any $\boldsymbol{\theta} : = (\theta_{0},..., \theta_{L}) \in \mathbb{R}^{L+1}$ (in this context, the "0" in the function is understood as the vector 0). Thus, our test will reject the null hypothesis that $\sum_{s=0}^{L} \theta_{s} d_{t-s} = 0$, if $\hat{Q}_{n}(0) > t(\alpha) + \delta$ where $\delta$ is the “infinitesimal correction" defined in Theorem (ref) and $t(\alpha) = \inf \{ t \colon \Pr( \left| \frac{ \bar{Bi} \left( \sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} , \tau \right) }{\sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} } \right| \geq t \mid (D^{t}_{i})_{i=1}^{n} ) \geq 1- \alpha \}$ where $\bar{Bi}(n,p)$ is a Binomial random variable with $n$ trials and probability of success $\tau$ centered around $np$. We observe that in this case, $t(\alpha)$ can be obtained without the need of simulations as it the smallest $t$ such that $\Pr \left( -t \sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} \leq \bar{Bi} \left( \sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} , \tau \right) \leq t \sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} \right) \geq 1-\alpha $; this observation can save computational time when computing the test and the confidence regions.\footnote{The dependence of $t(\alpha)$ on $(D^{t}_{i})_{i=1}^{n}$ is left implicit for the sake of the presenation.}
The confidence region is given by
\begin{align*}
CR_{n}(\alpha): = \left\{ \boldsymbol{\theta} \in\mathbb{R}^{L+1} \colon \hat{Q}_{n}(\boldsymbol{\theta}) \leq t(\alpha) + \delta \right\},
\end{align*}
which in this case it boils down to all the $\boldsymbol{\theta} \in\mathbb{R}^{L+1}$ such that $s(\boldsymbol{\theta} ) : = \sum_{s=0}^{L} \theta_{s} $ satisfies
\begin{align*}
\tau - t(\alpha) - \delta \leq \frac{\sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} 1\{ Y_{t,i} \leq \hat{\kappa}_{n} + s(\boldsymbol{\theta} ) + \sum_{s=1}^{P} \hat{\gamma}_{n,s} Y_{t-s,i} \} }{\sum_{i=1}^{n} 1\{ D^{t}_{i} = 1^{t} \} } \leq \tau + t(\alpha) + \delta.
\end{align*}
The next proposition shows that it has correct asymptotic confidence level.
\begin{proposition}
Suppose all the assumptions in Theorem (ref) hold. Let $(\theta_{s})_{s=0}^{L}$ be the true coefficients. Then, under the null hypothesis $\sum_{s=0}^{L} \theta_{s} = 0$ and $ \liminf_{n \rightarrow \infty} P(CR_{n}(\alpha) \ni (\theta_{s})_{s=0}^{L} ) \geq 1-\alpha $.
\end{proposition}
\begin{proof}
See Appendix (ref).
\end{proof}
$\triangle$
Special case: Inference for Heterogeneous treatment effects under staggered adoption
The purpose of this section is to illustrate that our framework and modeling assumptions are sufficiently general to allow for difference in difference (DiD) designs. In addition, we illustrate how our results can offer an alternative inference method for heterogeneous treatment effects under staggered adoption. This literature has mainly focused on identification (and failure of it) of this treatment effects; see ChaiseMartin2020, SUN2020, callaway2018difference, athey2018designbased and references therein. Even though some of these papers also focus on estimation, to the best of our knowledge, no paper in this branch of the literature considered inferential methods allowing for a small number of treated units. Given the dynamic nature of the problem, this last feature may arise in some applications, and thus, the results in this section may be of interest to practitioners.
We take the setup of callaway2018difference in which there is staggered adoption and irreversibility of treatment. That is, for each individual $i$, the treatment profile, $D^{t}_{i}$, is such that $D_{i,t} \leq D_{i,t+1}$. Henceforth, let $G_{i}$ denote the first time individual $i$ is treated; i.e., $G_{i} : = \min \{ s : D_{i,s} =1 \}$. We assume that $G_{i} > 1$, i.e., nobody is treated in the first period, and also, we set $G_{i} = 0$ if $D_{i}^{T} = 0$. Another restriction commonly made in this literature is that the potential outcomes only depend on the date of initiating the treatment, and not on the whole profile, i.e., $Y_{t}(d^{t}) : = Y_{t}( \min \{ s : d_{s} =1 \} ) $. The parameter of interest is thus given by
align*[align* omitted — 56 chars of source]
for all $g \leq t$ and all units $i$. That is, the ATE is allowed to have heterogeneous effects across time and across the starting period of the treatment.
The model for the counterfactual outcomes is as follows:
align[align omitted — 343 chars of source]
in which $\kappa_{i}$ is a individual fixed effect, $\delta_{t}$ is a time fixed effect, and it is assumed that $E[ \zeta_{i,t}(g) - \zeta_{i,t}(0)] = 0 $, for all $(i,t,g)$. Note that $\theta_{0,0} = 0$ as we assume nobody is treated at period $t=0$.
From this model and defining $\Delta$ as the $t$-difference operator, i.e., $\Delta Z_{t} : = Z_{t} - Z_{0}$ for any process $(Z_{t})_{t}$, we can show that
align*[align* omitted — 127 chars of source]
Hence, we can learn the $ATE_{t,g}$ by learning $\theta_{t,g}$, for any $t \geq g$.
Next, we show that this model can be studied with our framework by using the equations for the difference of outcomes. Formally, and with an abuse of notation, let $\Delta Y_{t}( \min\{ s \colon d_{s} =1 \} )$ denote the $d^{t}$ potential outcome in our theoretical framework. From equations (ref)-(ref), it follows that
align[align omitted — 136 chars of source]
in which $c : = \delta_{t} - \delta_{0}$ and $\varepsilon_{i,t}(g) : = \zeta_{i,t}(g) - \zeta_{i,0}(g)$. For any $g$, let $F$ be the conditional CDF of $\varepsilon_{t}(g)$ given $G$, we assume it is increasing and it satisfies the restriction that there exists a $\tau \in (0,1)$ such that
align[align omitted — 99 chars of source]
This condition is akin to the parallel trends assumption that is ubiquitious in these type of models. The main difference is that parallel trends restricts the conditional mean of $\varepsilon_{t}(g)$, whereas condition (ref) restrict a conditional quantile. Indeed, if $\varepsilon_{t}(g)$ is symmetric and $\tau = 0.5$, then the assumptions coincide.\footnote{We observe that symmetry of $\varepsilon_{t}(g)$ follows from the standard assumption that $(\zeta_{t}(g))_{t}$ are IID. }
It follows that the RHS of equation (ref) can be cast as $\mu_{t}(d^{t},U_{i,t}(g))$ where $g = \min\{ s \leq t \colon d_{s} =1 \}$, $U_{t}(d^{t}) : = F(\varepsilon_{t}(g) \mid g )$ and
align*[align* omitted — 143 chars of source]
As $\mu_{t}$ is really a function of the period of first receiving treatment, $g$, and not the entire treatment profile, henceforth we simply write $(g,u) \mapsto \mu_{t}(g,u) $. Assumption (ref) is satisfied as $u \mapsto F^{-1}(u|g)$ is increasing. Assumption (ref) holds because for any $d^{t}$ (that generates a particular group identity $g$),
align*[align* omitted — 222 chars of source]
where the first equality holds because there is no need to condition on past outcome nor on covariates in this example; the second equality follows from the definition of $U_{t}(d^{t})$ and the fact that is enough to condition on $G^{t}$ and not the whole treatment profile; finally, the third equality follows by the “parallel trends" condition (ref), note that by definition the last expression equals $F( F^{-1} (\tau \mid G^{t} ) \mid G^{t} ) = \tau$.
remark[On the role of Assumption (ref)]
The verification of Assumption (ref) in this example illustrates how this assumption does not present restrictions that are above and beyond the standard set of assumptions used in these models, namely the “parallel trends" assumption in expression (ref). Moreover, this application also illustrates that, while Assumption (ref) is by no means innocuous, it allows --- in conjuction with our setup --- for commonly used designs like the DiD one. In fact, given the form of Assumption (ref), it is easy to see that we can also cover generalizations of the model in (ref) - (ref) to allow, for instance, for time-varying covariates (including past outcomes) and a richer structure for potential outcomes (and thus a richer heterogeneity structure for the ATE) where the potential outcomes not only depend on the initial treatment date but on the whole treatment history.
$\triangle$
Our null hypothesis allow us to test whether the ATE is naught for different treatment groups. For instance, for any $g>1$, $H_{0} : \mu_{t}(g,\tau) = \mu_{t}(0,\tau)$ is equivalent to $\theta_{t,g} = 0$. We can also consider joint hypothesis that the ATE is naught for different groups, i.e., $\theta_{t,g[1]} = \theta_{t,g[2]} = ... = \theta_{t,g[M]} = 0$ for different $g[1],...,g[M]$.
Monte Carlo Simulations
The goal of this section is to evaluate the finite sample performance --- in particular size and power --- of our procedure for different distributions of the residuals in a simple regression model:
align*[align* omitted — 61 chars of source]
where $D_{i} \sim Ber(\pi)$ and $U$ is assumed to be IID-$F_{U}$.\footnote{We refer the reader to the paper by ferman2019inference where it is shown in a similar MC design that standard methods do not perform well when the number of treated units is small.} We consider different forms for $F_{U}$ to see how the model behaves to different specifications of this distribution, in particular the Kurtosis and heterosckedasticity. To be precise for $F_{U}$, we consider (i) “Gaussian": $U \sim N(0,1)$; (ii) “Heterosckedasticity": $U \sim N(0, 1+\sqrt{Z}+ | \sin(\pi Z)| )$ where $Z \sim U(0,1)$; (iii) “Uniform": $U \sim U(-\sqrt{3},\sqrt{3})$ and (iv) a distribution constructed as follows: With probability $1-\epsilon$, $U \sim U(-v,v)$ and with probability $\epsilon$, $U \in \{ -A, A \}$ with probability 0.5 where $v : = 0.1$, $\epsilon : = 0.04$ and $A: = \sqrt{ (1 - (1-\epsilon) (2 v)^2/12)/\epsilon} \approx 5$; the purpose of this last model is to considers residuals with “fat tails" distributions and thus we call this case “fat tails”. The true parameters are chosen to be $\theta_{0} = \theta_{1} = 0$. In all cases we consider the $\tau = 0.5$ quantile which is zero, and 10000 MC repetitions.
We first look at the size of our procedure for testing the null hypothesis of $\theta_{1} = 0$. As test statistic, we consider the square of the sum of the control and treated moments, i.e., $\left( \sum_{i=1}^{n} \frac{ (1-D_{i}) 1\{ Y_{i} \leq \hat{\theta}_{0} \}}{\sum_{i=1}^{n} (1-D_{i}) } -0.5 + \frac{ D_{i} 1\{ Y_{i} \leq \hat{\theta}_{0} \} }{ \sum_{i=1}^{n} D_{i} } - 0.5 \right)^{2}$ where $\hat{\theta}_{0}$ was estimated by running a median regression using only observations with $D_{i}=0$. By Theorem (ref), the asymptotic distribution of our test coincides with that of $\left( \sum_{i=1}^{n} \frac{ (1-D_{i}) B_{i} }{\sum_{i=1}^{n} (1-D_{i}) } - 0.5 + \frac{ D_{i} B_{i} }{ \sum_{i=1}^{n} D_{i} } - 0.5 \right)^{2}$ where $B_{i}$ is IID Bernoulli $0.5$; to compute the associated distribution we do 500 draws.
We report these results in Table (ref) for different samples sizes, $n \in \{25, 50, 200, 800\}$ and few treated units in each. To simulate few treated units for each sample size we choose $\pi$ so that the expected number of treated units is $N_{1} \in \{ 5, 10, 20 \}$; i.e., for each $n$ and $N_{1}$, $\pi$ is chosen such that $N_{1} = (1-\pi)n$. For samples sizes of $200$ and $800$ our procedure provides rejection rates that are close to the nominal values for all levels, even for very small number of treated units. For $n = 50$ and few treated units --- $N_{1} \in \{5,10\}$ --- our procedure still provides rejection rates that are close to the nominal values, except for $N_{1} = 5$ at 1% level where our procedure is somewhat conservative. For the case $n=50$ and $N_{1} = 20$ our procedure yields rejection rates that are lower than those in the other cases, around 7%, 3% and 0.5% for levels 10%, 5% and 1% respectively. It is worth noting, however, that in this case $n$ and $N_{1}$ are both small and proportional to each other, so our theory may not provide an accurate description of this case. Similarly, we believe the case for $n=25$ is also outside the purview of our theory as the sample size is small; we present it nonetheless to see how our procedure behaves in this type of cases.\footnote{In fact, $N_{1} =20$ implies that the number of control units is (on average) actually smaller than the number of treated units, and thus is not considered.} In this case our procedure behaves like in the case $(n=50,N_{1}=20)$, yielding rejection rates that are around 2-3 percentage points below the nominal size.
To summarize, for sample sizes of $n=50$ or larger, our test seems to perform well, yielding rejection rates that are close to the respective nominal values even when the number of treated units is small. In cases where the number of control units is small --- e.g., the case $(n=50,N_{1}=20)$ and $n=25$ --- our test is conservative, yielding rejection rates around 7%, 3% and 0.2% for levels 10%, 5% and 1% respectively.
table[table omitted — 2,815 chars of source]
In Figure (ref) we show the power function of our test for the case $\alpha = 0.05$, $n = 200$ and $N_{T} = 10$ (and $\pi =0.95$). To generate this we consider a DGP wherein $\theta_{1}$ ranges from 0 to 5.4. We can see that in all cases the power function asymptotes to 1, but at different rates depending on the DGP.
figure[figure omitted — 901 chars of source]
In order to gain intuition regarding this last fact and the magnitude of $\theta_{1}$ observe that the moment for the treated units is given by $(\sum_{i=1}^{n} D_{i} )^{-1} \sum_{i=1}^{n} D_{i} (1\{ U_{i}/\sigma_{U} \leq (\hat{\theta}_{0} - \theta_{0})/\sigma_{U} - \theta_{1}/\sigma_{U} \} - 0.5 ) $, where $\hat{\theta}_{0}$ is the estimator --- computed using the moment for the control units --- of $\theta_{0}$. Roughly speaking, $(\hat{\theta}_{0} - \theta_{0})/\sigma_{U} \approx 1/(n - \sum_{i=1}^{n} D_{i})^{0.5} \approx 1/n^{0.5}$, thus for values of $ \theta_{1}/\sigma_{U} $ in the range $[0,n^{-1/2}] \approx [0,0.07]$ the power function is expected to be close to 0 and without an specific shape. For values of $ \theta_{1}/\sigma_{U} $ in the range $[n^{-1/2},(\sum_{i=1}^{n} D_{i} )^{-1/2}]$ (which is approximately given by $[0.07,0.35]$) the power function is expected to be increasing, but not necessarily far from 0 as the deviations of the null are of smaller order than the variability of the moment. Hence, we expect our test to have non-trivial power only for deviations that are larger than $(\sum_{i=1}^{n} D_{i} )^{-1/2} \approx 0.35 $. Moreover, how fast the power function increases it depends on the variability of $U_{i}/\sigma_{U}$, this heterogeneity among the different models is depicted in Figure (ref).
We conclude by pointing out that while this is an apt description of the “average" (across MC repetitions) power function, for particular MC repetitions its behavior can be quite different. In fact, if the number of treated units is “extremely" small our test could reject with, essentially, zero probability and have a power function flat at zero. This feature stems from an “integer problem" in cases where the number of treated units is “extremely" small, rather than from the design of our test. To shed light on this fact lets consider a simplified version of the test statistic: One rejects the null of $\theta_{1} = 0$ if $\left| \sum_{i=1}^{n} D_{i} 1\{ Y_{i} \leq \hat{\theta}_{0} \} /\sum_{i=1}^{n} D_{i} - \tau \right| \geq t(\alpha,(D_{i})_{i=1}^{n} ) +\delta$ where
align*[align* omitted — 156 chars of source]
where $ \mathcal{B}(\sum_{i=1}^{n} D_{i}) : = \left| Bi(\tau , \sum_{i=1}^{n} D_{i} )/\sum_{i=1}^{n} D_{i} - \tau \right| $ and $Bi(p, M )$ is a Binomial random variable with probability of success $p$ and number of trials $M$. It turns out that given a $\alpha \in (0,1)$, there exists a cutoff $N_{1}$ such that for any $\sum_{i=1}^{n} D_{i} \leq N_{1}$, $t(\alpha,(D_{i})_{i=1}^{n} )$ will equal the highest value in the support of $ \mathcal{B}(\sum_{i=1}^{n} D_{i})$, and hence, the highest value in the support of $\left| \sum_{i=1}^{n} D_{i} 1\{ Y_{i} \leq \hat{\theta}_{0} \} /\sum_{i=1}^{n} D_{i} - \tau \right|$ as well. So, for any such number of treated units, our test will reject with probability 0.\footnote{If our test statistic also uses the moment corresponding to the “control" group, then this result may no longer hold, but the fact that the power function will be flat and not asymptote to one remains.} For the case of $\tau =0.5$ and $\alpha = 0.05$, such cutoff number is $5$. Therefore, while our theory indicates the proposed test will not over-reject for any number of units in the treatment groups, the power of the test can, mechanically, be very low in cases where the number of units in the treatment groups is extremely low.
Overall, the MC study shows that for moderate and even small sample sizes our test has rejection rates that are close to nominal ones, although being somewhat conservative for some designs. It also has non-trivial power for deviations of the null that are approximately larger than $(\sum_{i=1}^{n} D_{i} )^{-1/2}$. Finally, while our methodology can handle any number of treated units, for extremely low numbers our test will essentially reject with probability zero. This seems to be a feature of the problem rather than an idiosyncratic characteristic of our test.
Empirical application: The Effect of Weather on Growth
In this section we apply our method to gauge the effect of large temperature changes on the cross-sectional distribution of GDP growth. We use the data from dell2012temperature and complement their analysis in two directions. First, we study the effect of temperature changes on the 0.25, 0.50 and 0.75 quantiles of GDP growth. Second, we construct treatment profiles that allows us to estimate a saturated model as opposed to the linear regression model used by dell2012temperature and that was presented in Example (ref). The purpose of this exercise is to compare the conclusions we obtain from doing inference based on standard asymptotics to what we obtain when using the method proposed in this paper.
More precisely, growth of country $i$ in period $1990-1994$, $Y_{i}$, satisfies the following equation
align[align omitted — 114 chars of source]
in which $E[1\{ U \leq 0 \} | M , X ] = \tau$ with $\tau \in \{ 0.25, 0.50 , 0.75 \}$ and $M_{i}(l)$ takes value one if the history of weather events $(D_{1990},...,D_{1994})$ for $i$ equals $l$ and 0 otherwise. The variable $D_{i,t}$ is defined as $1$ if the change in temperature in country $i$ between period $t-1$ and period $t$ was larger than one degree Celsius. Following the preferred specification in dell2012temperature, $X_i$ includes region fixed effects, and the amount of rainfall of country $i$ in 1994. We restrict the sample to countries that are below the median of the GDP distribution in 1950, since those are the countries for which dell2012temperature find significant effect of temperature changes on GDP.
In Table (ref) we show the size of each treatment group defined by $(D_{1990},...,D_{1994})$. We can see that the control group, $(0,0,0,0)$, has 49 units whereas the other treatment groups has less, oscillating between 1 to 6 observations. Hence, standard inferential theory based on t-statistics or F-statistics can be unreliable and untrustworthy.
table[table omitted — 622 chars of source]
We report the estimated coefficients in Table (ref). In column 1 we show that, when estimating the coefficients in (ref) by OLS and using standard heterosckedastic robust SE, only contemporaneous temperature seems to matter for growth, i.e., the profile $00001$. The same result holds for quantiles 0.25, 0.5, and 0.75, which we show in columns 2-4. However, when using our method to test the null hypothesis that $\theta_{0001} = 0$ we cannot reject it at a $95\%$ level. Moreover, when testing the joint null hypothesis of no effect from any temperature shock during the 1990-1994 period, i.e., $\theta_{0001} = \theta_{00011} = ... = \theta_{1000} = 0$ we also cannot reject this null.\footnote{The test statistic we use to test $\theta_{0001} = 0$ is analogous to the one used in the Monte Carlo study. The test statistic we use to test $\theta_{0001} = \theta_{00011} = ... = \theta_{1000} = 0$ is also analogous but we constructed the treated moment by pooling the observations for all the profiles (other than $00000$).}
{{
table[table omitted — 2,333 chars of source]
}}