Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
86,158 characters · 11 sections · 59 citation commands
Improving Inference from Simple Instruments through Compliance Estimation
\xdef\@thefnmark\@footnotetext{Stephen Coussens, Mailman School of Public Health, Columbia University, [email removed]. Jann Spiess, Graduate School of Business, Stanford University, [email removed]. We thank Kirill Borusyak, Amanda Kowalski, Greg Lewis, and Ashesh Rambachan for helpful comments.}
Instrumental variables (IV) regression can recover consistent estimates of treatment effects in quasi-experimental studies and in randomized controlled trials (RCTs) where treatment randomization is infeasible, but there is at least some exogenous variation in treatment exposure generated by an instrument. Yet in practice, IV estimates are often noisy. With this in mind, in this paper we discuss how the precision of IV estimates can be improved when there is predictable variation in the strength of the instrument, provide valid inference for a specific implementation that can leverage machine learning algorithms, and document potential precision improvements in a simulation study.
IV estimation frequently involves the use of a two-stage estimator: the first stage predicts variation in treatment exposure using the instrumental variable(s), while the second stage regresses the outcome of interest on the first-stage predicted values. Better prediction of exogenous variation in treatment exposure can therefore improve the precision of treatment effect estimates. The proliferation of powerful machine learning algorithms designed to make predictions from high-dimensional data has thus spawned efforts to improve IV precision by combining a large number of relatively weak instruments into a single, strong instrument Belloni2012-nu,Hansen2014-wy. Yet in practice, valid instruments are typically hard to find, so applied research is frequently conducted with only a single simple instrument that takes on a limited number of values -- such as when randomized assignment to either a treatment group or a control group is used as a binary instrument for treatment. \footnote{ As we explore in this paper, even when only a single simple instrument is available, we can often create many more by interacting this instrument with covariates. Methods for high-dimensional instruments therefore remain applicable even in the simple-instrument case. In contrast to the literature on high-dimensional instruments, we explicitly consider the specific structure of instruments created by the interaction of a simple instrument with a function of covariates. }
We consider an IV estimator for simple instruments that improves efficiency by weighting each observation according to its estimated conditional probability of being a complier (that is, its probability of being affected by the instrument) in the case where the instrument is independent of baseline covariates. The underlying idea of weighting observations in instrumental variables estimation according to their estimated compliance is not new -- in the biostatistics literature, it can be traced back to at least Joffe2003-uq. Moreover, the implementation that we consider is close to one considered by Huntington-Klein as a tool to reduce finite-sample bias, and is similar to proposals by abadie2019instrumental and Borusyak2020-xu in linear IV models. However, compliance-weighted estimators have seen very limited adoption in applied research. \footnote{ Perhaps the most closely related applied research can be found in einav2018long, which reduces the variance of IV estimates by allowing the first stage coefficient to vary across quintiles of estimated compliance.}
In this paper, we aim to fill gaps between the idea of compliance weighting and its implementation by showing how compliance weighting arises naturally from the (constrained) solution of the first-stage prediction problem implicit to IV; by deriving the large-sample properties of a weighted IV estimator in the familiar potential outcomes rubin1977assignment and local average treatment effect (LATE) imbens1994identification frameworks; and by providing tools for valid inference that remain applicable when the first stage is estimated nonparametrically, allowing the researcher to leverage machine learning techniques to flexibly estimate compliance as a function of potentially high-dimensional covariates.
We begin by pointing out that when the instrument is simple (specifically, binary), then constructing efficient instruments by interacting them with covariates represents a prediction problem under constraints. Namely, we want to predict treatment from instruments and covariates, while also ensuring that the instrument remains valid. In recent work, Borusyak2020-xu tackle this challenge by recentering an unconstrained prediction of treatment from instruments and covariates so that only the variation in treatment that is correlated with the instrument remains. We instead characterize interacted instruments that are valid by construction and do not require recentering, provided that the covariates and the instrument are independent. We show that choosing instruments from this constrained class is equivalent to choosing weight functions that depend on covariates only. This approach is particularly applicable in high-dimensional data or when using nonparametric estimation techniques, in which case recovering valid instruments from more complex unconstrained predictions may be inefficient.
Next, we show that finding the efficient weights implicit in the IV estimator with first-stage interactions requires solving a heterogeneous treatment effect estimation problem. In a homogeneous and homoscedastic baseline case, the efficient weight for IV estimation under first-stage monotonicity is known to be the probability of compliance Joffe2003-uq, which is the causal effect of the instrument on treatment. Solving for the efficient instrument in this case is therefore equivalent to estimating the conditional average treatment effect of the instrument on treatment (i.e., the conditional probability of compliance).
In order to briefly illustrate the important role that compliance estimation plays in reducing the variance of IV estimators, consider a simple regression on a sample of size $N$ with binary instrument and treatment, where treatment effects are homogeneous, variance is homoscedastic, and the average probability of compliance (i.e. the first-stage coefficient) is $\alpha < 1$. In this case, the variance of the standard two-stage least-squares (2SLS) estimator is $\frac{\sigma^2}{N p(1-p) \alpha^2}$, where $\sigma^2$ is the variance of the regression residual in the reduced-form equation that relates the outcome to the instrument, and $p$ is the instrument mean. Now suppose that it is possible to perfectly identify all $\alpha N$ compliers in this sample and estimate the treatment effect using this smaller yet perfectly compliant sub-sample. Under this hypothetical scenario, the variance of the 2SLS estimator would fall to $\frac{\sigma^2}{N p(1-p) \alpha}$, a sizable reduction that would be equivalent to increasing the imperfectly compliant sample size from $N$ to $\frac{N}{\alpha}$. While in practice we will not generally be able to perfectly identify compliance -- an attribute that cannot be directly observed -- we can nonetheless estimate the probability of compliance conditional on covariates follmann2000effect.
Having argued in favor of compliance-weighting in a simple baseline case, we relax the assumptions of homoscedasticity and treatment effect homogeneity, and characterize the asymptotic distribution of a weighted IV estimator in this more general setting. We show that weighting the IV estimator changes the estimand when treatment effects are heterogeneous, and alters the asymptotic variance under heteroscedasticity. We point out that the weighted estimand can be interpreted as a convex combination of conditional LATEs, and then characterize the conditions under which this weighted IV estimator improves precision relative to the unweighted IV estimator, even when the unweighted unconditional LATE is the parameter of interest. We also consider an extension of this compliance-weighting approach that can further improve precision by also accounting for second-stage heteroscedasticity.
We then discuss an implementation of compliance-weighted IV estimation using machine learning. If the first stage can be nonparametric, then the causal problem of estimating compliance is different from an unconstrained prediction problem, in that naive predictions from covariates and the instrument would yield inefficient estimates of treatment effects. Like Huntington-Klein, we therefore study the use of recent tools from the intersection of causal inference and machine learning that are specifically designed to estimate conditional average treatment effects. Our implementation uses a cross-fitting scheme that repeatedly splits the sample to avoid biases from overfitting and enable valid inference even for high-dimensional estimation tools.
We go on to address the issue of inference when weights are estimated through such nonparametric procedures, which is a non-standard problem since the estimated weights affect the asymptotic distribution. In order to keep the assumptions placed on nonparametric first-stage estimation to a minimum, our framework considers estimation relative to an empirically-weighted estimand that varies with each sample. In this framework, we demonstrate that the estimator is asymptotically Normal, and that commonly-used approaches to estimating robust standard errors will provide for valid inference. These results stem from a general result on the convergence of empirically weighted sums that relies on sample splitting and may be of independent interest.
In simulations, we demonstrate that augmenting simple instruments through compliance estimation meaningfully reduces the variance of IV estimates when first-stage heterogeneity is present, and that these improvements typically outweigh biases relative to the unweighted IV estimand or ineffciencies from heteroscedastic errors. We present a simulation study that systematically varies treatment effects, residual errors, first-stage heterogeneity, and weight estimation methods, comparing the performance of compliance-weighted IV to that of a baseline 2SLS estimator. Our results suggest that in a variety of scenarios, applied researchers stand to substantially improve the precision of their IV estimates through the use of compliance weights.
\paragraph{Related literature.} This work is most closely related to research in biostatistics and econometrics that proposes weighting or selection by compliance Joffe2003-uq,coussens2018,abadie2019instrumental,Huntington-Klein, interacts instruments with covariates Bond2007-fd,Borusyak2020-xu, employs complier classification to analyze and improve the external validity of IV estimates Kennedy2018-fe, or explicitly discusses the role of exogenous covariates in linear IV models Chen2020-xg.
Joffe2003-uq propose weighting observations in an experiment according to their probability of compliance. abadie2019instrumental discard groups of observations with low estimated compliance, and employ an interacted IV estimator among the remaining groups that implicitly weights each group according to compliance. Their approach considers a grouped linear simultaneous equations model with an increasing number of groups and a higher-order expansion of second-stage mean-squared error. Huntington-Klein independently considers a variety of compliance grouping and weighting schemes in an effort to tackle a different problem: the small-sample bias of IV estimators in a linear model with heterogeneous first- and second-stage effects. Like us, Huntington-Klein characterizes estimands in terms of weighted causal effects, and uses the causal forest wager2018estimation to estimate compliance weights. The estimator that we consider is similar to one of Huntington-Klein's (Huntington-Klein) proposed implementations, but differs primarily in terms of sample splitting, which allows us to derive its large-sample properties and provide formal justifications for valid inference.
Our approach also relates to recent, independent work by Borusyak2020-xu, who consider non-random exposure to exogenous variation. Their work tackles the more complex task of parametrically constructing valid recentered instruments when exposure is itself non-random. Despite these different approaches, their construction based on chamberlain1987asymptotic effectively yields a similar efficient, compliance-weighted IV estimator when considered under our stronger exogeneity assumptions, and the authors also offer a supplemental interpretation in a potential outcomes framework similar to our setup in (ref).
Finally, Chen2020-xg independently consider the role of exogenous covariates in the construction of instruments using machine learning, and provide tools for inference that remain applicable in that case. Also invoking the characterization of efficient instruments in chamberlain1987asymptotic, Chen2020-xg use a cross-fitting construction to implement the optimal instrument, and characterize estimands in terms of weighted averages of marginal treatment effects. Unlike us, the authors do not assume that covariates and instrumental variables are independent, and specifically tackle complications that arise from the presence of covariates in that case.
\paragraph{Contribution.} Relative to existing work on leveraging compliance information, we formally motivate compliance weighting as the solution to a constrained, fully nonparametric first-stage prediction problem. We then interpret the estimand and analyze the properties of the estimator in a potential outcomes framework that does not impose any linearity assumptions. As our main technical contribution, we derive large-sample limits and propose inference on weighted local average treatment effects for a specific cross-fitting implementation that remain valid with arbitrary treatment effect heterogeneity and when the first-stage estimation of compliance is fully nonparametric.
Despite methodological advances in leveraging first-stage compliance information, we are not aware of widespread adoption of compliance-weighting approaches in applied research. Instead, attempts to harness first-stage heterogeneity have typically been limited to ad-hoc analysis of subgroups with higher estimated compliance, a practice that tends to result in inflated IV estimates and $t$-statistics abadie2019instrumental. We hope that we can contribute to the adoption of such methods by providing valid inference when using estimated compliance in IV models.
How can we exploit covariates to improve instrumental variable estimation using only a single, simple instrument? To fix ideas, consider the simple homogeneous linear instrumental variables model
where both the outcome $Y$ and the treatment variable $D$ are real, univariate random variables, and the univariate, real-valued instrument $Z$ is independent of the error term $\varepsilon$ and the regressors $X$.\footnote{Although independence is a strong assumption, this assumption is often fulfilled by design, e.g., using random assignment in an RCT as an instrument for treatment.} Since $\mathop{}\!\textnormal{Cov}(\varepsilon,Z) = 0$, if $\mathop{}\!\textnormal{Cov}(D,Z) \neq 0$ then the standard IV estimator
where we write $\mathop{}\!\mathbb{C}\textnormal{ov}_n(\cdot,\cdot)$ for the empirical covariance in an iid random sample of size $n$ from that distribution, is a $\sqrt{n}$-consistent and asymptotically normal estimator of $\tau$ under standard regularity conditions. In effect, this estimator divides the “reduced-form” OLS coefficient in the linear regression of $Y$ on $Z$ by the “first-stage” OLS coefficient in the linear regression of $D$ on $Z$; it is also equivalent to the 2SLS estimator that first regresses $D$ on $Z$ to form predicted values $\hat{D}$ and then regresses $Y$ on $\hat{D}$. However, this estimator does not make use of covariates $X$.
One commonly practiced way to incorporate covariates $X$ into IV estimation is as control variables in order to reduce the residual variation in $Y$. For example, if we assume that
then we can reduce the variance of the IV estimator by residualizing with respect to $X$ first. Given the exogeneity of $Z$, this is not required for consistency, but can nonetheless improve efficiency.
In this paper, we pursue a different, complementary use of pre-assignment covariates: in addition to controlling for covariates, we can also use them to construct new instruments by combining an existing instrument with these variables. In (ref), we relate such interacted instruments to the solution of a constrained prediction problem in a homogeneous and homoscedastic baseline case, which we solve for binary instruments. In (ref), we interpret the resulting estimator as a weighting estimator. Since in many applications researchers may not wish to assume homogeneous treatment effects or homoscedasticity, in (ref) we discard these assumptions and discuss the properties of this weighted IV estimator in a potential outcomes framework. In (ref), we discuss extensions of these results to non-binary instruments.
In this section, we consider the construction of augmented instruments by interacting them with functions of pre-assignment covariates Bond2007-fd. When covariates $X$ and error term $\varepsilon$ are independent of the instrument as in (ref), then there exist transformations of the form $h(Z,X)$ that also fulfill $\mathop{}\!\textnormal{Cov}(\varepsilon,h(Z,X)) = 0$ and can thus serve as valid instruments. This suggests the estimator
This augmented IV estimator $\hat{\tau}^h$, estimated from an iid sample, is consistent for the treatment effect $\tau$ if $\mathop{}\!\textnormal{Cov}(D,h(Z,X)) \neq 0$ holds and some mild regularity conditions guarantee the existence of sufficient population moments.
How would one choose among instruments $h(Z,X)$? To fix ideas, we make the additional homogeneity and homoscedasticity assumption that the residuals in (ref) a.s.\ fulfill $\mathop{}\!\textnormal{E}[\varepsilon^2|X] = \sigma^2$. This assumption (which we drop in (ref) below) is restrictive -- it limits treatment effects, conditional baseline expectations, and residuals. Under this assumption, the asymptotic distribution of $\hat{\tau}^h$ is
as $n \rightarrow \infty$. The asymptotic variance in this case has a particularly intuitive interpretation. Indeed, if we normalize $h(Z,X)$ to $\mathop{}\!\textnormal{E}[h(Z,X)] = \mathop{}\!\textnormal{E}[D]$ and $\mathop{}\!\textnormal{Var}(h(Z,X)) = \mathop{}\!\textnormal{Cov}(D,h(Z,X))$ -- which is without loss since $\hat{\tau}^h$ and thus also $V^h$ are invariant to shift and scale transformations \footnote{Formally, we could define
which is the best linear predictor of $D$ from $h(Z,X)$. } -- then
where $R_h^2$ is the (population) coefficient of determination in predicting $D$ using $h(Z,X)$.
This expression for the asymptotic variance connects efficient IV estimation to optimal prediction. (ref) shows that the asymptotic variance $V^h$ is strictly increasing in first stage mean-squared error. Moreover, $V^h$ is always weakly greater than the variance that would be obtained by regressing the outcome $Y$ directly on treatment $D$, a generally infeasible benchmark that could only be achieved if $D=h(Z,X)$ a.s. This scenario would imply full compliance with the instrument $h(Z,X)$, which of course would obviate the need for IV estimation in the first place.
The correspondence between low prediction loss and low variance of the implied IV estimator in (ref) has motivated the use of powerful machine-learning prediction algorithms to estimate instruments of the form $h(Z,X) = f(Z)$, where $Z$ is often assumed to be high-dimensional, for example using the LASSO Belloni2012-nu or ridge regression Hansen2014-wy. This literature has primarily focused on minimizing $V^h$ by solving the standard prediction problem of minimizing $\mathop{}\!\textnormal{E}[(D - h(Z,X))^2]$ through transformations of complex (or many) instruments $Z$. In applied research, where valid instruments are often simple and scarce while covariates are comparably rich and plentiful, these results can still be applied by considering instruments obtained by interaction with the covariates. Here, we explicitly consider such interacted instruments $h(Z,X)$ for which $Z$ is simple, but $X$ may be high-dimensional, and exploit the specific structure of $h(Z,X)$ in that case.
When we consider augmented instruments of the form $h(Z,X)$ to minimize the variance $V^h$ in (ref), we face an additional challenge: we have to ensure that the instrument remains valid -- that is, we want to solve (in the population distribution)
When is the constraint fulfilled? For a simple real-valued instrument $Z$ with $\mathop{}\!\textnormal{E}[Z] = p$, setting $h(Z,X) = w(X) \: (Z-p)$ for some conformal $w(\cdot)$ is sufficient. Indeed, if $Z$ is binary, then (up to trivial transformation) any instrument that ensures orthogonality by construction has this form:
When $Z$ is not binary, there are additional instruments; we discuss this case along with a generalization of (ref) to finitely supported instruments in (ref), yielding a linear combination of augmented instruments of the above form.
For the binary case, we can now solve for the augmented instrument of the form $h(Z,X) = w(X) \: (Z-p)$ that minimizes the prediction loss
maintaining the assumption that $\mathop{}\!\textnormal{E}[\varepsilon^2|X] \equiv \sigma^2$ a.s. The optimal interaction term takes a particularly simple form:
The optimal interaction term
is the local strength of the instrument (that is, the net causal effect of the instrument on treatment). This result directly mirrors abadie2019instrumental and Huntington-Klein who consider compliance that varies by groups, in which case a group-interacted estimator yields compliance weighting. Any other choice for $w(X)$ leads to a prediction loss of
in (ref), so solving for a low-variance estimator corresponds to the problem of estimating the treatment effect of $Z$ on $D$ with low mean-squared error. Recognizing that the minimization of IV variance is in fact a question of (first-stage) treatment effect estimation leads us to straightforwardly adapt machine learning tools previously developed to estimate heterogeneous treatment effects, which we discuss in (ref) and (ref).
Assuming that $w(X) \geq 0$ with $\mathop{}\!\textnormal{E}[w(X)] > 0$, we can shed additional light on the resulting estimator by noting that it can be understood as the weighted estimator
where $\mathop{}\!\mathbb{P}_n^w$ denotes empirical measure with weights $w(X_i)$ (so $\mathop{}\!\mathbb{E}_n^w[\circ] = \frac{\mathop{}\!\mathbb{E}_n[w(X) \: \circ]}{\mathop{}\!\mathbb{E}_n[w(X)]}$, etc.). Alternatively, we can consider the fully weighted IV estimator
that embeds weighting within the empirical covariance. While $\hat{\tau}^h$ with $h(Z,X) = w(X) \: (Z-p)$ and $\hat{\tau}_w$ differ slightly in how they handle baseline variation, once we control for the weight $w(X)$ in the IV regression, they are equivalent:
These two results show the equivalence of certain interacted instruments Bond2007-fd to weighted instruments Joffe2003-uq, which are also considered in Huntington-Klein. This connection allows us to obtain efficiency in specific cases. For example, when $D$ is binary and there are no defiers, then $\alpha(X)$ is the conditional probability of compliance. (We will make this language precise below using a standard potential outcomes framework.) In this setting, Joffe2003-uq present conditions under which compliance weighting is efficient in the homoscedastic and homogeneous case. For the remainder of this paper, we will impose the assumptions of this proposition, so we write $\hat{\tau}_w$ for the equivalent estimators, and call it the “$w$-weighted IV estimator”.
In this section, we consider the case of a binary instrument $Z$ and binary treatment $D$ in a potential outcomes framework, dropping the assumptions of homoscedasticity and treatment effect homogeneity. We adopt the standard assumptions for identification of the local average treatment effect (LATE) for the population distribution, also including covariates $X$ that are not affected by the instrument:
The last assumption implies that there are no defiers (units with $D(1)=0 < 1 = D(0)$), which allows us to identify conditional local average treatment effects
Monotonicity is not essential to these results, although it simplifies interpretation. For instance, if this assumption were violated, we could still consider estimating $\tau(x) = \frac{\mathop{}\!\textnormal{E}[Y| Z=1, X=x] - \mathop{}\!\textnormal{E}[Y| Z=0, X=x]}{\mathop{}\!\textnormal{E}[Y| D=1, X=x] - \mathop{}\!\textnormal{E}[Y| D=0, X=x]}$ for those values of $x$ with $\mathop{}\!\textnormal{E}[D| Z=1, X=x] > \mathop{}\!\textnormal{E}[D| Z=0, X=x]$. However, the parameter loses its causal interpretation if treatment effects differ for compliers and defiers with $X=x$ angrist1996.
Given an iid sample of size $n$ from the population distribution of $(Y,D,Z,X)$, the standard Wald estimator of the local average treatment effect $\tau = \mathop{}\!\textnormal{E}[Y(1) - Y(0) | D(1) > D(0)]$ is the IV estimator
where we write $\mathop{}\!\mathbb{E}_n,\mathop{}\!\mathbb{C}\textnormal{ov}_n$ for the expectation and covariance operators associated with the empirical (sample) measure $\mathop{}\!\mathbb{P}_n$ (So $\mathop{}\!\mathbb{E}_n[\circ] = \frac{1}{n} \sum_{i=1}^n \circ_i$, etc.). This is also the IV estimator in the model
where $\mathop{}\!\textnormal{Cov}(Z,\varepsilon) = 0$. (Throughout, we assume the existence of sufficient moments to allow for the consistent estimation of $\tau$ in this manner.)
When conditional treatment effects $\tau(X)$ vary, then the $w$-weighted IV estimator $\hat{\tau}_w$ is not necessarily consistent for the overall LATE $\tau = \mathop{}\!\textnormal{E}[Y(1) - Y(0) |D(1) > D(0)] = \mathop{}\!\textnormal{E}[\tau(X)]$. Instead, the estimand corresponding to this estimator now represents a weighted average of treatment effects among compliers:
In the previous section we followed Joffe2003-uq in arguing in favor of weighting by the compliance probability
However, the LATE is already a compliance-weighted average of conditional LATEs angrist1995two, which we write as
The compliance-weighted IV estimator $\hat{\tau}^* = \hat{\tau}_{\alpha}$ reduces the contribution of always- and never-takers to the variance by “doubling down” on that weighting, estimating the compliance-weighted LATE
corresponding to a “Super-Local Average Treatment Effect” in the linear model considered by Huntington-Klein. Furthermore, heterogeneity and heteroscedasticity can also affect the asymptotic distribution around the estimand:
The conditional residual variation $\sigma^2(x) = (1-p) \mathop{}\!\textnormal{E}[\varepsilon^2|Z=1,X=x] + p \mathop{}\!\textnormal{E}[\varepsilon^2|Z=0,X=x]$ comes from three sources: from variation of conditional LATEs $\tau(x)$ around their weighted average $\tau_w$; from variation in the outcome that can be predicted from pre-assignment characteristics, which can be addressed by residualizing the outcome with respect to (potentially nonparametric) functions of $X$; and from irreducible noise that cannot be explained by $X$.
While we leave a more thorough treatment of these additional sources of variation (and their mitigation) to future versions, we note here that the Cauchy--Schwartz inequality immediately implies that the variance-minimizing weights are given by
and a natural extension to the estimators in this paper is to also estimate $\sigma^2(X)$ when determining weights Miller2018-xt, in addition to compliance $\alpha(X)$. That said, an estimator that weights by compliance alone may prove to be desirable in many empirical settings due to concerns regarding the finite-sample performance of inverse-variance estimators, analogous to the reason that an applied researcher may prefer OLS to the asymptotically efficient GLS.
We also note a parallel to Borusyak2020-xu, in which the authors develop methods to improve IV efficiency by recentering the instrument to account for heterogeneous treatment exposure. In a more general setting where the given instrument is not valid unconditionally, they extend the framework of chamberlain1987asymptotic, and propose an optimal instrument that can be re-written in our notation and under our assumptions as
corresponding to the efficient weighting function $w^*(x)$. Motivated by the homoscedastic and homogeneous case where $\sigma^2(X) \equiv \sigma^2$, we focus for now on the weight choice $w(x) = \alpha(x)$, which is optimal in this case. This has the additional benefit of not requiring any information about the distribution of outcome $Y$, while also giving the weights in $\tau_\alpha$ an immediately intuitive meaning.
How should $\hat{\tau}_\alpha$ be interpreted in practice? Under assumptions of treatment effect homogeneity and homoscedasticity, it is a consistent estimator of the LATE, and more efficient than traditional IV estimators (e.g., 2SLS). Moreover, consistency for the LATE also extends to settings in which treatment effects are heterogeneous but uncorrelated with the compliance weights. For example, consider an RCT designed to estimate the effectiveness of a medical intervention administered at an outpatient health clinic. Suppose that the distance between study participants' homes and the clinic is highly predictive of their compliance with random assignment, but that the effectiveness of the treatment is largely determined by the participants' latent genetic characteristics. In such settings, it seems unlikely that estimated compliance would be meaningfully correlated with treatment effects, in which case compliance-weighted estimates could be simply interpreted as approximations to the LATE.
However, the estimand $\tau_w$ and the LATE $\tau$ are not generally equivalent when $\tau(x)$ are both heterogeneous and correlated with $w(x)$; in this setting, the estimand $\tau_\alpha$ represents a convex combination of conditional LATEs Huntington-Klein. Yet even when the unconditional LATE is the estimand of interest, we point out that in many applied scenarios, the compliance-weighted IV estimator $\hat{\tau}_\alpha$ will nonetheless be more precise for the LATE than the traditional IV estimator $\hat{\tau}$. To do so, we consider an asymptotic framework with relatively small treatment effects. When treatment effects are fixed in our asymptotic framework, then we can distinguish the LATE arbitrarily well from any alternative in a large sample. So instead, we focus on the case where treatment effects are of an order of magnitude that makes them hard to distinguish from noise, a common scenario in applied research. \footnote{Note that this approach is similar to the weak instrument analysis of staiger1997instrumental in that we adopt a local-to-zero regime, but complementary in that rather than applying it to first-stage effects, we apply it to treatment effects, a separate and generally less problematic source of weak signal.}
Hence, if the unconditional LATE $\tau$ is in fact the estimand of interest, the $w$-weighted estimator has an asymptotic bias $B_w$ that is equal to the covariance of conditional LATEs and relative weights $w(X)/\mathop{}\!\textnormal{E}[w(X)]$ among only the compliers. This result is intuitive: since only compliers contribute to the conditional LATEs, the extent to which $\tau_w$ can deviate from $\tau$ is governed by differences in the relative weighting of compliers. This also highlights the fact that when weighting by compliance (i.e., $w(x) = \alpha(x)$), $B_\alpha = 0$ both when $X$ is not predictive of compliance (in which case $w(X) \equiv \alpha$), as well as when $X$ is perfectly predictive of compliance (in which case $w(x) = 1$ for all compliers, and $w(x) = 0$ for all non-compliers).
In principle, the fit from estimating the unweighted LATE $\tau$ by the $w$-weighted IV estimator $\hat{\tau}_w$ can be asymptotically better or worse than using the standard unweighted IV estimator $\hat{\tau}^{\text{IV}}$. Indeed, the expected square of the asymptotic approximation to $\sqrt{n} (\hat{\tau}_w - \tau)$ is
representing a bias--variance trade-off that can make it smaller or larger than the variance of the large-sample limit of $\sqrt{n} (\hat{\tau}^{\text{IV}} - \tau)$, which is $\mathop{}\!\textnormal{E}\left[\sigma^2(X)\right] / (p(1-p) \mathop{}\!\textnormal{P}^2(D(1)>D(0)))$.
A particularly simple estimator that navigates this bias--variance trade-off is the estimator $\hat{\tau}^*(\lambda)$ that implements weights
for $\lambda \in [0,1]$, interpolating between equal weighting ($\lambda=0$) and compliance weights ($\lambda=1$). This weighting scheme can be interpreted as a shrinkage estimator in the population compliance prediction problem. When compliance probabilities do not co-vary too much with treatment effects and conditional variances, then the resulting estimator has better fit for the LATE than the standard IV estimator:
This condition holds, in particular, under homoscedasticity or when the variation of conditional variances is small.
As an alternative to this analysis of the estimator relative to the LATE, compliance-weighted estimators could be analyzed within the marginal treatment effect (MTE) frameworks heckman2005structural,heckman2006understanding,brinch2017beyond,mogstad2018using, which use IV to bound (or extrapolate to) treatment effect parameters of interest that cannot be point-identified, such as the ATE. In applications of these frameworks kowalski2016doing,kowalski2018examine,kowalski2018reconciling, compliance-weighted estimators may similarly improve precision in estimating treatment-effect bounds.
In this section we analyze the asymptotic distribution of the weight-augmented IV estimator when weights are estimated, rather than fixed ex-ante, and consider inference. Under regularity conditions, we find that weight estimation does not change the asymptotic distribution relative to a specific sample-dependent estimand, and that commonly used robust standard errors remain valid after first-stage estimation. Notably, our assumptions allow for valid inference relative to this data-dependent estimand even when compliance weights are estimated nonparametrically.
In the previous section, we considered IV estimation with instruments of the form $h(Z,X)$ for a simple instrument $Z$ that is independent of covariates $X$, focusing on cases where $h(Z,X) = w(X) \: (Z-p)$ for $p = \mathop{}\!\textnormal{E}[Z]$. In the homogeneous and homoscedastic baseline case with a binary instrument $Z$, we justified the specific choice
for the weight $w(X)$, corresponding to the probability of compliance in an instrumental-variable setting with binary treatment and no defiers. Yet in practice, such weights are unknown to the researcher, and must be estimated instead. We therefore analyze the asymptotic distributions of the augmented IV estimator $\hat{\tau}^{\hat{h}}$ and the weighted IV estimator $\hat{\tau}_{\hat{w}}$, where
for the empirical covariance $\mathop{}\!\mathbb{C}\textnormal{ov}_n$ and weighted empirical covariance $\mathop{}\!\mathbb{C}\textnormal{ov}_n^{\hat{w}}$.
We note that the notion of convergence of $w_{-j}$ to $w$ as well as the cross-fitting approach we employ for weights are similar to the assumption and construction Chen2020-xg employ for instruments with a linear second-stage model, while we focus specifically on compliance estimation in a completely nonparametric model.
While (ref) establishes consistency, we also care about the asymptotic distribution. $\hat{\tau}^{\hat{h}}, \hat{\tau}_{\hat{w}}$ can generally differ and have different asymptotic distributions. We now residualize $Y_i$ and $D_i$ linearly with respect to a constant and the weight $\hat{w}_{-j(i)}(X_i)$ (and potentially additional pre-assignment control variables) and invoke (ref), which applies to the estimators with estimated weights as well, to conclude that $\hat{\tau}^{\hat{h}} = \hat{\tau}_{\hat{w}}$ in that case. Assuming this residualization throughout, we can thus focus on $\hat{\tau}_{\hat{w}}$.
Under arbitrary treatment effect heterogeneity and estimated weights, there is little hope of establishing $\sqrt{n}$-consistency relative to the $w$-weighted LATE $\tau_w$ without imposing stronger assumptions on the estimation of weights and/or the heterogeneity of conditional LATEs. Instead of imposing such assumptions ex-ante, we consider an estimand that is itself a function of the data, namely the cross-fitted hybrid estimand
where conditional expectations consider a draw of the data independent of $\hat{w}_{-j}$. In other words, this estimand represents a $\hat{w}^{\text{CF}}(\cdot) = \frac{1}{k} \sum_{j=1}^k \hat{w}_{-j}(\cdot)$-weighted LATE that depends itself on the data through the cross-fitting splits and weight functions. Relative to this estimand, we obtain an asymptotic distribution that is equivalent to that for fixed weights:
This result is related to the idea in Chernozhukov2017-ug of separating out variation conditional on sample splits from variation through sample splits themselves. We simply focus on the former and take the sample splits -- and corresponding weights -- as given, while ignoring variation through sample splits and weight estimation itself. The “variational” inference approach from Chernozhukov2017-ug could complement this purely conditional inference. As an alternative, we could have considered the empirically-weighted estimand
reducing the asymptotic variance further to the residual in a regression of $Y - \tau(X) D$, but ignoring additional within-fold variation.
When treatment effects have variation that vanishes sufficiently quickly -- for example in the case where treatment effects are themselves local to zero -- we obtain $\sqrt{n}$-consistency relative to the $w$-weighted LATE:
When we aim to obtain compliance weights $\alpha(x) = \mathop{}\!\textnormal{E}[D|Z=1, X=x] - \mathop{}\!\textnormal{E}[D|Z=0, X=x]$ for the sake of minimizing the empirical variance $V_w$ in the homogeneous and homoscedastic case, we face a causal estimation problem, since the compliance score $\alpha(X)$ is not directly observed -- for each observation $i$, we only observe treatment $D=D_i$ given the realization $Z=Z_i$, but not for the counterfactual assignment $Z=1 - Z_i$. We therefore face a first stage heterogeneous treatment effect estimation problem (where the instrument $Z$ represents treatment, and the treatment variable $D$ is the outcome of interest). Two natural estimation strategies in this case are:
In general, with a binary instrument, any algorithm for the consistent estimation of heterogeneous treatment effects from experimental data is also a tool for the estimation of compliance weights in our setting, and better estimation translates into lower asymptotic variance under homogeneity and homoscedasticity. Additional shrinkage that goes beyond the regularization employed to reduce out-of-sample mean-squared error in prediction may be helpful to mitigate the impact of heterogeneous treatment effects when the local average treatment effect $\tau$ is the estimand of interest.
Finally, for inference on treatment effects, we also want to estimate the asymptotic variance
where $\varepsilon$ are the residuals in a linear regression of $Y - D \tau_w$ on the control variables, including the weight $w(X)$. We can consistently estimate the asymptotic variance by the sample analogue (where we assume that the instrument is binary):
Standard heteroscedasticity-robust variance estimators in the weighted IV regression (after residualization) with estimated weights similarly remain valid, where $\mathop{}\!\textnormal{P}(Z{=}1), \mathop{}\!\textnormal{P}(Z{=}0)$, and $\mathop{}\!\textnormal{Var}(Z)$ are typically estimated with weights (which are asymptotically equivalent when $X$ and $Z$ are independent, as in our setting).
This section presents a simulation study that systematically compares the performance of the compliance-weighted IV estimator under a number different weight estimation procedures to that of unweighted IV. We consider a number of data generating processes that vary the degree of heterogeneity in compliance and treatment effects, as well as the heteroscedasticity of errors.
In this section, we demonstrate the finite-sample performance of the estimation procedures discussed in (ref). Let $\delta_i, \varepsilon_i, \tau_i, \sim \mathcal{N}(\mathbf{0}, \Sigma)$, with
where $\delta_i$ is the latent tendency to receive treatment, $\tau_i$ is the treatment effect, and $\varepsilon_i$ is the baseline untreated potential outcome for individual $i$.
Using the framework outlined in (ref), let potential treatment indicators be defined as
where $S_{AT}$ and $S_{NT}$ represent the share of always-takers and never-takers in the population, respectively. Realized outcomes and treatments are thus defined as
in which scalar parameter $\zeta$ determines the degree of heteroscedasticity exhibited in the variance of the untreated outcome. When $\rho_{\delta\varepsilon} \neq 0$, observations in this simulation exhibit selection into treatment that is correlated with untreated potential outcomes, a setting in which IV methods can prove useful. Moreover, when $\rho_{\delta\tau} \neq 0$, there will be differences between the unweighted and $w$-weighted IV estimands, as discussed in (ref).
In order to explore how the performance of the compliance-weighted estimator relates to the researcher's ability to predict compliance, we employ a framework that allows us to vary the irreducible error in predicting treatment. Let
represent a noisy continuous predictor of treatment receipt, where $\eta_i \sim \mathcal{N}(0, \sigma_{\eta}^2)$ is a random error. Note that while $X$ is a linear function of $\delta$, compliance rate $\alpha(X) = \mathop{}\!\textnormal{E}[D|Z=1] - \mathop{}\!\textnormal{E}[D|Z=0]$ is not, making estimation a non-trivial exercise. We demonstrate the importance of selecting appropriate compliance estimation techniques for improving IV estimate precision and conducting valid inference in (ref) below.
Using this framework, we produce simulations under four different data generating processes (DGPs). In DGP 1, the baseline case, there is a constant treatment effect and homoscedasticity. In DGP 2, there are heterogeneous treatment effects that are positively correlated with the latent tendency to receive treatment ($\rho_{\delta\tau} = 0.5$), consistent with a Roy-type model of selection on heterogeneous treatment effects. In DGPs 3 and 4, errors are heteroscedastic and positively ($\zeta = 0.25$) and negatively ($\zeta = -0.25$) correlated with covariates, respectively. For each DGP, we draw $M = 1,000$ Monte Carlo samples of $N = 1,000$ observations and generate treatment effect estimates using both unweighted IV and the compliance-weighted IV estimators while varying irreducible error $\sigma_{\eta} \in \{0.5, 1, 2\}$. In each simulation, always-takers, compliers, and never-takers comprise $0.05$, $0.25$, and $0.70$ of the population, respectively.
To demonstrate the role of compliance estimation in the performance of the weighted estimator, we estimate compliance weights using three different feasible estimation procedures: in-sample OLS, cross-fitted OLS, and cross-fitted causal forest. In-sample weights are the compliance estimates generated by fitting a prediction function to the sample from which it was constructed. On the other hand, cross-fitted weights are out-of-sample fits produced using 5 randomly-partitioned folds, as discussed in (ref).
As mentioned above, the compliance score $\alpha(X)$ is a non-linear function of $X$. So when using OLS to generate weights, we flexibly approximate $\alpha(X)$ by splitting $X$ into an ordinal series of bins, then estimating the compliance rate within each bin. Specifically, let $\hat{t}_{jz}(J)$ represent a function that predicts treatment rates conditional on the instrument and a coarse ranking of $X_i$ that assigns individuals into one of $J<<N$ bins of equal size partitioning the distribution of $X_i$
where $F_N(\cdot)$ is the empirical distribution function for a sample of size $N$. Predicted rates are thus the estimated mean treatment rates among those in group $z$ and bin $j$, and the complexity of the function is increasing in $J$. The bin-specific compliance weight estimates are therefore
Note that these $w_j$ are equivalent to the coefficients from an OLS regression of $D$ on $Z$ interacted with dummy indicators for each bin. We can then control the complexity of this compliance prediction function, and therefore its tendency to over-fit in small samples, by varying $J$.
Alternatively, when using the causal forest algorithm wager2018estimation to estimate the weights, compliance is nonparametrically and “honestly" estimated by predicting $D$ using $Z$ and $X$ directly (rather than the $J$-bin partitioning of $X$ used in OLS estimation), with regularization parameters chosen to minimize out-of-bag RMSE. In this context, “honest" estimation means that tree splits on $X$ are selected by the algorithm to maximize compliance heterogeneity using only a randomly-drawn subset of the data, while compliance estimates are fitted using only the observations excluded from this subset. This allows for asymptotically unbiased estimation of compliance conditional on $X$.
(ref) presents the analytically-derived oracle (i.e., population) compliance weights $\alpha(X)$ across the cumulative distribution of $X$ for each value of irreducible error $\eta$.\footnote{See (ref) for derivations of the oracle compliance weight distribution.} When irreducible error is relatively small, compliance weights are heavily concentrated among compliers, and a substantial share of observations receives no weight whatsoever. However, as the irreducible error increases, the distribution of compliance weights flattens, placing relatively less weight on compliers. Throughout the simulation results presented below, the compliance-weighted IV estimates generated using these oracle weights represent an upper-bound on the performance of the compliance-weighted IV estimator possessing full (population) information of the joint distribution of observed variables $X$, $D$, and $Z$.
(ref) presents both unweighted and compliance-weighted IV estimate distributions corresponding to each of the DGPs described above using oracle weights. Similarly, (ref) presents the RMSE performance of the compliance-weighted IV estimator relative to unweighted IV in estimating the LATE. First, in order to temporarily abstract away from the role of compliance weight estimation error, consider the case in which oracle weights are used in each of the simulations. The role of irreducible error first-stage prediction problem (reflected by $\sigma_{\eta}$) is immediately evident in each of these figures. Regardless of the data-generating process, when the error is small (e.g., $\sigma_{\eta} = 0.5$), the reduction in RMSE relative to 2SLS is large (roughly 40% in DGP 1, the baseline homogeneous case). As $\sigma_{\eta}$ grows, the information content in $X$ weakens, reducing prediction quality and leading to more modest improvements in estimator performance over unweighted IV.
Next, consider the performance of the compliance-weighted estimator using the feasible OLS weight estimation schemes, considering two cases: one in which the complexity of the prediction function is low ($J = 10$), and one in which the complexity of the prediction function is high ($J = 50$). When $J = 10$, there 100 observations (including an average of 25 compliers) per bin, allows for relatively precise estimation of the bin-specific compliance weights. In this case, weights estimated in-sample generally offer better performance than those estimated through cross-fitting, yielding substantially lower variance and minimal bias.
However, when the complexity of the OLS prediction function is high ($J = 50$), there are only 20 observations (including an average of 5 compliers) per bin. In this case, the potential for bias due to over-fitting is much greater. Indeed, the results in (ref) demonstrate that when $X$ is an insufficiently strong predictor of compliance (e.g. when $\sigma_{\eta} = 2$) and weights are over-fitted, the in-sample compliance-weighted IV estimator can exhibit substantial bias. This results in coverage below the nominal rate, as shown in (ref) and (ref), which present coverage rates for the compliance-weighted IV estimators with respect to the population LATE and compliance-weighted LATEs, respectively. However, consistent with the analytical results in (ref), cross-fitted compliance-weighted IV estimators allow for valid inference regardless of the level of irreducible error or model complexity. Nonetheless, while the bias resulting from over-fitted weight estimates can be eliminated through the use of cross-fitting, this may introduce variance that exceeds the (squared) bias that it eliminates, potentially resulting in greater MSE.
Yet we need not restrict ourselves to using only unpenalized linear models to generate compliance rates. As described in (ref), the asymptotic variance of the IV estimator for DGP 1 depends on the prediction error of the instrument. This suggests that in many settings, superior performance can be obtained by exploiting the bias-variance trade-off in compliance prediction through regularization. The results in (ref) and (ref) show that compliance-weighted IV estimates using weights generated by a regularized prediction function such as the causal forest have substantially lower variance than those using unpenalized cross-fitted weights.
Note that regardless of the degree of noise in the first stage ($\sigma_{\eta}$) or the weight estimation procedure employed, the relative performance of the $w$-weighted estimator exhibits a similar pattern across the four DGPs. Relative to baseline case DGP 1 (homogeneous and homoscedastic), performance under heteroscedasticity depends on the manner in which the squared residuals are correlated with the compliance weights. When this correlation is positive (DGP 3), then performance declines, since the variance among compliers is above average; when this correlation is negative (DGP 4), then performance improves, since the variance among compliers is below average. Similarly, the RMSE performance dips slightly when treatment effects are heterogeneous and correlated with the weights (DGP 2), since in this case the compliance-weighted estimand differs from the LATE, as discussed in (ref). However, the results in DGP 2 strongly suggest that when the heterogeneity in treatment effects is limited, compliance-weighted estimates can in practice be interpreted as approximations to the LATE.
In this article, we contribute to a recent literature in econometrics coussens2018, abadie2019instrumental,Huntington-Klein,Borusyak2020-xu and build upon prior results in biostatistics Joffe2003-uq that consider improving IV estimation through weighting (or selecting) samples according to the probability of compliance. Specifically, we relate the construction of interacted instruments to weighting in the IV estimator, consider a potential outcomes framework with a nonparametric first stage to clarify the resulting estimands, and derive the large-sample properties of the corresponding estimators. Moreover, we provide tools for valid inference even when instruments are estimated using nonparametric techniques such as machine learning algorithms for the estimation of heterogeneous treatment effects. We then document the improvements in precision that can be obtained from using these methods in a theoretical illustration and a simulation study. We hope that the framework deployed in this article will provide practitioners with helpful insights and practical methods to improve inference in the often-encountered scenarios in which conventional IV estimates are noisy.