Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
38,045 characters · 9 sections · 36 citation commands
Valid Causal Inference with (Some) Invalid Instruments
Instrumental variable (IV) methods are a powerful approach for estimating treatment effects: they are robust to unobserved confounders and they are compatible with a variety of flexible nonlinear function approximators newey2003instrumental, darolles2011nonparametric, hartford2017deep, lewis2018adversarial, Singh2019, bennett2019deep, thereby allowing nonlinear estimation of heterogeneous treatment effects.
In order to use an IV approach, one must make three assumptions. The first, relevance, asserts that the treatment is not independent of the instrument. This assumption is relatively unproblematic, because it can be verified with data. The second assumption, unconfounded instrument, asserts that the instrument and outcome do not share any common causes. This assumption cannot be verified directly, but in some cases it can be justified via knowledge of the system; e.g. the instrument may be explicitly randomized or may be the result of some well understood random process. The final assumption, exclusion, asserts that the instrument's effect on the outcome is entirely mediated through the treatment. This assumption is even more problematic; not only can it not be verified directly, but it can be very difficult to rule out the possibility of direct effects between the instrument and the outcome variable. Indeed, there are prominent cases where purported instruments have been called into question for this reason. For example, in economics, the widely used “judge fixed effects" research design kling2006incarceration uses random assignment of trial judges as instruments and leverages differences between different judges' propensities to incarcerate to infer the effect of incarceration on some economic outcome of interest frandsen2019judging. mueller2015criminal points out that exclusion is violated if judges also hand out other forms of punishment (e.g. fines, a stern verbal warning etc.) that are not observed. Similarly, in genetic epidemiology “Mendelian randomization" davey2003mendelian uses genetic variation to study the effects of some exposure on an outcome of interest. For example, given genetic markers that are known to be associated with a higher body mass index (BMI), we can estimate the effect of BMI on cardiovascular disease. However, this only holds if we are confident that the same genetic markers do not influence the risk of cardiovascular disease in any other ways. The possibility of such “direct effects”---referred to as “horizontal pleiotropy" in the genetic epidemiology literature---is regarded as a key challenge for Mendelian randomization hemani2018evaluating.
It is sometimes possible to identify many candidate instruments, each of which satisfies the relevance assumption; in such settings, demonstrating exclusion is usually the key challenge, though in principle unconfounded instrument could also be a challenge. For example, many such candidate instruments can be obtained in both the judge fixed effects and Mendelian randomization settings, where individual judges and genetic markers, respectively, are treated as different instruments. Rather than asking the modeler to gamble by choosing a single candidate about which to assert these untestable assumptions, this paper advocates making a weaker assumption about the whole set of candidates. Most intuitively, we can assume majority validity: that at least a majority of the candidate instruments satisfy all three assumptions, even if we do not know which candidates are valid and which are invalid. Or we can go further and make the still weaker assumption of modal validity: that the modal relationship between instruments and response is valid. Observe that modal validity is a weaker condition because if a majority of candidate instruments are valid, the modal candidate--response relationship must be characterized by these valid instruments. Modal validity is satisfied if, as Tolsoy might have said, “All happy instruments are alike; each unhappy instrument is unhappy in its own way.”
This paper introduces ModeIV, a robust instrumental variable technique that we show is asymptotically valid when modal validity holds. ModeIV allows the estimation of nonlinear causal effects and lets us estimate conditional average treatment effects that vary with observed covariates. ModeIV is a black-box method in the sense that it is compatible with any valid IV estimator, which allows it to leverage any of the recent machine learning-based IV estimators. We experimentally validated ModeIV using both a modified version of the hartford2017deep demand simulation and a more realistic Mendelian randomization example. In both settings---even when we generated data with a very low signal-to-noise ratio---we observed ModeIV to be robust to exclusion-restriction bias and to accurately recover conditional average treatment effects (CATE).
We are interested in estimating the causal effect of some treatment variable, $t$, on some outcome of interest, $y$. The treatment effect is confounded by a set of observed covariates, $x$, as well as unobserved confounding factors, $\epsilon$, which affect both $y$ and $t$. With unobserved confounding, we cannot rely on conditioning to remove the effect of confounders; instead we use an instrumental variable, $z$, to identify the causal effect.
Instrumental variable estimation can be thought of as an inverse problem: we can directly identify the causal\footnote{Strictly, non-causal instruments suffice but identification and interpretation of the estimates can be more subtle swanson2018challenging.} effect of the instrument on both the treatment and the response before asking the inverse question, “what treatment--response mappings, $f:t\rightarrow y$, could explain the difference between these two effects?" The problem is identified if this question has a unique answer. If the true structural relationship is of the form, $y = f(t, x) + \epsilon$, one can show that, $ E[y|x,z] = \int f(t,x) d F(t |x, z), $ where $E[y|x,z]$ gives the instrument--response relationship, $F(t | x,z)$ captures the instrument--treatment relationship, and the goal is to solve the inverse problem to find $f(\cdot)$. In the linear case, $f(t, x) = \beta t + \gamma x$, so the integral on the right hand side of reduces to $\beta E[t| x, z] + \gamma x$ and $\beta$ can be estimated using linear regression of $y$ on the predicted values of $t$ given $x$ and $z$ from a first stage regression. This procedure is known as Two-Stage Least Squares angrist2008mostly. More generally, the causal effect is identified if the integral equation has a unique solution for $f$ newey2003instrumental.
A number of recent approaches have leveraged this additive confounders assumption to extend IV analysis beyond the linear setting. newey2003instrumental and darolles2011nonparametric proposed the first nonparametric procedures for estimating these structural equations, based on polynomial basis expansion. These methods relax the linearity requirement, but scale poorly in both the number of data points and the dimensionality of the data. To overcome these limitations, recent approaches have adapted deep neural networks for nonlinear IV analyses. DeepIV hartford2017deep fits a first-stage conditional density estimate of $\hat{F}(t | x, z)$ and uses it to solve the above integral equation. Both lewis2018adversarial and bennett2019deep adapt generalized method of moments hansen1982 to the nonlinear setting by leveraging adversarial losses, while Singh2019 proposed a kernel-based approach. Puli2019 showed that constraints on the structural equation for the treatment also lead to identification.
\paragraph{Inference with invalid instruments in linear settings} Much of the work on valid inference with invalid instruments is in the Mendelian Randomization literature, where violations of the exclusion restriction are common. For a recent survey see hemani2018evaluating. There are two broad approaches to valid inference in the presence of bias introduced by invalid instruments: averaging over the bias, or eliminating the bias with ideas from robust statistics. In the first setting, valid inference is possible under the assumption that each instrument introduces a random bias, but that the mean of this process if zero (although this assumption can be relaxed, c.f. Bowden2015, kolesar2015identification). Then, the bias tends to zero as the number of instruments grow. Methods in this first broad class have the attractive property that they remain valid even if none of the instruments is valid, but they rely on strong assumptions that do not easily generalize to the nonlinear setting considered in this paper. The second class of approaches to valid inference assumes that some fraction of the instruments are valid. Then, biased instruments are outliers whose effect can be removed by leveraging the robustness of the median kang2016instrumental and the mode hartwig2017robust. In this paper, we use the same robust statistics insights, generalizing them to the nonlinear setting.
\paragraph{Ensemble models} Ensembles are widely used in machine learning as a technique for improving prediction performance by reducing variance Breiman1996 and combining the predictions of weak learners trained on non-uniformly sampled data freund1995desicion. These ensemble methods frequently use modal predictions via majority voting among classifiers, but they are designed to reduce variance. Both the median and mode of an ensemble of models have been explored as a way of improve robustness to outliers in the forecasting literature stock_combination_2004, kourentzes_neural_2014, but we are not aware of any prior work that explicitly uses these aggregation techniques to eliminate bias from an ensemble.
\paragraph{Mode estimation} If a distribution admits a density, the mode is defined as the global maximum of the density function. More generally, the mode can be defined as the limit of a sequence of modal intervals---intervals of width $h$ that contains the largest proportion of probability mass---such that $x_{\text{mode}}=\lim_{h\rightarrow 0}\operatorname*{arg\,max}_x F([x-h/2, x+h/2])$. These two definitions suggest two estimation methods for estimating the mode from samples: either one may try to estimate the density function and the maximize the estimated function parzen_estimation_1962, or one might search for midpoints of modal intervals from the empirical distribution functions. To find modal intervals, one can either fix an interval width, $h$, and choose $x$ to maximize the number of samples within the modal interval chernoff_estimation_1964, or one can solve the dual problem by fixing the target number of samples to fall into the modal interval and minimizing $h$ dalenius_mode--neglected_1965, venter_estimation_1967. We use this latter dalenius_mode--neglected_1965--venter_estimation_1967 approach as the target number of samples can be parameterized in terms of the number of valid instruments, thereby avoiding the need to select a kernel bandwidth $h$.
In this paper, we assume we have access to a set of $k$ independent\footnote{This independence requirement can be weakened to conditionally independent, conditional on some variable $c$. See (ref) for details.} candidate instrumental variables, $\mathcal{Z}=\{z_1, \dots, z_k\}$, which are `valid' if they satisfy relevance, exclusion and unconfounded instrument, and `invalid' otherwise. Denote the set of valid instruments, $\mathcal{V}:=\{z_i: z_i \perp y | x,t, \epsilon \}$, and the set of invalid instruments, $\mathcal{I} = \mathcal{Z} \setminus \mathcal{V}$. We further assume that each valid instrument identifies the causal effect. This amounts to assuming that the unobserved confounder's affect on $y$ is additive, $y = f(t, x, z_{i:i\in \mathcal{I}}) + \epsilon$ for some function $f$ and $E[y|x,z_{i:i\neq j}, z_j] = \int f(t,x,z_{i:i\neq j}) d F(t |x,z_{i:i\neq j}, z_j)$ has a unique solution for all $j$ in $\mathcal{Z}$.
The ModeIV procedure requires the analyst to specify a lower bound $V \geq 2$ on the number of valid instruments and then proceeds in three steps.
The idea is that the estimates from the valid instruments will tend to cluster around the true value of the effect, $E[y|\text{do}(t), x]$. We assume that the most common effect instrument is a valid one; i.e., that the modal effect is valid. To estimate the mode, we look for the tightest cluster of points. These are the points contained in $\hat{\mathcal{I}}_{\mathrm{mode}}$. Intuitively, each estimate in this interval should be approximately valid and hence approximates the modal effect. We take the average of these estimates to gain statistical strength.
The next theorem formalizes this intuition and shows, in particular, that ModeIV asymptotically identifies and consistently estimates the causal effect.
We defer the proof to the supplementary material.
Of course, the ModeIV procedure can be generalized to allow different estimators of the mode than the one used in Steps 2 and 3. The particular choice we make here has the advantage of being straightforward, statistically stable, computationally inexpensive, and relatively insensitive to the choice of $V$. The procedure as a whole is, however, $k$ times more computationally expensive than running single estimation procedure at both training and test time.
We studied ModeIV empirically in two simulation settings. First, we investigated the performance of ModeIV for non-linear effect estimation as the proportion of invalid instruments increased for various amounts of direct effect bias. Second, we applied ModeIV to a realistic Mendelian randomization simulation to estimate heterogeneous treatment effects. For all experiments, we use DeepIV hartford2017deep as the nonlinear estimator. Full experimental details are given in the appendix.
We evaluated the effect of invalid instruments on estimation by modifying the low dimensional demand simulation from hartford2017deep to include multiple candidate instruments. The hartford2017deep demand simulation models a scenario where the treatment effect varies as a nonlinear function\footnote{$\psi(t)=2\left((t-5)^{4} / 600+\exp \left[-4(t-5)^{2}\right]+t / 10-2\right)$. See the appendix for a plot of the function} of time $\psi(t)$, and observed covariates $x$.
We highlight the differences between our data generating process and the hartford2017deep data generating process in red: we have $k$ instruments whose effect on the treatment is parameterized by $\beta^{(zx)}$, instead of a single instrument in the original; we include an exclusion violation term which introduces bias into standard IV approaches whenever $\gamma$ is non-zero. The vector $\beta^{(zy)}$ controls the direct effect of each instrument: invalid instruments have nonzero $\beta^{(zy)}_i$ coefficients, while valid instrument coefficients are zero.
We fitted an ensemble of $k$ different DeepIV models that were each trained with a different instrument $z_i$. In Figure (ref), we compare the performance of ModeIV with three baselines: DeepIV with oracle access to the set of valid instruments (DeepIV-opt); the ensemble mean (Mean); and a naive approach that fit a single instance of DeepIV treating all instruments as valid (DeepIV-all). The $x$-axis of the plots indicates the scaling factor $\gamma$, which scales the amount of bias introduced via violations of the exclusion restriction.
All methods performed well when all the instruments were valid. Once the methods had to contend with invalid instruments, Mean and DeepIV-all performed significantly worse than ModeIV because of both methods' sensitivity to the biased instruments. ModeIV's mean squared error closely tracked that of the oracle method as the number of biased instruments increased, and the raw mean squared errors of both methods also increased as the number of valid instruments in the respective ensembles correspondingly fell.
\paragraph{Sensitivity} When using ModeIV, one key practical question that an analyst faces is choosing $V$, the lower bound on the number of valid instruments. We evaluated the importance of this choice in figure (ref) (left) by testing the performance of ModeIV across the full range of choices for $V$ with different numbers of biased instruments. We found that, as expected, the best performance was achieved when $V$ equaled the true number of valid instruments, but also that similar levels of performance could be achieved with more conservative choices of $V$. That said, with only 5 valid instruments, ModeIV tended to perform worse when $V$ was set too small. To see why this is the case, notice that in figure (ref) (right), there are a number of regions of the input space where the invalid instruments agreed by chance (e.g. $t\in [-1, 0.]$), so these regions bias ModeIV for small mode set sizes. Overall, we observed that setting $V$ to half the number of instruments tended to work well in practice.
Asymptotically, ModeIV remains consistent when fewer than half of the instruments are valid, but when this is the case there are far more ways that Assumption 2 of Theorem (ref) can be violated. This is illustrated in (ref) (right) which shows that there are a number of regions where the bias instruments agree by chance. Because of this, we recommend only using ModeIV when one can assume that the majority of instruments are valid, unless one has prior knowledge to justify modal validity without assuming the majority of instruments are valid\footnote{For example, if direct effects are strictly monotone and disagree, chance agreements among invalid instruments can only occur in a finite number of locations.}.
\paragraph{Selecting instruments?} ModeIV constitutes a consistent method for making unbiased predictions but, somewhat counter-intuitively, it does not directly offer a way of inferring the set of valid instruments. For example, one might imagine identifying the set of candidates that most often form part of the modal interval $\hat{\mathcal{I}}_{\mathrm{mode}}$. The problem is that while candidates that fall within the modal interval $\hat{\mathcal{I}}_{\mathrm{mode}}$ tend to be close to the mode, the interval can include invalid instruments that yielded an effect close to the mode by chance. \todo[color={green!20}, inline]{KLB: Why does it have to be by chance? Couldn't it also be because an adversary set it up that way? I.e., we don't have a model of the invalid instruments as having to be random, do we? The paper is written in a way that seems to equivocate on this point; I think the real result is robust to adversarial manipulation, but much of the discussion is written as though invalid instruments must be uncorrelated.} \todo[color={green!20}, inline]{KLB: But if it is by chance, give the intuition about why the same invalid instrument can be chosen across most $x$ values. (I think this requires being more formal about what's meant by "by chance".)} Since these invalid estimates are close to the truth, they do not hurt the estimate. We can see this in (ref) (right) where invalid instruments form part of the modal interval in the region $t\in [-3.5, -2]$, without introducing bias. \todo[color={green!20}, inline]{KLB: I see that you've put this paragraph here because the last bit refers to the figure. But in general, this feels much more like discussion. It's tempting to move this to Section 6, though I guess then Figure 2 (right) would be unexplained unless it were moved to Section 6 also.}
We evaluate our approach on the simulated data adapted from hartwig2017robust, which is designed to reflect violations of the exclusion restriction in Mendelian randomization studies.
Instruments, $z_i$, represent SNPs---locations in the genetic sequence where there is frequent variation among people---modeled as random variables drawn from a Binomial$(2, p_i)$ distribution corresponding to the frequency with which an individual gets one or both rare genetic variants. The treatment and response are both continuous functions of the instruments with Gaussian error terms. The strength of the instrument's effect on the treatment, $\alpha_i$, and direct effect on the response, $\delta_i$, are both drawn from Uniform$(0.01, 0.2)$ distributions for all $i$. For all experiments we used 100 candidate instruments and varied the number of valid instruments from 50 to 100 in increments of 10; we set $\delta_i$ to 0 for all valid instruments. More formally,
In the original hartwig2017robust simulation, the treatment effect $\beta$ was fixed for all individuals. Here, we make the treatment effect vary as a function of observable characteristics to model a scenario where treatments may affect different sub-populations differently. We simulate this by making the treatment effect, $\beta(x)$, a sparse linear function of observable characteristics, $x\in R^{10}$, where 3 of the 10 coefficents, $\gamma_i^{(xt)}$ were sampled from $U(0.2, 0.5)$ and the remaining $\gamma_i^{(xt)}$ were set to $0$. We introduce non-linearity by rounding to the nearest 0.1, which makes the learning problem harder, while making it easier to visually show the differences between the fitted functions and their true targets.\\
Mendelian randomization problems tend to have low signal-to-noise ratios, where the treatment explains only 1-3% of the response variance.\todo[color={green!20}, inline]{KLB: Support this with a citation?} This makes the setting challenging for neural networks, which tend to perform best on low-noise regimes.\todo[color={green!20}, inline]{KLB: Again, a cite, or an argument? I've certainly seen neural networks do great in the presence of lots of noise.} To address this, we leveraged the inductive bias that the data is conditionally linear in the treatment effect, using a neural network to parameterize the slope of the treatment variable rather than outputting the response directly. So, for these problems, we defined $\hat{f}(t,x) = g(\phi(x)) t + h(\phi(x))$, where $g(\cdot)$ and $h(\cdot)$ are linear layers that act on a shared representation $\phi(x)$.
The general trends we observed on the Mendelian randomization simulation, summarized in Table (ref), were similar to those we observed in the biased demand simulation: DeepIV-all performed poorly; ModeIV closely tracked the performance of our oracle, DeepIV-opt. On this simulation the mean ensemble Mean achieved stronger performance, but still significantly underperformed ModeIV.
\paragraph{Conditional average treatment effects} Figure (ref) shows the predicted dose--response curves for a variety of different levels of the true treatment effect. The six plots correspond to six different subspaces of $x$ which all have the same true conditional treatment effect. Each of the light blue lines shows ModeIV's prediction for a different value of $x$. The model is not told that the true $\beta$ is constant for each of these sub-regions, but instead has to learn that from data so there is some variation in the slope of each prediction. Despite this, the majority of predicted curves match the sign of the treatment effect and closely match the ground truth slope. The average absolute bias in ModeIV's conditional average treatment effect estimation was 0.01 larger than that of DeepIV on the valid instruments, for true effect sizes that range between -0.3 and 0.3 (see Table (ref) in the appendix for details).
The conventional wisdom for IV analysis is: if you have many (strong) instruments and sufficient data, you should use all of them so that your estimator can maximize statistical efficiency by weighting the instruments appropriately. This remains true in our setting---indeed, DeepIV trained on the valid instruments typically outperformed any of the ensemble techniques---but of course requires a procedure for identifying the set of valid instruments. In the absence of such a procedure, falsely assuming that all candidate instruments are valid can lead to large biases, as illustrated by the poor performance of DeepIV-all on simulations that included bias. ModeIV gives up some efficiency by filtering instruments, but it gains robustness to invalid instruments and in practice we found that the loss of efficiency was negligible. Of course, that empirical finding will vary across settings. A useful future direction would find a procedure for recovering the set of valid instruments to further reduce the efficiency trade-offs. \todo[color={green!20}, inline]{KLB: This is where I'd put the earlier discussion and experimental results about whether we can identify the valid instruments.}
There are, however, some important settings where ModeIV either will not work or require more careful assumptions. First, our key assumption was that each valid instrument consistently estimates the same function, $f(t,x)$. In settings with discrete treatments, one typically only identifies a “(conditional) local average treatment effect” (CLATE / LATE respectively) for each instrument. The LATE for instrument $i$ can be thought of as the average treatment effect for the sub-population that change their behavior in response to a change in the value of instrument $i$; if the LATEs differ across instruments, this implies that each instrument will result in a different estimate of $E[\hat{f}_i(t,x)]$ regardless of whether any of the instruments are invalid. In such settings, ModeIV will return the average of the $V$ closest $\hat{f}_i(t,x)$'s, but one would need additional assumptions on how these estimates cluster relative to biased estimates to apply any causal interpretation to this quantity. The alternative is the approach that we take here: assume that a common function $f(t,x)$ is shared across all units and allow for heterogeneous treatment effects by allowing the treatment effect to vary as a function of observed covariates $x$. This shared heterogeneous effect assumption is weaker than prior work on robust IV, which requires a “constant effect” effect assumption that every individual responds in exactly the same way to the treatment via a linear parameter, $\beta$.
Second, we have focused on settings where each instrument is independent. In principle ModeIV extends to settings where some instruments are confounded or all the instruments share a common cause, but the conditions for valid inference are more delicate because one has to ensure all backdoor paths between the instruments and response are blocked (see (ref)).
IV methods are important tools in the causal inference toolbox. These methods have been applied to study causal effects across a wide range of settings, spanning economic policies, phenotypes that may cause disease, and recommendation algorithms. Typically, analysts use their best expert judgment to assess whether various candidate instruments satisfy the exclusion restriction and then proceed with effect estimation. However, this judgement is both difficult to make and highly consequential: biased instruments can invalidate the analyst's conclusions.
This paper offers the analyst a potentially easier alternative: assuming that some fixed proportion of instruments are valid. With this, the paper provides a method for estimating causal effects in the presence of invalid instruments, backed by theoretical guarantees. Because ModeIV captures nonlinear effects, it may be better suited to applications in genetics and economics than previous methods.
Of course, caveats still apply. As with all causal inference methods, ModeIV still requires an analyst to make assumptions and to assess potentially delicate, mathematical statements. Although we have striven to make the limitations of our method explicit and to provide guidelines where possible, negative impacts could arise if analysts apply our method without prudence and thus obtain invalid causal conclusions, particularly if these are offered as policy suggestions. To mitigate the potential for such negative impacts, we have clearly noted when ModeIV should not be applied. Beyond that, we recommend that analysts considering applying this or any other causal estimation procedure conduct as much sensitivity analysis and external validation as possible.