Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
72,569 characters · 16 sections · 54 citation commands
Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancing
\if11 \fi
\if01 {
} \fi
{\it Keywords:} unobserved confounding, partial identification, quantile regression
Estimating treatment effects from observational data is difficult because “treated" and “control" samples typically differ on many characteristics besides treatment status. For example, consumers of nutritional supplements may be wealthier or more health-conscious than those not taking supplements. One popular tool for adjusting for such baseline imbalances is Inverse Propensity Weighting (IPW) ipw_review, ATTViaIPW. This technique re-weights treated and untreated samples to be similar along all observed characteristics and then compares outcomes in the weighted samples. The crucial assumption underlying this approach is that the weighted samples do not systematically differ along important unobserved characteristics. This “unconfoundedness" assumption is untestable, and often implausible.
This paper studies how much can be learned when unconfoundedness does not hold, but one can bound the plausible degree of unobserved confounding. In particular, given a “sensitivity assumption" controlling the degree of selection, we aim to answer two questions:\\
The specific sensitivity assumption used in this paper is the “marginal sensitivity model" of tan2006, which is a variant of Rosenbaum's famous “$\Gamma$ sensitivity model" RosenbaumDesign, Rosenbaum1987, rosenbaum2002 that is better suited for IPW analyses. This sensitivity assumption is quite popular in causal inference; see tan2006, confounding_robust_policy_improvement, kallus2020confoundingrobust, kallus_zhou2020, kallus2018interval, zsb2019, causal_rule_ensemble, rosenman2020combining, rosenman2021designing, soriano2021interpretable for an incomplete list of references. As we will see, it lends itself to computationally-efficient sensitivity analyses which are simple enough to explain to any practitioner comfortable with IPW.
Recently, Zhao, Small, and Bhattacharya zsb2019 (hereafter ZSB) introduced an interpretable IPW sensitivity analysis for the marginal sensitivity model that has been largely responsible for the recent resurgence of interest in this sensitivity assumption. However, they did not answer the partial identification question, leaving open the possibility that more informative bounds could be obtained from the same data and assumptions. Indeed, there are no existing partial identification results for the marginal sensitivity model that can be used to benchmark a sensitivity analysis.
The first main contribution of this paper is to provide a complete answer to the partial identification question (ref). We derive closed-form expressions for the largest and smallest values of the “usual" estimands (e.g. average treatment effect) compatible with the marginal sensitivity assumption. These expressions show that the ZSB bounds are essentially always conservative because they ignore an infinite collection of constraints implied by the distribution of observed characteristics. tan2006 also identified these constraints, but deemed it intractable to incorporate them all in a sensitivity analysis. In contrast, our partial identification results show that this collection can actually be reduced to a single constraint which is easy to incorporate.
Our second main contribution is to introduce a new IPW sensitivity analysis, which we call the quantile balancing method. The method is a simple refinement of the ZSB sensitivity analysis, and has several desireable features:\\
We apply the quantile balancing method in several simulated examples and one real-data application, and find that it can substantially tighten the ZSB bounds when the covariates are good predictors of the outcome. We also extend our analysis to Augmented IPW (AIPW) estimators. That analysis shows that a slight refinement of the ZSB method is sharp under “additive-noise" data generating processes, though the refinement makes little difference in practice. One shortcoming we will mention up-front is that our statistical guarantees assume the outcome is continuously-distributed in order to enable quantile regression. Since our partial identification results also apply to discrete outcomes, we conjecture that the quantile balancing procedure could be modified to give sharp bounds in that setting too.
We consider the Neyman-Rubin potential outcomes model with a binary treatment neyman, rubin1974. We observe i.i.d. samples $(X_i, Y_i, Z_i)$ from a distribution $P$, where $X_i \in \mathcal{X} \subseteq \mathds{R}^d$ is a vector of covariates, $Z_i \in \{ 0, 1 \}$ is a binary treatment assignment indicator, and $Y_i \in \mathds{R}$ is a real-valued outcome.
We assume that each sample $(X_i, Y_i, Z_i)$ is obtained by coarsening a “full data" sample $(X_i, Y_i(0), Y_i(1), Z_i, U_i)$. Here, $Y_i(0)$ and $Y_i(1)$ are potential outcomes and $U_i$ is a vector of unobserved confounders of unspecified dimension. The observed outcome is related to the potential outcomes through the consistency relation $Y_i = Z_i Y_i(1) + (1 - Z_i) Y_i(0)$.
The goal is to use the observed data to draw inferences about a causal estimand $\psi_0$. For the purposes of exposition, we initially focus on the counterfactual means $\psi_{\textup{T}} = \mathbb{E}[ Y(1)]$ and $\psi_{\textup{C}} = \mathbb{E}[ Y(0)]$, although the examples of most practical interest are the average treatment effect (ATE) and the average treatment effect on the treated (ATT).
With minor modification, our identification results can also be applied to more complex estimands, including policy values athey2017efficient, confounding_robust_policy_improvement and weighted average treatment effects. However, we do not present those extensions in this paper.
Under the unconfoundedness assumption $(Y(0), Y(1)) \, \rotatebox[origin=c]{90}{$\models$} \, Z \mid X$, all of the above quantities can be consistently estimated from the observed data using inverse propensity weighting. IPW estimators work by reweighting the observed sample by some function of the propensity score $e(x) := P(Z = 1 | X = x)$. For example, if the estimand of interest is $\psi_{\textup{T}}$, the (stabilized) IPW estimator is given by ((ref)):
Here, $\hat{e}(\cdot)$ is an estimate of the propensity score $e(\cdot)$ and $\mathbb{E}_n[\cdot]$ is shorthand for $\tfrac{1}{n} \sum_{i = 1}^n [\cdot]_i$. An unstabilized version of $\hat{\psi}_{\textup{T}}$ which uses only the numerator of ((ref)) is also common. Related estimators for the other estimands considered will be denoted by $\hat{\psi}_{\textup{C}}, \hat{\psi}_{\text{ATE}}$, and $\hat{\psi}_{\text{ATT}}$. See the articles by ipw_review or ATTViaIPW for their exact formulas.
We will assume some conditions which are required for identification and estimation under unconfoundedness: overlap ($0 < e(X) < 1$ almost surely) and one outcome moment ($\mathbb{E}_{P}[|Y|] < \infty$). However, we will not assume unconfoundedness.
The marginal sensitivity model introduced by tan2006 is a relaxation of unconfoundedness which has been applied in many causal inference problems. This one-parameter sensitivity assumption allows for the existence of unobserved confounders $U$, but limits the degree of selection bias that can be attributed to these confounders.
The statement of the marginal sensitivity model presented in tan2006 and zsb2019 uses the potential outcomes $(Y(0), Y(1))$ in place of the unobserved variable $U$. However, as pointed out by a referee, these assumptions are equivalent.
To avoid confusion between $e_0$ and $e$, we will follow kallus_zhou2020 and refer to $e_0$ as the “true propensity score" and $e$ as the “nominal propensity score."
Like Rosenbaum's famous “$\Gamma$ sensitivity model", Assumption (ref) controls the degree of unobserved confounding with a single parameter. When $\Lambda = 1$, measuring additional confounders cannot change the odds of treatment at all, i.e. treatment assignment is unconfounded. As $\Lambda$ increases, stronger forms of confounding are allowed. For advice on how to choose this parameter, see hsu_small2013. For more on the relationship between this and Rosenbaum's model, see zsb2019 Section 7.1. The marginal sensitivity assumption is “nonparametric" in the sense that no assumptions are needed about how $e_0$ depends on $u$. Even the dimension of the vector $U$ does not need to be specified.
To see how Assumption (ref) can be used for sensitivity analysis, begin by considering how an oracle statistician who observed the confounders $U_i$ might estimate $\psi_{\textup{T}}$. One strategy would be to use the IPW estimator ((ref)), which is consistent under weak assumptions.
In reality, $\{ U_i \}_{i \leq n}$ are not observed, but under Assumption (ref), it is possible to bound the true propensity scores $e_0(X_i, U_i)$. In particular, the vector $(e_0(X_1, U_1), \cdots, e_0(X_n, U_n))$ must belong to the ZSB constraint set $\mathcal{E}_n(\Lambda)$ defined in ((ref)).
ZSB proposed bounding the oracle statistician's IPW estimator ((ref)) with the largest and smallest IPW estimates that can be obtained using putative propensities in $\mathcal{E}_n(\Lambda)$.
Since the interval ((ref)) contains the consistent estimator $\hat{\psi}_{\textup{T}}^*$, the distance between the true estimand $\psi_{\textup{T}}$ and the nearest point in the sensitivity interval tends to zero. ZSB show that this conclusion holds even if the nominal propensity score $e$ is replaced by a suitably consistent estimate $\hat{e}$ in the definition of $\mathcal{E}_n(\Lambda)$, which is important for practical applications as $e$ is typically not known in observational studies.
This simple idea is intuitive enough to explain to any practitioner who is comfortable with IPW and has been extended to estimands other than $\psi_{\textup{T}}$. ZSB also consider $\psi_{\text{ATE}}$ and $\psi_{\text{ATT}}$. Related work by confounding_robust_policy_improvement, kallus2020confoundingrobust, kallus2018interval, causal_rule_ensemble takes the idea substantially further. tan2006 applied a similar idea to a different propensity-score-based estimator and AronowLeeInterpretable, MiratrixEtAl, tudballZhaoEtAl2019interval used similar approaches in survey sampling problems.
The aforementioned works do not address the asymptotic optimality of the interval $[\hat{\psi}_{\textup{T,ZSB}}^-, \hat{\psi}_{\textup{T,ZSB}}^+]$. Does it converge to a limiting set containing all values of $\psi_{\textup{T}}$ compatible with Assumption (ref) and no others? Sensitivity analyses with this asymptotic optimality property are called “sharp" in the partial identification literature.
Sharpness is important for interpreting the results of a sensitivity analysis. If the primary analysis finds a positive treatment effect but the bounds associated with a very small value of $\Lambda$ include zero, one might be tempted to conclude that the primary analysis is sensitive to unobserved confounding. However, unless the bounds are known to be sharp, this inference is not warranted even in large samples. Perhaps the bounds were just too conservative.
Despite its attractive features, the ZSB sensitivity analysis is not sharp. It can be arbitrarily conservative. To illustrate this, consider a simple joint distribution of observables:
Suppose that a data analyst receives i.i.d. samples $(X_i, Y_i, Z_i)$ from this distribution and is willing to posit that Assumption (ref) is satisfied with $\Lambda = 2$. Let $\phi(\cdot)$ and $z_{\tau}$ denote the density and $\tau$-th quantile of the standard normal distribution, respectively. The following result, which follows from Theorem (ref) in Section (ref), writes the set of values of $\psi_{\textup{T}}$ compatible with Assumption (ref) explicitly in terms of these quantities and shows that this “partially identified" set is smaller than the limiting ZSB interval.
The precise meaning of (ref) is the following: for any $\psi_{\textup{T}} \in [\pm \tfrac{3}{4} \phi(z_{2/3})]$, it is possible to construct a distribution $Q$ for the full data $(X, Y(0), Y(1), Z, U)$ which marginalizes to ((ref)), satisfies Assumption (ref) with $\Lambda = 2$, and has $\mathbb{E}_{Q}[ Y(1)] = \psi_{\textup{T}}$. On the other hand, for any $\psi_{\textup{T}}$ not in this interval, it is impossible to construct such a distribution.
Corollary (ref) implies that the ZSB interval typically includes many values of $\psi$ which cannot possibly be reconciled with the data. The explanation for this conservatism is that the odds-ratio bound ((ref)) does not capture all of the restrictions on the true propensity score $e_0$. Additional information can be found in the marginal distribution of the observed characteristics. For example, in the context of Corollary (ref), consider the putative propensity score ((ref)).
This certainly satisfies the odds-ratio bound ((ref)) --- and is therefore a possible value of $\bar{e}$ in the ZSB optimization problem ((ref)) --- but it could not possibly be the true propensity score $e_0$. If it were, we would observe $P(Z = 1 | X \geq 0) = \tfrac{2}{3}$, while the observed data distribution $P$ demands that $P(Z = 1 | X \geq 0) = \tfrac{1}{2}$. Another way of saying this is that $\bar{e}$ does not marginalize to the nominal propensity score:
In short, this choice of $\bar{e}$ is allowed in the domain of the ZSB optimization problem but is incompatible with the distribution of observed data.
This example suggests that it should be possible to improve upon the ZSB bounds by only optimizing over the subset of $\mathcal{E}_n(\Lambda)$ which is “data compatible." However, this is easier said than done, because the observed data distribution actually imposes an infinite number of constraints on putative propensity scores $\bar{e}$. For example, the true $e_0$ “balances" all integrable functions $h : \mathcal{X} \rightarrow \mathds{R}$:
Every such $h$ gives rise to a testable “balancing constraint" ((ref)) which can be used to rule out incompatible values of $\bar{e}$.
In other words, any sharp sensitivity analysis must contend with an infinite number of constraints, which is typically computationally intractable BeresteanuEtAl, DaveziesDHault. Previous works have considered relaxing these constraints by balancing only a finite set of functions tan2006, tudballZhaoEtAl2019interval, but the resulting bounds are generally not sharp.
While this paper proceeds under the “superpopulation" model of causal inference, the idea that observable quantities can constrain unobserved variables can also be applied in the “finite population" model. See tudballZhaoEtAl2019interval for an application of this idea to partial identification in survey sampling problems.
In this section, we show that at the population level, it is possible to characterize the sharp bounds for $\psi_0 \in \{ \psi_{\textup{T}}, \psi_{\textup{C}}, \psi_{\textup{ATT}}, \psi_{\textup{ATE}} \}$ without ignoring or relaxing any of the infinitely many balancing constraints on the true propensity score. We apply these partial identification results to finite-sample sensitivity analysis in Section (ref).
To state these results formally, we need a few pieces of additional notation. Recall that Assumption (ref) requires the true propensity score $e_0(X, U)$ to satisfy the following odds-ratio bound:
Therefore, it is natural to define $\mathcal{E}_{\infty}(\Lambda)$ to be the set of all random variables $\bar{E}$ which satisfy the same condition:
This can be viewed as the “population" version of the ZSB constraint set $\mathcal{E}_n(\Lambda)$.
Additionally, we define the conditional distribution function $F(y | x, z)$ and quantile function $Q_t(x, z)$ by:
Since these functions only refer to observed quantities, they are identified from the observed-data distribution.
Our first partial identification result shows that to compute optimal bounds for $\psi_{\textup{T}}$, the infinitely-many balancing constraints described in Section (ref) can actually be reduced to a single constraint. In particular, it suffices to minimize/maximize the function $\bar{E} \mapsto \mathbb{E}[YZ/\bar{E}]$ over the set of putative propensity scores $\bar{E} \in \mathcal{E}_{\infty}(\Lambda)$ that “balance" a particular conditional quantile of $Y$.
We will highlight a few important takeaways from this theorem. First, if one adds additional balancing constraints of the form $\mathbb{E}[ h(X) Z / \bar{E}] = \mathbb{E}[ h(X)]$ in ((ref)) and ((ref)), the value of these problems will not change. Thus, for the purposes of computing population-level bounds, the quantile balancing constraints in Theorem (ref) capture all the information in the observed data. Second, the fact that only a single conditional quantile appears in each of the sharp bounds for $\psi_{\textup{T}}$ reflects a special advantage of the marginal sensitivity model. For alternative sensitivity assumptions, sharp bounds often involve distinct quantiles $Q_{\tau(x)}$ for each covariate level LeeSelection, MastenPoirerSharp, complicating estimation by potentially requiring estimates of the entire conditional quantile process masten2020assessing, semenova2020better. Third, this result shows that the ZSB sensitivity analysis for IPW can only be sharp when the conditional quantiles of $Y$ do not depend on $X$ at all, and can therefore be refined outside pathological cases. AIPW-based variants of the ZSB sensitivity analysis will generally refine the IPW bounds since some of the variability in the quantiles of $Y$ will be absorbed by the regression function. We discuss AIPW sensitivity analysis in Section (ref).
We can extend the theorem to other estimands. To bound $\psi_{\textup{C}}$, exchange the labels “treated" and “control" and apply Theorem (ref). Sharp bounds on $\psi_{\textup{C}}$ can be translated into sharp bounds on $\psi_{\textup{ATT}}$ using the relation $\psi_{\text{ATT}} = \tfrac{\mathbb{E}[ Y ] - \psi_{\textup{C}}}{P(Z = 1)}$.
Sharp bounds for $\psi_{\textup{ATE}}$ can be obtained by subtracting sharp bounds for $\psi_{\textup{T}}$ and $\psi_{\textup{C}}$. Equivalently, these bounds can be obtained by solving optimization problems with two quantile balancing constraints. Although this result is superficially similar to Theorem (ref) and Corollary (ref), its proof requires a novel construction, which we discuss in Section (ref).
In certain special cases, the partially identified set for $\psi_{\textup{ATE}}$ can be computed more explicitly. These explicit bounds are useful for gaining intuition about the main factors that make a causal estimate more or less robust to unobserved confounding. Corollary (ref), which is a corollary of our later work, gives such bounds in the Gaussian outcome model ((ref)).
For a fixed bound $\Lambda$ on the degree of unobserved confounding, the formula ((ref)) shows that two key features map the observed data distribution to robustness. The first is the magnitude of the nominal ATE: all else equal, larger nominal effects are more robust. The second is the average noise level $\mathbb{E}[ \sigma(X)]$: the better the measured variables predict the outcome, the less unobserved confounding can affect our estimates. In the extreme case where $X$ and $Z$ perfectly predict $Y$, then the ATE remains point-identified no matter how large $\Lambda$ is, as long as overlap holds. These insights are not specific to the marginal sensitivity model. In alternative sensitivity models, they have also been observed by rosenbaum2005, hsu_small_rosenbaum2013, CinelliHazlett, and others. rosenbaum2005, hsu_small_rosenbaum2013, CinelliHazlett, and others.
Although the qualitative implications of Corollary (ref) are plausible, we nevertheless find the quantile balancing formulas of Section (ref) to be counterintuitive. After all, it is certainly not true that every random variable $\bar{E} \in \mathcal{E}_{\infty}(\Lambda)$ satisfying $\mathbb{E}[ Q_{\tau}(X, 1) Z / \bar{E}] = \mathbb{E}[ Q_{\tau}(X, 1)]$ could plausibly be the true propensity score $e_0(X, U)$. Indeed, the constraints of the quantile-balancing optimization problems do not even enforce that $\mathbb{E}[ Z / \bar{E}] = 1$. Our intuition for why the ZSB procedure is conservative suggests the quantile balancing formulas should be conservative as well.
To explain how these results are possible, we begin by characterizing which random variables $\bar{E}$ could plausibly be the true propensity score $e_0(X, U)$. The calculation ((ref)) indicates that $\bar{E}$ should at least satisfy $\mathbb{E}[ h(X) Z / \bar{E}] = \mathbb{E}[ h(X)]$ for all integrable $h$, or equivalently, $\mathbb{E}[Z / \bar{E} | X] = 1$. Proposition (ref) shows that for the purposes of bounding $\psi_{\textup{T}}$, this is actually the only constraint on $\bar{E}$ implied by the distribution of observables. Similar results appear in BirminghamJRSSB, Robins_etal_2000, tan2006, graham2011, hristache_patilea2017, FranksEtAl, zsb2019.
In short, this result says that $\mathbb{E}[ YZ / \bar{E}]$ is a plausible value of $\psi_{\textup{T}}$ as long as $\mathbb{E}[ Z / \bar{E} | X] = 1$. It is not hard to show that the converse also holds: if $\psi$ is a plausible value of $\psi_{\textup{T}}$, then $\psi = \mathbb{E}[ YZ / \bar{E}]$ for some random variable $\bar{E}$ satisfying $\mathbb{E}[ Z / \bar{E} | X] = 1$. As a result, the optimal bounds for $\psi_{\textup{T}}$ can be obtained by solving the variational problems in Corollary (ref).
Even though the variational problems ((ref)) and ((ref)) can be infinite-dimensional optimization problems with infinitely-many constraints, they have several nice features that enable them to be solved explicitly. Some straightforward algebraic manipulation shows that the problem ((ref)) can be written as:
Not only is this problem linear in the decision “variable" $1/\bar{E}$, it also separates across levels of $X$. Therefore, it suffices to separately solve ((ref)) for each $x \in \mathcal{X}$.
The problem ((ref)) requires us to maximize one expectation subject to an equality constraint on another expectation. This resembles the problem solved by the Neyman-Pearson lemma, and in fact is a special case of the generalization due to dantzig_wald_1951. The optimization problems posed in Theorem (ref) also fall in this class. It turns out that both of these problems have a common solution, given in Proposition (ref).
The form of the propensity score $\bar{E}_+$ gives us insight into the confounding structure which maximizes $\psi_\textup{T}$: in the worst case, all observations with “high" values of $Y$ are unlikely to be treated and thus receive large propensity weight, while all observations with “low" values of $Y$ are likely to be treated and thus receive small propensity weight. The cutoff between high and low is chosen to satisfy the data-compatibility condition $\mathbb{E}[ Z / \bar{E}_+ | X ] = 1$.
This argument presented in this section extends immediately to $\psi_{\textup{C}}$ by swapping treatment and control labels, extends to $\psi_{\textup{ATT}}$ by the argument given in Section (ref), and can extend to other sensitivity models of the form $e_{\min}(X) \leq e_0(X, U) \leq e_{\max}(X)$ by modifying the constraints of ((ref)).
To extend the argument from Section (ref) to the ATE requires additional care. Although $\psi_{\textup{ATE}}^+ = \psi_{\textup{T}}^+ - \psi_{\textup{C}}^-$ is certainly a valid upper bound for the partially identified set for $\psi_{\textup{ATE}}$, it is not obviously a sharp one. Proposition (ref) only implies that there exists a distribution $Q$ matching the observed-data distribution which has $\mathbb{E}_{Q}[Y(1)] = \psi_{\textup{T}}^+$ and another distribution $Q'$ which has $\mathbb{E}_{Q'}[ Y(0)] = \psi_{\textup{C}}^-$, but these distributions need not be the same. In other words, the two bounds may not be simultaneously achievable.
Theorem (ref) indicates that the worst-case bounds on the counterfactual means are simultaneously achievable in the marginal sensitivity model. This is a surprising result, given that simultaneous achievability is not expected to hold in the closely-related Rosenbaum sensitivity model. In that model, yadlowsky2018bounds derived sharp bounds on $\psi_{\textup{T}}$ and $\psi_{\textup{C}}$ but required an extra symmetry assumption on the distribution of potential outcomes to establish sharpness of the resulting ATE bounds.
The key to our bounds on $\psi_{\textup{ATE}}$ is the following claim, which strengthens Proposition (ref).
Unlike Proposition (ref), this result does not follow from the existing data-compatibility characterizations of BirminghamJRSSB, Robins_etal_2000, tan2006, zsb2019 and instead requires an original construction. Given this result, one can derive Theorem (ref) as a consequence of Theorem (ref) and Corollary (ref).
In this section, we give our proposals for translating the population-level partial identification results of Section (ref) into practical sensitivity analyses. Our main proposal, which we call the quantile balancing method, conducts a sensitivity analysis for IPW estimators by modifying the ZSB proposal to incorporate the sufficient constraints derived in Section (ref). We also discuss extensions of our sensitivity analysis to the AIPW estimator of robins_rotnitzky_zhao1994 which are simpler to implement but only sharp under homoscedasticity.
Throughout this section, we take $\Lambda \geq 1$ to be fixed and set $\tau = \Lambda/(\Lambda + 1)$.
We begin by describing our IPW sensitivity analysis for the average treated potential outcome. Theorem (ref) implies that the largest value of $\psi_{\textup{T}}$ compatible with Assumption (ref) solves the optimization problem ((ref)):
In the above display, we have included an additional constraint $\mathbb{E}[ Z / \bar{E}] = \mathbb{E}[ Z / e(X)]$ which motivates our finite-sample procedure without affecting the optimization problem value.
Our proposal is to estimate $\psi_{\textup{T}}^+$ by replacing all of the unknown quantities in ((ref)) with empirical counterparts. We estimate $\psi_{\textup{T}}^-$ by following the same principle. To translate these estimates into confidence intervals, we employ the same simple percentile bootstrap scheme as ZSB.
We will be concrete about what optimization problem we are proposing to solve. Let $\hat{Q}_{\tau}(x, z)$ be an estimate of the conditional quantile function of $Y$ obtained by some kind of quantile regression (e.g. generalized_random_forests, koenker_bassett_1978, quantile_random_forest, stone1977). Let $\hat{e}$ be the data analyst's estimate of the nominal propensity score $e$ from their primary analysis. We define $\hat{\psi}_{\textup{T}}^+$ as the solution to the empirical maximization problem ((ref)).
The lower bound $\hat{\psi}_{\textup{T}}^-$ is defined similarly, but with maximization replaced by minimization and $\hat{Q}_{\tau}(x, z)$ replaced by another quantile estimate $\hat{Q}_{1 - \tau}(x, z)$. We call $\hat{\psi}_{\textup{T}}^+$ and $\hat{\psi}_{\textup{T}}^-$ the quantile balancing bounds for $\psi_{\textup{T}}$.
Two features of this proposal require some explanation. The first feature to explain is the inclusion of the constraint $\mathbb{E}_n[Z / \bar{e}] = \mathbb{E}_n[ Z / \hat{e}(X)]$ in ((ref)). At the population level, Theorem (ref) shows that only the constraint $\mathbb{E}[ Q_{\tau}(X, 1) Z / \bar{E}] = \mathbb{E}[ Q_{\tau}(X, 1) Z / e(X)]$ is relevant. However, in finite samples, this additional constraint improves robustness when $\hat{Q}_{\tau}$ is an inaccurate estimate of $Q_{\tau}$ and also simplifies the associated computation. The second feature to explain is why the right-hand side of the constraints in ((ref)) have an “IPW" form (i.e. $\mathbb{E}_n[ \hat{Q}_{\tau}(X, 1) Z / \hat{e}(X)]$) rather than a “sample average" form (i.e. $\mathbb{E}_n[ \hat{Q}_{\tau}(X, 1)]$). If $\mathbb{E}_n[ \hat{Q}_{\tau}(X, 1) Z/ \hat{e}(X)] \neq \mathbb{E}_n[ \hat{Q}_{\tau}(X, 1)]$, then a sample average version of ((ref)) may have no feasible propensities. With the IPW form, $\bar{e}_i = \hat{e}(X_i)$ is always feasible.
Now that we have explained our proposed sensitivity analysis, we will collect several immediate properties of the quantile balancing bounds:\\
The property (ref) leads us to call quantile balancing a “sensitivity analysis for IPW." One can also apply quantile balancing to unstabilized IPW estimators at the cost of properties (ref) and (ref). See Appendix B for computational details, including for Augmented IPW estimators.
The quantile balancing idea extends easily to other causal estimands. To compute bounds for $\psi_{\textup{C}}$, one only needs to exchange the definitions of “treated" and “control" and solve the same optimization problem. Subtracting the bounds for $\psi_{\textup{T}}$ and $\psi_{\textup{C}}$ gives bounds for $\psi_{\textup{ATE}}$, and bounds for $\psi_{\textup{ATT}}$ follow from a similar principle (see Appendix A for the exact formula).
To form confidence intervals based on quantile balancing, we follow zsb2019 and propose using the percentile bootstrap. If $[\hat{\psi}_b^-, \hat{\psi}_b^+]$ are quantile balancing bounds estimated in the $b^{\text{th}}$ of $B$ bootstrap samples, we report the quantile balancing $1 - \alpha$ confidence interval as:
As is standard for bootstrap-based IPW inference, we require re-estimating the nominal propensity score separately in each bootstrap replication. That requirement does not extend to the conditional quantiles. While the conditional quantiles can be re-estimated within bootstraps, our inference results will also apply if they are taken from the main dataset. This helps keep inference computationally tractable.
The quantile balancing sensitivity analysis described above requires the data analyst to perform several quantile regressions. Our partial identification results imply that, in certain “additive-noise" data generating processes, a data analyst whose primary analysis was conducted using the AIPW estimator can perform sharp sensitivity analysis without performing any quantile regressions.
To explain how, we begin by describing the modeling assumption. Suppose the observed outcome $Y$ has the following signal-plus-noise representation:
Such models frequently arise in the regression applications esl2001 and fit quite well in the real-data example we present in Section (ref) below.
The additive-noise assumption ((ref)) implies that the conditional quantiles of the residuals $\epsilon$ are constant. In particular, the assumption implies $Q_\tau(x, z) = \mu(x, z) + Q_\tau(\epsilon)$, where $Q_\tau(\epsilon)$ is the $\tau$-th quantile of the noise. Therefore, Theorem (ref) and some algebra imply that the sharp upper bound for $\psi_{\textup{T}}$ has the following formula:
Similar formulas can be derived for $\psi_{\textup{T}}^-, \psi_{\textup{C}}^+, \psi_{\textup{C}}^-$. This formula is convenient after an AIPW primary analysis, which requires estimates of all the nuisance parameters in this equation.
A natural estimate of $\psi_{\textup{T}}^+$ is the finite-sample analogue of ((ref)).
The estimated bound $\hat{\psi}_{\textup{T,AIPW}}^+$ grows with $\Lambda$ and recovers the original (stabilized) AIPW estimator when $\Lambda = 1$. One can also not divide by $\mathbb{E}_n[Z/\bar{e}]$ in ((ref)) to recover the unstabilized AIPW estimator at $\Lambda = 1$.
The estimator ((ref)) slightly modifies the proposal in Section 6.2 of zsb2019 to include the balancing constraint $\mathbb{E}_n[ Z / \bar{e}] = \mathbb{E}_n[Z / \hat{e}(X)]$. In theory, this constraint is necessary to achieve sharpness in the additive-noise model ((ref)). However, the simulations presented in Section (ref) find that when the additive-noise model holds, this constraint scarcely refines the stabilized point estimates while somewhat degrading the coverage of bootstrap confidence intervals.
We now state some theoretical properties of the quantile balancing bounds $[\hat{\psi}^-, \hat{\psi}^+]$ which apply when the outcome $Y$ has a continuous distribution. In short, the bounds are sharp when quantiles are estimated consistently and are valid even when quantiles are estimated inconsistently. Moreover, the percentile bootstrap yields valid confidence intervals if standard IPW inference conditions are satisfied and quantiles are estimated parametrically.
To obtain these results, we need a few conditions. The first condition collects some standard IPW consistency requirements which we expect the data analyst to have already assumed in their primary analysis.
The second condition requires that the outcome $Y$ has a bounded conditional density which is positive near the relevant conditional quantiles. This is a common identification condition for quantile regression generalized_random_forests, BelloniEtAl2019. However, it means our theoretical guarantees do not apply when $Y$ is discrete.
Finally, we make some assumptions about how the quantiles are estimated. For the standard linear quantile regression method of koenker_bassett_1978, one only needs to check that the regressors in the quantile regression have finite variance. We cover generic (possibly nonlinear) methods by requiring sample splitting to avoid overfitting. The specific form of sample splitting analyzed in our proofs is “cross-fitting" schick1986, newey_robins_crossfitting, doubleML, but leave-one-out or out-of-bag quantile estimates perform similarly in simulations. Based on our practical experience, we recommend using some kind of sample splitting even when the quantile model is linear.
Condition (ref) is essentially “algorithmic," and neither (ref) nor (ref) impose any accuracy requirements on the estimated conditional quantiles. The appendix conditions in (ref) are technical to state but very mild. Under Conditions (ref) and (ref), they are satisfied by quantile estimates based on nearest-neighbors stone1977, kernels kernel_quantile1990, and random forests generalized_random_forests, quantile_random_forest.
Under these conditions, we have the following result on the asymptotic sharpness of the quantile balancing bounds.
The same conclusions hold for the AIPW-based bounds introduced in Section (ref) when the outcome regression is estimated by linear regression, i.e. sharpness under an additive-noise model and validity in general. However, while AIPW is doubly-robust under unconfoundedness, the validity of the corresponding AIPW quantile balancing bounds relies on correct specification of the nominal propensity score.
The result (ref) shows that even when quantiles are not estimated consistently, the quantile balancing bounds are still valid; we will offer some intuition on why this novel robustness property holds. At the population level, the worst-case propensity score $\bar{E}_+$ defined in Proposition (ref) “balances" all integrable function of $X$, so intuitively, we should expect that it “nearly" balances the estimated quantile function $\hat{Q}_{\tau}(\cdot, 1)$ in finite samples even if $\hat{Q}_{\tau}(\cdot, 1)$ is not particularly close to $Q_{\tau}(\cdot, 1)$. That suggests a vector of propensities very close to the true worst-case propensity vector will belong to the feasible set $\mathcal{E}_n(\Lambda)$. Since the quantile balancing upper bound $\hat{\psi}_{\textup{T}}^+$ is defined as a maximum over the feasible set, it will be at least as large as a quantity close to $\psi_{\textup{T}}^+$. This roughly explains why validity holds even under misspecification.
The validity of the confidence interval ((ref)) follows under stronger parametric assumptions. We prove an inference result assuming the nominal propensity score is estimated by a correctly-specified parametric model and the conditional quantiles are estimated by a (potentially misspecified) parametric model.
We have found that these confidence intervals can under-cover the identified set in finite samples when the quantiles are correctly specified. In our simulations, the use of cross-fit conditional quantile estimates largely resolves the issue with minimal effect on point estimates, so we advocate for the use of such estimators in practice. Although we do not have theoretical support for the confidence interval $\text{CI}(\alpha)$ when quantiles are estimated by a nonlinear model, we find that approach performs reasonably well in the simulations of Section (ref) as long as cross-fit quantiles are used.
In this section, we illustrate the finite-sample performance of our proposed sensitivity analyses on several simulated datasets and one real-data example.
We consider two data-generating processes (DGPs) in our simulated examples. The two DGPs differ in the conditional distribution of $Y$ given $(X, Z)$, but otherwise can be described as follows:
In the first DGP, we use $\mu(x) = x_1 + \cdots + x_5$ and $\sigma(x) = 1$. In the second DGP, we use $\mu(x) = \tfrac{3}{2} \text{sign}(x_1) + \text{sign}(x_2)$ and $\sigma(x) = 2 + \text{sign}(x_3) + \text{sign}(x_4)$. The estimand of interest is the ATE and we fix $\Lambda = 2$, i.e. unobserved confounders can double or halve the odds of treatment.
We compare five methods for obtaining bounds on the partially identified set:
All methods estimate the nominal propensity score by logistic regression. We use 5-fold cross-fitting in all of our quantile regressions. We do not re-estimate quantiles or random forest models within bootstraps.
Figure (ref) shows the distribution of upper and lower bound point estimates from each of these five methods, estimated using 2,000 simulations with $n = 1,000$ observations each. Simulations at other sample sizes are presented in Appendix B. Dashed lines indicate the true partially identified region. The results conform to the asymptotic predictions of Section (ref): (i) when the quantile models are “correctly specified," the quantile balancing point estimates are nearly unbiased; (ii) under misspecification, the range of QB point estimates is too wide rather than too narrow; (iii) the ZSB range of point estimates is too wide in both cases; and (iv) AIPW-based methods give nearly-sharp bounds in the additive-noise DGP1 but conservative bounds in the heteroscedastic DGP2. We also find that the +1 constraint in AIPW+1, which is necessary for sharpness in theory, has minimal practical impact in either DGP.
Figure (ref) shows the coverage for 95% bootstrap confidence intervals based on each of the five methods. In DGP1, both quantile balancing methods have nearly nominal coverage, but AIPW-based methods undercover and the +1 constraint exacerbates the undercoverage. In DGP2 the QB-Forest method achieves nearly nominal coverage, while all other methods overcover. The ZSB method overcovers for both DGPs.
In this section, we apply our proposed sensitivity analysis to a subsample of data from the 1966-1981 National Longitudinal Survey (NLS) of Older and Young Men. We wish to estimate the impact of union membership on wages. Specifically, we consider the ATE of union membership on log wages. For illustrative reasons, we focus on the 1978 cross-section of Young Men and restrict our attention to craftsmen and laborers not enrolled in school. Our estimates are thus based on a sample of 668 respondents with measurements of wages, union membership, and eight covariates.
For our primary analysis, we use IPW to adjust for baseline imbalances in covariates between union and nonunion samples. Table (ref) reports the covariate balance between union and nonunion samples before and after weighting by the (estimated) inverse propensity score. On several important characteristics, inverse propensity weighting dramatically improves balance across the two samples.
The IPW point estimate of the ATE is 0.23 with an associated 90% confidence interval of $[0.18, 0.27]$. Thus, our primary analysis concludes that union membership has a positive effect on wages, at least on average among craftsmen and laborers. Both the point estimate and the confidence interval are in agreement with prior literature studying the same problem using cross-sectional data. See jakubson1991, johnson1975 for overviews. An AIPW-based primary analysis gives the same point estimate and confidence interval, up to rounding.
freemn1984_unions, mellow1981, and many other economists have argued that cross-sectional estimates of the union premium overestimate the true causal effect because higher-skill workers are simultaneously more likely to be selected for union jobs and earn higher wages. Here, “skill" refers to an unobserved confounder which is only partially captured by the measured covariates. Is it plausible that the positive effect we find in the IPW analysis could be entirely due to selection on skill? A sensitivity analysis may help address this question.
Figure (ref) reports point estimate ranges and 90% bootstrap confidence intervals from quantile balancing, the ZSB-IPW method, and the ZSB-AIPW method for several values of the sensitivity parameter $\Lambda$. For quantile balancing, we estimate conditional quantiles using linear quantile regression with five-fold cross fitting. For AIPW, we use linear regression for the outcome model.
All three sensitivity analyses show that the positive effect found in the primary analysis is fairly robust to unobserved confounding, but quantile balancing and ZSB-AIPW refine the baseline ZSB-IPW interval. Even if the odds of union membership for “skilled" workers were nearly double ($\Lambda = 1.9$) the odds for “typical" workers with the same observed covariates, the quantile balancing and AIPW sensitivity analyses analysis would still find a statistically significant positive treatment effect. Meanwhile, when $\Lambda = 1.8$, the ZSB confidence intervals already include the null. In this application, quantile balancing only slightly refines the ZSB range. Moreover, quantile balancing and ZSB-AIPW yield very similar ranges and confidence intervals. This is to be expected from the discussion in Section (ref), as an “additive noise" model appears quite plausible in this application.
To put these sensitivities in context, we follow kallus_zhou2020 and compute the degree to which the (estimated) odds of union membership could change if measured confounders were omitted from the dataset. Caveats to this approach and more sophisticated empirical calibration strategies are discussed in hsu_small2013, zhang_small2020, CinelliHazlett. No measured confounders except Laborer and South were able to nearly double or halve the odds of union membership for any respondent. We interpret these results as showing that the qualitative conclusions of the primary analysis are fairly robust to unobserved confounding by skill.
Incidentally, longitudinal estimates of union wage effects --- which control for individual-specific effects like “skill" --- come to similar conclusions as the one suggested by our sensitivity analysis. Although treatment effect estimates from longitudinal studies are generally smaller than those from cross-sectional studies, they still find evidence in favor of the “union premium" CHAMBERLAIN1982, jakubson1991, freemn1984_unions.
We have shown that quantile balancing --- a simple modification of the popular ZSB sensitivity analysis --- is feasible, robust, and sharp. This new sensitivity analysis for IPW is based on novel partial identification results for tan2006's marginal sensitivity model.
We will point to several interesting directions for future work. While our partial identification results focus on counterfactual means and a few treatment effects, it should be possible to extend our partial identification results to more complex estimands of the type considered in kallus2018interval, kallus2020confoundingrobust, confounding_robust_policy_improvement, kallus_zhou2020, causal_rule_ensemble. Perhaps a similarly compact sensitivity analysis could even apply to dynamic treatment regimes. Future work could also investigate data-compatibility in the finite population model. In addition, while our IPW identification arguments generalize to any sensitivity assumption that only restricts the propensity score in a pointwise fashion (i.e. $e_{\min}(x) \leq e_0(x, u) \leq e_{\max}(x)$), the practicality of our sensitivity analysis and its theoretical properties rely on the marginal sensitivity model quite heavily. It would be interesting to see if a practical and sharp sensitivity analysis could be developed for other sensitivity assumptions in this class.