Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
49,354 characters · 11 sections · 34 citation commands
Estimating Treatment Effects with Observed Confounders and Mediators
Causal effects are not, in general, identifiable from observational data alone. The fundamental insight of causal inference is that given structural assumptions on the data generating process, causal effects may become expressible as functionals of the joint distribution over observed variables. The do-calculus, introduced by pearl1995causal, provides a set of three rules that can be used to convert causal quantities into such functionals. We are motivated by the observation that, for some causal graphs, treatment effects may be over-identified. Here, applications of the do-calculus produce distinct functionals, all of which, subject to positivity conditions, yield consistent estimators of the same causal effect. Consider a causal graph (see Figure (ref)) for which the treatment $X$, mediator $M$, confounder $W$, and outcome $Y$ are all observable. Using the backdoor adjustment, we can express the average treatment effect of $X$ on $Y$ as a function of $P(X,W,Y)$, while the frontdoor adjustment expresses that same causal quantity via $P(X,M,Y)$ pearl1995causal. In our experiments, we work with a real-world dataset that contains both confounders and mediators. Faced with the (fortunate) condition of overidentification, our focus shifts from identification: is our effect estimable?, to optimality: which among multiple valid estimators dominates from a standpoint of statistical efficiency?
In this paper, we address this very graph, focusing our analysis on the linear causal model wright1934, a central object of study in causal inference and econometrics and also explore the semiparametric setting. Over-identification can arise in many other causal graphs (e.g. multiple backdoor adjustment sets, multiple instrumental variables, etc.). However, we focus on this graph because the frontdoor estimator is a canonical example of a novel identification result derived using graphical models. It is central in the causality literature pearl2018book, imbens2019potential and is a natural first step in the study of over-identified causal models. Deriving the finite sample variance of the backdoor and frontdoor estimators, and precisely characterizing conditions under which each dominates, we find that either may outperform the other to an arbitrary degree depending on the underlying model parameters. These expressions can provide guidance to practitioners for assessing the suitability of each estimator. For example, one byproduct of our analysis is to characterize what properties make for the “ideal mediator”. Moreover, in the data collection phase, if one has a choice between collecting data on the mediator or the confounder, these expressions, together with the practitioner's beliefs about likely ranges for model parameters, can be used to decide what data to collect.
Next, we propose techniques that leverage both observed confounders and mediators. For the setting where we simultaneously observe both the confounder and the mediator, we introduce an estimator that optimally combines all information. We prove theoretically that this method achieves lower mean squared error (MSE) than both the backdoor and frontdoor estimators, for all settings of the underlying model parameters. Moreover, the extent to which this estimator can dominate the better of the backdoor and frontdoor estimators is unbounded. Subsequently, we consider the partially-observed setting in which two datasets are available, one with observed confounders (but not mediators) $\{(X, W, Y)\}_{i=1}^n$, and another with observed mediators (but not confounders) $\{(X, M, Y)\}_{i=1}^m$. Interestingly, the likelihood is convex given simultaneous observations but non-convex under partially-observed data. We introduce an estimator that is guaranteed to achieve higher likelihood than either the backdoor or frontdoor estimators. Finally, we evaluate our methods on synthetic, semi-synthetic, and real datasets. Our proposed estimators that combine confounders and mediators always exhibit lower MSE than the backdoor and frontdoor estimators when our model assumptions are satisfied.
Our principal contributions are the following:
The backdoor adjustment formalizes the practice of controlling for known confounders and is widely applied in statistics and econometrics pearl2009causality, pearl2010foundations,perkovic2015complete. The frontdoor adjustment, which leverages observed mediators to identify causal effects even amid unobserved confounding, has seen increasing application in real-world datasets bellemare2019paper, glynn2018front, glynn2017front, chinco2016misinformed,cohen2014friends.
In the most similar work to ours, glynn2018front compare the frontdoor and backdoor adjustments, computing bias (but not variance) formulas for each and performing sensitivity analysis. Exploring a real-world job training dataset, they demonstrate that the frontdoor estimator outperforms its backdoor counterpart (in terms of bias). The finite sample variance of the frontdoor estimator for the linear Gaussian case was previously derived by kuroki2000selection. ramsahai2012supplementary compare the frontdoor and backdoor estimators based on their asymptotic variances and also show that the combined estimator's variance cannot be higher than the other two. kuipers2020variance derive the finite-sample variances of two possible adjustments in a three-variable binary causal graph and show that the optimal estimator depends on the model parameters. henckel2019graphical introduce a graphical criterion for comparing the asymptotic variances of adjustment sets for the backdoor criterion in linear causal models. rotnitzky2019efficient extend this work, showing that the same graphical criterion is valid for non-parametric causal models. They also present a semi-parametric efficient estimator that exploits the conditional independencies in a causal graph.
Researchers have also worked to generalize the frontdoor criterion. bareinboim2019causal introduce the conditional frontdoor criterion, allowing for both treatment-mediator confounders and mediator-outcome confounders. fulcher2020robust propose a method for including observed confounders along with a mediator with discrete treatments.
The study of overidentified models dates at least back to koopmans1950identification. sargan1958estimation, Hansen1982 formalized the result that in the presence of overidentification, multiple estimators can be combined to improve efficiency. This was extended to the non-parametric setting by chen2018overidentification. A related line of work considers methods for combining multiple datasets for causal inference. bareinboim2016causal study the problem of handling biases while combining heterogeneous datasets, while jackson2009bayesian present Bayesian methods for combining datasets with different covariates and some common covariates.
In this work, we work within the structural causal model (SCM) framework due to pearl2009causality, formalizing causal relationships via directed acyclic graphs (DAGs). Each $X \rightarrow Y$ edge in this DAG indicates that the variable $X$ is (potentially) a direct cause of variable $Y$. All measured variables are deterministic functions of their parents and a set of jointly independent per-variable noise terms.
Linear Gaussian SCM \quad In linear Gaussian SCMs, each variable is assumed to be a linear function of its parents. The noise terms are assumed to be additive and Gaussian. In this paper, the finite sample results are derived for the linear Gaussian SCM for the overidentified confounder-mediator graph (Figure (ref)), where the structural equations can be written as
Here, $w_i, x_i, m_i$, and $y_i$ are realized values of the random variables $W, X, M, Y$, respectively, and $u^w_i, u^x_i, u^m_i, u^y_i$ are realized values of the corresponding noise terms. The zero mean assumption in Eq. (ref) simplifies analysis, but is not necessary for the results presented in this paper.
The effect of a treatment $X$ is expressible in terms of the post-intervention distributions of the outcome $Y$ for different values of the treatment $X=x$. An intervention $do(X=x)$ in a causal graph can be expressed via the mutilated graph that results from deleting all incoming arrows to $X$, setting $X$'s value to $X=x$ for all instances, while keeping the SCM otherwise identical. This distribution is denoted as $P(Y|do(X=x))$.
The backdoor and frontdoor adjustments pearl2009causality express treatment effects as functionals of the observational distribution. Consider our running example of the causal model in Figure (ref). We denote $X$ as the treatment, $Y$ as the outcome, $W$ as a confounder, and $M$ as a mediator. Our goal is to estimate the causal quantity $P(Y|do(X=x))$.
Backdoor Adjustment \quad When all confounders of both $X$ and $Y$ are observed---in our example, $W$---then the causal effect of $X$ on $Y$, i.e., $P(Y|do(X=x))$ can be written as
Frontdoor Adjustment \quad This technique applies even when the confounder $W$ is unobserved. Here we require access to a mediator $M$ that (i) is observed; (ii) transmits the entire causal effect from $X$ to $Y$; and (iii) is not influenced by the confounder $W$ given $X$. The effect of $X$ on $Y$ is computed in two stages. We first find the effect of $X$ on $M$, then the effect of $M$ on $Y$ as:
We can then write the causal effect of $X$ on $Y$ as
In this section, we analyze the backdoor and frontdoor estimators and characterize the regimes where each dominates. We work with the linear SCM described in Eq. (ref). Throughout, our goal is to estimate the causal effect of $X$ on $Y$. In terms of the underlying parameters of the linear SCM, the quantity that we wish to estimate is $ac$. Absent measurement error, both estimators are unbiased (see proof in Appendix (ref)) and thus we focus our comparison on their respective variances.
Variance of the Backdoor Estimator \quad The backdoor estimator requires only that we observe $\{X, Y, W\}$ (but not necessarily the mediator $M$). Say we observe the samples $\{x_i, y_i, w_i\}^n_{i=1}$. We can estimate the causal effect $ac$ by taking the coefficient on $X$ in an OLS regression of $Y$ on $\{X, W\}$. This controls for the confounder $W$ and corresponds naturally to the adjustment described in Eq. (ref).
The finite sample and asymptotic variances of the backdoor estimator are (see proof in Appendix (ref))
Variance of the Frontdoor Estimator \quad The frontdoor estimator is used when $\{X, Y, M\}$ samples are observed. Say we observe the samples $\{x_i, y_i, m_i\}^n_{i=1}$. First, we estimate $c$ by taking the coefficient on $X$ in an OLS regression of $M$ on $X$. Let the estimate be $\widehat{c}$. This corresponds to the adjustment in Eq. (ref). Then, we estimate $a$ by taking the coefficient on $M$ in an OLS regression of $Y$ on $\{M, X\}$. Let the estimate be $\widehat{a}_f$. This corresponds to the adjustment in Eq. (ref).
The finite sample variances of $\widehat{c}$ and $\widehat{a}_f$ are (see proof in Appendix (ref))
Using the facts that $\text{Cov}(\widehat{a}_f, \widehat{c}) = 0$ and $\text{Cov}(\widehat{a}_f^2, \widehat{c}^2) = \text{Var}(\widehat{a}_f)\text{Var}(\widehat{c})$, the finite sample variance of the frontdoor estimator is (see proof in Appendix (ref))
And the asymptotic variance, which does not require Gaussianity, is (see proof in Appendix (ref))
The Ideal Frontdoor Mediator \quad A natural question then arises: what properties of a mediator make the frontdoor estimator most precise? We can see that $\text{Var}(\widehat{a}_f \widehat{c})$ is non-monotonic in the mediator noise $\sigma_{u_m}$. Eq. (ref) provides us with guidance. $\text{Var}(\widehat{a}_f \widehat{c})$ is a convex function of $\sigma^2_{u_m}$. The ideal mediator will have noise variance $\sigma^{2*}_{u_m}$ which minimizes Eq. (ref). That is,
where $D = d^2 \sigma^2_{u_w} + \sigma^2_{u_x}$.
Comparison of Backdoor and Frontdoor Estimators \quad The relative performance of the backdoor and frontdoor estimators depend on the underlying SCM's parameters. Using Eqs. (ref) and (ref), the ratio of the backdoor to frontdoor variance is
where $D = (d^2 \sigma^2_{u_w} + \sigma^2_{u_x})$ and $E = (b^2 \sigma^2_{u_w} \sigma^2_{u_x} + \sigma^2_{u_y} D)$. The backdoor estimator dominates when $R_{\text{Var}} < 1$ and vice versa when $R_{\text{Var}} > 1$. Note that there exist parameters that cause any value of $R_{\text{Var}} > 0$. In particular, as $\sigma^2_{u_x} \rightarrow 0$, $R_{\text{Var}} \rightarrow \infty$ and as $\sigma^2_{u_x} \rightarrow \infty$, $R_{\text{Var}} \rightarrow 0$, regardless of the sample size $n$. Thus, either estimator can dominate the other by any arbitrary constant factor.
Having characterized the performance of each estimator separately, we now consider optimal strategies for estimating treatment effects in the overidentified regime, where we observe both the confounder and the mediator simultaneously. Say we observe $n$ samples $\{x_i, y_i, w_i, m_i\}_{i=1}^{n}$. We show that the maximum likelihood estimator (MLE) is strictly better than the backdoor and frontdoor estimators. The MLE will be optimal since our model satisfies the necessary regularity conditions for MLE optimality (by virtue of being linear and Gaussian). The combined estimator is unbiased (see Appendix (ref)) and thus we focus on the variance.
Let the vector $\mathbf{s_i} = [x_i, y_i, w_i, m_i]$ denote the $i^{\text{th}}$ sample. Since the data is multivariate Gaussian, the log-likelihood of the data is $\mathcal{LL} = -\frac{n}{2} \left[\log{(\det{\Sigma})} + \operatorname{Tr}{(\widehat{\Sigma} \Sigma^{-1})}\right]$, where $\Sigma = \text{Cov}([X, Y, W, M])$ and $\widehat{\Sigma} = \frac{1}{n} \sum_{i=1}^{n} \mathbf{s_i} \mathbf{s_i}^\top$. The MLE for a Gaussian graphical model is $\Sigma^{\text{MLE}} = \widehat{\Sigma}$ uhler2019gaussian. Let the MLE estimates for parameters $c$ and $a$ be $\widehat{c}$ and $\widehat{a}_c$, respectively. Then
The MLE estimate for $c$ in Eq. (ref) is the same as for the frontdoor---the coefficient of $X$ in an OLS regression of $M$ on $X$. The MLE estimate for $a$ in Eq. (ref) is the coefficient of $M$ in an OLS regression of $Y$ on $\{M, W\}$. The finite sample variance of $\widehat{a}_c$ is (see proof in Appendix (ref))
The variance of $\widehat{c}$ is the same as the frontdoor case as in Eq. (ref). Let $r_1 = \sqrt{\frac{n-3}{n-5}}$, $r_2 = \sqrt{\frac{3(n-2)}{n-4}}$, and $L = \left( \frac{c^2 \sigma^2_{u_y}}{c^2 \sigma^2_{u_x} + \sigma^2_{u_m}} + \frac{a^2 \sigma^2_{u_m}}{d^2 \sigma^2_{u_w} + \sigma^2_{u_x}} \right)$. We can bound the finite sample variance of the combined estimator as
The lower bound is derived using the Cramer-Rao theorem (since the estimator is unbiased) and for the upper bound, we use the Cauchy-Schwarz inequality. The complete proof is in Appendix (ref). And the asymptotic variance, which does not require Gaussianity, is (see proof in Appendix (ref))
The Ideal Mediator \quad Just as with the frontdoor estimator, we can ask what makes for an ideal mediator in this case. Eq. (ref) shows that $\lim_{n \to \infty} \text{Var}(\sqrt{n}\widehat{a}_c\widehat{c})$ is a convex function of $\sigma^2_{u_m}$. The ideal mediator will have noise variance $\sigma^{2*}_{u_m}$ which minimizes the variance in Eq. (ref). We use the asymptotic variance here since we only have finite-sample bounds on the variance of the combined estimator. This means that
We can compare Eqs. (ref) and (ref) too see that, asymptotically, the combined estimator has lower variance than the backdoor estimator for all values of model parameters. That is, as $n \rightarrow \infty$, $\text{Var}(\sqrt{n}\widehat{a}_c \widehat{c}) \leq \text{Var}(\sqrt{n}\widehat{ac})_{\text{backdoor}}$. Similarly, we can compare Eqs. (ref) and (ref) to see that, asymptotically, the combined estimator is always better than the frontdoor estimator for all values of model parameters. That is, as $n \rightarrow \infty$, $\text{Var}(\sqrt{n}\widehat{a}_c \widehat{c}) \leq \text{Var}(\sqrt{n}\widehat{a}_f \widehat{c})$.
In the finite sample case, using Eqs. (ref) and (ref), we can see that for all model parameters, for a large enough $n$, the combined estimator will dominate the backdoor. That is, $\exists N, \text{s.t.}, \forall n > N, \text{Var}(\widehat{a}_c \widehat{c}) \leq \text{Var}(\widehat{ac})_{\text{backdoor}}$, where the dependence of $N$ on the model parameters is stated in Appendix (ref). We can make a similar argument for the dominance of the combined estimator over the frontdoor estimator. Using Eqs (ref) and (ref), it can be shown that $\exists N, \text{s.t.}, \forall n > N, \text{Var}(\widehat{a}_c \widehat{c}) \leq \text{Var}(\widehat{a}_f \widehat{c})$, where the dependence of $N$ on the model parameters is stated in Appendix (ref).
Next, we show that the combined estimator can dominate the better of the backdoor and frontdoor estimators by an arbitrary amount. That is, we show that the quantity $R = \frac{\min \left\{ \text{Var}(\widehat{ac})_\text{backdoor}, \text{Var}(\widehat{a}_f\widehat{c}) \right\}}{\text{Var}(\widehat{a}_c\widehat{c})}$ is unbounded. Consider the case when $\text{Var}(\widehat{ac})_\text{backdoor} = \text{Var}(\widehat{a}_f\widehat{c})$. This condition holds for certain settings of the model parameters (see Appendix (ref) for an example). Here,
where $D = d^2 \sigma^2_{u_w} + \sigma^2_{u_x}$, $E = c^2 \sigma^2_{u_x} + \sigma^2_{u_m}$, $r_1 = \sqrt{\frac{n-3}{n-5}}$, $r_2 = \sqrt{\frac{n-2}{n-4}}$, $F = (n-3) a^2 \sigma^2_{u_m} E$, $G = r_1 \frac{\sigma_{u_m}}{ \sqrt{(n-2) D} }$, $H = r_1 r_2 + |c| (n-2) D \left( |c| + G \right)$ and, in Eq. (ref), we used Eq. (ref). We can see that as $\sigma_{u_x} \rightarrow 0$, $R \rightarrow \infty$ and thus $R$ is unbounded. This shows that, even in finite samples, combining confounders and mediators can lead to an arbitrarily better estimator than the better of the backdoor and frontdoor estimators.
fulcher2020robust derive the efficient influence function and semi-parametric efficiency bound for a generalized model with discrete treatment and non-linear relationships between the variables. While they allow for confounding of the treatment-mediator link and the mediator-outcome link, the graph in Figure (ref) has additional restrictions. As per chen2018overidentification, this graph is locally overidentified. This suggests that it is possible to improve the estimator by fulcher2020robust (which we refer to as IF-Fulcher). In our model, there are two additional conditional independences compared to the graph studied in fulcher2020robust: $Y \perp\!\!\!\!\perp X | (M, W)$, and $M \perp\!\!\!\!\perp W | X$. We incorporate these conditional independences in IF-Fulcher by using $\mathbb{E}[Y|M,W,X]=\mathbb{E}[Y|M,W]$, and $f(M|X,W)=f(M|X)$ to create an estimator we refer to as IF-Restricted:
where, if $\widehat{f}, \widehat{P}$, and $\widehat{\mathbb{E}}$ are consistent estimators, then $\widehat{\Psi} \overset{p}{\to} \mathbb{E}[Y|do(X=x^*)]$. By double robustness of the given estimator, if $\widehat{f}, \widehat{P}$, and $\widehat{\mathbb{E}}$ are correctly specified, then IF-Restricted has identical asymptotic distribution as IF-Fulcher. But using the additional restrictions improves estimation of nuisance functions. Thus we expect the proposed semi-parametric estimator to perform better in finite samples. rotnitzky2019efficient, in contemporaneous work, analyzed the same graph and showed that, in addition, the efficient influence function is also changed when imposing these conditional independences (see Example 10 in their paper) (we refer to the estimator for this influence function as IF-Rotnitzky). For our experiments with binary treatments, we use linear regression for $\widehat{f},\, \widehat{\mathbb{E}}$ and logistic regression for $\widehat{P}$. Another way to adapt IF-Fulcher is for the case when we do not observe the confounders (as in the frontdoor adjustment). In this case, we can set $W_i = \varnothing$ and apply $\widehat{\Psi}$. We call this special case IF-Frontdoor.
We now consider a situation in which the practitioner has access to two datasets. In the first one, the confounders are observed but the mediators are unobserved. In the second one, the mediators are observed but the confounders are unobserved. This situation might arise if data is collected by two groups, the first selecting variables to measure to apply the backdoor adjustment and the second selecting variables to apply the frontdoor adjustment. Given the two datasets, we wish to optimally leverage all available data to estimate the effect of $X$ on $Y$.
A naive approach would be to apply the backdoor and frontdoor estimator to the first and second dataset, respectively, and take a weighted average of the two estimates. However, in this case, the variance will be between that of the frontdoor and backdoor estimator. We analyze the MLE, showing that this estimator has lower asymptotic variance than both the backdoor and frontdoor estimators.
Combined Log-Likelihood under Partial Observability \quad Say we have $P$ samples of $\{x_i, y_i, w_i\}^P_{i=1}$. Let each such sample be denoted by the vector $\mathbf{p}_i = [x_i, y_i, w_i]$. Moreover, say we have $Q$ samples of $\{x_i, y_i, m_i\}^Q_{i=1}$. Let each such sample be denoted using the vector $\mathbf{q}_j = [x_j, y_j, m_j]$. Let the observed data be represented as $D$. That is, $D = \left\{ \mathbf{p}_1, \mathbf{p}_2, \hdots, \mathbf{p}_P, \mathbf{q}_1, \mathbf{q}_2, \hdots, \mathbf{q}_Q \right\}$. Let $N = P + Q$ and let $k = \frac{P}{N}$. Since the data is multivariate Gaussian, the conditional log-likelihood given $k$ can be written as
where $\Sigma_p = \text{Cov}([X, Y, W])$, $\Sigma_q = \text{Cov}([X, Y, M])$, $\widehat{\Sigma}_p = \frac{\sum_{i=1}^P \mathbf{p}_i \mathbf{p}_i^\top}{P} $ and $\widehat{\Sigma}_q = \frac{\sum_{i=1}^Q \mathbf{q}_i \mathbf{q}_i^\top}{Q}$.
Cramer-Rao Lower Bound \quad To compute the variance of the estimate of $e = ac$, we compute the Cramer-Rao variance lower bound. We first compute the Fisher information matrix $\mathbf{I}$ as $\mathbf{I} = - \mathbb{E} \left[ \nabla^2_\theta \mathcal{LL} \right]$, where $\theta$ represents the eight model parameters. Let $\widehat{e}$ be the MLE. Since regularity holds for our model (due to linearity and Gaussianity), the MLE is asymptotically normal. Using the Cramer-Rao theorem, for constant $k$, as $N \to \infty$, we have $\sqrt{N}(\widehat{e} - e) \overset{d}{\to} \mathcal{N}(0, V_e)$, where $V_e$ is a function of $\mathbf{I}^{-1}$. The closed form expression for $V_e$ is given in Appendix (ref).
For any fixed $k \in (0, 1)$, $(V_e - \text{AVar}(\sqrt{P}\widehat{ac})_{\text{backdoor}}) < 0$ and $(V_e - \text{AVar}(\sqrt{Q}\widehat{a}_f \widehat{c})) < 0$, where $\text{AVar}$ is asymptotic variance. This shows that the combined estimator always has lower asymptotic variance than that of the backdoor and frontdoor estimators on the individual datasets. Moreover, we also find cases where the combined estimator outperforms both the backdoor and frontdoor estimators even when the total number of samples are the same. That is, there exist model parameters such that $(V_e - \text{AVar}(\sqrt{N}\widehat{ac})_{\text{backdoor}}) < 0$ and $(V_e - \text{AVar}(\sqrt{N}\widehat{a}_f \widehat{c})) < 0$ for some $k \in (0, 1)$. This means that is these cases, it is better to collect a mix of confounders and mediators rather than only collecting mediators or confounders. Despite having access to the same number of samples, a mix of confounders and mediators can lead to lower variance. This happens when the variances of the backdoor and frontdoor estimators are close to each other. In Figure (ref), we present two examples of causal graphs where having a mix of confounders and mediators leads to the lowest asymptotic variance (see Appendix (ref) for parameter values).
The Maximum Likelihood Estimator \quad Computing an analytical solution for the model parameters that maximizes the log-likelihood turns out to be intractable. As a result, we update our estimated parameters to maximize the likelihood numerically. The likelihood in Eq. (ref) is non-convex. So we intialize the parameters using the two datasets (see Appendix (ref) for details) and run the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm fletcher2013practical to maximize the likelihood. In our experiments, the non-convexity of the likelihood never proved a practical problem. When we find the global minimum, this estimator is optimal and dominates both the backdoor and frontdoor estimators.
Synthetic Data \quad To show that the empirical variance of the various estimators is close to the theoretical variance (Table (ref)), we randomly initialize parameters and for each instance, we compute the Mean and Standard Deviation of Absolute Percentage Error of theoretical variance as a predictor of empirical variance (see Appendix (ref)). Next, we compare the estimators under different settings of the model parameters. Unless stated otherwise, the model parameter values we use for experiments are $a = 10, b = 4, c = 5, d = 5, \sigma^2_{u_w} = 1, \sigma^2_{u_x} = 1, \sigma^2_{u_m} = 1, \sigma^2_{u_y} = 1$.
The quantity of interest is the causal effect $ac = 50$. For Figure (ref), we set $\{\sigma^2_{u_x} = 0.05, \sigma^2_{u_m} = 0.05\}$, which makes the backdoor estimator better as predicted by Eq. (ref). For Figure (ref), we set $\{\sigma^2_{u_w} = 2, \sigma^2_{u_x} = 0.01, \sigma^2_{u_m} = 0.1\}$ which makes the frontdoor estimator better as predicted by Eq. (ref). The plots in Figure (ref) and (ref) corroborate these predictions at different sample sizes. Furthermore, the optimal combined estimator always outperforms both the backdoor and frontdoor estimators.
Next, we evaluate the procedure for combining datasets described in Section (ref), generating two datasets with equal numbers of samples. In the first, only $\{X, Y, W\}$ are observed. In the second, only $\{X, Y, M\}$ are observed. We set $\{\sigma^2_{u_x} = 0.05, \sigma^2_{u_m} = 0.05\}$, which makes the backdoor estimator better (Figure (ref)), and then set $\{\sigma^2_{u_w} = 2, \sigma^2_{u_x} = 0.01, \sigma^2_{u_m} = 0.1\}$, which makes the frondoor estimator better (Figure (ref)). The plots show that the combined estimator has lower MSE than either for various sample sizes (Figure (ref)), supporting our theoretical claims.
IHDP Dataset \quad hill2011bayesian constructed a dataset from the Infant Health and Development Program (IHDP). This semi-synthetic dataset has been used for benchmarking in causal inference shi2019adapting, shalit2017estimating. The dataset is is based on a randomized experiment to measure the effect of home visits from a specialist on future cognitive test scores of children. The treatment is binary and the covariates contain both continuous and categorical variables representing measurements on the child and their mother. We use samples from the NPCI package Dorie2016. We converted the randomized data into an observational study by removing a biased subset of the treated group. This set contains $747$ samples with $25$ covariates.
We use the covariates and the treatment assignment from the real study. We use a procedure similar to hill2011bayesian to simulate the mediator and the outcome. The mediator $M$ takes the form $M \sim \mathcal{N}(cX, \sigma^2_{u_m})$, where $X$ is the treatment. The response $Y$ takes the form $Y \sim \mathcal{N}(aM + w^\top \mathbf{b}, 1)$ where $w$ is the vector of standardized (zero mean and unit variance) covariates and values in the vector $\mathbf{b}$ are randomly sampled (0, 1, 2, 3, 4) with probabilities (0.5, 0.2, 0.15, 0.1, 0.05). The ground truth causal effect is $c\times a$.
We evaluate our estimators and the four IF estimators: IF-Fulcher (IF-Fulc), IF-Restricted (IF-Restr), IF-Frontdoor (IF-FD), and IF-Rotnitzky (IF-Rotz) (Section (ref)). We test the estimators on two settings of the model parameters (Table (ref), the Complete dataset setting). The MSE values are computed across 1000 instantiations of the dataset created by simulating the mediators and outcomes. We first evaluate the estimators on the complete dataset of 747 samples. We see that for Setting $1$ (S1): $a=10, c=5, \sigma_{u_m} = 1$, the backdoor estimator dominates the frontdoor estimator whereas for Setting $2$ (S2): $a=10,c=1,\sigma_{u_m} = 2$, the frontdoor estimator is better. In both cases, the combined estimator (Section (ref)) outperforms both estimators. Furthermore, we see that IF-Restricted outperforms IF-Frontdoor, showing the value of leveraging the covariates. Moreover, IF-Restricted also outperforms IF-Fulcher, suggesting that incorporating the additional model restrictions improves performance. Next, we randomly split the data into two sets, one with the confounder observed and the other with the mediator observed, finding that the estimator that combines the datasets (Section (ref)) outperforms the frontdoor and backdoor estimator (Table (ref), the Partial dataset setting). We compute the MSE over 1000 realizations of the dataset.
National JTPA Study \quad The National Job Training Partnership Act (JTPA) Study evaluates the effect of a job training program on future earnings. We use the dataset from glynnData2019. The binary treatment $X$ represents if a participant signed up for the program. The outcome $Y$ represents future earnings. The collected covariates (like race, study location, age) are the confounders $W$. The covariates contain both categorical and continuous variables. There was non-compliance among the treated units. The binary mediator $M$ represents compliance, that is, whether the participant make use of JTPA services after signing up. The study contained a randomized component which allowed us to compute the ground truth treatment effect, which was $862.74$. glynn2018front showed that the backdoor estimator has high bias, suggesting that there was unmeasured confounding, so we omit the backdoor estimator in our results. They also justify the assumptions required for the frontdoor estimator and show that it works well for this study.
A comparison of the frontdoor estimator, the combined estimator (Section (ref)), IF-Restricted, IF-Frontdoor and IF-Fulcher (Section (ref)) is shown in Table (ref) (the “C” data setting). For IF-Restricted, we only use the $f(M|X,W)=f(M|X)$ restriction and do not use the $\mathbb{E}[Y|M,W,X]=\mathbb{E}[Y|M,W]$ restriction since it is not valid. We compute the variance and MSE using 1000 bootstrap iterations. The combined estimator has lower variance and MSE than the frontdoor estimator. IF-Restricted outperforms IF-Frontdoor, reinforcing the utility of combined estimators. Furthermore, IF-Restricted outperforms IF-Fulcher, showing that using model restrictions is valuable. Next, we evaluate our procedure for the partially-observed setting (Section (ref)). We compute variance and MSE across $1000$ bootstrap iterations. At each iteration, we randomly split our dataset into two datasets of equal size, one with revealed confounders, one with revealed mediators. The combined estimator does not outperform the frontdoor estimator (Table (ref), the “P” data setting). This is expected since the backdoor adjustment works poorly and the revealed-confounder data is unlikely to help. Despite this, the combined estimator does not suffer too badly and has low bias despite the required assumptions for one of the identification strategies not holding.
In this paper, we studied over-identified graphs with confounders and mediators, showing that the two identification strategies can lead to estimators with arbitrarily different variances. We show that having access to both confounders and mediators (either simultaneously or in separate datasets) can give (unbounded) performance gains. We also show that our results qualitatively apply to general non-linear settings.
\paragraph{Future Work} We see several promising lines for future work, including (i) extensions to more general graphs; (ii) online data collection subject to some cost structure over the observations; and (iii) leveraging overidentification to mitigate errors due to measurement and confounding. Our experiments show the applicability of our methods in the frontdoor-backdoor graph, with combined estimators yielding gains in both linear and non-linear settings. We expect these insights to extend to other over-identified settings (e.g. graphs with multiple instrumental variables, multiple confounders, etc.) and we hope next to extend the results to more general over-identified causal graphs. Additionally, we plan to analyze the online data collection setting. Here, subject to budget constraints, a practitioner must choose which variables to observe at each time step. This direction seems especially important in medical applications (where each test may be costly) and survey studies (with a cap on the number of questions). Our current results suggest that the optimal strategy must depend on the model parameters. At each step, the revealed data will improve our estimates of the model parameters, in turn impacting what we collect in the future.
One potential limitation of the method is that situations where there exist multiple valid identification formulas may be uncommon in practice, when finding a single source of identification can already be difficult. However, we believe that in reality, many identification approaches are often available, but members of the community find flaws in each of the proposed estimators. In these cases, with multiple imperfect estimators of the same causal effect, we believe that overidentification might be leveraged to create robust combined estimators.