Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
14,392 characters · 3 sections · 10 citation commands
a framework for generalization and transportation of causal estimates under covariate shift
\email{{\href{[email removed]}{[email removed]}}}
\email{{[email removed]}}
\email{{[email removed]}}
Randomized experiments are an excellent tool for estimating internally valid causal effects with the sample at hand, but their external validity is frequently questioned. While classical results on the estimation of Population Average Treatment Effects (PATE) implicitly assume random selection into experiments, this is typically far from true in many medical, social-scientific, and industry experiments. When the experimental sample is different from the target sample along observable or unobservable dimensions (termed covariate shift in the causal learning literature), experimental estimates may be of limited use for policy decisions. We cast this as a sample selection problem and propose methods to re-weight the doubly-robust scores from experimental subjects to estimate treatment effects in the overall sample (=: generalization) or in an alternate target sample (=: transportation). We implement these estimators in the open-source package \href{https://github.com/Netflix-Skunkworks/causalTransportR}{causalTransportR}\footnote{Available at \href{https://github.com/Netflix-Skunkworks/causalTransportR}{https://github.com/Netflix-Skunkworks/causalTransportR}} and illustrate its performance in a simulation study and discuss diagnostics to evaluate its performance.
We observe $n$ iid copies of $(\vee{X}_i, S_i, S_i A_i, S_i Y_i)_{i = 1}^n$, where covariates $\vee{X}_i \in \ensuremath{\mathbb{R}}^p$, treatment $A_i \in \mathcal{A} := \left\{ 0, \dots, K \right\}$, outcome $Y_i \in \ensuremath{\mathbb{R}}$, and selection indicator $S_i \in \left\{ 0, 1 \right\}$ is a function of pre-treatment variables and is not affected by treatment. In other words, we observe $(\vee{X}_i, A_i, Y_i)_{i = 1}^{N_1}$ for observations with $S_i = 1$ (henceforth the {\it study} sample $\mathcal{S}_1$), and only $(\vee{X}_i)_{i = N_1 + 1}^N$ for observations with $S_i = 0$ (henceforth the {\it external} sample $\mathcal{S}_0$). The {\it overall } sample is $\mathcal{S} := \mathcal{S}_1 \cup \mathcal{S}_0$.
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} {\normalfont}*{Estimands}
We write counterfactual means as $\phi = \mathbb{E}\left[Y^{a, S = 1}\right]$ for generalizability and $\mathbb{E}\left[Y^a | S = 0\right]$ for transportability, and contrasts between such counterfactual means under any two treatment levels $a, a'$ represent the average treatment effects (ATE). `Standard' estimation of effects in the study sample under unconfoundedness is a well-studied and largely resolved problem (see Kennedy2022-jj for a review). We study the generalization and transportation problems in the present paper. To this end, we make the following assumptions:
Under assumptions 1,2,3 and 4a, causal quantities of interest in the overall sample are identified Bia2020-lu,Dahabreh2019-cr, while under 1,2,3, and 4b, causal quantities of interest are identified in the target sample Dahabreh2020-ys. While prior work focused on binary treatments, we establish identification and estimation for counterfactual means and causal contrasts for multiple discrete treatments, which is the norm at Netflix and other industry settings.
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} {\normalfont}*{Estimators}
Our preferred estimators are efficient influence function (EIF) based that take the form of sample averages $\widehat{\psi} = \frac{1}{n} \ensuremath{\sum_{i=1}^n} \varphi(W_i)$ where $W_i = (A_i, \vee{X}_i, Y_i, S_i)$. The influence function obeys $n^{1/2} (\widehat{\psi} - \psi) = n^{-1/2} \ensuremath{\sum_{i=1}^n} \varphi(W_i) + o_p(1)$. This form characterizes Regular and Asymptotically Linear (RAL) estimators and allows us to construct valid confidence intervals using the sample variance of the influence function. These estimators rely on the estimation of three nuisance functions: (1) Outcome model $\mu^a(\vee{x}) = \mathbb{E}\left[Y | A = a , \vee{X = x}\right]$, (2) Treatment Propensity score $\pi^a(\vee{x}) = \mathbf{Pr}\left(A = a | \vee{X = x} , S = 1\right)$, and (3) Selection propensity score $\rho(x) = \mathbf{Pr}\left(S = 1 | \vee{X = x}\right)$. Their sample analogues $\widehat{\alpha}$ are fit using machine learning estimators with cross-fitting and is implemented in the package with regularized regressions and generalized random forests.
The estimators under consideration take on one of three forms outlined in table (ref). Outcome Modeling (OM) is a pure transfer-learning approach that involves fitting conditional response surfaces $\mathbb{E}\left[Y | A = a, S = 1\right]$ over the observations with nonmissing $Y$ and extrapolating these over the relevant samples. Inverse Selection Weighting (ISW) involves modeling the selection probability into the source sample with covariates, and reweighting observations to mimic the target samples. Augmented ISW (AISW) combines the ISW and OM approaches by augmenting the outcome model with an weighted average of residuals ($Y - \mu(\cdot)$) and possesses double-robustness properties (from outcome of both propensity models) analogous to the classical Augmented IPW estimator robins1994estimation. Both ISW and AISW can be stabilized using a Hajek normalization term equal to the sum of weights in each treatment level $a$.
If only summary statistics are available for the target sample, AISW is infeasible since it requires individual level covariates for all observations to construct weights. In such cases, a `calibration' approach based on solving for balancing weights that ensure balance between two population is feasible and has appealing properties in finite samples , and is also implemented using entropy loss Hainmueller2012-rd in the package.
We study the generalization estimators' performance in a simulation study, where we simulate four scenarios where covariates $x_1, \dots, x_{10} \sim \mathsf{U}[-1, 1]$ and treatment is randomly assigned with probability $0.5$, and the true (selection / outcome) models are (linear / nonlinear), and nuisance functions are estimated using regularized linear regressions with $\lambda$ set to minimise CV-MSE. The selection model dictates how the study sample is selected from the target population, and whenever this is a function of covariates, the experimental estimate of the average treatment effect (SATE) is biased for the treatment effect in the target population (PATE).
We report the performance of the above estimators in figure (ref), which displays the RMSE, Bias, Coverage rate, and runtime across 500 replications. We find that reweighting estimators (red, blue, and purple) consistently outperform `naive' SATE estimates (green). Consistent with the analogous results in the unconfoundedness literature, we find that among the generalization estimators, the augmented inverse selection weighting estimator (red) performs best along MSE, bias, and variance dimensions.
In this work, we provide a concise multi-treatment framework for causal generalization and transportation, and provide a performant computational implementation for it. In current practice, practitioners often informally perform `naive' extrapolation of the Sample Average Treatment Effect (SATE) to the Target Average Treatment Effect (TATE), which may result in erroneous conclusions arising from three distinct sources of bias generated by differences between study and external samples : (1) unequal distribution of effect modifier covariates in the two samples, (2) Lack of covariate overlap between the two samples, and (3) differences between the effect modification functions between the two samples\footnote{We provide a fuller derivation of these in appendix (ref)}.
Naive extrapolation can be improved upon by using the framework presented in the present paper, which shows that causal generalization and transportation involves (1) estimating strata-specific Conditional Average Treatment Effects (CATEs) $\tau_s(\vee{X})$, (2) asserting outcome model stability wherein the heterogeneity function $\tau(\vee{X})$ is stable across the study and external sample, and (3) reweighting CATEs to match the covariate distribution $p(\vee{X})$ in the target population. The feasibility of each of these steps may be problem-specific, and practitioners are advised to carefully consider potential problems in each step in their particular application.
While an enormous literature has emerged on CATE estimation (see Knaus2020-ah for a review), evaluating the quality of the resultant estimates remains a challenging problem. Omnibus tests for systematic treatment effect heterogeneity Ding2019-nr(implemented as dfmTest in the package) and chernozhukov2020generic are strongly recommended prior to the use of generalization estimators. CATE estimation for settings with small treatment effects as is typical in industry settings is a growing area of research athey2021semiparametric and progress on this problem can readily be integrated to improve upon effect transportation.
Outcome model stability across the study and external samples is inherently untestable, and must be justified from substantive knowledge of the study and target population, as well as the nature of the experimental manipulation. For example, a video compression treatment might have systematic and stable treatment effects across subpopulations, while a recommendation algorithm may not because of preference heterogeneity that isn't adequately captured by covariates. Sensitivity analyses in the vein of chernozhukov2022long,dorn2022sharp,Nie2021-ku that assess the magnitude of violations of outcome model stability that overturn transportation conclusions are a fruitful avenue for future research.
Finally, the effectiveness of selection weights in reducing the imbalance between the study and external populations can be evaluated using the suite of tools developed for propensity score weights. Plotting standardized mean differences between the source and target samples is a reasonable first check for whether the weights are effective at reducing imbalance, and is implemented as the plot method for the model object in the package. These figures are also also intended to help choose between `indirect' balancing via propensity score modelling versus calibration approaches that target balance directly.
\setcounter{table}{0} \setcounter{figure}{0}
\printbibliography