The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
14,302 characters
a framework for generalization and transportation of causal estimates under covariate shift
\title[Framework for Causal Bridging]{a framework for generalization and transportation of causal estimates under covariate shift}
\author{Apoorva Lal}
\email{{\href{[email removed]}{[email removed]}}}
\author{Wenjing Zheng}
\email{{[email removed]}}
\author{Simon Ejdemyr}
\email{{[email removed]}}
\keywords{experimentation, generalization, transportation, bridging}
\date{\today}
\maketitle
Randomized experiments are an excellent tool for estimating
\emph{internally valid} causal effects with the sample at hand, but
their \emph{external} validity is frequently questioned. While
classical results on the estimation of Population Average Treatment
Effects (PATE) implicitly assume random selection into experiments,
this is typically far from true in many medical, social-scientific,
and industry experiments. When the experimental sample is different
from the target sample along observable or unobservable dimensions
(termed \emph{covariate shift} in the causal learning literature),
experimental estimates may be of limited use for policy decisions. We
cast this as a sample selection problem and propose methods to
re-weight the doubly-robust scores from experimental subjects to
estimate treatment effects in the overall sample (=:
\emph{generalization}) or in an alternate target sample (=:
\emph{transportation}). We implement these estimators in the
open-source package
\href{https://github.com/Netflix-Skunkworks/causalTransportR}{\texttt{causalTransportR}}\footnote{Available
at
\href{https://github.com/Netflix-Skunkworks/causalTransportR}{https://github.com/Netflix-Skunkworks/causalTransportR}}
and illustrate its performance in a simulation study and discuss
diagnostics to evaluate its performance.
\vspace{-6mm}
\section*{Methods}
\vspace{-6mm}
We observe $n$ iid copies of $(\vee{X}_i, S_i, S_i A_i, S_i Y_i)_{i =
1}^n$, where covariates $\vee{X}_i \in \ensuremath{\mathbb{R}}^p$, treatment $A_i \in
\mathcal{A} := \left\{ 0, \dots, K \right\}$, outcome $Y_i \in \ensuremath{\mathbb{R}}$, and selection
indicator $S_i \in \left\{ 0, 1 \right\}$ is a function of pre-treatment
variables and is not affected by treatment. In other words, we observe
$(\vee{X}_i, A_i, Y_i)_{i = 1}^{N_1}$ for observations with $S_i = 1$
(henceforth the {\it study} sample $\mathcal{S}_1$), and only
$(\vee{X}_i)_{i = N_1 + 1}^N$ for observations with $S_i = 0$
(henceforth the {\it external} sample $\mathcal{S}_0$). The {\it overall
} sample is $\mathcal{S} := \mathcal{S}_1 \cup \mathcal{S}_0$.
\vspace{-3mm}
\@startsection{paragraph}{4}
\z@\z@{-\fontdimen2\font}
{\normalfont\bfseries}*{Estimands}
We write counterfactual means as $\phi = \mathbb{E}\left[Y^{a, S = 1}\right]$ for generalizability and $\mathbb{E}\left[Y^a | S = 0\right]$ for transportability, and contrasts between such counterfactual means under any two treatment levels $a, a'$ represent the average treatment effects (ATE). `Standard' estimation of effects in the study sample under unconfoundedness is a well-studied and largely resolved problem (see \parencite{Kennedy2022-jj} for a review). We study the generalization and transportation problems in the present paper. To this end, we make the following assumptions:
\begin{enumerate}[noitemsep,topsep=0pt]
\item Consistency / SUTVA : $Y_i = \mathds{1}_{A_i = a} Y^a_i$
\item Ignorability of Treatment: $Y^0, \dots, Y^a \protect\mathpalette{\protect\independenT}{\perp} A | \ve{X = x}, S = 1$
\item Overlap
\begin{enumerate}
\item Treatment overlap: $0 < \mathbf{Pr}\left(A = a | \vee{X = x} , S = 1\right) < 1$
\item Selection overlap: $0 < \mathbf{Pr}\left(S = 1 | \vee{X = x}\right) < 1$
\end{enumerate}
\item Selection
\begin{enumerate}
\item $Y^0, \dots, Y^a \protect\mathpalette{\protect\independenT}{\perp} S | \vee{X = x}$ Ignorability of Selection.
\item $\mathbb{E}\left[Y|A, \vee{X}, S = 1\right] = \mathbb{E}\left[Y|A, \vee{X}, S = 0\right]$. The outcome model is stable across S strata.
\end{enumerate}
\end{enumerate}
Under assumptions 1,2,3 and 4a, causal quantities of interest in the overall sample are identified \parencite{Bia2020-lu,Dahabreh2019-cr}, while under 1,2,3, and 4b, causal quantities of interest are identified in the target sample \parencite{Dahabreh2020-ys}. While prior work focused on binary treatments, we establish identification and estimation for counterfactual means and causal contrasts for multiple discrete treatments, which is the norm at Netflix and other industry settings.
\@startsection{paragraph}{4}
\z@\z@{-\fontdimen2\font}
{\normalfont\bfseries}*{Estimators}
Our preferred estimators are efficient influence function (EIF) based that take the form of sample averages $\widehat{\psi} = \frac{1}{n} \ensuremath{\sum_{i=1}^n} \varphi(W_i)$ where $W_i = (A_i, \vee{X}_i, Y_i, S_i)$. The influence function obeys $n^{1/2} (\widehat{\psi} - \psi) = n^{-1/2} \ensuremath{\sum_{i=1}^n} \varphi(W_i) + o_p(1)$. This form characterizes Regular and Asymptotically Linear (RAL) estimators and allows us to construct valid confidence intervals using the sample variance of the influence function. These estimators rely on the estimation of three nuisance functions: (1) Outcome model $\mu^a(\vee{x}) = \mathbb{E}\left[Y | A = a , \vee{X = x}\right]$, (2) Treatment Propensity score $\pi^a(\vee{x}) = \mathbf{Pr}\left(A = a | \vee{X = x} , S = 1\right)$, and (3) Selection propensity score $\rho(x) = \mathbf{Pr}\left(S = 1 | \vee{X = x}\right)$. Their sample analogues $\widehat{\alpha}$ are fit using machine learning estimators with cross-fitting and is implemented in the package with regularized regressions and generalized random forests.
The estimators under consideration take on one of three forms outlined in table~\ref{tab:table-estimators}. \textbf{Outcome Modeling (OM)} is a pure transfer-learning approach that involves fitting conditional response surfaces $\mathbb{E}\left[Y | A = a, S = 1\right]$ over the observations with nonmissing $Y$ and extrapolating these over the relevant samples. \textbf{Inverse Selection Weighting (ISW)} involves modeling the selection probability into the source sample with covariates, and reweighting observations to mimic the target samples.
\textbf{Augmented ISW (AISW)} combines the ISW and OM approaches by augmenting the outcome model with an weighted average of residuals ($Y - \mu(\cdot)$) and possesses double-robustness properties (from outcome of both propensity models) analogous to the classical Augmented IPW estimator \parencite{robins1994estimation}. Both ISW and AISW can be stabilized using a Hajek normalization term equal to the sum of weights in each treatment level $a$.
If only summary statistics are available for the target sample, AISW is infeasible since it requires individual level covariates for all observations to construct weights. In such cases, a `calibration' approach based on solving for balancing weights that ensure balance between two population is feasible and has appealing properties in finite samples , and is also implemented using entropy loss \parencite{Hainmueller2012-rd} in the package.
\begin{table}[H]
\centering
\caption{\label{tab:table-estimators} Estimators. Difference between marginal means $\widehat{\psi}_a - \widehat{\psi}_{a'}$ yields causal contrasts $\tau(a, a')$. Standard errors are computed as $\sqrt{\hat{\sigma}^2/n}$ where $\widehat{\sigma^2}$ is the sample variance of the influence function of interest (marginal mean or causal contrast) for AISW, or via the nonparametric or bayesian bootstrap for other estimators}
\centerline{
\begin{tabular}{|l | L | L |}
\hline
&
\makecell{
\textbf{Generalization} \\
\sum_x \mathbb{E}\left[Y | A = a, S = 1, \vee{X}\right] P(\vee{X})
}
&
\makecell{
\textbf{Transportation} \\
\sum_x \mathbb{E}\left[Y | A=a, S = 1,\vee{X}\right] P(\vee{X}| S = 0)
}
\\
\hline
OM &
\frac{1}{n} \sum_{i} \widehat{\mu}^{a}(\vee{X}_i) &
\frac{1}{\mathop{\rm abs}{\mathcal{S}_0}} \sum_i (1 - S_i) \widehat{\mu}^{a}(\vee{X}_i) \\
ISW &
\frac{1}{n} \sum_{i} \frac{S_i}{\widehat{\rho}(\vee{X}_i)}
\frac{\mathds{1}_{A = a}}{\widehat{\pi}^a(\vee{X}_i)} Y_i &
\frac{1}{n}
\sum_i
\frac{1}{\widehat{\mathbb{E}}[S_i = 0]}
\frac{S_i(1 - \widehat{\rho}(\vee{X}_i))} { \widehat{\rho}(\vee{X}_i)}
\frac{\mathds{1}_{A = a}}{\widehat{\pi}^a(\vee{X}_i)} Y_i \\
AISW &
\frac{1}{n} \sum_{i}
\widehat{\mu}^{a}(\vee{X}_i) +
\frac{S_i}{\widehat{\rho}(\vee{X}_i)}
\frac{\mathds{1}_{A = a}}{\widehat{\pi}^a(\vee{X}_i)} (Y_i - \widehat{\mu}^a(\vee{X}_i) ) &
\frac{1}{n} \sum_{i}
\frac{1}{\widehat{\mathbb{E}}[S_i = 0]}
\left(
(1-S_i) \widehat{\mu}^{a}(\vee{X}_i) +
\frac{S_i (1 - \widehat{\rho}(\vee{X}_i))}{\widehat{\rho}(\vee{X}_i)}
\frac{\mathds{1}_{A = a}}{\widehat{\pi}^a(\vee{X}_i)} (Y_i - \widehat{\mu}^a(\vee{X}_i) )
\right )\\
\hline
\end{tabular}
}
\end{table}
\section*{Simulation Study}
We study the generalization estimators' performance in a simulation study, where we simulate four scenarios where covariates $x_1, \dots, x_{10} \sim \mathsf{U}[-1, 1]$ and treatment is randomly assigned with probability $0.5$, and the true (selection / outcome) models are (linear / nonlinear), and nuisance functions are estimated using regularized
linear regressions with $\lambda$ set to minimise CV-MSE. The selection model dictates how the study sample is selected from the target population, and whenever this is a function of covariates, the experimental estimate of the average treatment effect (SATE) is biased for the treatment effect in the target population (PATE).
We report the performance of the above estimators in figure~\ref{fig:simfig}, which displays the RMSE, Bias, Coverage rate, and runtime across 500 replications. We find that reweighting estimators (red, blue, and purple) consistently outperform `naive' SATE estimates (green). Consistent with the analogous results in the unconfoundedness literature, we find that among the generalization estimators, the augmented inverse selection weighting estimator (red) performs best along MSE, bias, and variance dimensions.
\begin{figure}[htbp]
\caption{Simulation Results}
\label{fig:simfig}
\subfloat[RMSE]{
\includegraphics[width=.5\textwidth]{sim_RMSE.pdf}
}
\subfloat[Bias]{
\includegraphics[width=.5\linewidth]{sim_MAD.pdf}
}
\\
\subfloat[95\% CI coverage]{
\includegraphics[width=.5\linewidth]{sim_COV.pdf}
}
\subfloat[Runtime]{
\includegraphics[width=.5\linewidth]{sim_RUNTIME.pdf}
}
\end{figure}
\section*{Discussion}
In this work, we provide a concise multi-treatment framework for
causal generalization and transportation, and provide a performant
computational implementation for it. In current practice,
practitioners often informally perform `naive' extrapolation of the
Sample Average Treatment Effect (SATE) to the Target Average Treatment
Effect (TATE), which may result in erroneous conclusions arising from
three distinct sources of bias generated by differences between study
and external samples : (1) unequal distribution of effect modifier
covariates in the two samples, (2) Lack of covariate overlap between
the two samples, and (3) differences between the effect modification
functions between the two samples\footnote{We provide a fuller
derivation of these in appendix~\ref{appdx:biasDecomp}}.
Naive extrapolation can be improved upon by using the framework
presented in the present paper, which shows that causal generalization
and transportation involves (1) estimating strata-specific Conditional
Average Treatment Effects (CATEs) $\tau_s(\vee{X})$, (2) asserting
outcome model stability wherein the heterogeneity function
$\tau(\vee{X})$ is stable across the study and external sample, and
(3) reweighting CATEs to match the covariate distribution $p(\vee{X})$
in the target population. The feasibility of each of these steps may
be problem-specific, and practitioners are advised to carefully
consider potential problems in each step in their particular
application.
While an enormous literature has emerged on CATE estimation (see
\parencite{Knaus2020-ah} for a review), evaluating the quality of the
resultant estimates remains a challenging problem. Omnibus tests for
systematic treatment effect heterogeneity
\parencite{Ding2019-nr}(implemented as \texttt{dfmTest} in the
package) and \parencite{chernozhukov2020generic} are strongly
recommended prior to the use of generalization estimators. CATE
estimation for settings with small treatment effects as is typical in
industry settings is a growing area of research
\parencite{athey2021semiparametric} and progress on this problem can
readily be integrated to improve upon effect transportation.
Outcome model stability across the study and external samples is
inherently untestable, and must be justified from substantive
knowledge of the study and target population, as well as the nature of
the experimental manipulation. For example, a video compression
treatment might have systematic and stable treatment effects across
subpopulations, while a recommendation algorithm may not because of
preference heterogeneity that isn't adequately captured by covariates.
Sensitivity analyses in the vein of
\parencite{chernozhukov2022long,dorn2022sharp,Nie2021-ku} that assess
the magnitude of violations of outcome model stability that overturn
transportation conclusions are a fruitful avenue for future research.
Finally, the effectiveness of selection weights in reducing the
imbalance between the study and external populations can be evaluated
using the suite of tools developed for propensity score weights.
Plotting standardized mean differences between the source and target
samples is a reasonable first check for whether the weights are
effective at reducing imbalance, and is implemented as the plot method
for the model object in the package. These figures are also also
intended to help choose between `indirect' balancing via propensity
score modelling versus calibration approaches that target balance
directly.
\setcounter{table}{0}
\setcounter{figure}{0}
\printbibliography