The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
22,864 characters
Does Regression Produce Representative Causal Rankings?
\maketitle
\begin{abstract}
We examine the challenges in ranking multiple treatments
based on their estimated effects when using linear regression or its
popular double-machine-learning variant, the Partially Linear
Model (PLM), in the presence of treatment effect heterogeneity. We
demonstrate by example that overlap-weighting performed by linear
models like PLM can produce Weighted Average Treatment Effects (WATE)
that have rankings that are inconsistent with the rankings of the
underlying Average Treatment Effects (ATE). We define this as ranking
reversals and derive a necessary and sufficient condition for ranking
reversals under the PLM. We conclude with several simulation studies
conditions under which ranking reversals occur.
\end{abstract}
\@startsection{section}{1}
\z@{.7\linespacing\@plus\linespacing}{.5\linespacing}
{\normalfont\Large\bfseries\centering }{Introduction}
In both the public and private sector, ranking treatments based on
their causal effects is crucial for decision-making. In commercial
applications, it is common to rank user actions by estimating their
effect on a target metric, and subsequently seeking to encourage
actions with large estimated effects, which are deemed `high value'.
An increasingly popular approach is to use Partially Linear Models
(PLM) to flexibly condition on a large set of confounders as part of
estimating causal effects of treatments while relaxing the stringent
form assumptions
\parencite{Chernozhukov2018-fl}. This estimator is rooted in the
seminal Frisch-Waugh-Lovell theorem and is extremely popular in
practice, and is viewed as \emph{the} Double Machine Learning (DML)
estimator by applied users\footnote{This is not strictly correct,
since DML is in fact a recipe for constructing Neyman-orthogonal
estimators for a wide variety of causal and structural parameters.
However, due to the prevalence of conditional-ignorability-based
identification assumptions and the popularity of linear regression,
the PLM has become synonymous with DML. \textcite{Chernozhukov2022-se}
study Neyman-orthogonal estimators for a wide variety of causal and
structural parameters.}.
However, under treatment effect heterogeneity, it is well known from
that linear regression performs overlap-weighting. As a result, it is is biased for the Average Treatment Effect (ATE), but instead estimates a
conditional-variance Weighted average of treatment effects (WATE).
So, when unbiased estimation of treatment effects is the goal, practitioners opt for direct estimation methods such as IPW (Inverse Propensity Weighting) or its Augmented variety (AIPW), or regression imputation / g-modelling. However, in many cases, practioners seek to rank treatment effects instead, and performance of common estimators for ranking purposes is less well-understood. We first construct an example with two treatments where the ranking of Weighted Average Treatment Effects (WATEs) produced by the PLM is the opposite of the true ranking of underlying Average treatment Effects (ATEs), which we formalize as a `ranking reversal' property that is undesirable for downstream decision-making. This implies that decision-makers that seek to rank treatments based on the treatment effects may therefore form incorrect rankings if they use PLM coefficients to form these rankings.
We then derive a decomposition relating the WATE and ATE, which gives rise to a necessary and sufficient condition for ranking
reversals, and provide economic intuition for it. We
find that ranking reversals require substantial treatment effect
heterogeneity and covariances between regression weights and treatment
effects to be of opposite signs across the treatments being ranked. We conclude with an array of simulation designs that mimic realistic DGPs that comport with our theoretical findings about the likelihood of rank reversals under different heterogeneity patterns.
\@startsection{section}{1}
\z@{.7\linespacing\@plus\linespacing}{.5\linespacing}
{\normalfont\Large\bfseries\centering }{A Simple Numerical Example}
Consider a binary covariate $x \sim \text{Bernoulli}(0.5)$ and two
binary treatments $W_1, W_2$ with the following propensity scores:
\begin{table}[h]
\centering
\begin{tabular}{c|cc}
& $W_1 = 0$ & $W_2 = 1$ \\
\hline
$X = 0$ & 0.01 & 0.5 \\
$X = 1$ & 0.5 & 0.01
\end{tabular}
\end{table}
The true treatment effects are:
\begin{table}[h]
\centering
\begin{tabular}{c|cc}
& $\tau_1$ & $\tau_2$ \\
\hline
$X = 0$ & -3 & -2 \\
$X = 1$ & 3 & 3 \\
\hline
ATE & 0 & 0.5
\end{tabular}
\end{table}
With linear propensity scores, we can plug the above two sets of
numbers into \ref{eqn:watedecomp} and \ref{eqn:cvwt} to construct PLM
regression coefficients
\begin{align*}
\tilde{\tau}_1 & = \frac{-3 \cdot 0.01 \cdot 0.99 + 3 \cdot 0.5 \cdot
0.5}{ 0.01 \cdot 0.99 + 0.5 \cdot 0.5 } = 2.7714 \\
\tilde{\tau}_2 & = \frac{-2 \cdot 0.5 \cdot 0.5 + 3 \cdot 0.01 \cdot 0.99}{0.01 \cdot 0.99 + 0.5 \cdot 0.5} = -1.8095
\end{align*}
\begin{figure}
\centering
\includegraphics[width=0.5\linewidth]{figs/example_fig.png}
\caption{Strata-level and overall true effects, and estimated effects from PLM and AIPW}
\label{fig:basicsim}
\end{figure}
In contrast, IPW or AIPW correctly recovers the ATEs. These results
demonstrate that PLM leads to incorrect ranking of treatments, while
AIPW provides the correct ranking based on ATEs. This is an admittedly
contrived example; in the next section, we formalize the properties of
this example that yielded the poor ranking performance of PLM.
\@startsection{section}{1}
\z@{.7\linespacing\@plus\linespacing}{.5\linespacing}
{\normalfont\Large\bfseries\centering }{Methodology}
We consider a setting with multiple binary treatments where for each
unit $i$, we observe an outcome $Y_i \in \mathbb{R}$, treatment
assignment $W_i \in \{1,...,K\}$ indicating which of $K$ treatments
was received (with $W_i = 0$ denoting control), and pre-treatment
covariates $\mathbf{X}_i \in \mathbb{R}^d$. Our goal is to rank
treatments according to their average treatment effects relative to
control, defined as $\tau_j \vcentcolon= \mathbb{E}[Y_i(j) - Y_i(0)]$ for
each treatment $j$. We seek to form a poset ordering $(\leq, \ve{\tau}_j)$, and want to estimate $\ve{\tau}_j$s using standard techniques under selection-on-observables assumptions [Unconfoundedness and Overlap \parencite{Imbens2004-ir}].
\begin{defi}[Partially Linear Model]
For each treatment $W_i$, the PLM approach models the outcome as:
\begin{equation}
Y_i = \tau W_i + g(\mathbf{X}_i) + \varepsilon_i
\end{equation}
Estimation typically involves a residuals-on-residuals regression:
\begin{equation}
Y_i - \mathbb{E}[Y_i | \mathbf{X}_i] = \widehat{\tau} (W_i - \mathbb{E}[W_i | \mathbf{X}_i])
+ \eta_i \label{eqn:ror}
\end{equation}
Where the conditional expectations $\mathbb{E}\left[Y \mid \mathbf{X}\right] =\vcentcolon \mu(\mathbf{X})$ and $\mathbb{E}\left[W
\mid \mathbf{X}\right] =\vcentcolon p(\mathbf{X})$ are estimated using flexible non-parametric regression
methods and cross-fit to avoid over-fitting to satisfy the technical
requirements in Chernozhukov et al (2018).
\end{defi}
\begin{thm}[Conditional Variance weighting property of linear
regression]\label{thm:cvwt}
Under treatment effect heterogeneity, PLM estimates a weighted average treatment effect:
$$
\widehat{\tau} = \frac{\mathbb{E}\left[\omega_i \tau_i\right]}{\mathbb{E}\left[\omega_i\right]}
$$
where $\omega_i \vcentcolon= (W_i - \mathbb{E}\left[W_i \mid X_i\right])^2$
\parencite{Angrist1998-ok,Angrist1999-sp,Aronow2016-nn}.Defining normalized weights
$\gamma_i = \omega_i/\mathbb{E}\left[\omega_i\right]$ and working
(without loss of generality) with discrete $\mathbf{X}$ lets us rewrite
the above as
\begin{equation}
\text{plim } \hat{\tau} = \mathbb{E}[\gamma(\mathbf{X})\tau(\mathbf{X})] =\vcentcolon \text{WATE} \label{eqn:watedecomp}
\end{equation}
where $\gamma(\mathbf{X})$ are (normalized) weights that depend on the
propensity scores. The weights take the following form
\begin{equation}
\gamma(\mathbf{X}) = \frac{\mathbb{V}[D \mid \mathbf{X}]}{\mathbb{E}[\mathbb{V}[D \mid \mathbf{X}]]}
=
\frac{p(\mathbf{X}) (1-p(\mathbf{X}))}{\mathbb{E}[p(\mathbf{X})(1-p(\mathbf{X}))]}
\label{eqn:cvwt}
\end{equation}
where the second equality uses the fact that each treatment is binary
and substitutes in the expression for binomial variance. Proof in
\ref{sec:p1}.
\end{thm}
This means that in the presence of treatment effect heterogeneity
(i.e. $\tau(\mathbf{X})$ is not a constant function $=\tau$), the
probability limit of the regression coefficient is no longer the
Average Treatment Effect (ATE $\vcentcolon= \mathbb{E}[\tau(\mathbf{X})$) but is
instead the above Weighted Average Treatment Effect (WATE), with
weights $\gamma$ implicitly chosen by the regression specification.
These weights are largest for propensity scores close to 0.5, which
results in OLS performing `overlap-weighting' where it down-weights
strata with extreme propensity scores, and discards strata with no
overlap (with propensity scores equal to 0 or 1).
An interesting alternative but complementary decomposition is studied
by \textcite{SloczynskiUnknown-kg}, who shows that the regression
coefficient $\hat{\tau}$ can also be decomposed into the ATT (Average
Treatment Effect on the Treated) and ATU (Average Treatment Effect on
the Untreated), with weights that are inversely proportional to group
sizes. In other words, the larger the share of the treated group, the
lower weight it receives, and vice versa.
\subsection{Rank Reversal: definition and conditions}
With these weights in hand, we can define the property observed in the
previous section.
\begin{defi}[Rank Reversal]\label{defn:rrev}
For any two treatments $j$ and $k$, a
ranking reversal implies that we have $\text{ATE}_j > \text{ATE}_k$
but $\text{WATE}_j < \text{WATE}_k$. This occurs when
\begin{align}
\overbrace{\mathbb{E}\left[\tau_j(\mathbf{X})\right]}^{\text{ATE$_j$}} & >
\overbrace{\mathbb{E}\left[\tau_k(\mathbf{X})\right]}^{\text{ATE$_k$}} \label{eqn:atejgk} \\
\underbrace{\mathbb{E}\left[\gamma_j(\mathbf{X}) \tau_j(\mathbf{X})\right]}_{\text{WATE$_j$}} & < \underbrace{\mathbb{E}\left[\gamma_k(\mathbf{X})\tau_k(\mathbf{X})\right]}_{\text{WATE$_k$}} \label{eqn:wagejlk}
\end{align}
\end{defi}
We first derive an expression relating the ATE and WATE. For any
treatment $g$, we can decompose the WATE using the definition of
covariance ($\text{Cov}\left[a, b\right] = \mathbb{E}\left[ab\right] -
\mathbb{E}\left[a\right]\mathbb{E}\left[b\right]$)
$$\mathbb{E}[\gamma_g(\mathbf{X})\tau_g(\mathbf{X})] =
\mathbb{E}[\gamma_g(\mathbf{X})]\mathbb{E}[\tau_g(\mathbf{X})] +
\text{Cov}(\tau_g(\mathbf{X}), \gamma_g(\mathbf{X}))
$$
Note that by construction of regression weights $\gamma_g(\mathbf{X}) \vcentcolon= \mathbb{V}\left[W\mid\mathbf{X}\right]/\mathbb{E}\left[\mathbb{V}\left[W \mid \mathbf{X}\right]\right]$ have an expected value of 1. So, we arrive at the following decomposition
\begin{equation}
\underbrace{\mathbb{E}[\gamma_g(\mathbf{X})\tau_g(\mathbf{X})]}_{\text{WATE$_g$}}
=
\underbrace{\mathbb{E}[\tau_g(\mathbf{X})]}_{\text{ATE$_g$}} +
\text{Cov}(\tau_g(\mathbf{X}), \gamma_g(\mathbf{X})) \label{eqn:watedecomp}
\end{equation}
We provide three simple examples numerically illustrating the above decomposition with negative, zero, and positive covariance between the regression weights and treatment functions in figure~\ref{fig:treatment-effects-sim}.
\begin{figure}[t]
\centering
\begin{tikzpicture}[scale=1.2]
\tikzset{
every node/.style={font=\small},
estimate/.style={align=left, text width=3.5cm}
}
\foreach \i in {0,1,2} {
\begin{scope}[yshift=-4*\i cm]
\draw[gray!30] (0,0) grid[step=1] (4,2.5);
\draw[->,thick] (0,0) -- (4.2,0);
\draw[->,thick] (0,0) -- (0,2.7) ;
\foreach \x in {0,1,2,3,4} {
\draw (\x,2pt) -- (\x,-2pt) node[below] {\x/4};
}
\foreach \y in {0,0.65,1.3,2.5} {
\draw (2pt,\y) -- (-2pt,\y) node[left] {\y};
}
\end{scope}
}
\node[above] at (2,2.7) {Negative Covariance};
\node[estimate] at (6.5,1.3) {
WATE = 1.28 \\
= ATE (1.56) \\
+ Cov($\tau,\gamma$) (-0.29)
};
\node[above] at (2,-1.3) {Zero Covariance};
\node[estimate] at (6.5,-3) {
WATE = 1.00 \\
= ATE (1.00) \\
+ Cov($\tau,\gamma$) (0.00)
};
\node[above] at (2,-5.3) {Positive Covariance};
\node[estimate] at (6.5,-6.7) {
WATE = 0.93 \\
= ATE (0.64) \\
+ Cov($\tau,\gamma$) (0.29)
};
\draw[purple, thick] (0,2) -- (1,1.8) -- (2,0.2) -- (3,1.8) -- (4,2);
\draw[green!60!black, thick] (0,0.1) -- (1,0.15) -- (2,0.5) -- (3,0.85) -- (4,0.9);
\draw[orange, thick] (0,0.66) .. controls (1,0.93) and (2,1.83) .. (2,1.83) .. controls (3,0.93) and (4,0.66) .. (4,0.66);
\begin{scope}[yshift=-4cm]
\draw[purple, thick] (0,1) -- (4,1);
\draw[green!60!black, thick] (0,0.2) -- (1,0.35) -- (2,0.5) -- (3,0.65) -- (4,0.8);
\draw[orange, thick] (0,0.78) .. controls (1,1.11) and (2,1.22) .. (2,1.22) .. controls (3,1.11) and (4,0.78) .. (4,0.78);
\end{scope}
\begin{scope}[yshift=-8cm]
\draw[purple, thick] (0,0.2) -- (1,0.4) -- (2,2.0) -- (3,0.4) -- (4,0.2);
\draw[green!60!black, thick] (0,0.1) -- (1,0.15) -- (2,0.5) -- (3,0.85) -- (4,0.9);
\draw[orange, thick] (0,0.66) .. controls (1,0.93) and (2,1.82) .. (2,1.82) .. controls (3,0.93) and (4,0.66) .. (4,0.66);
\end{scope}
\begin{scope}[yshift=-9.0cm]
\draw[purple, thick] (1,0) -- (2,0) node[right] {Treatment Effect $\tau(x)$};
\draw[green!60!black, thick] (1,-0.4) -- (2,-0.4) node[right] {Propensity Score $p(x)$};
\draw[orange, thick] (1,-0.8) -- (2,-0.8) node[right] {Regression Weight $\gamma(x)$};
\end{scope}
\end{tikzpicture}
\caption{Treatment effect heterogeneity and regression weights under negative, zero, and positive scenarios for the $\text{Cov}\left[\tau_g(\mathbf{X}), \gamma_g(\mathbf{X}\right]$ term in \ref{eqn:watedecomp}. We have a single covariate $X$ with $5$ discrete strata with equal probability, and vary propensity scores and treatment effects according to the green and red functions specified above, which gives rise to the orange regression weights function. The right panel for each scenario shows how the weighted average treatment effect (WATE) estimated using regression decomposed into the true average treatment effect (ATE) and the covariance between treatment effects and regression weights.}
\label{fig:treatment-effects-sim}
\end{figure}
This decomposition immediately illustrates how rank reversals may arise in practice: when the second term in \ref{eqn:watedecomp} is large enough to offset the first, rank-reversals may occur.
\begin{prop}[Necessary and Sufficient Condition for Rank Reversal]
The following condition yields rank-reversal between treatments $j$ and $k$
\begin{equation}
\mathbb{E}\left[\tau_j(\mathbf{X}\right] + \text{Cov}\left[\tau_j(\mathbf{X}, \gamma_j(\mathbf{X}\right] < \mathbb{E}\left[\tau_k(\mathbf{X})\right] + \text{Cov}\left[\tau_k(X), \gamma_k(\mathbf{X}\right]
\label{eqn:cond}
\end{equation}
This is an immediate implication of the decomposition \ref{eqn:watedecomp}. Proof in \ref{appdx:directproof}. We also provide slightly more transparent sufficient conditions that parametrises the magnitudes of the two covariances in \ref{eqn:cond} in appdx~\ref{appdx:interpretablesuff}.
\end{prop}
\@startsection{paragraph}{4}
\z@\z@{-\fontdimen2\font}
{\sffamily \bfseries }{When can we expect PLM coefficients to yield correct
rankings?}
\begin{enumerate}
\item Constant treatment effects ($\tau(\mathbf{X}) = \tau$): Here, PLM,
IPW, and AIPW all estimate the same quantity. This is rare in practice
but serves as a useful benchmark.
\item Uncorrelated weights and effects
($\text{Cov}\left[\gamma(\mathbf{X}),\tau(\mathbf{X})\right] \approx 0$): This can happen
when:
\begin{enumerate}
\item Treatment assignment is relatively balanced ($p(\mathbf{X}) \approx 0.5$)
\item Treatment effects vary independently of variables that predict
treatment
\item As-good-as-random assignment: if units don't have the
opportunity to sort into treatment based on private information
about their own treatment effects $\tau(\mathbf{X})$, this covariance
will be more likely to be small.
\end{enumerate}
\item Uniform selection on gains: If units sort into treatments $j$ and
$k$ based on private information about their expected gains
$\tau_j(\mathbf{x}), \tau_k(\mathbf{x})$, the covariance
$\text{Cov}\left[\gamma_g(\mathbf{X}), \tau_g(\mathbf{X})\right]$ will be of the same sign
for $g \in \left\{ j,k \right\}$, which would not flip the rankings between the ATEs.
\item Similar propensity score distributions: When $p_j(\mathbf{X})$ and
$p_k(\mathbf{X})$ have similar distributions, $\gamma_j(\mathbf{X})$ and
$\gamma_k(\mathbf{X})$ will be similar, reducing the chance of rank
reversals. This suggests observational studies with very different
propensity scores across treatments are more prone to rank reversals
\item Moderate treatment effect heterogeneity: If heterogeneity in
treatment effects is modest, and this is known to agents, it is less
likely that they actively seek or avoid treatments (which pushes
$p_g(\mathbf{X})$ towards 0 or 1) based on this information, which
weakens the magnitude of $\text{Cov}\left[\tau(\cdot), \gamma(\cdot)\right]$, which
in turn makes it less likely that the covariances for different
treatments are of contrasting signs to result in rank reversals.
\end{enumerate}
A practical implication of the above is that when treatment effects
are suspected to be highly heterogeneous with units selecting into
treatments, researchers should prefer AIPW over PLM for ranking.
\begin{defi}[Augmented Inverse-Propensity Weighting (AIPW) Estimators]
An alternative to the PLM that does not fall prey to the ranking
reversal property is the AIPW estimator, which involves construction
of a `pseudo-outcome' $\Gamma_i^{j}$ that is the estimated potential outcome under treatment $j$ \parencite{Cattaneo2010-oc,Chernozhukov2018-fl}
\begin{align*}
\widehat{\Gamma}_i^{j} = \widehat{\mu}^{j,-k}(\mathbf{X}_i) +
&
\frac{\mathds{1}\{W_i = j\}}{\widehat{p}^{j,-k_i}(\mathbf{X}_i)} \left( Y_i - \widehat{\mu}^{j,-k_i}(\mathbf{X}_i) \right ) \\
\widehat{\tau}^{\text{AIPW}, a, b} = \frac{1}{n} \sum_{i}^n\left( \widehat{\Gamma}_i^{a} - \widehat{\Gamma}_i^{b} \right )
\end{align*}
where we first partition data by assigning each observation into $k_i
\in \mathsf{U} \left[ K \right ]$ folds, and cross-fit nuisance functions
$\widehat{\mu}(\cdot)$ (an outcome regression within treatment level $j$) and $\widehat{p}$ (a multi-class propensity score that models the probability of treatment level $j$) so that their predictions for unit $i$ are produced from models that were not trained on the $k_i-$th
fold. The above estimator is consistent for the ATE regardless of the
level of heterogeneity in the underlying treatment effect function
$\tau(\mathbf{X})$, which implies that it does not exhibit rank-reversal
properties, but conversely may have poor empirical performance in the
presence of extreme propensity scores.
\end{defi}
\@startsection{section}{1}
\z@{.7\linespacing\@plus\linespacing}{.5\linespacing}
{\normalfont\Large\bfseries\centering }{Numerical Experiments}
\subsection{Simulation Design}
We conduct Monte Carlo simulations to evaluate the performance of PLM
and AIPW estimators under various data generating processes (DGPs).
Each DGP is characterized by:
\begin{itemize}
\item A binary covariate $X \sim \text{Bernoulli}(0.5)$
\item Two binary treatments $W_1, W_2$ with stratum-specific
propensity scores $p_j(X)$
\item Heterogeneous treatment effects $\tau_j(X)$ for each treatment
\end{itemize}
We consider five scenarios that vary in their degree of effect
heterogeneity and propensity score distributions:
\begin{enumerate}
\item \textbf{Extreme Heterogeneity}: Large differences in treatment
effects across strata with extreme propensity scores
\item \textbf{Constant Effects}: Homogeneous effects within treatments
but different across treatments
\item \textbf{Uncorrelated}: Moderate heterogeneity with balanced
propensity scores
\item \textbf{Selection on Gains}: Treatment probability correlated
with treatment effects
\item \textbf{Balanced}: Equal propensity scores across strata with
heterogeneous effects
\end{enumerate}
For each scenario, we simulate 1,000 datasets with 10,000 observations
each. We evaluate the estimators on three dimensions:
\begin{itemize}
\item Distribution of point estimates
\item Bias relative to true effects
\item Proportion of correct rankings between treatments
\end{itemize}
We report figures for each of these settings in appendix
\ref{appdx:simfigs}. We find that with the exception of the extreme
heterogeneity setting that expands upon the example in section 2 (fig
\ref{fig:extremehet}), the rankings produced by the PLM are largely
consistent with the AIPW estimator, and conform with the sufficient
conditions derived in the previous section.
\@startsection{section}{1}
\z@{.7\linespacing\@plus\linespacing}{.5\linespacing}
{\normalfont\Large\bfseries\centering }{Conclusion}
This note highlights the importance of using appropriate methods for
estimating and ranking treatment effects in the presence of
heterogeneity. We show using an example that commonly used Partially
Linear Models can lead to biased estimates and incorrect rankings. We
then define a notion of ranking reversals and derive a decomposition relating the WATE and ATE, which gives rise to a necessary and sufficient
condition for ranking reversals in linear regression. Finally, we
propose interpretations for these conditions and recommend using
Augmented Inverse Probability Weighting estimator as a general
solution for ranking in the presence of substantial heterogeneity.
Our findings have important implications for decision-making in
various fields, including digital platforms and policy evaluation,
where accurate ranking of treatments is crucial. Future work could
explore the performance of these methods in more complex settings with
multiple treatments and high-dimensional covariates.
\printbibliography
\pagebreak