Does Regression Produce Representative Causal Rankings?

Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.

22,864 characters · 2 sections · 6 citation commands

Rendered from LaTeX for readability, not typeset faithfully. Citation keys are highlighted; maths is left as source; figures, tables and equation environments are summarised rather than reproduced; unrecognised commands are greyed out so nothing is silently dropped. Email addresses are removed.

abstractWe examine the challenges in ranking multiple treatments based on their estimated effects when using linear regression or its popular double-machine-learning variant, the Partially Linear Model (PLM), in the presence of treatment effect heterogeneity. We demonstrate by example that overlap-weighting performed by linear models like PLM can produce Weighted Average Treatment Effects (WATE) that have rankings that are inconsistent with the rankings of the underlying Average Treatment Effects (ATE). We define this as ranking reversals and derive a necessary and sufficient condition for ranking reversals under the PLM. We conclude with several simulation studies conditions under which ranking reversals occur.

\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Introduction}

In both the public and private sector, ranking treatments based on their causal effects is crucial for decision-making. In commercial applications, it is common to rank user actions by estimating their effect on a target metric, and subsequently seeking to encourage actions with large estimated effects, which are deemed `high value'. An increasingly popular approach is to use Partially Linear Models (PLM) to flexibly condition on a large set of confounders as part of estimating causal effects of treatments while relaxing the stringent form assumptions Chernozhukov2018-fl. This estimator is rooted in the seminal Frisch-Waugh-Lovell theorem and is extremely popular in practice, and is viewed as the Double Machine Learning (DML) estimator by applied users\footnote{This is not strictly correct, since DML is in fact a recipe for constructing Neyman-orthogonal estimators for a wide variety of causal and structural parameters. However, due to the prevalence of conditional-ignorability-based identification assumptions and the popularity of linear regression, the PLM has become synonymous with DML. Chernozhukov2022-se study Neyman-orthogonal estimators for a wide variety of causal and structural parameters.}.

However, under treatment effect heterogeneity, it is well known from that linear regression performs overlap-weighting. As a result, it is is biased for the Average Treatment Effect (ATE), but instead estimates a conditional-variance Weighted average of treatment effects (WATE). So, when unbiased estimation of treatment effects is the goal, practitioners opt for direct estimation methods such as IPW (Inverse Propensity Weighting) or its Augmented variety (AIPW), or regression imputation / g-modelling. However, in many cases, practioners seek to rank treatment effects instead, and performance of common estimators for ranking purposes is less well-understood. We first construct an example with two treatments where the ranking of Weighted Average Treatment Effects (WATEs) produced by the PLM is the opposite of the true ranking of underlying Average treatment Effects (ATEs), which we formalize as a `ranking reversal' property that is undesirable for downstream decision-making. This implies that decision-makers that seek to rank treatments based on the treatment effects may therefore form incorrect rankings if they use PLM coefficients to form these rankings. We then derive a decomposition relating the WATE and ATE, which gives rise to a necessary and sufficient condition for ranking reversals, and provide economic intuition for it. We find that ranking reversals require substantial treatment effect heterogeneity and covariances between regression weights and treatment effects to be of opposite signs across the treatments being ranked. We conclude with an array of simulation designs that mimic realistic DGPs that comport with our theoretical findings about the likelihood of rank reversals under different heterogeneity patterns.

\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{A Simple Numerical Example}

Consider a binary covariate $x \sim \text{Bernoulli}(0.5)$ and two binary treatments $W_1, W_2$ with the following propensity scores:

table[table omitted — 142 chars of source]

The true treatment effects are:

table[table omitted — 156 chars of source]

With linear propensity scores, we can plug the above two sets of numbers into (ref) and (ref) to construct PLM regression coefficients

align*[align* omitted — 255 chars of source]

figure[figure omitted — 210 chars of source]

In contrast, IPW or AIPW correctly recovers the ATEs. These results demonstrate that PLM leads to incorrect ranking of treatments, while AIPW provides the correct ranking based on ATEs. This is an admittedly contrived example; in the next section, we formalize the properties of this example that yielded the poor ranking performance of PLM.

\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Methodology}

We consider a setting with multiple binary treatments where for each unit $i$, we observe an outcome $Y_i \in \mathbb{R}$, treatment assignment $W_i \in \{1,...,K\}$ indicating which of $K$ treatments was received (with $W_i = 0$ denoting control), and pre-treatment covariates $\mathbf{X}_i \in \mathbb{R}^d$. Our goal is to rank treatments according to their average treatment effects relative to control, defined as $\tau_j \vcentcolon= \mathbb{E}[Y_i(j) - Y_i(0)]$ for each treatment $j$. We seek to form a poset ordering $(\leq, \ve{\tau}_j)$, and want to estimate $\ve{\tau}_j$s using standard techniques under selection-on-observables assumptions [Unconfoundedness and Overlap Imbens2004-ir].

defi[Partially Linear Model] For each treatment $W_i$, the PLM approach models the outcome as: \begin{equation} Y_i = \tau W_i + g(\mathbf{X}_i) + \varepsilon_i \end{equation} Estimation typically involves a residuals-on-residuals regression: \begin{equation} Y_i - \mathbb{E}[Y_i | \mathbf{X}_i] = \widehat{\tau} (W_i - \mathbb{E}[W_i | \mathbf{X}_i]) + \eta_i \end{equation} Where the conditional expectations $\mathbb{E}\left[Y \mid \mathbf{X}\right] =\vcentcolon \mu(\mathbf{X})$ and $\mathbb{E}\left[W \mid \mathbf{X}\right] =\vcentcolon p(\mathbf{X})$ are estimated using flexible non-parametric regression methods and cross-fit to avoid over-fitting to satisfy the technical requirements in Chernozhukov et al (2018).

thm[Conditional Variance weighting property of linear regression] Under treatment effect heterogeneity, PLM estimates a weighted average treatment effect: $$ \widehat{\tau} = \frac{\mathbb{E}\left[\omega_i \tau_i\right]}{\mathbb{E}\left[\omega_i\right]} $$ where $\omega_i \vcentcolon= (W_i - \mathbb{E}\left[W_i \mid X_i\right])^2$ Angrist1998-ok,Angrist1999-sp,Aronow2016-nn.Defining normalized weights $\gamma_i = \omega_i/\mathbb{E}\left[\omega_i\right]$ and working (without loss of generality) with discrete $\mathbf{X}$ lets us rewrite the above as \begin{equation} plim \hat{\tau} = \mathbb{E}[\gamma(\mathbf{X})\tau(\mathbf{X})] =\vcentcolon WATE \end{equation} where $\gamma(\mathbf{X})$ are (normalized) weights that depend on the propensity scores. The weights take the following form \begin{equation} \gamma(\mathbf{X}) = \frac{\mathbb{V}[D \mid \mathbf{X}]}{\mathbb{E}[\mathbb{V}[D \mid \mathbf{X}]]} = \frac{p(\mathbf{X}) (1-p(\mathbf{X}))}{\mathbb{E}[p(\mathbf{X})(1-p(\mathbf{X}))]} \end{equation} where the second equality uses the fact that each treatment is binary and substitutes in the expression for binomial variance. Proof in (ref).

This means that in the presence of treatment effect heterogeneity (i.e. $\tau(\mathbf{X})$ is not a constant function $=\tau$), the probability limit of the regression coefficient is no longer the Average Treatment Effect (ATE $\vcentcolon= \mathbb{E}[\tau(\mathbf{X})$) but is instead the above Weighted Average Treatment Effect (WATE), with weights $\gamma$ implicitly chosen by the regression specification. These weights are largest for propensity scores close to 0.5, which results in OLS performing `overlap-weighting' where it down-weights strata with extreme propensity scores, and discards strata with no overlap (with propensity scores equal to 0 or 1).

An interesting alternative but complementary decomposition is studied by SloczynskiUnknown-kg, who shows that the regression coefficient $\hat{\tau}$ can also be decomposed into the ATT (Average Treatment Effect on the Treated) and ATU (Average Treatment Effect on the Untreated), with weights that are inversely proportional to group sizes. In other words, the larger the share of the treated group, the lower weight it receives, and vice versa.

Rank Reversal: definition and conditions

With these weights in hand, we can define the property observed in the previous section.

defi[Rank Reversal] For any two treatments $j$ and $k$, a ranking reversal implies that we have $\text{ATE}_j > \text{ATE}_k$ but $\text{WATE}_j < \text{WATE}_k$. This occurs when \begin{align} \overbrace{\mathbb{E}\left[\tau_j(\mathbf{X})\right]}^{ATE$_j$} & > \overbrace{\mathbb{E}\left[\tau_k(\mathbf{X})\right]}^{ATE$_k$} \\ \underbrace{\mathbb{E}\left[\gamma_j(\mathbf{X}) \tau_j(\mathbf{X})\right]}_{WATE$_j$} & < \underbrace{\mathbb{E}\left[\gamma_k(\mathbf{X})\tau_k(\mathbf{X})\right]}_{WATE$_k$} \end{align}

We first derive an expression relating the ATE and WATE. For any treatment $g$, we can decompose the WATE using the definition of covariance ($\text{Cov}\left[a, b\right] = \mathbb{E}\left[ab\right] - \mathbb{E}\left[a\right]\mathbb{E}\left[b\right]$)

$$\mathbb{E}[\gamma_g(\mathbf{X})\tau_g(\mathbf{X})] = \mathbb{E}[\gamma_g(\mathbf{X})]\mathbb{E}[\tau_g(\mathbf{X})] + \text{Cov}(\tau_g(\mathbf{X}), \gamma_g(\mathbf{X})) $$

Note that by construction of regression weights $\gamma_g(\mathbf{X}) \vcentcolon= \mathbb{V}\left[W\mid\mathbf{X}\right]/\mathbb{E}\left[\mathbb{V}\left[W \mid \mathbf{X}\right]\right]$ have an expected value of 1. So, we arrive at the following decomposition

equation[equation omitted — 241 chars of source]

We provide three simple examples numerically illustrating the above decomposition with negative, zero, and positive covariance between the regression weights and treatment functions in figure (ref).

figure[figure omitted — 3,460 chars of source]

This decomposition immediately illustrates how rank reversals may arise in practice: when the second term in (ref) is large enough to offset the first, rank-reversals may occur.

prop[Necessary and Sufficient Condition for Rank Reversal] The following condition yields rank-reversal between treatments $j$ and $k$ \begin{equation} \mathbb{E}\left[\tau_j(\mathbf{X}\right] + Cov\left[\tau_j(\mathbf{X}, \gamma_j(\mathbf{X}\right] < \mathbb{E}\left[\tau_k(\mathbf{X})\right] + Cov\left[\tau_k(X), \gamma_k(\mathbf{X}\right] \end{equation} This is an immediate implication of the decomposition (ref). Proof in (ref). We also provide slightly more transparent sufficient conditions that parametrises the magnitudes of the two covariances in (ref) in appdx (ref).

\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} {When can we expect PLM coefficients to yield correct rankings?}

enumerate• Constant treatment effects ($\tau(\mathbf{X}) = \tau$): Here, PLM, IPW, and AIPW all estimate the same quantity. This is rare in practice but serves as a useful benchmark. • Uncorrelated weights and effects ($\text{Cov}\left[\gamma(\mathbf{X}),\tau(\mathbf{X})\right] \approx 0$): This can happen when: \begin{enumerate} • Treatment assignment is relatively balanced ($p(\mathbf{X}) \approx 0.5$) • Treatment effects vary independently of variables that predict treatment • As-good-as-random assignment: if units don't have the opportunity to sort into treatment based on private information about their own treatment effects $\tau(\mathbf{X})$, this covariance will be more likely to be small. \end{enumerate} • Uniform selection on gains: If units sort into treatments $j$ and $k$ based on private information about their expected gains $\tau_j(\mathbf{x}), \tau_k(\mathbf{x})$, the covariance $\text{Cov}\left[\gamma_g(\mathbf{X}), \tau_g(\mathbf{X})\right]$ will be of the same sign for $g \in \left\{ j,k \right\}$, which would not flip the rankings between the ATEs. • Similar propensity score distributions: When $p_j(\mathbf{X})$ and $p_k(\mathbf{X})$ have similar distributions, $\gamma_j(\mathbf{X})$ and $\gamma_k(\mathbf{X})$ will be similar, reducing the chance of rank reversals. This suggests observational studies with very different propensity scores across treatments are more prone to rank reversals • Moderate treatment effect heterogeneity: If heterogeneity in treatment effects is modest, and this is known to agents, it is less likely that they actively seek or avoid treatments (which pushes $p_g(\mathbf{X})$ towards 0 or 1) based on this information, which weakens the magnitude of $\text{Cov}\left[\tau(\cdot), \gamma(\cdot)\right]$, which in turn makes it less likely that the covariances for different treatments are of contrasting signs to result in rank reversals.

A practical implication of the above is that when treatment effects are suspected to be highly heterogeneous with units selecting into treatments, researchers should prefer AIPW over PLM for ranking.

defi[Augmented Inverse-Propensity Weighting (AIPW) Estimators] An alternative to the PLM that does not fall prey to the ranking reversal property is the AIPW estimator, which involves construction of a `pseudo-outcome' $\Gamma_i^{j}$ that is the estimated potential outcome under treatment $j$ Cattaneo2010-oc,Chernozhukov2018-fl \begin{align*} \widehat{\Gamma}_i^{j} = \widehat{\mu}^{j,-k}(\mathbf{X}_i) + & \frac{\mathds{1}\{W_i = j\}}{\widehat{p}^{j,-k_i}(\mathbf{X}_i)} \left( Y_i - \widehat{\mu}^{j,-k_i}(\mathbf{X}_i) \right ) \\ \widehat{\tau}^{AIPW, a, b} = \frac{1}{n} \sum_{i}^n\left( \widehat{\Gamma}_i^{a} - \widehat{\Gamma}_i^{b} \right ) \end{align*} where we first partition data by assigning each observation into $k_i \in \mathsf{U} \left[ K \right ]$ folds, and cross-fit nuisance functions $\widehat{\mu}(\cdot)$ (an outcome regression within treatment level $j$) and $\widehat{p}$ (a multi-class propensity score that models the probability of treatment level $j$) so that their predictions for unit $i$ are produced from models that were not trained on the $k_i-$th fold. The above estimator is consistent for the ATE regardless of the level of heterogeneity in the underlying treatment effect function $\tau(\mathbf{X})$, which implies that it does not exhibit rank-reversal properties, but conversely may have poor empirical performance in the presence of extreme propensity scores.

\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Numerical Experiments}

Simulation Design

We conduct Monte Carlo simulations to evaluate the performance of PLM and AIPW estimators under various data generating processes (DGPs). Each DGP is characterized by:

itemize• A binary covariate $X \sim \text{Bernoulli}(0.5)$ • Two binary treatments $W_1, W_2$ with stratum-specific propensity scores $p_j(X)$ • Heterogeneous treatment effects $\tau_j(X)$ for each treatment

We consider five scenarios that vary in their degree of effect heterogeneity and propensity score distributions:

enumerate• Extreme Heterogeneity: Large differences in treatment effects across strata with extreme propensity scores • Constant Effects: Homogeneous effects within treatments but different across treatments • Uncorrelated: Moderate heterogeneity with balanced propensity scores • Selection on Gains: Treatment probability correlated with treatment effects • Balanced: Equal propensity scores across strata with heterogeneous effects

For each scenario, we simulate 1,000 datasets with 10,000 observations each. We evaluate the estimators on three dimensions:

itemize• Distribution of point estimates • Bias relative to true effects • Proportion of correct rankings between treatments

We report figures for each of these settings in appendix (ref). We find that with the exception of the extreme heterogeneity setting that expands upon the example in section 2 (fig (ref)), the rankings produced by the PLM are largely consistent with the AIPW estimator, and conform with the sufficient conditions derived in the previous section.

\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Conclusion}

This note highlights the importance of using appropriate methods for estimating and ranking treatment effects in the presence of heterogeneity. We show using an example that commonly used Partially Linear Models can lead to biased estimates and incorrect rankings. We then define a notion of ranking reversals and derive a decomposition relating the WATE and ATE, which gives rise to a necessary and sufficient condition for ranking reversals in linear regression. Finally, we propose interpretations for these conditions and recommend using Augmented Inverse Probability Weighting estimator as a general solution for ranking in the presence of substantial heterogeneity.

Our findings have important implications for decision-making in various fields, including digital platforms and policy evaluation, where accurate ranking of treatments is crucial. Future work could explore the performance of these methods in more complex settings with multiple treatments and high-dimensional covariates.

\printbibliography