Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
15,699 characters · 5 sections · 16 citation commands
Sequential Validation of Treatment Heterogeneity
Consider a randomized controlled trial (RCT) on $i = 1, \, \ldots, \, n$ units with potential outcomes $Y_i(0), \, Y_i(1)$, pre-treatment covariates $Z_i$, and a binary treatment $D_i \cond Y_i(0), \, Y_i(1), \, Z_i \sim \text{Bernoulli}(\pi)$. The analyst observes $(Z_i, \, Y_i, \, D_i)$ where $Y_i = Y_i(D_i)$. As discussed by in the paper by \citet*[henceforth CDDF]{chernozhukov2024fisher}, given data from an RCT it is often of interest to estimate and understand treatment heterogeneity, and to use knowledge of such heterogeneity to design treatment-assignment rules.
It is well understood that the conditional average treatment effect (CATE),
is a natural metric for quantifying treatment heterogeneity under the potential outcomes model athey2016recursive,manski2004statistical; furthermore, in an RCT, the CATE admits a number of simple identification formulas in terms of observables, e.g., $\tau(z) = \mathbb{E}[Y_i \cond Z_i = z, D_i = 1] - \mathbb{E}[Y_i \cond Z_i = z, D_i = 0]$. What's less clear is how we should go about estimating the CATE. There's been considerable recent interest in developing machine-learning based methods for CATE estimation athey2019generalized,kunzel2019metalearners,hahn2020bayesian,foster2023orthogonal. However, consistency of these methods relies on often hard-to-verify properties of the data-generating distribution, and the same is true of the fundamental statistical difficulty of the CATE estimation task kennedy2022minimax.
Given the challenges in obtaining provably valid estimates of $\tau(z)$ without making regularity (e.g., smoothness) assumptions, CDDF recommend that applied researches refrain from directly relying on theoretical consistency guarantees for non-parametric CATE estimators. Instead, they recommend that researchers first make a best effort to estimate $\htau(\cdot)$ well (e.g., using a machine learning method), and then focus on evaluating how well the estimated $\htau(\cdot)$ captures the true treatment heterogeneity on held-out data. The main argument of CDDF is that such evaluation metrics allow for simple and transparent large-sample inference without making any smoothness assumptions on $\tau(\cdot)$.
I second this recommendation, and found CDDF to make a compelling case for it. When estimating the CATE one of course wants to try to be as accurate as possible. But then, when communicating results the transparency and credibility of any claimed inferences is key---and validating a proposed $\htau(\cdot)$ estimate on a fresh evaluation dataset is a much simpler task (and thus allows for more robust inferential strategies) than than trying to estimate $\tau(\cdot)$ itself. The incentives are also well aligned: If a researcher is able to start with a good estimate $\htau(\cdot)$, then such diagnostic exercises will often yield interesting insights; whereas a researcher starting with a poor estimate $\htau(\cdot)$ will simply find themselves unable to say anything at all about potential treatment heterogeneity.
CDDF propose interrogating the alignment of $\htau(x)$ with the true $\tau(x)$ using metrics which they refer to as the BLP and GATES.\footnote{The authors also discuss the CLAN metric, but the focus of the CLAN is about describing the estimate $\htau(x)$ rather than validating its alignment with $\tau(x)$. The main use of CLAN comes after we've already validated that $\htau(x)$ captures real and meaningful heterogeneity---thus making a description of how this heterogeneity was captured relevant.} The BLP metric estimates the regression coefficient of the true CATE $\tau(Z_i)$ on estimates $\htau(Z_i)$ (and a coefficient of 1 means that $\htau(\cdot)$ is well calibrated); whereas the GATES metric estimates average treatment effects across subgroups defined by stratifying $\htau(Z_i)$ on the evaluation set (and verifies whether subgroups with claimed larger treatment effects in fact do). Other relevant metrics that can be used for this task include the QINI curve from the marketing literature radcliffe2007using, the area-under-the-curve measure of zhao2013effectively, and generalizations which yadlowsky2021evaluating refer to as rank-weighted average treatment effects (RATEs).
All these methods can be used to produce test statistics that have expectation 0 when the estimates $\htau(Z_i)$ are unaligned with any true treatment heterogeneity, and can be used to assess a null of no (explained) treatment heterogeneity. CDDF construct confidence intervals for both the BLP and GATES that hold under minimal assumptions (essentially the same assumptions needed to build confidence intervals for the average treatment effect); large-sample inference results for the QINI curve and its generalizations are available in yadlowsky2021evaluating.
One notable challenge with this approach---and a key focus of CDDF---is that in many applications we seek to both provide an estimate $\htau(\cdot)$ and evaluate it, and that splitting the data into separate training and evaluation sets may be onerous. As such it is natural to explore cross-fold approaches. The usual cross-validation-like blueprint for cross-fold evaluation is as follows:
One can leverage BLP, GATES, or other related metrics in step 4.
Now, given this setup, each of the $T_k$ on its own can obviously be used to build a valid $p$-value against the null, $p_k = 2\Phi\p{-\abs{T_k}}$, where $\Phi(t)$ denotes the standard normal distribution. In practice, however, only using one of the $T_k$ may feel wasteful, and so it may be of interest to provide $p$-values that aggregate across all the $T_k$.
A first, na\"ive, approach to aggregation would be to treat the $K$ test statistics as independent, and use
This is, however, not valid, as the cross-fold construction doesn't imply independence of the $T_k$; see the numerical experiment below for an example where these $p$-values do not achieve nominal rejection rates. From a statistical perspective, the failures of of na\"ive aggregation become acute when there is no treatment heterogneity to be found: In this regime, the estimates $\htau^{(-k)}(\cdot)$ become highly responsive to the noise in the training folds, thus creating dependence between the $T_k$.\footnote{The usual argument deployed by chernozhukov2018double to justify cross-fitting does not hold here because these methods do not target a pathwise-differentiable parameter hirano2012impossibility.}
What CDDF propose is to consider the median $p$-value instead:
Theorem 4.1 of CDDF establishes validity of this procedure under a median-concentration condition. The median aggregation approach is attractive in that it allows us to aggregate test statistics across folds while maintaining validity. However, this approach still may be seen as leaving information on the table. Each of the test statistics $T_k$ is only computed using $n/K$ samples and so (especially when $K$ is large), any of the $T_k$ on their own may not be particularly powerful. Ideally, one might hope that aggregation would allow us to pool information across folds to recover power comparable to a sample size of $n$ as occurs with, e.g., cross-fitting methods for semiparametric inference chernozhukov2018double,schick1986asymptotically; however, with median-aggregation this is not the case.
luedtke2016statistical faced a similar challenge when studying inference for the value of the optimal treatment rule, and proposed addressing it using a martingale construction that goes through the folds in sequence and only uses data from past folds for estimation. In our setting, this construction can be adapted into the following sequential validation procedure:
Finally, we aggregate the $p$-values as with the na\"ive construction:
This sequential construction is able to pool information across folds but, unlike the na\"ive method described above, is valid. The result below is adapted from luedtke2016statistical; the proof hinges on noting that, under the assumptions made, the $T_k$ (approximately) form a martingale difference sequence with predictable variance.
To better understand power of different methods for aggregating test statistics, we consider a simple numerical evaluation on GATES with two-groups: Our test statistic is the difference in ATEs for the top- vs bottom-half of units as ordered by the estate $\htau(Z_i)$. We instantiate the workflows described above as follows:
We consider a simple RCT simulation with $n = 1,000$ units and $p = 10$ dimensional covariates. We generate $Z_i \sim \text{Unif}([-1, \, 1]^p)$, $Y_i(d) \cond Z_i \sim \nn\p{d\tau(z), \, 1}$, and $D_i \sim \text{Bernoulli}(\pi = 0.5)$. We consider two settings for evaluation: One with $\tau(z) = 0$ (so the null holds), and one with $\tau(z) = \p{z_1}_+$ (so treatment heterogeneity exists and is relatively easy to find).
Results are shown in Table (ref). The first row shows rejection rates at a nominal 5% level under a setting where the null holds. We see that the na\"ive cross-fold aggregation method fails, and gets a 10% failure rate which is greater than the nominal 5% level. In contrast, both the median cross-fold aggregation and sequential validation methods are valid (median aggregation is conservative, while sequential validation is exact to within the precision of our simulation experiment). The second row shows power in a setting where treatment heterogeneity exists. Here, we see that the sequential validation method is much more powerful than median aggregation. The reason for the improved power of sequential validation is simple: For example, if all the $\hdelta_k$ are positive but none of the $T_k$ reaches significance on its own, then the sequential validation can take this shared directionality into account whereas median aggregation cannot.
CDDF highlight the importance of robust and transparent validation tests for treatment heterogeneity, and discuss a number of nice options. This is a good recommendation, and I hope the CDDF paper will help promote the use of such tests. One seemingly esoteric and technical---yet in practice important---consideration is how to develop in-sample validation tests that allow for both estimating $\htau(\cdot)$ and validating it withing the same sample. Simple cross-fold aggregation, i.e., the na\"ive method, is not valid here. CDDF propose median aggregation, which recovers validity at the cost of lower power. Here, we argued that sequential validation appears more powerful than median aggregation, all while retaining validity. Understanding optimal performance for in-sample validation of treatment heterogeneity---and developing practical methods that achieve such performance---seems like a promising goal for future work.
I am grateful for many enlightening conversations with Susan Athey and Erik Sverdrup on how best to estimate and validate heterogeneous treatment effects in practice, and to Gene Katsevich for asking a very interesting question. This research was supported by NSF grant SES--2242876.