Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
51,986 characters · 17 sections · 51 citation commands
Treatment Effect Risk: Bounds and Inference
\email{[email removed]}
Policymakers and project managers regularly conduct randomized experiments (“A/B tests”) to assess potential changes to policy or product. A key metric is the average treatment effect (ATE), the difference in the population-average outcome when everyone or no one is treated. ATEs are easily estimated by differences in the sample-average outcome within treatment groups, barring interference. Estimation from observational data is also possible under appropriate assumptions, \eg, unconfoundedness imbens2015causal. Identifying an individual's outcome with their utility -- as we will throughout this paper -- the ATE is the difference in social welfare in these two counterfactual scenarios. By linearity, this coincides with the population-average of each individual's treatment effect, the difference in their own utility in the two counterfactual scenarios.
It is widely recognized, however, that treatment effects can vary widely between individuals heckman1997making,crump2008nonparametric. Thus, even if the ATE is positive, there is a risk that many individuals are harmed by the proposed change. Crucially, distributional treatment effects (DTEs), which compare the two counterfactual utility distributions beyond their means, cannot capture this risk. Indeed, imbens2009recent note “quantile effects are defined as differences between quantiles of the two marginal potential outcome distributions, and not as quantiles of the unit level effect.” They nonetheless advocate for the former because policy “choice should be governed by preferences of the policymaker over these distributions.” However, such rational-decision-making framing presumes a policymaker facing a choice between lotteries drawing at random from individual outcomes. Instead, concerned with equity beyond social welfare, we should worry about the individuals, not the policymaker. Hypothetically, harm to some individuals is possible even when the “treat-all” utility distribution first-order-dominates “treat-none” so that any expected-increasing-utility-function-maximizer would choose “treat-all.”
One way to gain further insight into heterogeneity and hence inequities is to consider conditional ATEs (CATEs) given pre-treatment covariates. For example, if we observe a discrete sensitive attribute (\eg, race), we can simply compare the CATE in each attribute-value group.\footnote{We may still make some inferences on these even if we do not observe such attributes; see chen2019fairness,kallus2021assessing.} But it may not always be clear what are relevant such attributes and whether we are omitting important ones. Given rich and continuous covariates, we can still reliably learn the CATE function by leveraging recent advances in causal machine learning slearner,xlearner,drlearner,rlearner,causaltree,causalforest. It may still not be clear, nonetheless, whether the covariates are relevant for fairness considerations, what groups are captured in this way, and/or how to summarize the many individual predictions of complex machine-learned CATEs.
It is therefore particularly appealing to focus directly on the distribution of individual treatment effects (ITEs), such as the average effects among the worst-affected 10%, 20%, \etc, corresponding to the conditional value at risk (CVaR) of this distribution. The challenge is that no ITE can ever be observed -- the so-called Fundamental Problem of Causal Inference. Nonetheless, regardless of whether covariates are meaningful for fairness considerations, if they control for heterogeneity, CATE may predict ITE well. In this paper, we leverage this to proxy these important but unidentifiable treatment-effect risk measures. Specifically, we provide the tightest-possible upper and lower bounds given by CATE on the CVaR of ITE. By construction these are functions of distributions of observables. What remains is inference from data, whether experimental or observational. Since the CATE function can be high-dimensional, especially if we use a lot of covariates to control for heterogeneity, inference is difficult and na\"ive plug-in approaches fail. We design debiased estimators and confidence intervals for our bounds that overcome this challenge by being exceedingly robust: given rough, machine-learned estimates of CATE and other nuisances, they behave as though we used perfect estimates; they remain consistent even when some nuisances are mis-estimated; and surprisingly they remain valid as bounds even when CATE is mis-estimated. We conclude by using our tools to illustrate treatment-effect risk in a case study of job-search-assistance benefits.
Each individual in the population is associated with two potential outcomes, $Y^*(0),\,Y^*(1)\in\Rl$, corresponding to individual utility under “treat-all” and “treat-none,” respectively, and baseline covariates (observable characteristics), $X\in\mathcal X$. The ITE, ATE, and CATE are, respectively,
We assume $\E\delta^2<\infty$ throughout.
Of interest is the average effect among the $(100\times\alpha)\%$-worst affected, formalized by $\cvarat\alpha(\delta)$, where for any $Z$ rockafellar2000optimization\footnote{CVaR is sometimes defined for the right tail, corresponding to our $-\cvarat\alpha(-Z)$.}
where $(u)_-=u\wedge0$. The $\sup$ is attained by $\beta$ equal the $\alpha$-quantile:
Provided $F_Z(F_Z^{-1}(\alpha))=\alpha$ (\eg, $Z$ continuous), then $\cvarat\alpha(Z)=\Eb{Z\mid Z\leq F_Z^{-1}(\alpha)}$. Otherwise, $\cvarat\alpha(Z)\in[\Eb{Z\mid Z< F_Z^{-1}(\alpha)},\,\Eb{Z\mid Z\leq F_Z^{-1}(\alpha)}]$, and, unlike these two endpoints, $\cvarat\alpha(Z)$ is continuous in $\alpha$ and coherent artzner1999coherent. It is therefore the correct generalization of “average of the $(100\times\alpha)\%$-lowest values” when ambiguous due to discontinuities.
We consider data from a randomized experiment or observational study. Each individual is associated with a treatment $A\in\{0,1\}$, and we observe the factual outcome $Y=Y^*(A)$ (never $Y^*(1-A)$). The data is $(X_i,A_i,Y_i)\sim(X,A,Y)$, $1\leq i\leq n$. We assume unconfoundedness throughout: $Y^*(a)\indep A\mid X$.\footnote{And $Y=Y^*(A)$ assumes non-interference rubin1986comment.} Randomized experiments (our focus) ensure this by design (often with $X\indep A$). Our results nonetheless extend to observational settings assuming unconfoundedness. Under unconfoundedness, ATE and CATE are identifiable, \ie, are functions of the $(X,A,Y)$-distribution: $\mu(X,a)=\E[Y\mid X,A=a]$, $\tau(X)=\mu(X,1)-\mu(X,0)$, $\bar\tau=\E\tau(X)$ ($=\E[Y\mid A=1]-\E[Y\mid A=0]$ if $X\indep A$). Define also the propensity score $e(X)=\Prb{A=1\mid X}$ and marginal-outcome regression $\bar\mu(X)=\E[Y\mid X]=e(X)\mu(X,1)+(1-e(X))\mu(X,0)$.
We now illustrate treatment-effect risk and its unidentifiability, which motivates us to consider the tightest-possible identifiable bounds ((ref)) and inference thereon ((ref)).
An upper bound on $\cvarat\alpha(\delta)$ is crucial: if negative or substantially below ATE, the change poses certifiable risk or inequity to an $(100\times\alpha)\%$-subpopulation.
Since $\tau(X)$ represents our best guess for $\delta$ (in squared error), imputing the unknown $\delta$ with $\tau(X)$ seems reasonable. (ref) shows this in fact provides an upper bound.\footnote{(ref) extends to any coherent risk by writing $\delta=\tau(X)+(\delta-\tau(X))$ and using sub-additivity.} If $\tau(X)$ is continuous, $\cvarat\alpha(\tau(X))=\Efb{\delta\mid \tau(X)\leq F_{\tau(X)}^{-1}(\alpha)}$, and (ref) is intuitive: $\cvarat\alpha(\delta)$ is worst average effect among all $(100\times\alpha)\%$-subpopulations, while $\cvarat\alpha(\tau(X))$ only among $X$-defined subpopulations. This bound is also tight: given just $\tau(X)$, it cannot be improved.\footnote{The bound need not be tight given the $(X,A,Y)$-distribution, which characterizes more than the mean of the $(\delta\mid X)$-distribution, as described by the Fr\'echet-Hoeffding bounds. We focus on best bounds given just by CATE, which is the common tool to understand effect heterogeneity in practice.}
(ref) implies an ordering:
Much as we try to best control for heterogeneity, disparate effect-predictiveness of covariates may mean some negative ITEs are averaged out and hidden while others are singled out. A remedy when concerned about disproportionate predictiveness among sensitive groups (\eg, race) would be to include these (or proxies) within $X$. But, we may always worry about missing something. A lower bound can provide assurances about what the upper bound may be missing.
This depends on how much residual heterogeneity remains. Our first set of lower bounds limit the range of residual heterogeneity, \ie, almost-sure bounds on $\delta-\tau(X)$, while our second set of lower bounds limit its variance, \ie, bounds on $\op{Var}(\delta\mid X)=\E(\delta-\tau(X))^2$.
The right-hand side of (ref) is the $\alpha$-CVaR of the equal-mixture distribution of $\tau(X)-b$ and $\tau(X)+b$. It reduces to $\cvarat\alpha(\tau(X))$ when $b=0$ (equivalent to $\delta=\tau(X)$). When $\alpha=1$, it becomes $\bar\tau$ for any $b\geq0$ (as necessary for tightness). The lower bound is established via weak semi-infinite duality and its tightness by exhibiting the equal-mixture distribution.
Since $(\tau(X)\pm b-\beta)_-\geq (\tau(X)-\beta)_--b$, (ref) upper bounds $\cvarat\alpha(\tau(X))-b$. This simpler bound is tight if we only assume a one-sided-bounded range.
The lower bound is immediate and its tightness given by exhibiting a skewed two-point-mass distribution. For $\alpha=1$, (ref) simply reads $\bar\tau\geq\bar\tau-b$, but for any $\alpha<1$, (ref) is actually tight.
Limiting residual heterogeneity within a range may be implausible, or plausible only with large constants, yielding a weak bound. We next explore the implication of the residual ITE-variance after controlling for $X$, which we can bound given observables.
The proof of (ref) leverages strong duality for convex semi-infinite optimization. Note (ref) equals $\cvarat\alpha(\tau(X))$ whenever $\bar\sigma^2(X)=0$ and $\bar\tau$ whenever $\alpha=1$. Since $\abs{\delta-\tau(X)}\leq b\implies\op{Var}(\delta\mid X)\leq b^2$, plugging $\bar\sigma^2(X)=b^2$ into (ref) must be looser than (ref) by tightness. Triangle inequality verifies this directly: $\sum_{\pm}(\tau(X)\pm b-\beta)_-=\tau(X)-\beta-\frac12\sum_{\pm}\abs{\tau(X)\pm b-\beta}\geq \tau(X)-\beta-\sqrt{(\tau(X)-\beta)^2+b^2}$.
A residual-variance bound is both more plausible and easier to calibrate than an absolute bound. Letting $\rho(X)=\op{Corr}(Y(0),Y(1)\mid X)\in[-1,1]$, we have
where all terms but $\rho(X)$ are identifiable. Thus, postulating different potential-outcome correlations, we obtain different bounds. (ref) is maximized for $\rho(X)=-1$, which is tight, as all correlations are realizable. Thus, plugging $\bar\sigma^2(X)=(\op{Var}^{1/2}(Y\mid X,A=0)+\op{Var}^{1/2}(Y\mid X,A=1))^2$ into (ref) yields a tight lower bound on ITE-CVaR, given conditional expectations and variances. We may obtain better bounds if we postulate larger $\rho(X)$.
(ref) also implies a simpler but looser bound.
(ref) more transparently bounds the slack in (ref) in terms of residual effect variance. However, it is not tight, as can be seen for $\alpha=1$. (ref) is even looser but appealing as it avoids $\op{Var}(Y\mid X,A)$, depending only on the root-mean-squared error of regressing $Y$ on $X$ for each $A\in\{0,1\}$ (\ie, the numerator of nonparametric $R^2$).
We next turn to estimating the bounds developed in (ref) and constructing confidence intervals. Recall our data $(X_i,A_i,Y_i)\sim(X,A,Y)$, $1\leq i\leq n$, may be experimental or observational. The only relevant technical difference between these two cases is whether propensity, $e(X)=\Prb{A=1\mid X}$, is known or not. While it matters not here, note that $e(X)$ is usually constant in experiments ($A\indep X$). In observational settings $e(X)$ may be estimated.
We focus here on inference on CATE-CVaR. We provide analogous procedures for the lower bounds of (ref) in (ref). Fix $\alpha$. Our inferential target is $$\Psi=\cvarat\alpha(\tau(X))=\beta^*+\frac1{\alpha}\E(\tau(X)-\beta^*)_-,\quad\text{where}~\beta^*=F_{\tau(X)}^{-1}(\alpha)=\inf\fbraces{\beta:\Prb{\tau(X)\leq\beta}\geq\alpha}.$$ Since $\tau(X)$ is not directly observed, the first step is fitting it. Fortunately, recent advances in causal machine learning provide excellent tools for this slearner,xlearner,drlearner,rlearner,causaltree,causalforest. Given an estimate $\hat\tau$, we might consider a plug-in approach: $\hat\Psi^\text{plug-in}=\sup_\beta\fprns{\beta+\frac1{n\alpha}\sum_{i=1}^n(\hat\tau(X_i)-\beta)_-}$. Unfortunately, the statistical behavior of $\hat\Psi^\text{plug-in}$ depends heavily on that of $\hat\tau$: if $\hat\tau$ converges slowly and/or has non-negligible bias, as occurs when fit by flexible machine-learning methods, both estimation rates and valid inference may be imperiled for $\hat\Psi^\text{plug-in}$.
Instead, we develop a debiasing approach that is insensitive to CATE-estimation, accommodating both misspecified parametric models and flexible-but-imprecise machine-learning CATE-estimators. The main challenge is estimating $\beta^*$, which cannot be expressed by an estimating equation in $X,Y(0),Y(1)$, so its efficient/orthogonal estimation is unclear, unlike the case of quantile/CVaR treatment effects kallus2019localized,belloni2017program,firpo2007efficient. Fortunately, we care only about $\Psi$, not $\beta^*$, and special optimization structure in $\Psi$ gives robustness to perturbations. so even rough estimates suffice. Our approach is therefore unique: we treat both $\tau$ and $\beta^*$ as nuisance parameters, together with $e,\mu$, and ensure simultaneous orthogonality to all four nuisances.
(ref) summarizes our procedure. It proceeds by approximating the sample average of $\Psi=\E\phi(X,A,Y,e,\mu,\tau,\beta^*)$, where, we define
We first estimate the unknown $(e,\mu,\tau,\beta^*)$. We do so using “cross-fitting” over $K$ even folds so that nuisance estimates are independent of samples where applied schick1986,doubleML,zheng2011cross.\footnote{We may avoid cross-fitting and fit nuisances once on the whole sample if we assume estimates belong to a Donsker class with probability tending to 1; we omit this option for brevity.} As we discuss in detail in (ref), we treat $\tau$ as a separate nuisance even though $\tau(x)=\mu(x,1)-\mu(x,0)$. For one, this enables the use of specialized CATE-learners. We also treat $\beta^*$ as a separate nuisance (not as a parameter as in kallus2019localized) and fit it as the quantile of $\hat\tau(X)$ in the out-of-fold data. As simple regressions, $e$ and $\mu$ can be fit by parametric regression or standard machine-learning methods such as random forests, gradient boosting, neural networks, \etc.
We now establish favorable guarantees for (ref). First, we show it is insensitive to slow but consistent estimation of nuisances, having first-order behavior as if we used true values.
We will need some minimal regularity.
The first condition ensures that the $X$-distributions of experimental groups overlap. It is usually guaranteed in randomized experiments by setting $e(X)$ constant ($A\indep X$). In unconfounded observational studies, it is a standard assumption. The second condition requires bounded outcomes and is largely technical to make analysis tractable. The third condition prohibits degeneracy of the quantile. The same is needed for asymptotic normality of sample quantiles of observed variables. If $\tau(X)$ is discrete, the condition may be replaced by $\exists \varepsilon>0:F^{-1}_{\tau(X)}(\alpha-\varepsilon)=F^{-1}_{\tau(X)}(\alpha+\varepsilon)$, yielding superefficient quantile estimation. The only problematic case is multiplicity of $\{\beta:F_{\tau(X)}(\beta)=\alpha\}$, but only finitely-many such “bad” $\alpha$'s exist. Since the focus is on $X$ being rich, we focus on the continuous case and the condition in (ref).
We first show how, under (ref), estimation rates for $\hat\tau^{(k)}$ translate to rates for $\hat\beta^{(k)}$.
We now show robust oracle-like behavior for $\hat\Psi$.
The rate assumptions on $e$ and $\mu$ are lax: it suffices to have $o_p(n^{-1/4})$-rates on both or no rate on $\mu$ at all if $e$ is known. This parallels standard conditions in double-machine-learning ATE-estimation, achievable by a variety of machine-learning methods doubleML. We explore the condition on $\tau$ in (ref).
(ref) guarantees good performance if all nuisances are estimated slowly, but still consistently. But even if nuisances are inconsistent, we perform well.
First, we establish a property mirroring doubly-robust ATE-estimation RRZDoubleRobust: even if $e$ or $\mu$ are inconsistent, we remain consistent, provided $\tau$ is consistently estimated, albeit slowly.
(ref) is particularly strong in experiments ($e$ known): we can get away with $\hat\mu^{(k)}=0$. We need only estimate CATE at $o_p(n^{-1/4})$-rates to ensure $O_p(n^{-1/2})$-consistency.
It would appear we must consistently estimate CATE to have hope of estimating its CVaR. While true, we next show that even if we mis-estimate CATE and also one of $e,\mu$, we still get an upper bound on CATE-CVaR (hence on ITE-CVaR). This appears to be the second finding of a double-validity property since being first documented in sensitivity analysis dorn2021doubly.
We first establish the population-level bound behavior and then state the implication for estimation.
(ref) guarantees extensive robustness and suggests a practical, blackbox-free approach in experimental settings: set $\hat\mu^{(k)}=0$ and use simple misspecified parametric models (\eg, linear) for CATE-estimation, and we still estimate a valid ITE-CVaR bound at fast $O_p(n^{-1/2})$-rates.
(ref) accepts separate learners for both $\mu$ and $\tau$. So, while $\tau(x)=\mu(x,1)-\mu(x,0)$, we need not have $\hat\tau^{(k)}(X)=\hat\mu^{(k)}(x,1)-\hat\mu^{(k)}(x,0)$, and in fact we should not. Recent work advocates and provides specialized methods for directly estimating CATE causaltree,causalforest,xlearner,slearner,rlearner,drlearner.
This is important because (ref) uses the $\mu$- and $\tau$-estimates differently and, correspondingly, our theoretical results impose different assumptions on each. The $\tau$-estimate is used for approximating the event $\indic{\tau(X)\leq\beta^*}$, which is crucial for targeting CVaR correctly. In contrast, the $\mu$-estimate is just used in order to estimate a weighted-average treatment effect, given the weights $\indic{\tau(X)\leq\beta^*}$, and is therefore interchangeable with propensity.
We next review different options for CATE-estimation and how these ensure the conditions of (ref). We emphasize that these need not be understood as exhaustive list of which learners to use: practically, the nuisance-estimation rates are high-level assumptions that generally say one may safely plug-in black-box machine-learning estimators to (ref): no restrictions are made but rates (no metric-entropy conditions), estimators can be flexible/nonparametric in that rates can be much slower than “parametric" $O_p(n^{-1/2})$-rates, and results are exceedingly robust to inconsistent estimation.
A major issue with CATE-estimation by differencing outcome regressions is that effect signals are easily lost. CATE is generally simpler and less variable than baseline mean outcomes, $\mu(X,0),\mu(X,1)$. For example, many variables often help predict outcomes, but few modulate the treatment effect. It is therefore imperative to learn CATE directly.
In experimental settings ($e$ known) we can construct a pseudo-outcome $\Delta=\frac{A-e(X)}{e(X)(1-e(X))}Y$ and, since $\tau(X)=\Eb{\Delta\mid X}$, learn CATE by regressing $\Delta$ on $X$, using any supervised-learning method. One case that theoretically ensures $\fmagd{\hat \tau^{(k)}-\tau}_\infty=o_p(n^{-1/4})$ is when $\tau(x)$ is more-than-$d/2$-smooth in $x\in\R d$ stoneglobal. Another option is $\tau(x)$ linear with $o(\sqrt{n}/\log d)$-nonzero coefficients belloni2017program. Note this works regardless of $\mu$ being nice.
Or, we may avoid black-box models (and cross-fitting) altogether by using simple linear regression of $\Delta$ on $X$ to obtain a valid bound per (ref).
To satisfy the other conditions, for (ref) we can set $\mu=0$, and for (ref) we need only estimate $\mu$ consistently without rate. We can either estimate $\mu$ directly or only estimate $\bar\mu(X)=\Eb{Y\mid X}$ and set $\hat\mu^{(k)}(X,A)=\hat{\bar\mu}^{(k)}(X)+(A-e(X))\hat\tau^{(k)}(X)$. Consistency for either is immediate from $\EY^2<\infty$ gyorfi2002distribution.
When $e$ is unknown, the pseudo-outcome-construction needs refinement. One option is DR-leaner drlearner: regress $\Delta=\hat\mu(X,1)-\hat\mu(X,0)+\frac{A-\hat e(X)}{\hat e(X)(1-\hat e(X))}(Y-\hat\mu(X,A))$ on $X$, where $\hat e,\hat \mu$ are appropriately cross-fitted. Another is R-learner rlearner: let $\hat\tau$ minimize the average of $\prns{Y-\hat{\bar\mu}(X)-(A-\hat e(X))\hat\tau(X)}^2$, where $\hat e,\hat{\bar\mu}$ are appropriately cross-fitted. drlearner provides rates for local-polynomial R-learners: if $e(x)$ is $s_e$-smooth in $x\in\R d$, $\bar\mu(x)$ $s_\mu$-smooth, and $\tau(x)$ more-than-$d/2$-smooth, then we obtain $o_p(n^{-1/4})$-rate pointwise error, provided $s_e\geq s_\mu,\,\frac{s_e+s_\mu}{2}>\frac{d}{8}$. To convert pointwise-error bounds to sup-norm-error bounds, we may follow the discretization approach of stoneglobal, incurring only logarithms. Or, we can simply use linear R- or DR-learners and get a valid bound per (ref).
We now demonstrate our bounds and inference.\footnote{Replication code is available at \url{https://github.com/CausalML/TreatmentEffectRisk}.} While we consider a program-evaluation example, we believe our results are also particularly relevant to A/B testing on online platforms, where, after testing, product innovations are usually either scrapped/reworked or broadly rolled out, and where ATEs are often small, creating an opportunity for many users to be negatively impacted despite positive average effects. Little such data is public, however.
behaghel2014private analyze a large-scale randomized experiment comparing assistance programs offered to French unemployed individuals. They compare three arms: individuals in the “control” arm receive the standard services of the Public Employment Services, in “public” receive an intensive counseling program run by a public agency, and in “private” a similar program run by private agencies.
We consider a hypothetical scenario where the private-run counseling program ($A=0$) is currently being offered to the unemployed and we consider the change to a public-run program ($A=1$).\footnote{Some individuals assigned to the additional counseling refused it. We nonetheless restrict our attention to intent-to-treat interventions, considering hypothetically making available either the public-run or private-run counseling to unemployed individuals, who may decline it.} We take reemployment within six months as our (binary) outcome.
The ATE is $1.22$ percentage points (90%-CI $[-0.35, 2.8]$), a $4.9\%$ increase in reemployment. This suggests a positive/neutral effect, so a policymaker might hypothetically consider this an acceptable policy change, \eg, if the public-run program provided cost savings.\footnote{behaghel2014private discuss why public-run programs fare better.}
To apply our methodology, we consider all pre-treatment covariates in table 2 of behaghel2014private, except we treat as numeric (rather than dichotomize) age, number children, years experience, salary target, assignment timing, and number unemployment spells. Other variables quantify education, employment level and type, gender, martial status, national origin, region, unemployment reason, and long-term-unemployment risk. The propensity is constant. As recommended in (ref), we fit CATE using a pseudo-outcome linear regression. We estimate $\mu$ using cross-fitted gradient-boosting machines.
(ref) presents inference on CATE-CVaR using (ref) for $\alpha\in\{0.01,0.02,\dots,1\}$. The line represents our point estimate, after rearrangement as recommended in (ref),\footnote{We present the figure without rearrangement in (ref).} and the shaded region represents point-wise 90%-confidence intervals. Note uncertainty grows for smaller $\alpha$.
We see that the ATE-estimate (right-most point) is positive with an interval containing zero. We find, however, that some 56%-sized $X$-defined-subpopulation has a negative effect at 90%-confidence.\footnote{Since outcome is binary, the largest fraction that can have a negative effect is $(50\times(1-\bar\tau))\%$, so either $\bar\tau<0$ or at most half may be negatively affected. The ATE interval indeed contains zero with confidence only 90%.} This strongly suggests that the change, if enacted could materially negatively impact a large portion of the population, despite the positive/neutral ATE. Thus, considering treatment effect risk provides a crucial metric not reflected in the ATE. This risk is also not reflected in DTEs: the binary potential-outcome distributions are fully specified by just $\E[Y(0)],\,\E[Y(1)]$.\footnote{In particular, the $\alpha$-quantile DTE is uselessly zero for all $\alpha\in[0,1]\backslash\{1-\E[Y(0)],1-\E[Y(1)]\}$ and the $\alpha$-CVaR DTE is $\frac1\alpha(\E[Y(1)]-1+\alpha)_+-\frac1\alpha(\E[Y(0)]-1+\alpha)_+$, which is not even monotonic. For illustration we plot it in (ref).}
In (ref) we focus on comparing CATE-CVaR to ATE following (ref). The only difference to (ref) is a slight vertical shift and that confidence intervals (correctly) shrink to a point as $\alpha\to1$, enabling more confident conclusions comparing subpopulations to the population.
In (ref) we consider CATE-CVaR when we capture less heterogeneity, using only age, high-school dropout, African national origin, and Paris-region resident as covariates ($X_1$). This detects no significant risk.
While the upper bounds show a significant subpopulation can be negatively harmed, being only bounds, it may be the subpopulation can be harmed even more or an even larger subpopulation can be harmed. Lower bounds help us understand how much greater the risk might be.
In (ref) we consider our lower bounds (vs ATE) when limiting the residual-heterogeneity range given by Theorems (ref) (two-sided range) and (ref) (one-sided range).
Since it may be hard to justify and calibrate a limited range, in (ref) we consider lower bounds given by (ref) by limiting residual-heterogeneity variance. For the former, we fit $\op{Var}(Y\mid A,X)$ using gradient-boosting machines and construct $\bar\sigma^2(X)$ per (ref) by varying constant values of $\rho(X)=\rho\in[-1,1]$. Recall $\rho=-1$ always yields an assumption-free bound. We use the same model to estimate the right-hand side of (ref). We compute the cross-validated root-mean-squared prediction error to estimate the right-hand side of (ref).
We observe that assuming perfectly-conditionally-correlated potential outcomes yields a lower bound very close to the upper bound. The bounds of (ref) appear loose; indeed they are not tight.
We study the average effect on those worst-affected by a proposed change as a measure of its risk, how to tightly bound it given covariates that explain some heterogeneity, and how to make robust inferences on these bounds even when this heterogeneity is roughly estimated. This provides very practical tools for assessing policy and product changes beyond their ATE and DTEs. We can safely use flexible yet biased/slow-to-converge machine learning, or we can avoid black-box models and easily get good bounds by considering only linear projections of heterogeneity . In the hypothetical case study this detected that, what appeared to be a positive/neutral change could actually very negatively impact a substantial subpopulation.
We focused on experimental (or, unconfounded observational) settings without interference, where risk is already unidentifiable despite randomization. A future direction is to consider the impact of interference johari2022experimental,athey2018exact or confounding tan2006, where even ATEs are unidentifiable and fairness is harder to assess kilbertus2020sensitivity,jung2020bayesian,kallus2018residual. Interestingly, for partial identification under tan2006' model, $X$-conditional outcome-CVaR plays a crucial role dorn2021doubly. Another direction may be to consider other risk measures, such as given by Kullback-Leibler ambiguity sets ahmadi2012entropic. Per (ref), the tight upper bound is still the risk measure applied to CATE, but it remains to compute lower bounds and design robust inference methods.