Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
55,079 characters · 16 sections · 29 citation commands
What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment
Before making changes to an online platform, product managers regularly conduct experiments (“A/B tests”) to make sure a change does not negatively impact users' experience. Similarly, extensive program evaluations, whether experimental or observational, are a corner stone of evidence-based policymaking and help avoid harmful policy changes. These tests and evaluations focus on assessing the causal effect of a change on key metrics, whether user engagement or retention on online platforms or outcomes like employment for social programs. Average treatment effects, heterogeneous treatment effects, and distributional treatment effects on these outcomes quantify the quality of the intervention in terms of aggregate impacts on the population or on subpopulations. This is in its essence a statistical solution to the fundamental problem of causal inference: assess aggregate effects since the impact on any one individual can never be observed. We can never know whether a user that churned under arm “B” would have been retained under arm “A,” but we can characterize the distribution of retention under each arm, even segmenting the population by observable features.
While unknowable, the impact of changes on individuals is clearly of material importance to the decision to implement a change or not. This individual impact can have downstream effects, whether user-behavioral, reputational, or operational. Moreover, quantifying how negative this impact can be is crucial to understanding fairness with respect to the welfare of the individuals, rather than just focusing on social (\ie, average) welfare, the welfare of the decision-making entity, or the welfare of observable groups. Specifically, even if the average treatment effect is zero, there can still be a sizable subpopulation that is negatively affected by the change. That is, a change with zero average effect cannot in earnest be called “harmless." In this paper we consider measuring the fraction that are negatively affected. The aim is to be able to flag changes that may individually harm many. Unfortunately this fraction cannot be identified from data, even experimental data, but we may still be able to bound it. For example, if, in an A/B test, half of users click (or buy, or watch, or renew, etc.) whether exposed to the standard experience “A" or a new one “B," hypothetically it could be because the change affects no one, because the change positively affects half the user population to go from no-click to click while negatively affecting the other half, or anything in between. Such a large bound on the range of possibilities, however, is very crude and uninformative.
In this paper, we derive the sharp bounds on the fraction of individuals negatively affected by treatment using all baseline features available, and we develop tools to conduct robust inferences on these bounds. That is, we characterize the tightest-possible interval that contains all values of this unknowable quantity consistent with all observable information, in particular crucially leveraging unit feature information. We consider both the fraction negatively affected under a wholesale change from one treatment regime to another as well as under an optimal policy that assigns each observable subpopulation the treatment with best average outcomes for the subpopulation. Then, we tackle how to actually assess these theoretical bounds based on actual data, namely, how to estimate the interval endpoints and construct confidence intervals (CIs) that account for sampling error, on top of the inherent epistemic uncertainty characterized by the bounds. The bounds involve potentially complex functions such as the conditional average treatment effect (CATE) function, so we develop inference methodology that is exceedingly robust to learning these functions. Namely, our inferences are valid and calibrated even when these complex functions are learned nonparametrically at slow rates, and our estimated bounds remain valid, albeit conservative, even when these are learned inconsistently, so that conclusions as to the presence of harm based on our tools can be highly credible. We demonstrate our tools in a simulation study and in a case study of French reemployment assistance programs.
The Population\quad Each individual in the population is associated with baseline covariates (observable characteristics), $X\in\Xcal$, and two binary potential outcomes, $Y^*(0),\,Y^*(1)\in\{0,1\}$, corresponding to each of two treatment options, $0$ and $1$. Examples include: two versions of product on an online platform and whether user is retained in the next quarter, or two versions of a job training program for the unemployed and whether the participant is reemployed within the following year. We will consider the generalization to non-binary outcomes in (ref).
The individual treatment effect is the difference in potential outcomes: ${\mathrm{ITE}}=Y^*(1)-Y^*(0).$\break We assume that an outcome value of $1$ corresponds to a better outcome (\eg, retained, reemployed) and $0$ to a worse outcome (\eg, churned, unemployed). Thus, if treatment 0 represents status quo and treatment 1 represents a proposed change, then individuals with ${\mathrm{ITE}}=0$ ($Y^*(0)=Y^*(1)=0$ or $Y^*(0)=Y^*(1)=1$) are unaffected by the change, individuals with ${\mathrm{ITE}}=1$ ($Y^*(0)=0,\,Y^*(1)=1$) benefit from the change, and individuals with ${\mathrm{ITE}}=-1$ ($Y^*(0)=1,\,Y^*(1)=0$) are harmed by the change. This list of potential-outcome combinations is exhaustive.
The Data\quad We consider data from either a randomized experiment or an observational study, wherein we never observe both $Y^*(0)$ and $Y^*(1)$ simultaneously. Each individual is associated with a treatment $A\in\{0,1\}$, and we observe the factual outcome $Y=Y^*(A)$ corresponding to $A$. We never observe the counterfactual $Y^*(1-A)$. That is, the data consists of observations of $(X,A,Y)$.
In particular, the distribution of the data only reflects part of the full population distribution involving both potential outcomes. Let $\PP^*$ denote the distribution of $(X,A,Y^*(0),Y^*(1))$. Define the coarsening function $\Ccal:(x,a,y_0,y_1)\mapsto(x,a,ay_1+(1-a)y_0)$, which has the pre-image $\Ccal^{-1}(x,a,y)=\{(x,a,y_0,y_1):ay_0+(1-a)y_1=y\}$. Then the distribution induced on $(X,A,Y)$ is given by $\PP=\PP^*\circ\Ccal^{-1}$ (as measures). That is, the data distribution $\PP$ is given by taking a draw from $\PP^*$ and coarsening it by $\Ccal$. The data then consists of $n$ independent draws, $(X_i,A_i,Y_i)\sim\PP$ for $1\leq i\leq n$. We use $\EE^*,\EE$ to denote expectations with respect to $\PP^*,\PP$, respectively. Note that both have the same $(X,A)$-distribution, so anything involving only $(X,A)$ will be the same. For any function $f$ of $X,A,Y$ we let $\|f\|_p$ be the $L_p$-norm with respect to $\PP$.
We assume throughout that all endogeneity is explained by $X$, known as unconfoundedness: $\PP^*(A=1\mid X)=\PP^*(A=1\mid Y^*(0), X)=\PP^*(A=1\mid Y^*(1), X)$. That is, controlling for $X$, treatment assignment is independent of an individual's idiosyncrasies as relevant to their potential outcomes. Randomized experiments ensure this by design, often with $X\indep A$ (complete randomization). Our results extend to observational settings assuming unconfoundedness, which is a common assumption in this setting imbens2015causal. For our purposes, the only technical difference between the experimental and observational settings is whether or not we exactly know the propensity score, defined as $e(X)=\PP\prns{A=1\mid X}.$ We assume throughout that $e(X)\in(0,1)$ almost surely, known as overlap. Note that assuming $Y=Y^*(A)$ also encapsulates non-interference rubin1986comment.
In the following it will also be useful to define the conditional mean of potential outcomes:
where the last equality follows from unconfoundedness and overlap, showing $\mu$ only depends on $\PP$. We also define the conditional average treatment effect (CATE) and sum (CATS) functions as
The primary parameter of interest we consider is the fraction negatively affected (FNA) by a wholesale change from treatment 0 to treatment 1: $${\mathrm{FNA}}=\PP^*\prns{{\mathrm{ITE}}<0}=\PP^*\prns{Y^*(0)=1,\,Y^*(1)=0}.$$ In particular, this quantity stands in contrast to the average treatment effect, ${\mathrm{ATE}}=\EE^*[{\mathrm{ITE}}]=\EE^*[Y^*(1)]-\EE^*[Y^*(0)].$ A zero or positive ATE is generally interpreted to indicate a neutral or favorable treatment. However, having ${\mathrm{ATE}}\geq0$ need not mean having ${\mathrm{FNA}}=0$. A simple example is $(Y(0),Y(1))\sim\operatorname{Unif}(\{0,1\}^2)$, which has ${\mathrm{ATE}}=0$ but ${\mathrm{FNA}}=1/4$.
Given we observe baseline covariates, we can more generally consider personalized treatment policies that assign treatments based on the value of $X$. Let $\pi_0,\pi_1:\Xcal\to\{0,1\}$ denote any two such policies and let us define the fraction harmed by a change from $\pi_0$ to $\pi_1$ as $${\mathrm{FNA}}_{\pi_0\to\pi_1}=\PP^*\prns{(\pi_1(X)-\pi_0(X))\cdot{\mathrm{ITE}}<0}=\PP^*\prns{Y^*(\pi_0(X))=1,\,Y^*(\pi_1(X))=0}.$$
This more general construct gives rise to several parameters of interest:
In this section, we derive sharp bounds on our parameters of interest. For the sake of generality we focus on the parameter ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ as it captures all of our parameters of interest using different $\pi_0,\pi_1$. The sharp bounds describe the set of values that ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ can take if all we know is the distribution of the data, $\PP$. This characterizes the most we can hope to learn at the limit of infinite data since this distribution is the most we can learn from draws from it, that is, our data.
To define this formally, recall that $\PP=\PP^*\circ\Ccal^{-1}$ is given by $\PP^*$ via coarsening. However, this mapping from $\PP^*$ to $\PP$ need not be invertible. Given a parameter $\psi(\PP^*)$ that depends on the full distribution $\PP^*$, we define its identified set as the set of all values consistent with $\PP$, that is, that could be explained by some $\PP^*$ that gives rise to the given data distribution and satisfies unconfoundedness: $$ \Scal(\psi;\PP)=\braces{\psi(\PP^*)~~:~~ \PP^*\circ\Ccal^{-1}=\PP,~\PP^*(A=1\mid X)=\PP^*(A=1\mid X,Y^*(a)),\,a=0,1}. $$ We are interested in the identified set of and sharp bounds on the parameter ${\mathrm{FNA}}_{\pi_0\to\pi_1}$.
When $\abs{\Scal(\psi;\PP)}=1$, we say that $\psi$ is identifiable because it is uniquely specified by the distribution of the data. Otherwise, we say it is unidentifiable: even given infinite data (equivalently, the distribution of the data), we cannot determine the value of $\psi(\PP^*)$ based on the data alone since many values are consistent with the observations. For example, ATE is identifiable because ${\mathrm{ATE}}=\EE[\mu(X,1)]-\EE[\mu(X,0)]$ and (ref) shows that $\mu(X,a)=\EE[Y\mid X,A=a]$ depends on $\PP$ alone.
Is ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ identifiable? Our first result characterizes exactly when, showing that generally, no.
(ref) shows that only in degenerate cases is ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ identifiable. Generally, there will exist a non-negligible (positive probability) set of $X$'s where the policies $\pi_0,\,\pi_1$ differ and neither conditional outcome distribution is constant. Then, by (ref), ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ would be unidentifiable.
We are now prepared to state our main theorem characterizing the sharp bounds on ${\mathrm{FNA}}_{\pi_0\to\pi_1}$. Since ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ is generally unidentifiable, the best we can do using the data is measure the set $\Scal({\mathrm{FNA}}_{\pi_0\to\pi_1};\PP)$. The sharp lower and upper bounds are $\inf(\Scal({\mathrm{FNA}}_{\pi_0\to\pi_1};\PP))$ and $\sup(\Scal({\mathrm{FNA}}_{\pi_0\to\pi_1};\PP))$, respectively, as these are precisely the largest (resp., smallest) lower (resp., upper) bounds on the set of possible values. The following result gives the sharp bounds and moreover shows the identified set is closed and convex so it is in fact an interval with the bounds as its endpoints.
The proof of (ref) proceeds by applying the Fr\'echet-Hoeffding bounds frechet1935generalisation,ruschendorf1981sharpness for each level of $X$ and showing this remains sharp. (ref) simplify a lot for the change from all-0 to all-1:
We can also simplify when considering the optimal policy. Plugging in the optimal policy we get:
Unlike most policies, it is always plausible that the optimal policy affects no one negatively: because ${\mathrm{FNA}}^-_{1-\pi^*\to\pi^*}=0$, we always have $0\in\Scal({\mathrm{FNA}}_{1-\pi^*\to\pi^*};\PP)$. On the other hand, it is generally also plausible that it does, that is, ${\mathrm{FNA}}^+_{1-\pi^*\to\pi^*}$ is generally nonzero. In agreement with (ref), it is in fact zero only when $\mu(X,0)\in\{0,1\}$ or $\mu(X,1)\in\{0,1\}$ for almost all $X$.
In the previous section we characterized the sharp (meaning, tightest possible) bounds on FNA given the distribution of the data, that is, the best we could do if we had infinite data. We now address how to estimate these from actual data and characterize the uncertainty. That is, how to do inference on the parameters ${\mathrm{FNA}}_{\pi_0\to\pi_1}^\pm$. Recall our data is $n$ independent draws $(X_i,A_i,Y_i)\sim\PP$ for $i=1,\dots,n$.
In order to systematically handle all of parameters of interest, we will develop a generic, robust method for a large class of parameters that include all of our parameters of interest as special cases.
We consider a class parameters we call Average Hinge Effects (AHEs), parametrized by a positive integer $m\in\NN$, a vector of signs $\rho\in\{-1,+1\}^m$, and a set of $3(m+1)$ functions $g_\ell(x)=(g_\ell\s{0}(x),g_\ell\s{1}(x),g_\ell\s{2}(x))\in[-1,1]^3$ for $\ell=0,\dots,m$:
The AHE is so called because $\min\{0,x\}$ is called the hinge function. AHEs cover all of our parameters of interest: ${\mathrm{FNA}}^{-}={\mathrm{AHE}}^{(-)}_{(0,0,0),(-1,1,0)}$, ${\mathrm{FNA}}^{+}={\mathrm{AHE}}^{(+)}_{(1,0,0),(-1,-1,1)}$, ${\mathrm{FNA}}^{-}_{\pi_0\to\pi_1}={\mathrm{AHE}}^{(-)}_{(0,0,0),(\pi_0-\pi_1,\,\pi_1-\pi_0,\,0)}$, ${\mathrm{FNA}}^{+}_{\pi_0\to\pi_1}={\mathrm{AHE}}^{(+)}_{(\pi_0(1-\pi_1),\,\pi_1-\pi_0,0),\,(\pi_0-\pi_1,\,\pi_0-\pi_1,\,-1)}$, and ${\mathrm{FNA}}^+_{1-\pi^*\to\pi^*}={\mathrm{AHE}}^{(+,+)}_{(1,0,0),(-1,1,0),(-1,-1,1)}$. For now, we focus on any given $m,\rho,g_0,\dots,g_m$.
The AHE is defined as an average of a function of $X$. The only unknown part of this function is $\mu(x,a)$. A na\"ive approach would be to estimate $\mu$ by $\hat\mu$, plug it into (ref) and replace $\E$ by a sample average over $X_1,\dots,X_n$. This estimate, however, will not behave like a sample average approximation of (ref). In particular, it has a nonzero derivative in $\hat\mu$, so that the errors in $\hat\mu$ directly translate to errors in AHE estimation. For example, even if $\hat\mu$ converges as fast as $O_p(1/\sqrt{n})$, this will introduce significant errors on the same order as the convergence of sample averages, which can imperil both efficiency and inference. And, if it converges more slowly, as would generally be the case when using flexible machine learning methods for estimating (as parametric ones will inevitably be misspecified), it can even affect the convergence rate of the final estimate. Worst yet, if $\hat\mu$ is inconsistent for $\mu$ we have no hope of consistency for our estimate or an understanding of the direction of its bias. Generally, we would like to avoid depending on how exactly we fit $\hat\mu$, provide guarantees even when it is estimated slowly and regardless of what method is used to estimate it, and even provide guarantees when its estimation is inconsistent.
Our approach is based on focusing on an alternative formulation of AHE as a sample average where plugging in wrong values for some unknown functions does not affect the formulation too much.
Toward that end, let us first define $\eta_\ell:\Xcal\to[-3,3]$ by
Specifically, we have $\eta_1(X)=(\pi_1(X)-\pi_0(X))\tau_-(X)$ for ${\mathrm{FNA}}_{\pi_0\to\pi_1}^-$, $\eta_1(X)=(\pi_1(X)-\pi_0(X))(1-\tau_+(X))$ for ${\mathrm{FNA}}_{\pi_0\to\pi_1}^+$, and $\eta_1(X)=\tau_-(X),\,\eta_2(X)=1-\tau_+(X)$ for ${\mathrm{FNA}}_{1-\pi^*\to\pi^*}^+$.
Given $\check e,\check \mu,\check\eta_{1},\ldots,\check\eta_{m}$, understood as (possibly wrong) stand-ins for $e, \mu,\eta_{1},\ldots,\eta_{m}$, define
If we plug in the correct values of the nuisance parameters $e, \mu,\eta_{1},\ldots,\eta_{m}$, then we would have
Crucially, unlike the definition of AHE as an average in (ref), we will show in (ref) that this new representation as an average remains faithful even when we make small errors in the nuisances.
Given our special construction of $\phi_{g_0,\dots,g_m}^\rho$, our algorithm, presented in (ref), follows a simple recipe: we estimate the nuisances in a cross-fold fashion and plug them into (ref), approximating the expectation by a sample average. In (ref) we will show that the special structure of $\phi_{g_0,\dots,g_m}^\rho$ affords this procedure a lot of robustness. The use of cross-fitting ensures nuisance estimates are independent of samples applied thereto schick1986,doubleML,zheng2011cross. This allows our analysis to only require lax rate assumptions on the nuisance fitting without requiring any additional regularity on restricting how the fitting is done. (If we assume nuisance estimates belong to a Donsker class with probability tending to 1, all of our results will hold even without cross-fitting; we omit this option for brevity.)
Our algorithm requires methods for fitting the nuisances $e, \mu,\eta_{1},\ldots,\eta_{m}$. Fitting $e$ and $\mu$ amounts to binary regressions (probabilistic classification), which can be done with any of a variety of standard supervised learning methods, whether logistic regression, random forests, or neural nets.
There are different ways to fit $\eta_\ell$. In particular, despite the fact that $\eta_\ell$ is determined by $\mu$, we treat it as a separate nuisance to allow for specialized learners. Of course, the simplest way to fit it is to simply plugin an estimate for $\mu$ into the definition of $\eta_\ell$ in (ref), and that is a legitimate option. For all of our parameters of interest, all $\eta_\ell$ nuisances are given by learning just two functions: $\tau_-$ and $\tau_+$ as in (ref). Therefore, we may alternatively learn these directly and plug them into $\eta_\ell$. For example, there exists a wide literature specialized to learning $\tau_-$ motivated by the observation that effect signals are often more nuanced and can therefore be washed out by the noise in baselines if we simply difference $\mu$-estimates slearner,xlearner,drlearner,rlearner,causaltree,causalforest. We can similarly apply of all these approaches to learning $\tau_+$ by simply first mutating the outcome data as $Y\gets (2A-1)Y$. Notably, drlearner,rlearner give rates of convergence for $\tau_-$-learning, which we can use to satisfy our assumptions in the next section.
We now show our inference method has some nice robustness guarantees. This will depend on showing that using our special $\phi$ renders (ref) insensitive to errors in the nuisances. The specific level of errors allowed will depend on the sharpness of margin satisfied by $\eta_\ell$, as defined below.
Every $f$ trivially satisfies a 0-margin, but usually a sharper margin holds. If $f(X)$ is almost surely either zero or bounded away from zero, then $f$ satisfies an infinitely sharp margin:
In particular, if $X$ has finite support, then we can always set $t_0=\inf\{\abs{f(x)}:x\in\Xcal,f(x)\neq0\}$. But, this also works more generally. Consider $\eta_1(X)=(\pi_1(X)-\pi_0(X))\tau_-(X)$ for ${\mathrm{FNA}}_{\pi_0\to\pi_1}^-$. If $\inf\{\abs{\tau_-(x)}:\pi_1(x)\neq\pi_0(x),\tau_-(x)\neq0\}>0$, that is, the policies only differ either far from the boundary where $\tau_-$ is 0 or exactly on this boundary, then (ref) ensures sharpness $\infty$.
More generally, we should expect a margin with sharpness 1 in continuous settings.
Consider again the example of $\eta_1(X)=(\pi_1(X)-\pi_0(X))\tau_-(X)$. If $\tau_-(X)$ is a continuous random variable with a bounded density near $0$, then (ref) ensures a margin with sharpness 1. This holds for any two policies, including $\pi_1=1,\,\pi_0=0$, which differ everywhere. The same conclusion would also hold if the CDF of $\tau_-(X)$ was just continuously differentiable at $0$.
Armed with the notion of margin, we can state in what sense the representation (ref) is robust.
(ref) has several important implications. Our first result shows that, if we estimate all nuisances correctly but potentially slowly at nonparametric rates, then we can ensure our estimator looks like a sample average approximation of (ref) and our CIs are calibrated.
The conditions of (ref) allow that we learn $e,\mu,\eta_1,\ldots,\eta_m$ rather slowly. In particular, unlike na\"ively plugging in a $\mu$-estimate into (ref) and taking sample averages, we do not depend at all on how $\mu$ is estimated and the uncertainty therein, provided some slow nonparametric rates hold. For example, learning $e,\mu$ in with $o_p(n^{-1/4})$-rates in $L_2$-error suffices. Or, if $e$ is known, then consistently estimating $\mu$ without a rate suffices. For $\eta_\ell$, if an $\infty$-margin holds, it suffices to learn $\eta_\ell$ consistently (no rate) in $L_\infty$-error or at $o_p(n^{-1/4})$-rates in $L_2$-error. If a $1$-margin holds, it suffices to learn at $o_p(n^{-1/4})$-rates in $L_\infty$-error. While conditions like metric entropy imply $L_2$-error rates wainwright2019high, for many classes like smooth functions the $L_p$ error rates are even the same for all $p\in[1,\infty]$ stoneglobal.
In fact, under one more condition, our estimator achieves the semiparametric efficiency lower bound.
For all of our parameters of interest, the condition of (ref) holds if $\PP\prns*{\tau_-(X)=0}=\PP\prns*{\tau_+(X)=0}=0$ (\eg, $\tau_-(X),\tau_+(X)$ are continuous). Similarly, the condition that $\abs*{\hat\eta_\ell\s k(x)}\leq 3\II[\eta_\ell(x)\neq0]$ in (ref) is immediately satisfied if $\hat\tau_-\s k(X),\hat\tau_+\s k(X)$ are also almost never 0 (\eg, continuous).
Next we show that (ref) is very robust to incorrectly learning some nuisances.
The first equation (double robustness) in (ref) shows our estimator remain consistent for the AHE even if we incorrectly learn either $e$ or $\mu$, bot not both, as long as we correctly learn $\eta_1,\dots,\eta_m$.
The second two equations (double validity) in (ref) show what happens when we also learn $\eta_\ell$ incorrectly. For upper double validity (second equation), we show that, as long as the $\eta_\ell$ that are misestimated correspond to convex terms in the AHE ($\rho_\ell=+$), we remain consistent for an upper bound on the AHE, even if we also incorrectly learn either $e$ or $\mu$, bot not both. In particular, for ${\mathrm{FNA}}^+$ we only have convex terms, so we are guaranteed and upper bound on the upper bound, that is, we still estimate a valid upper bound on ${\mathrm{FNA}}$ when we misestimate the CATE $\tau_-$ (which is $\eta_1$ for ${\mathrm{FNA}}^+$) and one of $e$ or $\mu$, it just may not be sharp. We also have a symmetric result for lower double validity (third equation), and applying it to ${\mathrm{FNA}}^-$ we find that we will still estimate a valid upper bound, albeit possibly unsharp, on ${\mathrm{FNA}}$ when we misestimate $\tau_+$ and one of $e$ or $\mu$. The significance is that we can really trust the results of (ref) are truly bounds on FNA, even if we make mistakes, and therefore conclusions about harm based on our estimates and inferences can be highly credible.
We demonstrate our method in a simulation and a case study with data from a real experiment. Replication code is given in the supplement. Experiments were run on an AWS c5.24xlarge instance.
Simulation Study\quad We consider $7$-dimensional standard normal $X$ and set $\mu(x,a)=(1+\exp(-\beta(2\II[\xi(x)x_2>0]-1)(\II[\xi(x)x_1\leq0]+(2a-1)\II[\xi(x)x_1>0])))^{-1}$, where $\beta\geq0$ and $\xi(x)=2\operatorname{XOR}(\II[x_3>0],\ldots,\II[x_7>0])-1$. Here $\beta$ controls how much $X$ predicts $Y(a)$. The dashed lines in (ref) show the sharp bounds of (ref) for ${\mathrm{FNA}}$ and ${\mathrm{FNA}}_{1-\pi^*\to\pi^*}$ as we vary $\beta$. We can see that, as $X$ becomes more predictive, the bounds go from $[0,\,0.5]$ to a point at $0.25$. Note there is no true value for ${\mathrm{FNA}}$ as we are not specifying the joint distribution of potential outcomes; instead, per (ref), there are joint distributions with ${\mathrm{FNA}}$ equal to any point between the bounds.
Next, we estimate these true bounds. We fix $e(X)=(1+\exp(0.25-\II[x_3>0]+0.5\II[x_4>0])^{-1}$, and draw $n$ samples from $X\sim\mathcal N(0,I)$, $A\mid X\sim \operatorname{Bernoulli}(e(X))$, $Y\mid X,A\sim\operatorname{Bernoulli}(\mu(X,A))$. We apply (ref) to this data with $K=5$, estimating $e,\,\mu$ using random regression forests and $\tau_-,\,\tau_+$ using causal forests (all using R package grf with default parameters). The solid lines in (ref) show the point estimate and 95%-CI output by (ref) for a single draw of $n=12800$ data for each $\beta$. In (ref) we plot the average root mean squared error of point estimates and coverage of CIs over $100$ replications with $\beta=3$ and varying $n$. We compare the performance to the (cross-fitted) “plugin” estimator that just plugs in the (same) estimates for $\mu$ into (ref) and approximate the expectation by a sample average. Shaded intervals denote 95%-CIs for performance (\ie, due to finite replications).
Case Study\quad Using data from behaghel2014private (BSD license), we compare three different assistance programs offered to French unemployed individuals: the standard benefits (sta), access to public-run counseling (pub), and to private-run counseling (pri). Our binary outcome is reemployment within six months. For example, we can consider a hypothetical scenario where we change from a private to a public counseling provider. As reported by behaghel2014private, the ATE for this hypothetically change is slightly positive, so a policymaker might therefore enact it, especially if, for example, it offered cost savings or other operational benefits. We consider who may be harmed by this: how many who remain unemployed under the new public program that would have been reemployed under the old private program.
For any pair of arms, we consider the sharp bounds on the FNA from one to other or the other way and on the misclassification rate of the optimal policy choosing between the two. Following kallus2022treatment, we set $X$ to all pre-treatment covariates in table 2 of behaghel2014private., and we consider the sharp bounds either with these $X$ or given no $X$ at all. We fit $\mu$ using linear regression and $\tau_-,\tau_+$ using doubly-robust pseudo-outcome linear regression. Since the propensity is known, in light of the guarantees of (ref), we do not worry about misspecifying $\mu$, only using it for variance reduction by accounting for main effects, and we do not worry about misspecifying $\tau_-,\tau_+$, only hoping to tighten the bounds some from using no $X$. The results are shown in (ref). We see that, while covariates are somewhat uninformative, we are still able to tighten bounds compared to no $X$. In particular, for the change from sta to pri or pub or the change from pri to pub, without $X$ the lower bounds on FNA are 0, while with $X$ the lower bounds as well as the lower confidence bounds on these bounds are strictly positive. This means that, with $X$ and the methods developed we can prove that there must be some harm by this change, something we could not do otherwise and such a finding can bolster further work to investigate and address this harm.
Connections to partial identification\quad Partial identification of unknowable parameters has a long tradition in econometrics tamer2010partial,manski1997mixing,manski2003partial,manski1995identification,molinari2020microeconometrics,semenova2017debiased,chernozhukov2007estimation,king2013solution. Some approaches focus on average treatment effects in the presence of confounding cross2002regressions,dorn2021doubly,kallus2019interval,rosenbaum,tan2006,yadlowsky2018bounds,bonvini2021sensitivity,chernozhukov2021omitted, among which dorn2021doubly,yadlowsky2018bounds,bonvini2021sensitivity,chernozhukov2021omitted are notable for conducting semiparametric inference on bounds, but for confounding rather than for individual effects, as we do. Some others, like us, focus on understanding the joint distribution of potential outcomes heckman1997making,ding2019decomposing,abbring2007econometric,manski1997monotone,firpo2008bounds. Like us, these heavily leverage the Fr\'echet-Hoeffding bounds ruschendorf1981sharpness,frechet1935generalisation,cambanis1976inequalities,williamson1990probabilistic,ridder2007econometrics. Unlike these, we use efficient and robust inference that can aggregate across covariates to tighten bounds by leveraging machine learning of conditional-mean outcome functions.
Connections to fairness\quad Understanding the distribution of impact is a central problem in algorithmic fairness mitchell2021algorithmic,barocas-hardt-narayanan. Here, we specifically focused on the differential impact of interventions, which brought up issues of identification. In contrast, in algorithmic fairness one usually measures disparities in observed outcomes, such as in the form of the loss function of a model on a labeled example kearns2018preventing,bird2020fairlearn,ghosh2021characterizing,agarwal2018reductions. A line of work specifically focuses on identification issues in algorithmic fairness kallus2021assessing,chen2019fairness,kallus2019assessing,kallus2018residual,fogliato2020fairness,lakkaraju2017selective,coston2021characterizing,kleinberg2018human. Notably, kallus2021assessing also conduct semiparametric inference on sharp bounds derived from the Fr\'echet-Hoeffding bounds, but in the context of algorithm evaluation with unobserved protected labels rather than interventions with unobserved counterfactuals. And, kallus2019assessing do consider unobserved counterfactuals in assessing equality of opportunity hardt2016equality for intervention-prioritization policies, but (appropriately in their own context) they assume away our primary focus here by assuming an a priori known bound on FNA (possibly zero, \ie, monotone treatment response).
Limitations and Extensions\quad Restricting outcomes to be binary is one limitation of our work, and extensions to continuous settings is an interesting avenue of future research. Another limitation is that, while we can bound the FNA, it can still be hard to asses who is negatively affected. This is, unfortunately, impossible for the same reason FNA is unidentifiable, but a place to start such analyses may be to consider who are individuals with $\tau_-(X)\leq 0$ and even characterize that group via summary statistics compared to the population. Indeed, per (ref), the lower bound on FNA is exactly the (negative of the) ATE on this group. Another important consideration is whether the data is representative: FNA refers to the fraction of the studied population, which might differ from the population of interest. \Eg, if an experiment did not enroll a representative sample, we may be systematically excluding some groups from consideration. If such unrepresentativeness is explained by covariates $X$ (\ie, missing at random), the solution is simple: we reweight. If not explained by $X$ (\ie, missing not at random), then we need to also account for this additional source of unidentifiability. An avenue for future research is to combine such ambiguity with counterfactual ambiguity. Another concern is whether the outcome represents the impact we want to measure passi2019problem.
Conclusions\quad Our tools support drawing credible conclusions about the potential negative impact of interventions: they both account for ambiguity due to unobserved counterfactuals (while mitigating it using covariates) as well as strongly guard against slow or inconsistent learning of necessary nuisances functions. Robust inference on the lower bounds, in particular, crucially provides watertight demonstrations of negative impact, which can bolster efforts to mitigate harm and improve equity.
I am grateful for the helpful comments of the anonymous reviewers and for many insightful and thought-inspiring conversations with my colleagues at Netflix.