The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
55,079 characters
What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment
\maketitle
\begin{abstract}
The fundamental problem of causal inference -- that we never observe counterfactuals -- prevents us from identifying how many might be negatively affected by a proposed intervention. If, in an A/B test, half of users click (or buy, or watch, or renew, \etc), whether exposed to the standard experience A or a new one B, hypothetically it could be because the change affects no one, because the change positively affects half the user population to go from no-click to click while negatively affecting the other half, or something in between. While unknowable, this impact is clearly of material importance to the decision to implement a change or not, whether due to fairness, long-term, systemic, or operational considerations. We therefore derive the \emph{tightest-possible} (\ie, sharp) bounds on the fraction negatively affected (and other related estimands) given data with only factual observations, whether experimental or observational. Naturally, the more we can stratify individuals by observable covariates, the tighter the sharp bounds. Since these bounds involve unknown functions that must be learned from data, we develop a robust inference algorithm that is efficient almost regardless of how and how fast these functions are learned, remains consistent when some are mislearned, and still gives valid conservative bounds when most are mislearned. Our methodology altogether therefore strongly supports credible conclusions: it avoids spuriously point-identifying this unknowable impact, focusing on the best bounds instead, and it permits exceedingly robust inference on these. We demonstrate our method in simulation studies and in a case study of career counseling for the unemployed.
\end{abstract}
\section{Introduction}
Before making changes to an online platform, product managers regularly conduct experiments (``A/B tests'') to make sure a change does not negatively impact users' experience. Similarly, extensive program evaluations, whether experimental or observational, are a corner stone of evidence-based policymaking and help avoid harmful policy changes. These tests and evaluations focus on assessing the causal effect of a change on key metrics, whether user engagement or retention on online platforms or outcomes like employment for social programs. Average treatment effects, heterogeneous treatment effects, and distributional treatment effects on these outcomes quantify the quality of the intervention in terms of aggregate impacts on the population or on subpopulations. This is in its essence a statistical solution to the \emph{fundamental problem of causal inference}: assess aggregate effects since the impact on any one individual can never be observed. We can never know whether a user that churned under arm ``B'' would have been retained under arm ``A,'' but we can characterize the distribution of retention under each arm, even segmenting the population by observable features.
While unknowable, the impact of changes on individuals is clearly of material importance to the decision to implement a change or not.
This individual impact can have downstream effects, whether user-behavioral, reputational, or operational.
Moreover, quantifying how negative this impact can be is crucial to understanding fairness with respect to the welfare of the individuals, rather than just focusing on social (\ie, average) welfare, the welfare of the decision-making entity, or the welfare of observable groups.
Specifically, even if the average treatment effect is zero, there can still be a sizable subpopulation that is negatively affected by the change.
That is, a change with zero average effect cannot in earnest be called ``harmless."
In this paper we consider measuring the fraction that are negatively affected.
The aim is to be able to flag changes that may individually harm many.
Unfortunately this fraction cannot be identified from data, even experimental data, but we may still be able to bound it.
For example, if, in an A/B test, half of users click (or buy, or watch, or renew, etc.) whether exposed to the standard experience ``A" or a new one ``B," hypothetically it could be because the change affects no one, because the change positively affects half the user population to go from no-click to click while negatively affecting the other half, or anything in between. Such a large bound on the range of possibilities, however, is very crude and uninformative.
In this paper, we derive the \emph{sharp} bounds on the fraction of individuals negatively affected by treatment using all baseline features available, and we develop tools to conduct robust inferences on these bounds. That is, we characterize the \emph{tightest-possible} interval that contains all values of this unknowable quantity consistent with all observable information, in particular crucially leveraging unit feature information. We consider both the fraction negatively affected under a wholesale change from one treatment regime to another as well as under an optimal policy that assigns each observable subpopulation the treatment with best average outcomes for the subpopulation. Then, we tackle how to actually assess these theoretical bounds based on actual data, namely, how to estimate the interval endpoints and construct confidence intervals (CIs) that account for sampling error, on top of the inherent epistemic uncertainty characterized by the bounds. The bounds involve potentially complex functions such as the conditional average treatment effect (CATE) function, so we develop inference methodology that is exceedingly robust to learning these functions. Namely, our inferences are valid and calibrated even when these complex functions are learned nonparametrically at slow rates, and our estimated bounds remain valid, albeit conservative, even when these are learned inconsistently, so that conclusions as to the presence of harm based on our tools can be highly credible. We demonstrate our tools
in a simulation study and in a case study of French reemployment assistance programs.
\section{Problem Set Up and the Fraction Negatively Affected}
\textbf{The Population}\quad
Each individual in the population is associated with baseline \textbf{covariates} (observable characteristics), $X\in\Xcal$, and two \emph{binary} \textbf{potential outcomes}, $Y^*(0),\,Y^*(1)\in\{0,1\}$, corresponding to each of two treatment options, $0$ and $1$.
Examples include:
two versions of product on an online platform and whether user is retained in the next quarter,
or two versions of a job training program for the unemployed and whether the participant is reemployed within the following year.
We will consider the generalization to non-binary outcomes in \cref{remark:nonbin,thm:general}.
The \textbf{individual treatment effect} is the difference in potential outcomes: ${\mathrm{ITE}}=Y^*(1)-Y^*(0).$\break
We assume that an outcome value of $1$ corresponds to a better outcome (\eg, retained, reemployed) and $0$ to a worse outcome (\eg, churned, unemployed).
Thus, if treatment 0 represents status quo and treatment 1 represents a proposed change, then individuals with ${\mathrm{ITE}}=0$ ($Y^*(0)=Y^*(1)=0$ or $Y^*(0)=Y^*(1)=1$) are unaffected by the change, individuals with ${\mathrm{ITE}}=1$ ($Y^*(0)=0,\,Y^*(1)=1$) benefit from the change, and individuals with ${\mathrm{ITE}}=-1$ ($Y^*(0)=1,\,Y^*(1)=0$) are harmed by the change. This list of potential-outcome combinations is exhaustive.
\textbf{The Data}\quad
We consider data from either a randomized experiment or an observational study, wherein we never observe both $Y^*(0)$ and $Y^*(1)$ simultaneously. Each individual is associated with a \textbf{treatment} $A\in\{0,1\}$, and we observe the \textbf{factual outcome} $Y=Y^*(A)$ corresponding to $A$. We never observe the counterfactual $Y^*(1-A)$. That is, the data consists of observations of $(X,A,Y)$.
In particular, the distribution of the data only reflects part of the full population distribution involving both potential outcomes. Let $\PP^*$ denote the distribution of $(X,A,Y^*(0),Y^*(1))$.
Define the coarsening function $\Ccal:(x,a,y_0,y_1)\mapsto(x,a,ay_1+(1-a)y_0)$, which has the pre-image $\Ccal^{-1}(x,a,y)=\{(x,a,y_0,y_1):ay_0+(1-a)y_1=y\}$. Then the distribution induced on $(X,A,Y)$ is given by $\PP=\PP^*\circ\Ccal^{-1}$ (as measures).
That is, the data distribution $\PP$ is given by taking a draw from $\PP^*$ and coarsening it by $\Ccal$.
The data then consists of $n$ independent draws, $(X_i,A_i,Y_i)\sim\PP$ for $1\leq i\leq n$. We use $\EE^*,\EE$ to denote expectations with respect to $\PP^*,\PP$, respectively. Note that both have the same $(X,A)$-distribution, so anything involving only $(X,A)$ will be the same. For any function $f$ of $X,A,Y$ we let $\|f\|_p$ be the $L_p$-norm with respect to $\PP$.
We assume throughout that all endogeneity is explained by $X$, known as unconfoundedness: $\PP^*(A=1\mid X)=\PP^*(A=1\mid Y^*(0), X)=\PP^*(A=1\mid Y^*(1), X)$. That is, controlling for $X$, treatment assignment is independent of an individual's idiosyncrasies as relevant to their potential outcomes.
Randomized experiments ensure this by design, often with $X\indep A$ (complete randomization).
Our results extend to observational settings assuming unconfoundedness, which is a common assumption in this setting \citep{imbens2015causal}.
For our purposes, the only technical difference between the experimental and observational settings is whether or not we exactly know the \textbf{propensity score}, defined as
$e(X)=\PP\prns{A=1\mid X}.$
We assume throughout that $e(X)\in(0,1)$ almost surely, known as overlap.
Note that assuming $Y=Y^*(A)$ also encapsulates non-interference \citep{rubin1986comment}.
In the following it will also be useful to define the \textbf{conditional mean of potential outcomes}:
\begin{equation}\label{eq:muid}\mu(X,a)=\EE^*[Y^*(a)\mid X]=\EE[Y\mid X,A=a],\quad a=0,1,\end{equation}
where the last equality follows from unconfoundedness and overlap,
showing $\mu$ only depends on $\PP$.
We also define the \textbf{conditional average treatment effect (CATE) and sum (CATS) functions} as
\begin{align}\label{eq:cate}
\tau_-(X)=\EE^*[{\mathrm{ITE}}\mid X]=\mu(X,1)-\mu(X,0),\quad
\tau_+(X)=\mu(X,1)+\mu(X,0).
\end{align}
\subsection{Parameters of Interest}
The primary parameter of interest we consider is the \textbf{fraction negatively affected (FNA)} by a wholesale change from treatment 0 to treatment 1:
$${\mathrm{FNA}}=\PP^*\prns{{\mathrm{ITE}}<0}=\PP^*\prns{Y^*(0)=1,\,Y^*(1)=0}.$$
In particular, this quantity stands in contrast to the average treatment effect,
${\mathrm{ATE}}=\EE^*[{\mathrm{ITE}}]=\EE^*[Y^*(1)]-\EE^*[Y^*(0)].$
A zero or positive ATE is generally interpreted to indicate a neutral or favorable treatment. However, having ${\mathrm{ATE}}\geq0$ need not mean having ${\mathrm{FNA}}=0$. A simple example is $(Y(0),Y(1))\sim\operatorname{Unif}(\{0,1\}^2)$, which has ${\mathrm{ATE}}=0$ but ${\mathrm{FNA}}=1/4$.
Given we observe baseline covariates, we can more generally consider personalized treatment policies that assign treatments based on the value of $X$. Let $\pi_0,\pi_1:\Xcal\to\{0,1\}$ denote any two such policies and let us define the fraction harmed by a change from $\pi_0$ to $\pi_1$ as
$${\mathrm{FNA}}_{\pi_0\to\pi_1}=\PP^*\prns{(\pi_1(X)-\pi_0(X))\cdot{\mathrm{ITE}}<0}=\PP^*\prns{Y^*(\pi_0(X))=1,\,Y^*(\pi_1(X))=0}.$$
This more general construct gives rise to several parameters of interest:
\begin{enumerate}[leftmargin=*]
\item First, we recover our primary parameter of interest, ${\mathrm{FNA}}={\mathrm{FNA}}_{0\to1},$ where $0$ (or, $1$) stands for the constant policy taking value $0$ (or, $1$) everywhere.
\item Second, given a proposed personalized treatment policy $\pi:\Xcal\to\{0,1\}$, the quantity ${\mathrm{FNA}}_{0\to\pi}$ is the fraction negatively affected by changing from a status quo of 0 to deploying the proposed policy $\pi$, which intervenes to treat the subpopulation with covariates $X$ such that $\pi(X)=1$.
\item Third, we can recover the misclassification rate of the policy $\pi$: $$\ts{\mathrm{FNA}}_{1-\pi\to\pi}=\PP^*\prns*{\ts\pi(X)\notin\argmax_{a\in\{0,1\}}Y^*(a)}.$$ This represents the fraction of individuals negatively affected by being misclassified by $\pi$: they should be treated by one treatment but are being given the other.
\item Of particular interest is the misclassification rate of the \emph{optimal} personalized treatment policy, which assigns the treatment with larger conditional average outcome:
$$\ts\pi^*(X)\in\argmax_{a\in\{0,1\}}\EE^*\bracks{Y^*(a)\mid X}.$$
We are interested in ${\mathrm{FNA}}_{1-\pi^*\to\pi^*}$.
The policy $\pi^*$ can equivalently be characterized as the policy \emph{maximizing} social welfare, $\EE^*\bracks{Y^*(\pi(X))}$, or as \emph{minimizing} misclassification, ${\mathrm{FNA}}_{1-\pi\to\pi}$, both over \emph{all} policies $\pi$.
Given this, one hopes but cannot rule out that $\pi^*$ avoids negative impact.
\end{enumerate}
\section{Sharp Bounds}\label{sec:sharp}
In this section, we derive sharp bounds on our parameters of interest. For the sake of generality we focus on the parameter ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ as it captures all of our parameters of interest using different $\pi_0,\pi_1$. The sharp bounds describe the set of values that ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ can take if all we know is the distribution of the data, $\PP$. This characterizes the most we can hope to learn at the limit of infinite data since this distribution is the most we can learn from draws from it, that is, our data.
To define this formally, recall that $\PP=\PP^*\circ\Ccal^{-1}$ is given by $\PP^*$ via coarsening. However, this mapping from $\PP^*$ to $\PP$ need not be invertible.
Given a parameter $\psi(\PP^*)$ that depends on the full distribution $\PP^*$, we define
its \emph{identified set} as the set of all values consistent with $\PP$, that is, that could be explained by some $\PP^*$ that gives rise to the given data distribution and satisfies unconfoundedness:
$$
\Scal(\psi;\PP)=\braces{\psi(\PP^*)~~:~~
\PP^*\circ\Ccal^{-1}=\PP,~\PP^*(A=1\mid X)=\PP^*(A=1\mid X,Y^*(a)),\,a=0,1}.
$$
We are interested in the identified set of and sharp bounds on the parameter ${\mathrm{FNA}}_{\pi_0\to\pi_1}$.
When $\abs{\Scal(\psi;\PP)}=1$, we say that $\psi$ is \emph{identifiable} because it is uniquely specified by the distribution of the data. Otherwise, we say it is \emph{unidentifiable}: even given infinite data (equivalently, the distribution of the data), we cannot determine the value of $\psi(\PP^*)$ based on the data alone since many values are consistent with the observations.
For example, ATE \emph{is} identifiable because ${\mathrm{ATE}}=\EE[\mu(X,1)]-\EE[\mu(X,0)]$ and \cref{eq:muid} shows that $\mu(X,a)=\EE[Y\mid X,A=a]$ depends on $\PP$ alone.
Is ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ identifiable?
Our first result characterizes exactly when, showing that generally, \emph{no}.
\begin{theorem}[Identifiability]\label{thm:id}
${\mathrm{FNA}}_{\pi_0\to\pi_1}$ is identifiable if and only if, for almost all $X$, either $\pi_0(X)=\pi_1(X)$ or $\operatorname{Var}(Y\mid X,A=0)=0$ or $\operatorname{Var}(Y\mid X,A=1)=0$.
\end{theorem}
\Cref{thm:id} shows that only in degenerate cases is ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ identifiable.
Generally, there will exist a non-negligible (positive probability) set of $X$'s where the policies $\pi_0,\,\pi_1$ differ and neither conditional outcome distribution is constant. Then, by \cref{thm:id}, ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ would be unidentifiable.
We are now prepared to state our main theorem characterizing the sharp bounds on ${\mathrm{FNA}}_{\pi_0\to\pi_1}$.
Since ${\mathrm{FNA}}_{\pi_0\to\pi_1}$ is generally unidentifiable, the best we can do using the data is measure the set $\Scal({\mathrm{FNA}}_{\pi_0\to\pi_1};\PP)$.
The \emph{sharp lower and upper bounds} are $\inf(\Scal({\mathrm{FNA}}_{\pi_0\to\pi_1};\PP))$ and $\sup(\Scal({\mathrm{FNA}}_{\pi_0\to\pi_1};\PP))$, respectively, as these are precisely the largest (resp., smallest) lower (resp., upper) bounds on the set of possible values.
The following result gives the sharp bounds and moreover shows the identified set is closed and convex so it is in fact an interval with the bounds as its endpoints.
\begin{theorem}[Sharp Bounds on FNA]\label{thm:bounds}
Fix any $\pi_0,\pi_1:\Xcal\to\{0,1\}$.
Set
\begin{align}
\label{eq:boundslo}{\mathrm{FNA}}^-_{\pi_0\to\pi_1}=\EE[\max\{&(\pi_0(X)-\pi_1(X))\tau_-(X),\,0\}],\\
\label{eq:boundshi}{\mathrm{FNA}}^+_{\pi_0\to\pi_1}=\EE[\min\{&
\pi_1(X)(1-\pi_0(X))\mu(X,0)+\pi_0(X)(1-\pi_1(X))(1-\mu(X,0))
,
\\&
\notag\pi_1(X)(1-\pi_0(X))(1-\mu(X,1))+\pi_0(X)(1-\pi_1(X))\mu(X,1)
\}].
\end{align}
Then, the identified set is the interval
$\Scal({\mathrm{FNA}}_{\pi_0\to\pi_1};\PP)=[{\mathrm{FNA}}^-_{\pi_0\to\pi_1},\,{\mathrm{FNA}}^+_{\pi_0\to\pi_1}]$.
\end{theorem}
The proof of \cref{thm:bounds} proceeds by applying the Fr\'echet-Hoeffding bounds \citep{frechet1935generalisation,ruschendorf1981sharpness} for each level of $X$ and showing this remains sharp.
\Cref{eq:boundslo,eq:boundshi} simplify a lot for the change from all-0 to all-1:
\begin{align}\label{eq:bounds01}
&{\mathrm{FNA}}^-=-\EE[\min\{\tau_-(X),\,0\}],
\quad{\mathrm{FNA}}^+=\EE[\min\{
\mu(X,0),\,1-\mu(X,1)\}].
\end{align}
We can also simplify when considering the optimal policy.
Plugging in the optimal policy we get:
\begin{align}\label{eq:boundsopt}
\hspace{-0.8em}{\mathrm{FNA}}^-_{1-\pi^*\to\pi^*}=0,~
{\mathrm{FNA}}^+_{1-\pi^*\to\pi^*}=
\EE[\min\{\mu(X,0),\,1-\mu(X,0),\,\mu(X,1),\,1-\mu(X,1)\}]
.
\end{align}
Unlike most policies, it is always plausible that the optimal policy affects no one negatively: because ${\mathrm{FNA}}^-_{1-\pi^*\to\pi^*}=0$, we always have $0\in\Scal({\mathrm{FNA}}_{1-\pi^*\to\pi^*};\PP)$. On the other hand, it is generally also plausible that it does, that is,
${\mathrm{FNA}}^+_{1-\pi^*\to\pi^*}$
is
generally nonzero. In agreement with \cref{thm:id}, it is in fact zero \emph{only} when $\mu(X,0)\in\{0,1\}$ or $\mu(X,1)\in\{0,1\}$ for almost all $X$.
\begin{remark}[The Non-Binary Case]\label{remark:nonbin}
We can generalize \cref{thm:bounds} to the non-binary-outcome case as well. Consider now \emph{general} scalar outcomes $Y^*(0),Y^*(1)\in\Rl$, whether continuous, discrete, or mixed. Define the general parameter:
$$\psi_{\zeta,\delta}(\PP^*)=\PP^*(\zeta(X){\mathrm{ITE}}<\delta).$$
Then, ${\mathrm{FNA}}=\PP^*(Y^*(1)<Y^*(0))=\psi_{1,0}$, ${\mathrm{FNA}}_{\pi_0\to\pi_1}=\psi_{\pi_1-\pi_0,0}$, and ${\mathrm{FNA}}_{1-\pi\to\pi}=\psi_{2\pi-1,0}$.
\begin{theorem}\label{thm:general}
Fix $\zeta,\delta$. We then have
\begin{align*}
\sup(\Scal(\psi_{\zeta,\delta};\PP))&\ts=1+\E\inf_y\prns{\PP(\zeta(X)Y< y+\delta\mid X,A=1)-\PP(\zeta(X)Y\leq y\mid X,A=0)},\\
\inf(\Scal(\psi_{\zeta,\delta};\PP))&\ts=\E\sup_y\prns{\PP(\zeta(X)Y< y+\delta\mid X,A=1)-\PP(\zeta(X)Y\leq y\mid X,A=0)}.
\end{align*}
\end{theorem}
We recover \cref{eq:boundslo,eq:boundshi} by min/maximizing over $y\in\{-1,0,0.5,1\}$.
For inference, we focus on the binary case. The non-binary case can be handled similarly but is more complicated as it requires learning the functions $y(X)$ that optimize the above $\inf$ and $\sup$ for each $X$. (Nonetheless, even if we learn the wrong functions we will get valid, albeit not sharp, bounds; see \cref{thm:drdv,lemma:drdvonavg}.)
\end{remark}
\begin{remark}[Tail Expectations of Individual Treatment Effects]
\citet{kallus2022treatment} considers bounds on the conditional value at risk (CVaR) of individual treatment effects (ITEs): for $\alpha\in(0,1]$, $$\ts\operatorname{CVaR}_{\alpha}({\mathrm{ITE}})=\sup_\beta\EE^*\bracks{\beta+\alpha^{-1}\min\{{\mathrm{ITE}}-\beta,\,0\}}.$$ This is equal to the smallest subgroup-ATE among all $\alpha$-sized fractions of the population, that is, the average effect on the $(100\alpha)\%$-worst affected. It is generally unidentifiable from data.
While \citet{kallus2022treatment} considers general real-valued outcomes, in the special case of binary outcomes considered here, $\operatorname{CVaR}_{\alpha}({\mathrm{ITE}})$ is a function of just FNA and ATE:
\begin{lemma}\label{lemma:cvar} For $\alpha\in(0,1)$,
$\operatorname{CVaR}_{\alpha}({\mathrm{ITE}})=\max\{-1,\,-\alpha^{-1}{\mathrm{FNA}},\,1-\alpha^{-1}({\mathrm{ATE}}-1)\}$.
\end{lemma}
As a consequence of \cref{thm:bounds,lemma:cvar}, we immediately have that the identified set for $\operatorname{CVaR}_{\alpha}({\mathrm{ITE}})$ for $\alpha\in(0,1)$ is the
interval
$\Scal(\operatorname{CVaR}_{\alpha}({\mathrm{ITE}});\PP)=[\max\{-1,\,-\alpha^{-1}{\mathrm{FNA}}^+,\,1-\alpha^{-1}({\mathrm{ATE}}-1)\},
\,\max\{-1,\,-\alpha^{-1}{\mathrm{FNA}}^-,\,1-\alpha^{-1}({\mathrm{ATE}}-1)\}]$,
with ${\mathrm{FNA}}^\pm$ as defined in \cref{eq:bounds01}.
Consequently, the endpoints
are the sharp lower and upper bounds on $\operatorname{CVaR}_{\alpha}({\mathrm{ITE}})$.
\citet{kallus2022treatment} gives the upper bound $\operatorname{CVaR}_{\alpha}({\mathrm{ITE}})\leq \operatorname{CVaR}_{\alpha}(\tau_-(X))$ and shows it is tight given only $\tau_-(X)$ in the sense that is always realizable by some $\PP^*$ with the same distribution of $\tau_-(X)$.
However, this bound is generally not sharp, that is, it need not be realizable given $\PP$, which characterizes more than just $\tau_-(X)$.
In contrast, plugging ${\mathrm{FNA}}^-$ into \cref{lemma:cvar}
gives
the sharp upper bound in the special case of binary outcomes, that is, this bound fully uses \emph{all} the information in $\PP$.
Moreover, we have a finite sharp lower bound, whereas \citet{kallus2022treatment} shows that no lower bound exists when given only $\tau_-(X)$. Nonetheless, \citet{kallus2022treatment} crucially handles general real-valued outcomes. In particular, the CVaR curve as $\alpha$ varies is most interesting in settings with non-binary outcomes, since in binary settings, two numbers -- the ATE and FNA -- summarize all relevant information
per \cref{lemma:cvar}.
\end{remark}
\section{Inference Methodology}
In the previous section we characterized the sharp (meaning, tightest possible) bounds on FNA given the distribution of the data, that is, the best we could do if we had infinite data. We now address how to estimate these from actual data and characterize the uncertainty. That is, how to do inference on the parameters ${\mathrm{FNA}}_{\pi_0\to\pi_1}^\pm$. Recall our data is $n$ independent draws $(X_i,A_i,Y_i)\sim\PP$ for $i=1,\dots,n$.
In order to systematically handle all of parameters of interest,
we will develop a generic, robust method for a large class of parameters that include all of our parameters of interest as special cases.
\subsection{Average Hinge Effects}
We consider a class parameters we call Average Hinge Effects (AHEs), parametrized by a positive integer $m\in\NN$, a vector of signs $\rho\in\{-1,+1\}^m$, and a set of $3(m+1)$ functions $g_\ell(x)=(g_\ell\s{0}(x),g_\ell\s{1}(x),g_\ell\s{2}(x))\in[-1,1]^3$ for $\ell=0,\dots,m$:
\begin{align}
\label{eq:ahe}\ts{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}=\EE\bigl[&g_0\s{0}(X)\mu(X,0)+g_0\s{1}(X)\mu(X,1)+g_0\s{2}(X)\\&\ts+\sum_{\ell=1}^m\rho_\ell\min\{0,\,g_\ell\s{0}(X)\mu(X,0)+g_\ell\s{1}(X)\mu(X,1)+g_\ell\s{2}(X)\}\bigr].\notag
\end{align}
The AHE is so called because $\min\{0,x\}$ is called the hinge function.
AHEs cover all of our parameters of interest:
${\mathrm{FNA}}^{-}={\mathrm{AHE}}^{(-)}_{(0,0,0),(-1,1,0)}$,
${\mathrm{FNA}}^{+}={\mathrm{AHE}}^{(+)}_{(1,0,0),(-1,-1,1)}$,
${\mathrm{FNA}}^{-}_{\pi_0\to\pi_1}={\mathrm{AHE}}^{(-)}_{(0,0,0),(\pi_0-\pi_1,\,\pi_1-\pi_0,\,0)}$,
${\mathrm{FNA}}^{+}_{\pi_0\to\pi_1}={\mathrm{AHE}}^{(+)}_{(\pi_0(1-\pi_1),\,\pi_1-\pi_0,0),\,(\pi_0-\pi_1,\,\pi_0-\pi_1,\,-1)}$, and
${\mathrm{FNA}}^+_{1-\pi^*\to\pi^*}={\mathrm{AHE}}^{(+,+)}_{(1,0,0),(-1,1,0),(-1,-1,1)}$.
For now, we focus on any given $m,\rho,g_0,\dots,g_m$.
\subsection{Re-formulating the AHE}
The AHE is defined as an average of a function of $X$. The only unknown part of this function is $\mu(x,a)$. A na\"ive approach would be to estimate $\mu$ by $\hat\mu$, plug it into \cref{eq:ahe} and replace $\E$ by a sample average over $X_1,\dots,X_n$.
This estimate, however, will \emph{not} behave like a sample average approximation of \cref{eq:ahe}. In particular, it has a nonzero derivative in $\hat\mu$, so that the errors in $\hat\mu$ directly translate to errors in AHE estimation.
For example, even if $\hat\mu$ converges as fast as $O_p(1/\sqrt{n})$, this will introduce significant errors on the same order as the convergence of sample averages, which can imperil both efficiency and inference. And, if it converges more slowly, as would generally be the case when using flexible machine learning methods for estimating (as parametric ones will inevitably be misspecified), it can even affect the convergence rate of the final estimate.
Worst yet, if $\hat\mu$ is inconsistent for $\mu$ we have no hope of consistency for our estimate or an understanding of the direction of its bias.
Generally,
we would like to avoid depending on how exactly we fit $\hat\mu$, provide guarantees even when it is estimated slowly and regardless of what method is used to estimate it, and even provide guarantees when its estimation is inconsistent.
Our approach is based on focusing on an alternative formulation of AHE as a sample average where plugging in wrong values for some unknown functions does not affect the formulation too much.
Toward that end,
let us first define $\eta_\ell:\Xcal\to[-3,3]$ by
\begin{equation}\label{eq:eta}
\eta_\ell(x)=\,g_\ell\s{0}(X)\mu(X,0)+g_\ell\s{1}(X)\mu(X,1)+g_\ell\s{2}(X).
\end{equation}
Specifically,
we have $\eta_1(X)=(\pi_1(X)-\pi_0(X))\tau_-(X)$ for ${\mathrm{FNA}}_{\pi_0\to\pi_1}^-$,
$\eta_1(X)=(\pi_1(X)-\pi_0(X))(1-\tau_+(X))$ for ${\mathrm{FNA}}_{\pi_0\to\pi_1}^+$, and $\eta_1(X)=\tau_-(X),\,\eta_2(X)=1-\tau_+(X)$ for ${\mathrm{FNA}}_{1-\pi^*\to\pi^*}^+$.
Given $\check e,\check \mu,\check\eta_{1},\ldots,\check\eta_{m}$, understood as (possibly wrong) stand-ins for $e, \mu,\eta_{1},\ldots,\eta_{m}$, define
\begin{align*}
\phi^\rho_{g_0,\dots,g_m}(X,A,Y;&~\check e,\check \mu,\check\eta_{1},\ldots,\check\eta_{m})=\ts
g_0\s{2}(X)+\sum_{\ell=1}^m\rho_\ell\indic{\check\eta_\ell(X)\leq0}g_\ell\s{2}(X)
\\&\ts+
\prns*{g_0\s{0}(X)+\sum_{\ell=1}^m\rho_\ell\indic{\check\eta_\ell(X)\leq0}g_\ell\s{0}(X)}\frac{\prns{A-\check e(X)}\check\mu(X,0)+(1-A)Y}{1-\check e(X)}
\\&\ts+
\prns*{g_0\s{1}(X)+\sum_{\ell=1}^m\rho_\ell\indic{\check\eta_\ell(X)\leq0}g_\ell\s{1}(X)}\frac{\prns{\check e(X)-A}\check\mu(X,1)+AY}{\check e(X)}
.\end{align*}
If we plug in the correct values of the nuisance parameters $e, \mu,\eta_{1},\ldots,\eta_{m}$, then we would have \begin{equation}\label{eq:ahephi}{\mathrm{AHE}}_{g_0,\dots,g_m}^\rho=\EE[\phi_{g_0,\dots,g_m}^\rho(X,A,Y;e, \mu,\eta_{1},\ldots,\eta_{m})].\end{equation}
Crucially, unlike the definition of AHE as an average in \cref{eq:ahe}, we will show in \cref{sec:robustness} that this new representation as an average remains faithful \emph{even} when we make small errors in the nuisances.
\subsection{Inference Algorithm}
Given our special construction of $\phi_{g_0,\dots,g_m}^\rho$, our algorithm, presented in \cref{alg:est}, follows a simple recipe:
we estimate the nuisances in a cross-fold fashion and plug them into \cref{eq:ahephi}, approximating the expectation by a sample average.
In \cref{sec:robustness} we will show that the special structure of $\phi_{g_0,\dots,g_m}^\rho$ affords this procedure a lot of robustness.
The use of cross-fitting ensures nuisance estimates are independent of samples applied thereto \citep{schick1986,doubleML,zheng2011cross}.
This allows our analysis to only require lax rate assumptions on the nuisance fitting without requiring any additional regularity on restricting \emph{how} the fitting is done. (If we assume nuisance estimates belong to a Donsker class with probability tending to 1, all of our results will hold even without cross-fitting; we omit this option for brevity.)
\subsection{Nuisance Fitting}
Our algorithm requires methods for fitting the nuisances $e, \mu,\eta_{1},\ldots,\eta_{m}$. Fitting $e$ and $\mu$ amounts to binary regressions (probabilistic classification), which can be done with any of a variety of standard supervised learning methods, whether logistic regression, random forests, or neural nets.
There are different ways to fit $\eta_\ell$.
In particular, despite the fact that $\eta_\ell$ is determined by $\mu$, we treat it as a separate nuisance to allow for specialized learners.
Of course, the simplest way to fit it is to simply plugin an estimate for $\mu$ into the definition of $\eta_\ell$ in \cref{eq:eta}, and that is a legitimate option. For all of our parameters of interest, all $\eta_\ell$ nuisances are given by learning just two functions: $\tau_-$ and $\tau_+$ as in \cref{eq:cate}. Therefore, we may alternatively learn these directly and plug them into $\eta_\ell$. For example, there exists a wide literature specialized to learning $\tau_-$ motivated by the observation that effect signals are often more nuanced and can therefore be washed out by the noise in baselines if we simply difference $\mu$-estimates \citep{slearner,xlearner,drlearner,rlearner,causaltree,causalforest}. We can similarly apply of all these approaches to learning $\tau_+$ by simply first mutating the outcome data as $Y\gets (2A-1)Y$. Notably, \citep{drlearner,rlearner} give rates of convergence for $\tau_-$-learning, which we can use to satisfy our assumptions in the next section.
\begin{algorithm}[t!]\footnotesize
\caption{Point estimate and CIs for ${\mathrm{AHE}}^\rho_{g_0,\dots,g_m}$}\label{alg:est}
\begin{algorithmic}[1]
\STATEx\textbf{Input:} Data $\{(X_i,A_i,Y_i):i=1,\dots,n\}$, number of folds $K$, estimators for $e,\mu,\eta_1,\ldots,\eta_m$
\FOR{$k=1,\dots,K$}
\STATE{Estimate $\hat e^{(k)},\hat\mu^{(k)},\hat\eta_1^{(k)},\dots,\hat\eta_m^{(k)}$ using data $\{(X_i,A_i,Y_i):i\not\equiv k-1~\text{(mod $K$)}\}$}
\STATE{\textbf{for}~$i\equiv k-1~\text{(mod $K$)}$~\textbf{do}~set $\phi_{i}=\phi_{g_0,\dots,g_m}^\rho(X_i,A_i,Y_i;\hat e^{(k)},\hat \mu^{(k)},\hat \eta_1^{(k)},\ldots,\hat \eta_m^{(k)})
$\label{alg:est phistep}}
\ENDFOR
\STATE{Set $\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}=\frac1n\sum_{i=1}^n\phi_{i}$, $\hat{\operatorname{se}}=\sqrt{\frac1{n(n-1)}\sum_{i=1}^n(\phi_{i}-\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m})^2}$\label{alg:est psistep}}
\STATE{Return point estimate $\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}$ and $\beta$-CIs $[\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}\pm \Phi^{-1}((1+\beta)/2)\hat{\operatorname{se}}]$}
\end{algorithmic}
\end{algorithm}
\section{Robustness Guarantees for Inference}\label{sec:robustness}
We now show our inference method has some nice robustness guarantees. This will depend on showing that using our special $\phi$ renders \cref{eq:ahephi} insensitive to errors in the nuisances.
The specific level of errors allowed will depend on the \emph{sharpness} of margin satisfied by $\eta_\ell$, as defined below.
\begin{definition}
We say that a given function $f:\Xcal\to\RR$ satisfies a margin with sharpness $\alpha\in[0,\infty]$ (or, an $\alpha$-margin) if there exists $t_0>0$ such that $\PP\prns{0<\abs{f(X)}\leq t}\leq (t/t_0)^\alpha$ for all $t>0$. (We use the convention that $x^\infty=0$ if $0\leq x<1$, $1^\infty=1$, and $x^\infty=\infty$ if $x>1$.)
\end{definition}
Every $f$ trivially satisfies a 0-margin, but usually a sharper margin holds.
If $f(X)$ is almost surely either zero or bounded away from zero, then $f$ satisfies an infinitely sharp margin:
\begin{lemma}\label{lemma:inftymargin}
If $\Prb{\abs{f(X)}\geq t_0}=\Prb{f(X)\neq0}$ for some $t_0>0$ then $f$ satisfies an $\infty$-margin.
\end{lemma}
In particular, if $X$ has finite support, then we can always set $t_0=\inf\{\abs{f(x)}:x\in\Xcal,f(x)\neq0\}$.
But, this also works more generally. Consider $\eta_1(X)=(\pi_1(X)-\pi_0(X))\tau_-(X)$ for ${\mathrm{FNA}}_{\pi_0\to\pi_1}^-$.
If $\inf\{\abs{\tau_-(x)}:\pi_1(x)\neq\pi_0(x),\tau_-(x)\neq0\}>0$, that is, the policies only differ either far from the boundary where $\tau_-$ is 0 or exactly on this boundary, then \cref{lemma:inftymargin} ensures sharpness $\infty$.
More generally, we should expect a margin with sharpness 1 in continuous settings.
\begin{lemma}\label{lemma:1margin}
If the cumulative distribution function (CDF) of $f(X)$ is boundedly differentiable on $(-\epsilon,0)\cup(0,\epsilon)$ for some $\epsilon>0$, then $f$ satisfies a $1$-margin.
\end{lemma}
Consider again the example of $\eta_1(X)=(\pi_1(X)-\pi_0(X))\tau_-(X)$. If $\tau_-(X)$ is a continuous random variable with a bounded density near $0$, then \cref{lemma:1margin} ensures a margin with sharpness 1. This holds for \emph{any} two policies, including $\pi_1=1,\,\pi_0=0$, which differ everywhere.
The same conclusion would also hold if the CDF of $\tau_-(X)$ was just continuously differentiable at $0$.
Armed with the notion of margin, we can state in what sense the representation \cref{eq:ahephi} is robust.
\begin{theorem}\label{lemma:orthogonal}
Fix any
$\tilde e,\check e:\Xcal\to[\bar e,\,1-\bar e]$ with $\bar e>0$,
$\tilde \mu,\check \mu:\Xcal\times\{0,1\}\to[0,1]$,
$\tilde \eta_{\ell},\check\eta_\ell:\Xcal\to[-3,3]$ for $\ell=1,\dots,m$.
Fix any $p_1,\dots,p_\ell\in[1,\infty]$.
Suppose that either $\tilde e=e$ or $\tilde \mu=\mu$ (or both).
Set $\kappa_\ell=1$ if $\tilde \eta_\ell=\eta_\ell$ and otherwise set $\kappa_\ell=0$.
Further assume that $\Prb{\tilde\eta_\ell(X)=0,\,\check\eta_\ell(X)\neq0}=0$.
Finally, suppose that $\tilde\eta_\ell$ satisfies a margin with sharpness $\alpha_\ell$. Then, for some constant $c>0$, we have
\begin{align*}
&\ts c\abs{
\EE[\phi_{g_0,\dots,g_m}^\rho(X,A,Y;\check e, \check\mu,\check\eta_{1},\ldots,\check\eta_{m})]
-
\EE[\phi_{g_0,\dots,g_m}^\rho(X,A,Y;\tilde e, \tilde\mu,\tilde\eta_{1},\ldots,\tilde\eta_{m})]
}
\\[-1.5ex]&\ts\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq
\magd{ \check e- e}_2
\magd{ \check \mu- \mu}_2
+
\sum_{\ell=1}^m\magd{\check \eta_\ell-\tilde \eta_\ell}_{p_\ell}^{\frac{p_\ell(\kappa_\ell+\alpha_\ell)}{p_\ell+\alpha_\ell}},\\
&\ts c\magd{\phi_{g_0,\dots,g_m}^\rho(X,A,Y;\check e, \check\mu,\check\eta_{1},\ldots,\check\eta_{m})
-
\phi_{g_0,\dots,g_m}^\rho(X,A,Y;\tilde e, \tilde\mu,\tilde\eta_{1},\ldots,\tilde\eta_{m})}_2
\\[-1.5ex]&\ts\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq
\magd{\check e-\tilde e}_2+
\magd{\check \mu-\tilde \mu}_2
+
\sum_{\ell=1}^m\magd{\check \eta_\ell-\tilde \eta_\ell}_{p_\ell}^{\frac{p\alpha_\ell}{2p+2\alpha_\ell}}.
\end{align*}
In the above, we use the convention that $\frac{\infty a+b}{\infty c+d}=\frac{a}{c}$.
\end{theorem}
\subsection{Local Robustness, Confidence Intervals, and Efficiency}
\Cref{lemma:orthogonal} has several important implications.
Our first result shows that, if we estimate all nuisances correctly but potentially slowly at nonparametric rates, then we can ensure our estimator looks like a sample average approximation of \cref{eq:ahephi} and our CIs are calibrated.
\begin{theorem}\label{thm:ral}
Suppose $e(X)\in[\bar e,1-\bar e]$ with $\bar e>0$, $\eta_\ell$ satisfies an $\alpha_\ell$-margin, and for $k=1,\dots,K$ $\|\hat e\s k-e\|_2\|\hat \mu\s k-\mu\|_2=o_p(n^{-\frac12})$, $\|\hat\eta_\ell\s k-\eta_\ell\|_{p_\ell}=o_p(n^{-\frac{p_\ell+\alpha_\ell}{2p_\ell(1+\alpha_\ell)}})$ with $p_\ell\in[1,\infty]$, and $\PP\prns*{\bar e\leq\hat e\s k(x)\leq1-\bar e,\,0\leq\hat\mu\s k(x)\leq1,\,\abs*{\hat\eta_\ell\s k(x)}\leq 3\II[\eta_\ell(x)\neq0],\,\ell\leq m,\,\text{a.e.~$x$}}\to1$. Then,
\begin{align*}
&\ts\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}=\frac1n\sum_{i=1}^n\phi_{g_0,\dots,g_m}^\rho(X_i,A_i,Y_i;e, \mu,\eta_{1},\ldots,\eta_{m})+o_p(n^{-\frac12}),
\\
&\ts\PP\prns*{{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}\in\bracks*{\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}\pm\Phi^{-1}((1+\beta)/2)\hat{\operatorname{se}}}}\to\beta\quad\forall\beta\in(0,1).
\end{align*}
\end{theorem}
The conditions of \cref{thm:ral} allow that we learn $e,\mu,\eta_1,\ldots,\eta_m$ rather slowly.
In particular, unlike na\"ively plugging in a $\mu$-estimate into \cref{eq:ahe} and taking sample averages, we do not depend at all on \emph{how} $\mu$ is estimated and the uncertainty therein, provided some slow nonparametric rates hold.
For example, learning $e,\mu$ in
with $o_p(n^{-1/4})$-rates in $L_2$-error suffices. Or, if $e$ is known, then consistently estimating $\mu$ \emph{without} a rate suffices. For $\eta_\ell$, if an $\infty$-margin holds, it suffices to learn $\eta_\ell$ consistently (no rate) in $L_\infty$-error or at $o_p(n^{-1/4})$-rates in $L_2$-error. If a $1$-margin holds, it suffices to learn at $o_p(n^{-1/4})$-rates in $L_\infty$-error.
While conditions like metric entropy imply $L_2$-error rates \citep{wainwright2019high}, for many classes like smooth functions the $L_p$ error rates are even the same for all $p\in[1,\infty]$ \citep{stoneglobal}.
In fact, under one more condition, our estimator achieves the semiparametric efficiency lower bound.
\begin{theorem}\label{thm:eif}
Suppose $\PP\prns*{\eta_\ell(X)=0,g_\ell\s{a}(X)\neq0}=0$ for $\ell=1,\dots,m$, $a=0,1$.
The semiparametric efficiency lower bound for
${\mathrm{AHE}}_{g_0,\dots,g_m}^\rho$
is
$\operatorname{Var}(\phi_{g_0,\dots,g_m}^\rho(X,A,Y;e, \mu,\eta_{1},\ldots,\eta_{m}))$,
whether or not $e(X)$ is unknown (varies in the model) or known (fixed in the model).
\end{theorem}
\begin{corollary}\label{thm:efficient}
Under the assumptions of \cref{thm:ral,thm:eif}, $\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}$ is efficient: it is regular and achieves the minimum asymptotic variance among all regular estimators.
\end{corollary}
For all of our parameters of interest,
the condition of \cref{thm:eif} holds if $\PP\prns*{\tau_-(X)=0}=\PP\prns*{\tau_+(X)=0}=0$ (\eg, $\tau_-(X),\tau_+(X)$ are continuous).
Similarly, the condition that $\abs*{\hat\eta_\ell\s k(x)}\leq 3\II[\eta_\ell(x)\neq0]$ in \cref{thm:ral} is immediately satisfied if $\hat\tau_-\s k(X),\hat\tau_+\s k(X)$ are also almost never 0 (\eg, continuous).
\subsection{Double Robustness and Double Validity}
Next we show that \cref{alg:est} is very robust to incorrectly learning some nuisances.
\begin{theorem}\label{thm:drdv}
Fix any $\tilde\mu,\tilde e,\tilde\eta_1,\dots,\tilde\eta_m$ with $\tilde\mu(X)\in[0,1],\,\tilde e(X)\in[\bar e,1-\bar e]$ with $\bar e>0$. Set $\kappa_\ell=1$ if $\tilde \eta_\ell=\eta_\ell$ and otherwise set $\kappa_\ell=0$. Suppose $\tilde\eta_\ell$ satisfies a margin with sharpness $\alpha_\ell$, and for $k=1,\dots,K$ that $\|\hat e\s k-\tilde e\|_2=o_p(1)$, $\|\hat \mu\s k-\tilde\mu\|_2=o_p(1)$, $\|\hat\eta_\ell\s k-\tilde\eta_\ell\|_{p_\ell}=O_p(n^{-\frac{p_\ell+\alpha_\ell}{2p_\ell(\kappa_\ell+\alpha_\ell)}})$ for some $p_\ell\in[1,\infty]$, and $\PP\prns*{\bar e\leq\hat e\s k(x)\leq1-\bar e,\,0\leq\hat\mu\s k(x)\leq1,\,\abs*{\hat\eta_\ell\s k(x)}\leq 3\II[\tilde\eta_\ell(x)\neq0],\,\ell\leq m,\,\text{a.e.~$x$}}\to1$.
If either $\|\hat e\s k- e\|_2=O_p(n^{-1/2})$ or $\|\hat \mu\s k-\mu\|_2=O_p(n^{-1/2})$, then
\begin{align*}
&\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}={\mathrm{AHE}}^\rho_{g_0,\dots,g_m}+O_p(n^{-\frac12})~~\text{if $\kappa_1=\cdots=\kappa_m=1$,}&\text{(double robustness)}\\
&\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}\geq{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}+O_p(n^{-\frac12})~~\text{if $\rho_\ell=+$ whenever $\kappa_\ell=0$,}&\text{(double validity, upper)}\\
&\widehat{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}\leq{\mathrm{AHE}}^\rho_{g_0,\dots,g_m}+O_p(n^{-\frac12})~~\text{if $\rho_\ell=-$ whenever $\kappa_\ell=0$}.&\text{(double validity, lower)}
\end{align*}
\end{theorem}
The first equation (double robustness) in \Cref{thm:drdv} shows our estimator remain consistent for the AHE even if we \emph{incorrectly} learn either $e$ or $\mu$, bot not both, as long as we correctly learn $\eta_1,\dots,\eta_m$.
The second two equations (double validity) in \Cref{thm:drdv} show what happens when we also learn $\eta_\ell$ \emph{incorrectly}. For upper double validity (second equation), we show that, as long as the $\eta_\ell$ that are misestimated correspond to convex terms in the AHE ($\rho_\ell=+$), we remain consistent for an \emph{upper bound} on the AHE, even if we \emph{also} incorrectly learn either $e$ or $\mu$, bot not both. In particular, for ${\mathrm{FNA}}^+$ we only have convex terms, so we are guaranteed and upper bound on the upper bound, that is, we still estimate a valid upper bound on ${\mathrm{FNA}}$ when we misestimate the CATE $\tau_-$ (which is $\eta_1$ for ${\mathrm{FNA}}^+$) and one of $e$ or $\mu$, it just may not be sharp. We also have a symmetric result for lower double validity (third equation), and applying it to ${\mathrm{FNA}}^-$ we find that we will still estimate a valid upper bound, albeit possibly unsharp, on ${\mathrm{FNA}}$ when we misestimate $\tau_+$ and one of $e$ or $\mu$.
The significance is that we can really trust the results of \cref{alg:est} are truly bounds on FNA, even if we make mistakes, and therefore conclusions about harm based on our estimates and inferences can be highly credible.
\section{Empirical Investigation}
We demonstrate our method in a simulation and a case study with data from a real experiment. Replication code is given in the supplement. Experiments were run on an AWS c5.24xlarge instance.
\textbf{Simulation Study}\quad
We consider $7$-dimensional standard normal $X$ and set $\mu(x,a)=(1+\exp(-\beta(2\II[\xi(x)x_2>0]-1)(\II[\xi(x)x_1\leq0]+(2a-1)\II[\xi(x)x_1>0])))^{-1}$, where $\beta\geq0$ and $\xi(x)=2\operatorname{XOR}(\II[x_3>0],\ldots,\II[x_7>0])-1$. Here $\beta$ controls how much $X$ predicts $Y(a)$. The dashed lines in \cref{fig:sim bounds} show the sharp bounds of \cref{thm:bounds} for ${\mathrm{FNA}}$ and ${\mathrm{FNA}}_{1-\pi^*\to\pi^*}$ as we vary $\beta$. We can see that, as $X$ becomes more predictive, the bounds go from $[0,\,0.5]$ to a point at $0.25$. Note there is no \emph{true} value for ${\mathrm{FNA}}$ as we are not specifying the joint distribution of potential outcomes; instead, per \cref{thm:bounds}, there are joint distributions with ${\mathrm{FNA}}$ equal to any point between the bounds.
Next, we estimate these true bounds. We fix $e(X)=(1+\exp(0.25-\II[x_3>0]+0.5\II[x_4>0])^{-1}$, and draw $n$ samples from $X\sim\mathcal N(0,I)$, $A\mid X\sim \operatorname{Bernoulli}(e(X))$, $Y\mid X,A\sim\operatorname{Bernoulli}(\mu(X,A))$. We apply \cref{alg:est} to this data with $K=5$, estimating $e,\,\mu$ using random regression forests and $\tau_-,\,\tau_+$ using causal forests (all using \emph{R} package \texttt{grf} with default parameters). The solid lines in \cref{fig:sim bounds} show the point estimate and 95\%-CI output by \cref{alg:est} for a single draw of $n=12800$ data for each $\beta$. In \cref{fig:sim rmse,fig:sim cov} we plot the average root mean squared error of point estimates and coverage of CIs over $100$ replications with $\beta=3$ and varying $n$. We compare the performance to the (cross-fitted) ``plugin'' estimator that just plugs in the (same) estimates for $\mu$ into \cref{eq:ahe} and approximate the expectation by a sample average. Shaded intervals denote 95\%-CIs for performance (\ie, due to finite replications).
\textbf{Case Study}\quad\label{sec:casestudy}
Using data from \citep{behaghel2014private} (BSD license), we compare three different assistance programs offered to French unemployed individuals: the standard benefits (sta), access to public-run counseling (pub), and to private-run counseling (pri).
Our binary outcome is reemployment within six months.
For example, we can consider a hypothetical scenario where we change from a private to a public counseling provider.
As reported by \citep{behaghel2014private}, the ATE for this hypothetically change is slightly positive, so a policymaker might therefore enact it, especially if, for example, it offered cost savings or other operational benefits. We consider who may be harmed by this: how many who remain unemployed under the new public program that would have been reemployed under the old private program.
For any pair of arms, we consider the sharp bounds on the FNA from one to other or the other way and on the misclassification rate of the optimal policy choosing between the two.
Following \citet{kallus2022treatment}, we set $X$ to all pre-treatment covariates in table~2 of \citet{behaghel2014private}.,
and we consider the sharp bounds either with these $X$ or given no $X$ at all.
We fit $\mu$ using linear regression and $\tau_-,\tau_+$ using doubly-robust pseudo-outcome linear regression. Since the propensity is known, in light of the guarantees of \cref{thm:drdv}, we do not worry about misspecifying $\mu$, only using it for variance reduction by accounting for main effects, and we do not worry about misspecifying $\tau_-,\tau_+$, only hoping to tighten the bounds some from using no $X$. The results are shown in \cref{fig:behaghel}. We see that, while covariates are somewhat uninformative, we are still able to tighten bounds compared to no $X$. In particular, for the change from sta to pri or pub or the change from pri to pub, without $X$ the lower bounds on FNA are 0, while with $X$ the lower bounds as well as the lower confidence bounds on these bounds are strictly positive. This means that, with $X$ and the methods developed we can prove that there must be some harm by this change, something we could not do otherwise and such a finding can bolster further work to investigate and address this harm.
\begin{figure}[t!]\centering
\begin{minipage}{0.28\textwidth}
\centering
\includegraphics[width=\linewidth]{cisims}
\caption{Sharp bounds, estimates, and 95\%-CIs as $X$-predictiveness ($\beta$) varies}\label{fig:sim bounds}
\end{minipage}\hfill
\begin{minipage}{0.28\textwidth}
\centering
\includegraphics[width=\linewidth]{simsrmse}
\caption{Root mean squared error (RMSE) for estimating the sharp bounds as $n$ varies}\label{fig:sim rmse}
\end{minipage}\hfill
\begin{minipage}{0.28\textwidth}
\centering
\includegraphics[width=\linewidth]{simscov}
\caption{Coverage of 95\%-CIs using $\pm1.96/\sqrt{n}$ standard deviations as $n$ varies}\label{fig:sim cov}
\end{minipage}
\\[\floatsep]
\begin{subfigure}[m]{0.35\textwidth}\includegraphics[height=1.6cm]{beh_lb}\caption{Lower bound ${\mathrm{FNA}}^-$}\end{subfigure}
\begin{subfigure}[m]{0.35\textwidth}\includegraphics[height=1.6cm]{beh_ub}\caption{Upper bound ${\mathrm{FNA}}^+$}\end{subfigure}
\begin{subfigure}[m]{0.2\textwidth}\includegraphics[height=1.6cm]{beh_opt}\caption{${\mathrm{FNA}}^+_{1-\pi^*\to\pi^*}$}\end{subfigure}\hfill
\begin{subfigure}[m]{0.09\textwidth}\includegraphics[height=1.8cm]{beh_leg}\end{subfigure}
\caption{Estimated sharp bounds for FNA in the case study with covariates and without covariates}\label{fig:behaghel}
\end{figure}
\section{Connections, Limitations, Extensions, and Conclusions}\label{sec:discussion}
\textbf{Connections to partial identification}\quad
Partial identification of unknowable parameters has a long tradition in econometrics \citep{tamer2010partial,manski1997mixing,manski2003partial,manski1995identification,molinari2020microeconometrics,semenova2017debiased,chernozhukov2007estimation,king2013solution}. Some approaches focus on average treatment effects in the presence of confounding \citep{cross2002regressions,dorn2021doubly,kallus2019interval,rosenbaum,tan2006,yadlowsky2018bounds,bonvini2021sensitivity,chernozhukov2021omitted}, among which \citep{dorn2021doubly,yadlowsky2018bounds,bonvini2021sensitivity,chernozhukov2021omitted} are notable for conducting semiparametric inference on bounds, but for confounding rather than for individual effects, as we do. Some others, like us, focus on understanding the joint distribution of potential outcomes \citep{heckman1997making,ding2019decomposing,abbring2007econometric,manski1997monotone,firpo2008bounds}. Like us, these heavily leverage the Fr\'echet-Hoeffding bounds \citep{ruschendorf1981sharpness,frechet1935generalisation,cambanis1976inequalities,williamson1990probabilistic,ridder2007econometrics}. Unlike these, we use efficient and robust inference that can aggregate across covariates to tighten bounds by leveraging machine learning of conditional-mean outcome functions.
\textbf{Connections to fairness}\quad
Understanding the distribution of impact is a central problem in algorithmic fairness \citep{mitchell2021algorithmic,barocas-hardt-narayanan}. Here, we specifically focused on the \emph{differential} impact of interventions, which brought up issues of identification. In contrast, in algorithmic fairness one usually measures disparities in observed outcomes, such as in the form of the loss function of a model on a labeled example \citep{kearns2018preventing,bird2020fairlearn,ghosh2021characterizing,agarwal2018reductions}. A line of work specifically focuses on identification issues in algorithmic fairness \citep{kallus2021assessing,chen2019fairness,kallus2019assessing,kallus2018residual,fogliato2020fairness,lakkaraju2017selective,coston2021characterizing,kleinberg2018human}. Notably, \citep{kallus2021assessing} also conduct semiparametric inference on sharp bounds derived from the Fr\'echet-Hoeffding bounds, but in the context of algorithm evaluation with unobserved protected labels rather than interventions with unobserved counterfactuals. And, \citep{kallus2019assessing} do consider unobserved counterfactuals in assessing equality of opportunity \citep{hardt2016equality} for intervention-prioritization policies, but (appropriately in their own context) they assume away our primary focus here by assuming an a priori known bound on FNA (possibly zero, \ie, monotone treatment response).
\textbf{Limitations and Extensions}\quad
Restricting outcomes to be binary is one limitation of our work, and extensions to continuous settings
is an interesting avenue of future research. Another limitation is that, while we can bound the FNA, it can still be hard to asses \emph{who} is negatively affected. This is, unfortunately, impossible for the same reason FNA is unidentifiable, but a place to start such analyses may be to consider who are individuals with $\tau_-(X)\leq 0$ and even characterize that group via summary statistics compared to the population. Indeed, per \cref{eq:bounds01}, the lower bound on FNA is exactly the (negative of the) ATE on this group.
Another important consideration is whether the data is representative: FNA refers to the fraction of the \emph{studied population}, which might differ from the population of interest. \Eg, if an experiment did not enroll a representative sample, we may be systematically excluding some groups from consideration. If such unrepresentativeness is explained by covariates $X$ (\ie, missing at random), the solution is simple: we reweight. If not explained by $X$ (\ie, missing \emph{not} at random), then we need to also account for this additional source of unidentifiability. An avenue for future research is to combine such ambiguity with counterfactual ambiguity. Another concern is whether the outcome represents the impact we want to measure \citep{passi2019problem}.
\textbf{Conclusions}\quad Our tools support drawing credible conclusions about the potential negative impact of interventions: they both account for ambiguity due to unobserved counterfactuals (while mitigating it using covariates) as well as strongly guard against slow or inconsistent learning of necessary nuisances functions.
Robust inference on the lower bounds, in particular, crucially provides watertight demonstrations of negative impact, which can bolster efforts to mitigate harm and improve equity.
\section*{Acknowledgments}
I am grateful for the helpful comments
of the anonymous reviewers and for many insightful and thought-inspiring conversations with my colleagues at Netflix.
\bibliographystyle{plainnat}
\bibliography{lit}
\section*{Checklist}
\begin{enumerate}
\item For all authors...
\begin{enumerate}
\item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\answerYes{Provided sharp bounds and ways to do inference thereon}
\item Did you describe the limitations of your work?
\answerYes{See \cref{sec:discussion}}
\item Did you discuss any potential negative societal impacts of your work?
\answerYes{See \cref{sec:discussion}}
\item Have you read the ethics review guidelines and ensured that your paper conforms to them?
\answerYes{}
\end{enumerate}
\item If you are including theoretical results...
\begin{enumerate}
\item Did you state the full set of assumptions of all theoretical results?
\answerYes{}
\item Did you include complete proofs of all theoretical results?
\answerYes{}
\end{enumerate}
\item If you ran experiments...
\begin{enumerate}
\item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
\answerYes{}
\item Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
\answerYes{}
\item Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
\answerYes{}
\item Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
\answerYes{}
\end{enumerate}
\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
\begin{enumerate}
\item If your work uses existing assets, did you cite the creators?
\answerYes{}
\item Did you mention the license of the assets?
\answerYes{}
\item Did you include any new assets either in the supplemental material or as a URL?
\answerYes{}
\item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
\answerNA{}
\item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
\answerNA{}
\end{enumerate}
\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
\item Did you include the full text of instructions given to participants and screenshots, if applicable?
\answerNA{}
\item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
\answerNA{}
\item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
\answerNA{}
\end{enumerate}
\end{enumerate}
\clearpage