Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
92,993 characters · 20 sections · 67 citation commands
Externally Valid Policy Choice
\defaultbibliography{evtc} \defaultbibliographystyle{chicago}
There is increasing interest across many disciplines in personalized treatment policies.\footnote{In economics, the literature goes back to Manski2004 and includes BhattacharyaDupas, Stoye2012, KitagawaTetenov2018, MTM, and AtheyWager2021. References in other fields include QianMurphy2011, ZZRK, SwaminathanJoachims, and Kallus2017,Kallus2021, among others.} The typical objective is to estimate a policy mapping individual characteristics into a treatment choice to maximize overall welfare. The assumption underlying this literature is that the target population in which the policy is to be implemented is the same as the experimental (or training) population from which the data are sampled.\footnote{We refer to the distribution from which the data are sampled as “experimental”, with the understanding that the analyst could in fact be using observational (i.e., non-experimental) data.} While many policy learning algorithms have good welfare guarantees when the target and experimental populations are identical, the resulting policies can perform poorly if the target population differs from the experimental population.
There are several reasons why the target and experimental populations may differ, echoing well-known concerns about the external validity of experiments (see, e.g., BannerjeeDuflo2009,Deaton2010,Allcott2015). For instance, data may be collected under experimental conditions that differ from real-world settings where the policy is to be implemented. Additionally, delays between data collection and implementation can result in distribution shifts. Experiments may also use easily measurable variables (e.g., test scores) to define outcomes, whereas the policymaker may be concerned with more difficult to quantify outcomes (e.g., overall academic achievement). Furthermore, experiments may be run in selected sub-populations that are not representative of the broader target population.
We consider the problem of learning personalized treatment policies that are externally valid or generalizable: they perform well in target populations different from the experimental population from which the data are sampled. We allow for shifts in both the distribution of potential outcomes and characteristics between the experimental and target populations. We propose methods for estimating externally valid policies using experimental or observational data (where treatment may be endogenous). Beyond policy learning, our methods can be used as a stress test to evaluate the robustness of treatment policies to distribution shifts. Our findings also shed light on the important roles that treatment effect heterogeneity within the experimental population plays in shaping the generalizability of policies.
More formally, consider a policy $\tau$ mapping individual characteristics $X$ into a binary outcome $\tau(X)$, where $\tau(X) = 1$ indicates that treatment is to be assigned to an individual with characteristics $X$ and $\tau(X) = 0$ indicates otherwise.\footnote{As with much of the literature on policy learning, we do not consider randomized (or fractional) policies where $\tau(X)$ can also take intermediate values in $(0,1)$.} Following Manski2004, policies are typically evaluated using a social welfare criterion
where $Y_0$ and $Y_1$ denote the individual's untreated and treated potential outcomes, and $\mathbb{E}_P[\,\cdot\,]$ denotes expectation under the distribution $P$ of $(X,Y_0,Y_1)$ in the experimental population. It is also sometimes of interest to evaluate policies by their welfare gain relative to a policy in which no one is treated:
The typical objective is to learn a policy that maximizes ((ref)) or ((ref)) over a class $\mc T$ that may incorporate functional-form, budget, fairness, or other constraints. This literature relies on the assumption that the experimental and target populations are identical.
Our objective is to derive generalizable policies that deliver welfare guarantees over target populations that may differ from the experimental population. To this end, we replace criteria ((ref)) and ((ref)) with the robust welfare criterion
and the robust welfare gain criterion
where $\mc Q = \mc Q(P)$ is a set of target populations that are “close” to $P$ in a sense we make precise below. We propose methods to learn robust policies $\tau_{\tiny\text{Robust}}$ that maximize $\mathrm{RW}$ or $\mathrm{RWG}$ over $\mc T$. This max-min approach ensures that $\tau_{\tiny\text{Robust}}$ delivers welfare guarantees uniformly over all target populations $Q \in \mc Q$.
A novel aspect of our approach is to define $\mc Q$ using Wasserstein distance. This has several advantages. First, the size parameter $\varepsilon$ used to define $\mc Q$ is the maximum difference in the average treatment effect (ATE) between the experimental population $P$ and target populations $Q \in \mc Q$, as we show in Section (ref). This makes $\varepsilon$ very interpretable and easily calibrated by the analyst. Wasserstein neighborhoods also lead to tractable characterizations of the robust welfare objectives ((ref)) and ((ref)).
Section (ref) considers settings where the distribution of outcomes can shift between the experimental and target populations, while the distribution of characteristics remains fixed. Propositions (ref) and (ref) characterize the robust criteria ((ref)) and ((ref)) in this context. These characterizations imply that any policy that maximizes the usual welfare criteria ((ref)) or ((ref)) is robust to a certain class of shifts in the distribution of outcomes. Hence, policy learning methods with good statistical guarantees under criteria ((ref)) or ((ref)), see, e.g., Manski2004, QianMurphy2011, KitagawaTetenov2018, AtheyWager2021 and MTM, also enjoy good guarantees under the corresponding robust criteria ((ref)) or ((ref)). Appendix (ref) extends these findings to settings with a known shift in characteristics.
Section (ref) examines shifts in both outcomes and characteristics between the experimental and target populations. Proposition (ref) characterizes the robust criteria ((ref)) and ((ref)) in this setting. We then build on this characterization to show how external validity is shaped by two key factors: (i) the distribution of conditional average treatment effects (CATEs) relative to the treatment/non-treatment frontier, and (ii) unobserved heterogeneity in treatment effects within the experimental population. For the first, robust policies sort individuals by the magnitude of their CATEs: individuals with smaller CATEs should be near the treatment/non-treatment frontier, whereas those with larger CATEs should be further away. The second factor arises because nature can shift the distribution of characteristics to adversarially reallocate individuals across the treatment/non-treatment frontier. That is, individuals who benefit from treatment are moved to the non-treatment side, and vice versa for individuals with negative treatment effects. This adversarial shifting renders the policymaker's objective function concave in individual treatment effects. By contrast, neither source of heterogeneity affects the ranking of policies under the usual criteria ((ref)) and ((ref)), which are linear in CATEs/individual treatment effects. An important take-away of this section is to be mindful of treatment effect heterogeneity when recommending policies in scenarios where distribution shifts are possible.
Section (ref) discusses empirical implementation. Here the robust criteria depend on the joint distribution of characteristics and individual treatment effects in the experimental population. This distribution is not identified without further assumptions. We first derive sharp bounds on the robust welfare criterion in the absence of identifying assumptions. We then introduce nonparametric estimators of the bounds and establish consistency and convergence rates. Further, we provide a non-exhaustive set of methods for estimating robust policies under different identifying assumptions from the literature on distributional treatment effects (see AbbringHeckman2007 and references therein), allowing for experimental and observational data. We also provide convergence rates and regret guarantees for the estimated robust policies.
Finally, Section (ref) considers two variants of our framework. First, Section (ref) explores an alternative neighborhood construction in which we shut down the ability of nature to shift distributions in a way that exploits unobserved heterogeneity in treatment effects. Second, most of our paper operates under the conventional assumption that the decision maker is assumed to implement the (robust) welfare-maximizing policy irrespective of its cost. To this end, Section (ref) discusses extensions to settings with a constant per-unit treatment cost. For both variants, we characterize robust welfare and discuss the estimation of robust policies.
We conclude in Section (ref) with an application that revisits the experiment of MKP2021, which examined the effect of business training on rural Kenyan firms led by female entrepreneurs. Outcomes were measured over a four-year window following treatment, introducing a significant time lag between the experiment and any potential policy implementation. Our empirical findings corroborate some of our theoretical results about the role of heterogeneity in shaping generalizability. We also illustrate how the methods we develop can be used as a “stress test” for assessing the fragility or robustness of policies estimated using standard methods (e.g., empirical welfare maximization) to distribution shifts. For instance, researchers could report plots similar to those we present in Section (ref), which compare the robust welfare of policies across families of target populations of different sizes.
\@startsection{paragraph}{4}{\z@} {0pt \@plus1ex \@minus.2ex} {-1em} {\normalfont}{Related Literature.}
MoQiLiu and Spini2021 study robustness to shifts in the distribution of $X$ over $f$-divergence neighborhoods. A key and arguably restrictive assumption underlying their approach is that the conditional distribution of outcomes given characteristics does not change between the experimental and target populations. By contrast, we allow the distribution of outcomes to shift as well.
Like us, Kido2022, SZZB, and QiPangLiu2022 seek robustness with respect to shifts in both outcomes and characteristics. These works and ours use different robustness sets which lead to different insights and different rankings of policies. Kido2022 restricts the conditional distribution of outcomes given $X$ to Wasserstein neighborhoods (pointwise in $X$), and restricts the marginal distribution of $X$ to $f$-divergence (e.g., Kullback--Leibler divergence) neighborhoods. SZZB constrain the joint distribution of outcomes and $X$ to Kullback--Leibler neighborhoods. QiPangLiu2022 constrain the ratio of the densities in the experimental and target populations. These constructions all prohibit extrapolation beyond the support of the experimental population, which may be restrictive in applications. The notion of neighborhood size is also more difficult to interpret for these constructions, whereas it has a clear interpretation in our setting. Section (ref) discusses in more detail how Wasserstein distances address these shortcomings. Moreover, none of these prior works shed light on how treatment effect heterogeneity affects generalizability, which is arguably the most important take-away from our work.
Moreover, Kido2022 considers shifts in outcomes only as well as shifts in both outcomes and $X$, as do we. For the former, his work and ours produce the same ranking if outcomes are unbounded from below. For the latter, Kido2022 constructs neighborhoods sequentially (first outcomes shift conditional on $X$, then $X$ shifts) whereas we construct them jointly, allowing outcomes and $X$ to shift together. As a consequence, his robust criterion is invariant to unobserved heterogeneity in treatment effects whereas heterogeneity plays an important role in our characterization. A further interesting consequence is that robust policies in his setting can be learned using the methods of MoQiLiu, which account only for shifts in $X$.
LSW2023 assume study participants are a non-random sample from the target population and model sampling bias,\footnote{See also Stoye2012 in the context of aggregate (i.e., not individualized) treatment policies.} i.e., how participants select into the sample. Their robustness set $\mc Q$ consists of unknown target distributions that are consistent with the experimental population given selection. We instead do not take a stand on the cause of the shift between $P$ and $Q$, allowing for a general class of unstructured shifts in distributions which could be caused by many mechanisms. In our approach, all the researcher needs to commit to is the maximum shift in the ATE between $P$ and $Q$ to be considered. Conversely, if the researcher has a good reason for committing to a specific mechanism causing the shift (e.g., sampling bias), then sharper results could be derived by tailoring the robustness set $\mathcal Q$ accordingly.
Munro2023 considers individualized policies in environments where agents strategically report their characteristics. His analysis attributes the distribution shift to strategic reporting by agents, which does not give rise to a distributionally robust problem. While we consider classes of deterministic policies, optimal policies with strategic agents may involve randomization.
In this section, we describe the set of target populations we work with and show that the neighborhood size has a clear interpretation as the maximal change in the ATE between the experimental and target populations. We then compare our approach with recent approaches using $f$-divergences (such as Kullback--Leibler divergence).
Let $Z := (X,Y_0,Y_1)$ take values in $\mc Z$ and let $d : \mc Z \times \mc Z \to \mb R_+ \cup \{+\infty\}$ be a metric on $\mc Z$. The Wasserstein distance of order $p$ between $P$ and $Q$ is \[ d_{W,p}(P,Q) = \inf_{\pi \in \Pi(P,Q)} \mathbb{E}_\pi[d(Z,\tilde Z)^p]^{\frac{1}{p}}, \quad (1 \leq p < \infty), \] where $\Pi(P,Q)$ denotes all joint distributions for $(Z,\tilde Z)$ with marginals $P$ for $Z$ and $Q$ for $\tilde Z$. We will focus on the metric with $p=1$, so we drop the $p$ subscript and write $d_W(P,Q)$ in what follows. This is mainly to simplify presentation: Appendix (ref) presents generalizations to $p>1$.
We define Wasserstein neighborhoods using
for some metric $b$. We use different choices of $b$ to handle robustness to shifts in outcomes only (as in Section (ref)) and shifts in outcomes and characteristics (as in Section (ref)). We define neighborhoods as
where $\varepsilon > 0$ is a measure of neighborhood size. As we now show, the parameter $\varepsilon$ is the maximum change in the ATE as $Q$ varies over $\mc Q$. This interpretation holds irrespective of whether we seek robustness with respect to shifts in outcomes only or in both outcomes and characteristics.
First, we introduce some notation. Let $\mathcal Y$ denote the support of the potential outcomes.\footnote{By “support” we mean the set of all values the potential outcomes could conceivably take, as distinct from the measure-theoretic notion of support.} To simplify exposition, we present results for unbounded potential outcomes for which $\inf \mathcal Y = -\infty$ and $\sup \mathcal Y = +\infty$. See Appendix (ref) for the case where at least one of $\inf \mathcal Y$ or $\sup \mathcal Y$ is finite, which is relevant, for example, for binary outcomes or non-negative outcomes such as earnings. Let $\Delta = Y_1 - Y_0$ denote the individual treatment effect.
To give a quantitative example, consider the empirical application in Section (ref). We are interested in the effect of a business training program on firm profits, which were measured over a four-year window following treatment. Therefore, any policymaker deciding whether to implement such a training program would be relying on evidence from an experiment conducted (at least) four years ago. This raises the possibility that the experimental and target populations can differ due to changes in the profit distribution over time. To quantify the drift, note that average profits for untreated firms one- and three years after treatment are approximately 1,467 and 1,493 Kenyan Shillings (KSh), a difference of approximately 13 KSh per year. Remark (ref) suggests $\varepsilon \approx 50$ could be reasonable for a policymaker deciding whether to implement a policy four years after the experiment. In view of Proposition (ref), $\varepsilon \approx 50$ represents slightly less than one third of the treatment effect on profits one year out of 157 KSh.
We close this section by contrasting our approach based on Wasserstein distance with the more familiar $f$-divergence (e.g., Kullback--Leibler (KL) divergence) used in a number of recent works.\footnote{Specifically, SZZB constrain the joint distribution of $(X,Y_0,Y_1)$ using KL neighborhoods; MoQiLiu and Spini2021 both assume there is no shift in the conditional distribution of $(Y_0,Y_1)|X$, but allow the marginal distribution of $X$ to vary over $f$-divergence neighborhoods; and Kido2022 allows the distribution of $(Y_0,Y_1)|X$ to vary over Wasserstein neighborhoods and constrains the distribution of $X$ using KL divergence.}
Consider an experiment in which subjects are assigned to a conditional cash transfer (CCT) program aimed at improving children's education outcomes. GS2017 provide a meta-analysis of CCTs. Suppose $X$ is households' distance from school and $(Y_0, Y_1)$ are educational outcomes with and without receiving cash transfers. Let $P$ denote the joint distribution over $(X, Y_0, Y_1)$ for an experimental population in a middle-income country, $Q$ the joint distribution in a target population that is another middle-income country, and $Q'$ the joint distribution in a target population that is a low-income country.
Table (ref) presents a numerical example. In middle-income populations $P$ and $Q$, most students are 1 or 2 miles from their school, whereas most students in the low-income population $Q'$ are 10 miles from their school. All that matters for KL divergence is the relative probabilities over the support of the reference measure, not the actual values taken. Thus, both $Q$ and $Q'$ are the same KL-divergence from $P$:\footnote{Although we are considering shifts in the joint distribution of $(X,Y_0,Y_1)$, we have kept the values of $(Y_0,Y_1)$ fixed. This way we are effectively measuring shifts in the marginal distribution of $X$, so this critique is relevant for the constructions in MoQiLiu and Spini2021.} \[ KL(Q \|P) = KL (Q' \| P) \approx 1.456. \] However, it seems reasonable that low-income populations where 80% of students live 10 miles from school are inherently “further” from middle-income populations in which 90% of students live within 1 or 2 miles of their school. The fact that KL divergence doesn't reflect this difference limits its interpretability. However, this difference is reflected in Wasserstein distance: \[
\] where we have used the metric $b(x,\tilde x) = |x - \tilde x|$ in ((ref)).
Moreover, in the scenario where now the middle-income experimental population is given by $P'$, so that no student in the experimental population lives 10 miles from school, the distributions $Q$ and $Q'$ (and for that matter $P$) are not absolutely continuous with respect to $P'$. Correspondingly, we have \[ KL(Q\|P') = KL(Q'\|P') = KL(P\|P') = +\infty \] In other words, a robustness analysis using KL divergence centered at $P'$ would never guard against shifts to $Q$ and $Q'$ (or $P$), since these are not within any ball of finite KL-divergence around $P'$. Our approach based on Wasserstein distance would guard against such shifts, because the distance between these distributions and $P'$ remains well defined, with \[
\] While we have focused the above example on KL-divergences, the same critique applies to any $f$-divergence.
In this section we consider robustness with respect to shifts in the distribution of potential outcomes, holding characteristics fixed. This is relevant in a number of scenarios. First, it is relevant for designing policies when welfare is defined using outcomes that may differ slightly from those that are measured in the experimental population. For instance, the policymaker may care about overall educational attainment, while the experiment measures outcomes using a specific test score. It is also relevant when experimental conditions differ from the real-world setting in which the policy is to be implemented.\footnote{In these scenarios, we could think about individuals being characterized by $(X,Y_0,Y_1,Y_0^*,Y_1^*)$, where $(Y_0,Y_1)$ are the experimental-condition outcomes (or measured outcomes) and $(Y_0^*,Y_1^*)$ are the real-world setting outcomes (or policy-relevant outcomes). The distribution $P$ is the marginal for $(X,Y_0,Y_1)$, while $Q$ is the marginal for $(X,Y_0^*,Y_1^*)$.} Two extensions are presented in the appendix: see Appendix (ref) for bounded outcomes and Appendix (ref) for known shifts in characteristics.
We begin introducing an appropriate neighborhood construction to handle this case. To allow for shifts in the distribution of potential outcomes while holding the distribution of characteristics fixed, we use Wasserstein distance induced by the metric
This metric combines the $\ell^1$ norm to penalize the shifts in the distribution of outcomes and a discrete metric on the support $\mc X$ of $X$ to prohibit shifts in characteristics.
The following result characterizes the robust welfare criterion. To simplify the proofs, we assume that the support $\mc Y$ of potential outcomes is equispaced, i.e., there exists a finite $C> 0$ such that for any $y \in \mc Y$ the set $[y - C, y) \cap \mc Y$ is nonempty. Common supports such as $\mb R$ and $\mb Z$ satisfy this condition.
An analogous result holds for robust welfare gain.
We now discuss several implications of these results.
For intuition for this result, a “worst-case” distribution $Q \in \mathcal Q$ for which $W(\tau;Q) = \mathrm{RW}(\tau;P)$ is the image measure of $P$ under the map \[ T(x,y_0,y_1) =
\] That is, the distribution of $X$ is held fixed, but $Y_0$ is reduced by $\varepsilon$ for untreated individuals and $Y_1$ is reduced by $\varepsilon$ for treated individuals. The net effect is to simply reduce welfare by $\varepsilon$, which is why the ordering over policies is the same for the usual criteria and its robust counterpart. Interestingly, this is also a worst-case distribution in the related problem, where $\mc Q$ is defined as the set of distributions $Q$ for which the marginal distribution of $X$ is the same as under $P$ and, for each $x$, the conditional distribution for $(Y_0,Y_1)|X = x$ under $Q$ is constrained to a Wasserstein neighborhood of radius $\varepsilon$ of the conditional distribution under $P$, as in Kido2022.
We now turn to the problem of external validity when we allow for shifts in outcomes and characteristics. We begin in Section (ref) by deriving the robust criteria, and then explore their implications in Section (ref). In particular, we show how the distribution of CATEs and unobserved heterogeneity (within the experimental population) play distinct but important roles in shaping external validity.
Section (ref) discusses empirical implementation, allowing for experimental or observational data. Here the robust welfare criterion depends on the joint distribution of characteristics and individual treatment effects in the experimental population. As this distribution is not identified without further assumptions, we first derive sharp bounds on robust welfare without imposing identifying assumptions and show how the bounds can be estimated nonparametrically. We then provide approaches for estimating robust welfare under different identifying assumptions and establish convergence rates and regret guarantees for the estimated policies. As before, all results in this section are presented for the case in which potential outcomes are unbounded (i.e., $\inf \mathcal Y = -\infty$ and $\sup \mathcal Y = +\infty$ with $\mathcal Y$ the support of $Y_0$ and $Y_1$). See Appendix (ref) for the case where at least one of $\inf \mathcal Y$ or $\sup \mathcal Y$ is finite.
To allow for shifts in the distribution of potential outcomes and characteristics, we use Wasserstein distance induced by the metric
where $\|\cdot\|$ is a norm on $\mc X$.
The first thing to note is the presence of the “$\min$” operation inside the expectation. This term follows from the interpretation of Lagrangian duality as a two-player zero-sum game. Fix any $(X,Y_0,Y_1)$ under $P$. Adversarial nature receives $-Y_0$ from shifting the individual to the non-treatment region at a cost of $\eta h_0(X;\tau)$ for a net gain of $-Y_0 - \eta h_0(X;\tau)$. Similarly, $-Y_1 - \eta h_1(X;\tau)$ represents the net gain from shifting the individual to the treatment region. Nature chooses whichever is larger. Averaging across $P$ yields nature's payoff, which is the negative of the expectation in ((ref)). Also note in view of ((ref)) that both the robust welfare and robust welfare gain criteria induce the same ordering over policies.
The functions $h_0(x;\tau)$ and $h_1(x;\tau)$ represent the “distance to non-treatment” and “distance to treatment” under policy $\tau$ for an individual with characteristics $X = x$. Different policies $\tau$ induce different distances $h_0$ and $h_1$. We illustrate how to compute these for three popular classes of policies. To simplify exposition, we take $\mc X = \mb R^d$ and let $\|\cdot\|$ be the Euclidean norm.
We have so far been vague about the choice of norm $\|\,\cdot\|$ on $\mathcal X$. If the covariate vector $X$ contains variables whose units and scale are comparable to the outcomes (such as in our empirical application, where both are profits), our recommendation is to simply use the Euclidean norm. Otherwise, if $X$ contains variables whose units or scale is not comparable to outcomes, our recommendation is to use the weighted Euclidean norm \[ \|x - \tilde x\| = \sqrt{ (x - \tilde x)'\Sigma^{-1} (x - \tilde x) }, \] where \[ \Sigma = \frac{\mathrm{Cov}(X)}{\mathrm{Var}(Y_0)}. \] This weighting serves two purposes. First, it measures changes in $X$ on the same scale and in the same units as pre-treatment outcomes. Second, it is invariant to rescaling or, more generally, invertible linear transformations of the $X$ variables.
Before turning to empirical implementation, we first discuss some implications of the robust criteria derived in Proposition (ref). We highlight the different roles that treatment effect heterogeneity plays in the ranking of policies. We focus the discussion below on robust welfare, but the findings carry over to robust welfare gain also.
Different patterns of heterogeneity in CATEs can have very different implications for the ranking of policies under distribution shifts. This is most easily seen when individual treatment effects $\Delta = Y_1 - Y_0$ conditional on characteristics are constant, i.e., $\Delta |(X = x) =_{a.s.} \delta(x)$ holds for all $x$ under $P$. We emphasize that our methods do not require this assumption; it is simply to facilitate the following discussion. Under constant treatment effects, the social welfare criterion ((ref)) becomes \[ \mathrm{W}(\tau;P) = \mathbb{E}_P[Y_0] + \mathbb{E}_P[\delta(X)|\tau(X) = 1] P(\tau(X) = 1)\,. \] Consider ranking a policy $\tau$ relative to a policy in which no one is treated. As $P(\tau(X) = 1) > 0$, whether $\tau$ is preferred under criterion ((ref)) is entirely determined by the sign of the average effect of treatment on the treated. In particular, the criterion is invariant to whether individuals with large negative treatment effects who are not treated under $\tau$ are “almost” treated in the sense that $h_1(X;\tau)$ is small. Similarly, it is invariant to whether treated individuals benefitting most from treatment are almost not treated. If so, such a policy may generalize poorly: there are distributions “close” to $P$ in which characteristics of a small mass of individuals with large negative or positive treatment effects are moved across the treatment/non-treatment frontier.
By contrast, the robust criterion from Proposition (ref) becomes
Evidently, the robust criterion considers both the sign of treatment effects and their magnitudes relative to how close individuals are to the treatment/non-treatment frontier. “Robust” policies sort individuals by the size of their CATEs: individuals for whom $\delta(X)$ is small should be near the frontier, those for whom $\delta(X)$ is large should be far.
We illustrate this with a numerical example. Consider the populations $P_a$ and $P_b$ plotted in Figure (ref). Individual treatment effects (conditional on $(x_1,x_2)$) are constant so there is no unobserved heterogeneity. Individuals with $x_1 + x_2 \geq 0$ benefit from treatment whereas those with $x_1 + x_2 < 0$ are adversely affected by it. The EWM policy in $P_a$ and $P_b$ is to treat individuals if and only if $x_1 + x_2 \geq 0$:\footnote{Here we refer to the welfare-optimal policy as the empirical welfare maximizing policy since the population distribution is known.} \[ \tau_{\tiny\text{EWM}}(x) = \mb I[ x_1 + x_2 \geq 0]. \] The frontier between the treatment and non-treatment regions is represented by the dashed line in Figure (ref). This policy generates the same social welfare in $P_a$ and $P_b$, \[ \mathrm{W}(\tau_{\tiny\text{EWM}}; P_a) = \mathrm{W}(\tau_{\tiny\text{EWM}}; P_b), \] despite the different distributions of treatment effects in $P_a$ and $P_b$. A policy that treats everyone (“treat all”) generates welfare $0.25$ (the ATE) in both populations whereas a policy that treats no one (“treat none”) generates zero welfare since we have normalized $Y_0 = 0$ for all individuals in $P_a$ and $P_b$.
Figure (ref) plots the robust welfare of the EWM, treat-all, and treat-none policies. The performance of the EWM policy is relatively insensitive to small perturbations of $P_a$. For instance with $\varepsilon = 0.5$, which represents twice the ATE in $P_a$, the robust welfare of the EWM policy is positive and exceeds that of the treat-all and treat-none policies. But the ranking of the three policies changes dramatically in $P_b$, where the EWM policy eventually performs worse than both the treat-all and treat-none policies. For small $\varepsilon$, robust welfare of the EWM policy decreases at a rate of 5 times neighborhood size in $P_b$, double the rate of decrease in $P_a$.
The EWM policy can be viewed as a special case of linear eligibility score policies in Example (ref). We compute the robust policy that maximizes $\mathrm{RW}(\,\cdot\,;P)$ over this class for $P_a$ and $P_b$ and plot the resulting welfare in Figure (ref). For $P_a$, the EWM policy is very robust: it performs almost as well as the robust policies which use different thresholds to determine treatment status. The robust policy $\tau_{\tiny\text{Robust}}(x) = \mb I[0.314 + x_1 + x_2 \geq 0]$ for $\varepsilon < 0.7$, while for $\varepsilon \geq 0.7$ the robust policy is $\tau_{\tiny\text{Robust}}(x) = \mb I[0.471 + x_1 + x_2 \geq 0]$. For $P_b$, the robust policy is $\tau_{\tiny\text{Robust}}(x) = \mb I[0.157 + x_1 + x_2 \geq 0]$ for $\varepsilon \leq 0.3$ while for $\varepsilon > 0.3$ the robust policy is no longer individualized but rather the treat-all policy. Here the EWM policy is reasonably robust to small shifts but performs much worse than the robust policy over shifts of size $\varepsilon \geq 0.5$.
Overall, this example shows that individualized policies that leverage CATE heterogeneity within the experimental population may generalize poorly when there is a lot of heterogeneity near the treatment/non-treatment frontier.
A second important consequence of Proposition (ref) is that unobserved heterogeneity in treatment effects now plays a role. By contrast, the usual welfare criterion ((ref)) is entirely invariant to unobserved heterogeneity. To see this, note that the inner expectation in the robust criterion ((ref)) can be written
Consider populations $P_c$ and $P_d$ such that $(X,Y_0)$ has the same distribution in both, but $\Delta|(X = x) =_{a.s.} \delta(x)$ holds for all $x$ in $P_c$ whereas $\mathbb{E}_{P_d}[\Delta|X = x] = \delta(x)$ in $P_d$. Thus, the CATEs are the same in $P_c$ and $P_d$, but $P_d$ exhibits unobserved heterogeneity whereas $P_c$ does not. It follows from expression ((ref)) that \[ \mathrm{RW}(\tau;P_c) \geq \mathrm{RW}(\tau;P_d). \] The same is true for robust welfare gain. For intuition, nature's minimization over $Q$ transforms the objective from linear in $\Delta$ to concave in $\Delta$. Hence by Jensen's inequality, robust welfare is lower when there is unobserved heterogeneity. This is in contrast to the usual criteria ((ref)) and ((ref)) which are invariant to unobserved heterogeneity: \[ \mathrm{W}(\tau;P_c) = \mathrm{W}(\tau;P_d)\,, \quad \quad \mathrm{WG}(\tau;P_c) = \mathrm{WG}(\tau;P_d)\,. \]
To see how unobserved heterogeneity plays a distinct role from heterogeneity in CATEs, consider the population $P_a$ described in Figure (ref)(a). Individual treatment effects are degenerate (conditional on characteristics) in this population. We construct alternative populations with unobserved heterogeneity in treatment effects by adding independent mean-preserving spreads $\sigma W$ to individuals' treated outcomes, where $\sigma \geq 0$ and $P(W = 1) = P(W = -1) = \frac 12$. Hence, in these populations, the distributions of $X$ and CATEs are the same as Figure (ref)(a), all that changes is the variance of $Y_1$. Figure (ref)(a) plots the robust welfare of the EWM policy $\tau_{\tiny\text{EWM}}(x) = \mb I[ x_1 + x_2 \geq 0]$ for $\sigma \in \{0, 1, 2, 3\}$. When $\varepsilon = 0$, robust welfare is the same for all values of $\sigma$, reflecting the fact that the social welfare criterion ((ref)) is invariant to unobserved heterogeneity. But as the degree of unobserved heterogeneity $\sigma$ and/or $\varepsilon$ increases, the EWM policy eventually performs worse than the treat-all and treat-none policies.
For each $\varepsilon$ and $\sigma$, we solve for the robust policy over the class of linear eligibility score policies. We plot the results in Figure (ref)(b). The solid black line corresponds to the robust policy for $\sigma = 0$ (this is the line plotted in red in Figure (ref)(a)), shaded lines correspond to robust policies with $\sigma > 0$. For small $\sigma$ and/or small $\varepsilon$, robust policies are all of the form $\tau_{\tiny\text{Robust}}(x) = \mb I[\beta_0 + x_1 + x_2 \geq 0]$, where $\beta_0 \geq 0$ depends on $\sigma$ and $\varepsilon$. For certain $\sigma$ and $\varepsilon$, the optimal $\beta_0 = 0$, in which case $\tau_{\tiny\text{Robust}} = \tau_{\tiny\text{EWM}}$. For other $\sigma$ and $\varepsilon$, the optimal $\beta_0 > 0$, in which case the robust policy assigns treatment for a larger share of the population than does the EWM policy. For sufficiently large $\varepsilon$ and $\sigma$, the robust policy is no longer individualized but rather the treat-all policy.
We now present strategies for estimating externally valid policies given a finite data set drawn from the experimental population. The challenge, seen from ((ref)), is that the criterion involves the expectation of a nonlinear function of individual treatment effects $\Delta = Y_1 - Y_0$. Even in an experiment, the distribution of $(X,\Delta)$ is not identified without further assumptions. We first show how to estimate sharp bounds on robust welfare in the absence of identifying assumptions. We then show how to estimate robust welfare under different identifying assumptions. For brevity we confine most of the discussion that follows to the robust welfare criterion, though in view of ((ref)) the results carry over immediately to robust welfare gain.
In this section we derive sharp bounds on the robust criteria and show how they may be estimated. The bounds depend only on the joint distributions of $(X,Y_0)$ and $(X,Y_1)$ in the experimental population $P$. These two distributions are identified under standard unconfoundedness (selection on observables) and overlap assumptions. They can also be identified from observational data under various assumptions.
Our approach follows HeckmanSmithClements1997, Manski1997, and in particular FanPark2010 and Stoye2010 (see also ImbensMenzel2021). We seek to minimize and maximize $\mathrm{RW}(\tau;\tilde P)$ over all distributions $\tilde P$ for which the marginals for $(X,Y_0)$ and $(X,Y_1)$ are the same as under $P$. Different couplings of the conditional distributions of $Y_0|X$ and $Y_1|X$ induce different distributions for $\Delta|X$. The expectation appearing in ((ref)) may be written as \[ \mathbb{E}_P \left[ \mathbb{E}_P \left[ \left. \min \left\{ \Delta + \eta h_1(X;\tau) , \eta h_0(X;\tau) \right\} \right| X \right] \right]. \] Because the inner expectation is taken over a concave function of $\Delta$, it suffices to consider second-order stochastic dominance (SSD) relations among distributions induced by different couplings. FanPark2010 show that perfect positive dependence (or rank invariance) of $Y_0|X$ and $Y_1|X$ produces a distribution of $\Delta|X$ that SSDs all distributions of $\Delta|X$ induced by other couplings. Similarly, perfect negative dependence (or rank reversal) produces a distribution of $\Delta|X$ that is SSD by all distributions induced by other couplings. The upper and lower bounds on robust welfare are therefore achieved under perfect positive and negative dependence, respectively: \[ \mathrm{RW}(\tau; P^-) \leq \mathrm{RW}(\tau; P) \leq \mathrm{RW}(\tau; P^+) \quad \text{for all $\tau$}, \] where $P^-$ and $P^+$ denote the joint distribution of $(X,Y_0,Y_1)$ under perfect negative and positive dependence of $Y_0|X$ and $Y_1|X$, respectively, for $P$-almost every $X$. An analogous ranking holds for robust welfare gain.
\@startsection{paragraph}{4}{\z@} {0pt \@plus1ex \@minus.2ex} {-1em} {\normalfont}{Empirical Implementation.} Suppose we observe a random sample $(X_i,Y_i,D_i)_{i=1}^n$, where $D_i$ is a binary treatment indicator, $Y_i = D_i Y_{1i} + (1-D_i)Y_{0i}$, and the unconfoundedness condition
and overlap condition
for all $x$ in the support of $X$, both hold. Then both $F_0(y|x) = P(Y_0 \leq y|X = x)$ and $F_1(y|x) = P(Y_1 \leq y|X = x)$ are identified:
Standard nonparametric methods can be used to estimate $F_0$ and $F_1$ based on ((ref)). If unconfoundedness fails, then $F_0(y|x)$ and $F_1(y|x)$ may be identified using instrumental variables---see, e.g., VuongXu2017 and references therein.
To simplify exposition, suppose that $Y_0$ and $Y_1$ are continuously distributed conditional on $X$. Given $F_0$ and $F_1$, counterfactual mappings may be constructed as \[ \phi_x^+(\cdot) = F_1^{-1}(F_0(\,\cdot\,|x)|x) \,, \quad \quad \phi_x^-(\cdot) = F_1^{-1}(1-F_0(\,\cdot\,|x)|x) \,. \] The function $\phi_x^+$ maps an individual's untreated outcome to their treated outcome under perfect positive dependence; its inverse maps treated outcomes to untreated outcomes. The function $\phi_x^-$ and its inverse give the same counterfactual mappings under perfect negative dependence. Individual $i$'s treatment effects under perfect positive and negative dependence are therefore
Given estimates $\hat F_0$ and $\hat F_1$, we may construct estimates $\hat \phi_x^+$ and $\hat \phi_x^-$ of $\phi_x^+$ and $\phi_x^-$. We may then estimate individual treatment effects under both dependence assumptions using
Finally, given an estimate $\bar Y_0$ of $\mathbb{E}_P[Y_0]$, we construct an empirical robust welfare criterion under perfect positive dependence as follows:
The criterion $\mathrm{ERW}_n(\tau;P^-)$ for perfect negative dependence is constructed analogously, replacing $\hat \Delta_i^+$ with $\hat \Delta_i^-$ in the above expression. In view of ((ref)), an empirical counterpart to robust welfare gain can be constructed by setting $\bar Y_0 = 0$ in ((ref)). As such, Propositions (ref) and (ref) apply equally to robust welfare gain.
In practice, the optimization over $\eta$ may be efficiently performed using linear programming. For scalars $a_i,b_i,c_i$, $i = 1,\ldots,n$, we have
It is possible to relax boundedness of $\mc X$ and derive convergence rates under conditions on $\mc T$ and the estimators $\hat \Delta^{\!\pm}$ and $\bar Y_0$. Here we give one such result. Each $\tau \in \mc T$ is identified with a decision set $C$ such that $\tau(x) = \mb I [x \in C]$. Let $\mc C$ be the set of all decision sets for $\tau \in \mc T$. A standard assumption is that $\mc C$ is a VC class (i.e., has finite VC dimension),\footnote{We refer the reader to Chapter 2.6 of vanderVaartWellner for terminology.} which means $\mc T$ has the same complexity as a parametric class. We impose a related condition. Let $\ol C$ denote the closure of a set $C$ and for $\delta \geq 0$ let $\ol C^\delta = \{ x : \inf_{\tilde x \in \ol C} \|x - \tilde x\| \leq \delta\}$ denote its $\delta$-expansion. Let $\mc C^c = \{C^c : C \in \mc C\}$ and let $\mc C^* = \{ \ol C^\delta : C \in \mc C \cup \mc C^c, \delta \geq 0\}$. We require that $\mc C^*$ is a VC class, though we allow its VC dimension $v_n$ to grow with the sample size.
A simple approach to identifying $P$ is to assume treatment effects are constant conditional on $X$, i.e.,
for some function $\delta : \mc X \to \mb R$. Under this assumption, treatment effects can vary across individuals with different characteristics but are homogeneous for individuals with the same characteristics. While this assumption implies perfect positive dependence, it simplifies empirical implementation as the problem of estimating the counterfactual mapping $\phi_x^+$ is replaced by the (simpler) problem of estimating $\delta$.
\@startsection{paragraph}{4}{\z@} {0pt \@plus1ex \@minus.2ex} {-1em} {\normalfont}{Empirical Implementation.} Suppose we observe a random sample $(X_i,Y_i,D_i)_{i=1}^n$ where $D_i$ is a binary treatment indicator and $Y_i = D_i Y_{1i} + (1-D_i)Y_{0i}$. Then $\delta$ is identified under the unconfoundedness condition ((ref)) and the overlap condition ((ref)). We may estimate $\delta$ using a variety of nonparametric regression techniques. Alternatively, suppose we observe $(X_i,Y_i,D_i,Z_i)_{i=1}^n$ where $Z_i$ is an instrumental variable satisfying appropriate regularity conditions (see, e.g., Abadie2003). By ((ref)), $\delta$ is identified as the ratio \[ \delta(x) = \frac{\mathrm{Cov}(Y_i,Z_i|X_i = x)}{\mathrm{Cov}(D_i,Z_i|X_i = x)} \] and may be estimated using nonparametric instrumental variables methods.
In either case, given estimators $\hat \delta$ of $\delta$ and $\bar Y_0$ of $\mathbb{E}_P[Y_0]$, we take $\hat \Delta_i \equiv \hat \delta(X_i)$ to be our estimate of $\Delta_i \equiv \delta(X_i)$ and form the empirical robust welfare criterion
As before, the optimization over $\eta$ can be efficiently implemented via linear programming (see equation ((ref))). We can estimate policies that are robust to shifts in outcomes and characteristics by maximizing this criterion with respect to $\tau \in \mc T$. Asymptotic properties of the estimated policy $\hat \tau$ may be established under a variety of regularity conditions. The following high-level results allow for both experimental and observational data.
For the next result, we again identify each $\tau \in \mc T$ with a decision set $C$ such that $\tau(x) = \mb I[x \in C]$ and impose a complexity condition from Section (ref) on the sets.
An alternative identifying assumption is to posit the existence of a random variable $V$ such that all dependence between $Y_0$ and $Y_1$ given $X$ comes through $V$:
AbbringHeckman2007, and that a suitably modified version of unconfoundedness holds:
The variables $V$ may coincide with those in $X$, or may be distinct. For example, $Y_0$ and $Y_1$ may be conditionally independent given $X$ and a latent common factor $F$, and $V$ could be a perfect proxy for $F$ constructed from $X$ and other observables. A special case is group fixed effects, where potential outcomes (and possibly the assignment mechanism) has a group-specific component $\alpha_V$. Using a linear model for simplicity, we might have \[
\] with $C$ denoting the individual's group membership (assumed known) and where $u_0$ and $u_1$ are conditionally independent given $(X,V)$.
Suppose the analyst observes a random sample $(X_i,Y_i,D_i,V_i)_{i=1}^n$ in which ((ref)) and ((ref)) hold and a suitably modified version of the overlap condition ((ref)) holds. The conditional CDFs $F_0(\cdot|x,v)$ and $F_1(\cdot|x,v)$ are then nonparametrically identified as the conditional CDFs of $Y$ given $D = d,X = x,V=v$ for $d = 0,1$, respectively. Assumption ((ref)) then permits identification of the robust welfare criterion under $P$. Note the conditional CDF $G_{\eta,\tau}(\cdot|x,v)$ of $Z := \min\left\{ Y_0 + \eta h_0(X;\tau)\,, Y_1 + \eta h_1(X;\tau) \right\} $ given $X = x$ and $V = v$ is \[ G_{\eta,\tau}(z|x,v) = 1 - (1 - F_0(z-\eta h_0(x;\tau)|x,v))(1 - F_1(z-\eta h_1(x;\tau)|x,v)), \] where $F_0$ and $F_1$ are the conditional CDFs of $Y_0$ and $Y_1$ given $X = x$ and $V = v$. It follows by iterated expectations that \[ \mathbb{E}_P \left[ \min\left\{ Y_0 + \eta h_0(X;\tau)\,, Y_1 + \eta h_1(X;\tau) \right\} \right] \\ = \int \int z \, \mr d G_{\eta,\tau}(z|x,v) \, \mr d P_{X,V}(x,v)\,, \] where $P_{X,V}$ is the distribution of $(X,V)$ in the experimental population.
Given estimators $\hat F_0(y|x,v)$ and $\hat F_1(y|x,v)$, we estimate $G_{\eta,\tau}$ using \[ \hat G_{\eta,\tau}(z|x,v) = 1 - (1 - \hat F_0(z-\eta h_0(x;\tau)|x,v))(1 - \hat F_1(z-\eta h_1(x;\tau)|x,v))\,. \] A useful feature of $\hat G_{\eta,\tau}$ is that it is uniformly consistent in $(z,\eta,\tau)$ whenever $\hat F_0(y|x,v)$ and $\hat F_1(y|x,v)$ are uniformly consistent in $y$. We choose $\hat \tau$ by maximizing the empirical robust welfare criterion
with respect to $\tau \in \mc T$. The following result establishes consistency. Let $\bar F_d(y) = \frac{1}{n} \sum_{i=1}^n \hat F_{d}(y|X_i,V_i)$ for $d = 0,1$, let $a,A$ be finite positive constants, and let wpa1 denote “with probability approaching one”.
As discussed in Section (ref), heterogeneity in CATEs and unobserved heterogeneity both play distinct yet important roles in shaping generalizability. To better understand the separate roles they play, we can isolate the role of CATE heterogeneity and shut down the latter channel as follows.\footnote{We are grateful to an anonymous referee for this suggestion.} First, we can write the usual welfare objective ((ref)) as \[ \mathrm{W}(\tau;P) = \mathbb{E}_P[ D \, \tau(X) + M] , \] where $(D,M)|(X = x) =_{a.s.} (\delta(x), \mu_0(x))$ under $P$, with $\delta(x) = \mathbb{E}[ \Delta|X = x]$ and $\mu_0(x) = \mathbb{E}[Y_0|X = x]$. Suppose we wish to consider robustness with respect to shifts in the distribution of $(D, M, X)$. We constrain the class of distributions to Wasserstein neighborhoods centered at $P$ and defined using the metric \[ d((\delta,m,x),(\tilde \delta, \tilde m, \tilde x)) = |\delta - \tilde \delta| + |m - \tilde m| + \|x - \tilde x\|. \] The “worst-case” distributions that minimize robust welfare are also degenerate in $(D,M)|X = x$. By similar arguments to Proposition (ref), we arrive at the robust welfare objective
This is identical to the robust welfare formulation from Section (ref), except we now interpret $\delta(x)$ as the CATE. Given estimates $\hat \delta$ of $\delta$ and $\bar Y_0$ of $\mathbb{E}_P[Y_0]$, we can estimate policies that are robust to this class of distribution shifts by maximizing criterion ((ref)) with $\hat \Delta_i = \hat \delta(X_i)$. Theoretical properties of the estimated policies follow from Propositions (ref) and (ref).
In related work, Kido2022 uses a sequential neighborhood construction in which the distribution of $(Y_0,Y_1)|X$ can vary over a Wasserstein neighborhood, then the marginal distribution of $X$ can shift over a $f$-divergence neighborhood. That construction likewise eliminates the influence of unobserved heterogeneity. Moreover, it results in an objective function that is essentially equivalent to that of MoQiLiu, who fix the conditional distribution of $(Y_0,Y_1)|X$ between the experimental and target populations, but allow the marginal distribution of $X$ to vary over $f$-divergence neighborhoods. These approaches similarly produce a criterion function involving an expectation over a convex function of the CATE that is optimized with respect to a Lagrange multiplier representing the neighborhood constraint.
\@startsection{paragraph}{4}{\z@} {0pt \@plus1ex \@minus.2ex} {-1em} {\normalfont}{Example: Joint Shifts or Fixed-CATE Shifts?}
The construction here in essence holds the CATE fixed, whereas our earlier approach allows the joint distribution of $(X,Y_0,Y_1)$ to shift freely.
To understand the differences between these approaches, consider the numerical example presented in Section (ref). There we start with a given DGP with no unobserved heterogeneity, then add mean-preserving spreads $\sigma W$ to the treated outcome $Y_1$, where $\sigma \geq 0$ is a scale parameter and $P(W = 1) = P(W = -1) = \frac 12$. Increasing $\sigma$ leaves the CATE unchanged but increases unobserved heterogeneity.
Under criterion ((ref)), the conditional distribution of treatment effects around the CATE (i.e., unobserved heterogeneity) is irrelevant. Thus, for any $\sigma \geq 0$, criterion ((ref)) yields the same value of robust welfare and therefore the same ranking over treatment policies. By contrast, Figure (ref) shows that this is not so for the earlier criterion ((ref)): unobserved heterogeneity matters and different values of $\sigma$ can produce different rankings over treatment policies.
Why the difference? Criterion ((ref)) guards against distribution shifts where individuals may be adversarially reassigned (by shifting their $X$) into the treatment/non-treatment regions based on the size of their individual treatment effects. Criterion ((ref)) does not guard against shifts with this “selection” feature. Instead, it guards against a smaller class of shifts in which individuals with the same $X$ all undergo the same shift into the treatment/non-treatment region based on the size of their common CATE.
The literature dealing with empirical welfare maximization (EWM) typically focuses attention on policies that attain the highest reward while ignoring cost of treatment. We have followed this convention. However, our approach easily extends to allow for a fixed treatment cost $c$ per individual, as in KitagawaTetenov2018, by redefining $Y_1$ as the treated outcome net of that cost. Thus, a “cost-aware” version of the robust welfare objective ((ref)) is \[ \mathrm{RW}(\tau;P) = \sup_{\eta \geq 1} \mathbb{E}_P \left[ \min \left\{ Y_0 + \eta h_0(X;\tau) , Y_1 - c + \eta h_1(X;\tau) \right\} \right] - \eta \varepsilon . \] Similarly, cost-aware empirical robust welfare criteria can be obtained by replacing $\hat \Delta^+_i$ and $\hat \Delta_i$ in ((ref)) and ((ref)) with $\hat \Delta^+_i - c$ and $\hat \Delta_i - c$, respectively.
We now present an empirical application using data from MKP2021. The authors study the effect of the Gender and Enterprise Together business training program on various outcomes (e.g., profits) for a sample of rural Kenyan firms run by female entrepreneurs. The training focuses on both standard business topics (e.g., record keeping and pricing) and topics designed to overcome gender-related constraints (e.g., division of household and business tasks). Firm outcomes are evaluated over a four-year window following training, raising the possibility that distributions may drift over time.
The experiment used two layers of randomization: at the market level and again at the firm level. Markets were randomly assigned to treatment (at least one firm in the market is invited to training) or control (no firm is invited to training), conditional on strata. Strata were determined by geographic region and the number of firms in the market. Within treated markets, firms were randomly assigned treatment conditional on their baseline weekly profits, as reported in a pre-experiment market census. Thus, assignment is random conditional on both strata and baseline profits.
The outcome variable $Y$ is profit one year after training.\footnote{We convert profits in years after training to real values in the year of training using Kenyan CPI data in the MKP2021 replication files.} We take the experimental population to be firms in treatment markets. We drop the few firms that do not report subsequent profits, leaving a sample of 2,009 firms, of which 1,096 were assigned the treatment. The average treatment effect is approximately 157 Kenyan Shillings (KSh), roughly 11% of baseline profits.
We focus on a class of tree-based policies that assign treatment as a function of baseline profits (also measured in KSh). We consider trees of depth 2, so treatment is assigned depending on whether baseline profits fall in a certain interval. Within this class, the empirical welfare maximizing (EWM) policy is to treat if baseline profits exceed 40 KSh and are less than 3,034 KSh. Thus, only the least and most profitable firms are excluded from treatment under the EWM policy (see Figure (ref)).
To examine the robustness of this policy to adversarial shifts in outcomes and baseline profits, we proceed as in Section (ref). We impute counterfactual mappings using kernel estimates of the distribution of treated and untreated outcomes conditional on strata and baseline earnings.\footnote{We estimate the conditional CDFs of treated and untreated outcomes nonparametrically using the R package np with bandwidths chosen by cross validation, treating strata as categorical. } Although we use both baseline profits and strata to impute individual causal effects, strata do not factor into our neighborhood construction. Policies are solely a function of baseline profits, so only baseline profits are relevant for computing robust welfare. We take the lower bound on the support of outcomes to be $\underline Y =0$ as the post-treatment profit distribution appears truncated at zero. Similar results are obtained with $\underline Y = -\infty$.
First consider the case of perfect positive dependence. Figure (ref) plots empirical robust welfare $\mathrm{ERW}_n(\tau;P^+)$ from ((ref)) for the EWM policy and policies in which all firms or no firms are treated, for $\varepsilon \in \{0.05, 0.1, 0.5, 1, 5, 10, 25, 50, 75, 100\}$. For each $\varepsilon$ we also compute the robust policy $\hat \tau$ that maximizes $\mathrm{ERW}_n(\tau;P^+)$ in ((ref)).\footnote{To do so, we first ran a search over a fine grid of thresholds to select parameter values that approximately maximized criterion ((ref)). We then used these as starting values for gradient-free optimization algorithm and selected the parameter values that yielded the highest optimum.} To interpret $\varepsilon$, note that the estimated ATE is 157 KSh, so the neighborhood size of $\varepsilon$ includes populations with an ATE between $157-\varepsilon$ and $157+\varepsilon$ KSh. Remark (ref) also notes that $\varepsilon$ is the amount by which average pre-treatment outcomes can shift between the experimental and target populations. Mean reported profits for untreated firms one and three years after treatment are approximately 1,467 KSh and 1,493 KSh, a difference of approximately 26 KSh (or 13 KSh per year). Consider a policymaker deciding whether to implement the policy four years, say, after the initial treatment. An annual drift around 13 KSh suggests that $\varepsilon \approx 50$ might be reasonable. Figure (ref) shows that even for neighborhoods of size $\varepsilon \geq 1$, the EWM policy performs much worse than the policy in which everyone is treated.
The robust policy is different for different $\varepsilon$. For very small $\varepsilon$, the roust policy treats all firms with pre-intervention earnings below a higher threshold (3,299 with $\varepsilon = 0.1$ and 3,245 KSh with $\varepsilon = 0.5$) than the EWM policy. For larger values of $\varepsilon$ (say, $\varepsilon \geq 1$) the robust policy is the policy in which all firms are treated. For intuition, Figure (ref) plots the imputed individual treatment effects $\hat{\Delta}_i^+$ under perfect positive dependence against pre-treatment earnings. Under the EWM policy, all firms between the two vertical black dashed lines are treated, while those outside this interval are not treated. The upper threshold of 3,034 under the EWM policy is just to the right of a large mass of firms who are positively affected (on average), but with a lot of heterogeneity. Under adversarial shifts, some of these firms who are positively affected can be reallocated across the frontier into the non-treatment region, reducing welfare. The robust policy uses a higher threshold, so that “larger” shifts are needed to push these firms across the frontier. For $\varepsilon = 0.5$, this threshold is roughly midway between the previously mentioned mass of firms, and another mass with higher pre-treatment earnings but who are negatively affected (on average) by treatment. As there is still a lot of heterogeneity on both sides of this threshold, for larger values of $\varepsilon$ it is optimal to choose a policy in which all firms are treated. This finding is consistent with the numerical experiments we performed in Section (ref).
It is also striking in Figure (ref) that the robust welfare of the EWM policy decays rapidly over small neighborhoods. For instance, with $\varepsilon = 25$ the robust welfare of the EWM policy is over 100 KSh below its welfare in the experimental population. Figure (ref) shows this is explained by the large heterogeneity in individual treatment effects, so that only small shifts are required to reallocate firms near the boundary who benefit most from treatment into non-treatment regions and vice versa. Nevertheless, robust welfare of the EWM policy remains above that of the treat-none policy. Moreover, robust welfare gain of the EWM policy is positive up to a neighborhood of size 70 or so, roughly 45% of the experimental population ATE. Thus, the EWM policy should deliver welfare improvements relative to treating no one, even allowing for a broad class of distribution shifts.
On the other hand, no individualized policy delivers better robust welfare than a one-size-fits-all policy in which all firms are treated, except for the smallest values of $\varepsilon$. And even at these small values of $\varepsilon$, the difference in $\mathrm{RW}(\tau;P^+)$ for the robust policy and treat-everyone policy is less than 10 KSh, which represents only a few percent of the ATE. As 98% of the experimental population is treated under the EWM policy, the additional cost of implementing the robust policies seems marginal relative to that of the EWM policy.
We repeat the analysis under perfect negative dependence. Results are plotted in Figure (ref). Individual treatment effects are more heterogeneous under perfect negative dependence: the standard deviations of $\hat \Delta_i^-$ and $\hat \Delta_i^+$ are approximately 3,225 and 706 KSh, respectively. In view of the discussion in Section (ref), it is not surprising that robust welfare decays much faster as $\varepsilon$ increases. Indeed, the treat-all policy maximizes $\mathrm{RW}(\tau;P^-)$ for $\varepsilon \geq 0.5$. Overall, it appears preferable to adopt a policy providing training to all firms.
We consider the problem of learning personalized treatment policies that perform well in target populations beyond the experimental (or training) population from which the data are sampled. This paper makes two main contributions. First, we develop new methods for policy learning that are robust to shifts in the distribution of both outcomes and characteristics between the experimental and target populations. Second, we shed light on how heterogeneity in CATEs and unobserved heterogeneity in treatment effects (within the experimental population) play distinct but important roles in shaping external validity. In addition to policy learning, the methods developed in this paper may be used as a stress test to assess the fragility or robustness of treatment policies to distribution shifts.