Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
46,504 characters · 18 sections · 35 citation commands
Efficient Policy Learning from Surrogate-Loss Classification Reductions
Policy learning from observational data is an important but challenging problem because it requires reasoning about the effects of interventions not observed in the data. For example, if we wish to learn an improved policy for medical treatment assignment based on observational data from electronic health records, we must take care to consider potential confounding: since healthier patients who were already predisposed to positive outcomes were likely to have historically been assigned less invasive treatments, na\"ive approaches may incorrectly infer that a policy of always assigning less invasive treatments will obtain better outcomes.
Various recent work has recently tackled this problem, known as policy learning from observational (or, off-policy) data, by optimizing causally-grounded estimates of policy value such as inverse-propensity weighting (IPW), doubly robust (DR) estimates, or similar qian2011performance,beygelzimer2009offset,kitagawa2018should,swaminathan2015counterfactual,zhao2012estimating,zhou2017residual,jiang2019entropy,kallus2018policy,kallus2018balanced,kallus2017recursive. In particular, athey2017efficient,zhou2017residual, among others, highlight the importance of using efficient estimates of policy value as optimization objectives, i.e., having minimal asymptotic mean-squared error (MSE). Examples of efficient estimators are direct modeling or IPW when outcome functions or propensities are sufficiently smooth hirano2003efficient,hahn1998role, or DR leveraging cross-fitting chernozhukov2017double in more general non-parametric settings.
Regardless of which of the three estimates one uses, the resulting optimization problem amounts to a difficult binary optimization problem. Therefore many of the above leverage a reduction of this problem to weighted classification (for two actions; cost-sensitive classification more generally) and leverage tractable convex formulations that use surrogate loss functions for the zero-one loss, such as, for example, hinge loss zhao2012estimating,zhou2017residual and logistic loss jiang2019entropy. The recently proposed entropy learning approach of jiang2019entropy is particularly appealing, since the logistic regression-based surrogate loss is smooth and therefore allows for statistical inference on the estimated optimal parameters.
However, as we here emphasize, even if we use policy value estimates that are efficient, this does not imply that we obtain efficient estimation/learning of the optimal policy itself, even if the surrogate-loss model is well-specified. For example, in the case of logistic loss, we demonstrate that, although logistic regression is statistically efficient for actual binary classification when well-specified (as is well-known), in the case of policy learning via a weighted-classification reduction, well-specification only implies a semi-parametric model and therefore minimizing the empirical average of loss is not efficient in this case.
On the other hand, the implications of correct specification can be summarized as a conditional moment problem. Such problems are amenable to efficient solution using approaches based on the generalized method of moments hansen1982large. We demonstrate what an efficient such estimate would look like, in terms of the efficient instruments for our specific policy learning problem. We propose a particular implementation of solving our problem based on recent work on efficiently solving conditional moment problems using a reformulation of the efficient GMM solution as a smooth game optimization problem, which can be solved using adversarial training of neural networks bennett2019deep. In addition, we prove some results relating the efficiency of optimal policy estimation to the asymptotic regret of the surrogate loss, and also prove that under correct specification the regret of the surrogate loss upper bounds the true regret of policy learning.
We demonstrate empirically over a wide range of scenarios that our methodology indeed leads to greater efficiency, with lower MSE in estimating the optimal policy parameter estimates under correct specification. Furthermore, we demonstrate that in practice, both with and without correct specification, our methodology tends to learn policies with lower regret, particularly in the low-data regime.
Let $X$ denote the context of an individual, $T \in \{-1,1\}$ the treatment assigned to that individual, and $Y$ the resultant outcome. In addition let $Y(t)$ denote the counterfactual outcome that would have been obtained for the corresponding individual if, possibly contrary to fact, treatment $t$ had been assigned instead. We assume throughout that we have access to logged data consisting of $n$ iid observations, $\mathcal S_n=\{(X_i,T_i,Y_i):i\leq n\}$, of triplet $(X,T,Y)$ generated by some behavior policy.
We make standard causal assumptions of consistency and non-interference, which can be summarized by assuming that $Y = Y(T)$. Furthermore, as is standard in the above policy learning literature, we assume that $X$ encapsulates all possible confounders, that is, $Y(t)\bot T\mid X\ \forall t \in \{-1,1\}$, as would for example be guaranteed if the logging policy is a function of the observed individual context.
A policy $\pi$ denotes a mapping from individual context to treatment to be assigned. Concretely, given individual context $x$, let $\pi(x)\in\{-1,+1\}$ denote the treatment assigned by policy $\pi$ (we may also consider stochastic policies but since optimal policies are deterministic we focus on these).
Let
denote the expected value of following policy $\pi$, relative to complete randomization. Given the logged data and some policy class $\Pi$, our task is to learn an optimal policy from the class, defined by $\pi^* \in \operatorname*{arg\,max}_{\pi \in \Pi} J(\pi)$ (notice that offsetting by the complete randomization policy does not affect this optimization problem). In particular we consider policy classes where each policy $\pi$ is indexed by some utility function $g$ and is defined by $\pi(x) = \operatorname{sign}(g(x))$, where in turn the utility functions are parametrized by $\theta\in\Theta\subseteq\mathbb R^d$ as $\mathcal G = \{g_\theta : \theta \in \Theta\}$, so that $$\Pi=\{\operatorname{sign}(g_\theta(x)):\theta\in\Theta\}.$$ Correspondingly, we define $$J(\theta)=J(\operatorname{sign}(g_\theta(\cdot)))=\mathbb{E}\xspace[\operatorname{sign}(g_\theta(X))(Y(+1)-Y(-1))]$$ and $\theta^*\in\operatorname*{arg\,max}_{\theta\in\Theta}J(\theta)$. A prominent example is linear decision rules, where $g_\theta(x)=\theta^Tx$. Other examples include decision trees of bounded depth and neural networks.
We briefly review what it means to estimate the optimal policy parameters, $\theta^*$, efficiently. For simplicity, suppose that $\theta^*$ is unique. A model $\mathcal M$ is some set of distributions for the data-generating process (DGP), i.e., a set of probability distributions for the triplet $(X,T,Y)$.
A model is generally non-parametric in the sense that this set of distributions can be arbitrary, infinite, and infinite dimensional.
Consider any learned policy parameters $\hat\theta$, that is, a function of the data $\mathcal S_n$ with values in $\Theta$. Roughly speaking, we say that $\hat\theta$ is regular if, whenever the data is generated from $(X_i,T_i,Y_i)\sim p\in\mathcal M$, we have that $\sqrt{n}(\hat\theta-\theta^*)$ converges in distribution to some limit as $n\to\infty$ and this limit holds in a particular locally uniform sense in $\mathcal M$ (see van2000asymptotic for a precise definition). Semiparametric efficiency theory (see ibid.) then establishes that there exists a covariance matrix $V$ such that for any cost function $c:\mathbb R^d\to\mathbb R$ for which the sublevel sets are $\{v:c(v)\leq c_0\}$ are convex, symmetric about the origin, and closed, we have that
for any estimator $\hat\theta$ that is regular in $\mathcal M$. An important example is MSE, given by $c(v)=\|v\|_2^2$.
Efficient estimators are those for which (ref) holds with equality for all such functions $c$, which, by the portmanteau lemma, would be implied if the estimator has the limiting law $\sqrt{n}(\hat\theta-\theta^*)\Rightarrow\mathcal N(0,V)$. Regular estimators is a very general class of estimators so the bound in (ref) is rather strong. So much so that, in fact, (ref) holds in a local asymptotic minimax sense for all estimators (see ibid., Theorem 25.21).
Efficiency is important because, in observational data, we only have the data that we have and cannot experiment or simulate to generate more so we should use the data optimally. (ref) relates to the efficiency of estimating $\theta^*$. In (ref) we also relate this to regret.
There has been a variety of past work on the problem of policy learning from observational data. Much of this work considers formulating the objective of policy learning as a weighted classification problem beygelzimer2009offset,dudik2011doubly, and either minimizing the 0-1 loss directly using combinatorial optimization athey2017efficient,kitagawa2018should,zhou2018offline, using smooth stochastic policies to obtain a nonconvex but smooth loss surface swaminathan2015counterfactual, or replacing the 0-1 objective with a convex surrogate to be minimized instead zhao2012estimating,zhou2017residual,jiang2019entropy,beygelzimer2009offset,dudik2011doubly. In addition there is work that extends some of the above approaches to the continuous action setting kallus2018policy,krishnamurthy2019contextual,chernozhukov2019semi; our focus will be solely on binary actions. Of these methods the convex-surrogate approach has the advantage of computational tractability and, when the convex surrogate is smooth jiang2019entropy, the ability to perform statistical inference on the optimal parameters. Our paper extends this work by investigating how to solve the smooth surrogate problem efficiently. Although much of this past work has used objective functions for learning based on statistically efficient estimates of policy value dudik2011doubly,athey2017efficient,zhou2018offline,chernozhukov2019semi, to the best of our knowledge our paper is novel in investigating the efficient estimation of the optimal policy parameters themselves.
In addition there has been a variety of past work on solving conditional moment problems (see khosravi2019non,bennett2019deep and citations therein). Our paper builds on this work as it reformulates the problem of policy learning as a conditional moment problem, which we propose to solve using optimally weighted GMM hansen1982large and DeepGMM bennett2019deep.
In this section, we present the surrogate-loss reduction of policy learning and the implications of correct specification.
Many policy learning methods start by recognizing that the policy value can be re-written as
where $\psi$ is any of the following score variables, which all depend on observables:
where $e_t(x) = P(T=t \mid X=x)$ and $\mu_t(x) = \mathbb{E}\xspace[Y(t) \mid X=x]$. (ref) arises once we recognize that all of these satisfy $\mathbb{E}\xspace[\psi \mid X] = \mathbb{E}\xspace[Y(1) - Y(-1) \mid X]$.
Then we can approximate (ref) using its empirical version:
In particular, athey2017efficient,kitagawa2018should,zhou2018offline prove bounds of the form $\sup_{\theta\in\Theta}|J_n(\theta)-J(\theta)|\xspace=O_p(1/\sqrt{n})$ given that the policy class has bounded complexity. This shows that optimizing $\hat\theta\in\operatorname*{arg\,max}_{\theta\in\Theta}J_n(\theta)$ provides near-optimal solutions in the original policy learning problem, since $J(\theta^*)-J(\hat\theta)\leq J(\theta^*)-J(\hat\theta)+J_n(\hat\theta)-J_n(\theta^*)\leq 2\sup_{\theta\in\Theta}|J_n(\theta)-J(\theta)|\xspace$. Given that in practice the nuisance functions $e_t$ and $\mu_t$ are estimated from data, we denote the corresponding score variable when such estimates are plugged in as $\hat\psi$ to differentiate it from the variable $\psi$ that uses the true nuisance functions. We correspondingly let $\hat J_n(\theta)=\frac1n\sum_{i=1}^n\hat\psi_i \operatorname{sign}(g_\theta(X_i))$. When $\hat J_n(\theta)$ is efficient for $J(\theta)$ one can generally additionally prove that $\sup_{\theta\in\Theta}|\hat J_n(\theta)-J_n(\theta)|\xspace=o_p(1/\sqrt{n})$.
Given the non-convexity and non-smoothness of the empirical objective function (ref) it is not necessarily clear how to actually optimize it, however. Many works jiang2019entropy,zhao2012estimating,beygelzimer2009offset recognize that this optimization problem is actually equivalent to weighted binary classification (in our two-action case), since $\psi_i\operatorname{sign}(g_\theta(X_i))=|\psi_i|\xspace(1-2\mathbb I_{\operatorname{sign}(g_\theta(X_i)\neq\psi_i})$, so any classification algorithm that accepts instance weights can perhaps be used to address (ref). Specifically, many classification algorithms take the form of minimizing a convex surrogate loss:
where $l(g,s)$ acts as a surrogate for the zero-one loss $\mathbb I_{\operatorname{sign}(g_\theta(X_i)\neq\psi_i}$. Analogous to above, we let $L(\theta) = \mathbb{E}\xspace[|\psi|\xspacel(g_\theta(X),\operatorname{sign}(\psi))]$ denote the population version of this loss. For classification, bartlett2006convexity studies which losses are appropriate surrogates, i.e., are classification-calibrated. The population version of the surrogate loss, which $L_n(\theta)$ is approximating, is
Following jiang2019entropy, we will focus on the logistic (or, logit-cross-entropy) loss function and define $l(g,s)$ everywhere as:\footnote{All of our results actually extend to any twice-differentiable classification-calibrated loss. Logistic is the most prominent such loss.}
This loss is clearly smooth, and $l(g_\theta(x),s)$ is also convex in $\theta$ as long as $g_\theta(x)$ is convex in $\theta$ for each $x$. The loss is also classification-calibrated, which immediately yields the following, given an additional regularity assumption:
(ref) establishes that, under correct specification, if we minimize the population surrogate loss, $L(\theta)$, then we obtain the optimal policy. Therefore, a natural strategy for policy learning would be to directly minimize the empirical loss $L_n(\theta)$, as was done by the above. Although the above arguments indicate that this approach would be computationally tractable, and also consistent under mild regularity conditions that ensure that optimizers of $L_n(\theta)$ would converge to optimizers of $L(\theta)$, it is not clear that it is statistically efficient, even if we use an efficient score variable for policy value estimation.
In this section we establish a new interpretation of the surrogate-loss reduction as a conditional moment problem and we discuss the implications of this in terms of the model implied by correct specification. This will enable us to conduct efficiency analysis and to design algorithms with improved efficiency in the next section.
To make progress toward a characterization of efficiency under correct specification, we next establish an equivalent formulation of optimizing the population surrogate loss under correct specification as a conditional moment problem.
Define the derivative of $l$ with respect to $g$:
where $\sigma(g)=\exp(g)/(1+\exp(g))$ is the logistic function.
(ref) arises straightforwardly from the observation that, under correct specification, $g_\theta(x)$ minimizes $\mathbb{E}\xspace[|\psi_i|\xspace l(g_\theta(X), \operatorname{sign}(\psi))\mid X=x]$ for almost every $x$. Using smoothness and convexity, this latter observation is restated using first-order optimality conditions. Dominated convergence theorem allows us to exchange differentiation and expectation and we obtain the result. (ref) provides an alternative characterization of $\theta^*$ as solving a conditional moment problem.
Notice that (ref) is equivalent to the statement that, for any square integrable function $f$ of $X$, we have the moment restriction
This alternative characterization makes the problem amenable to efficiency analysis.
Notice that by first-order optimality, if $\theta^*\in\operatorname{interior}(\Theta)$, optimizing $L(\theta)$ in (ref) exactly corresponds to solving the set of $d$ moment equations given by $\mathbb{E}\xspace[m(X;\theta)h_\theta(X)\xspace]=0$. Similarly, optimizing the empirical loss $L_n(\theta)$ in (ref) corresponds to solving these $d$ equations with population averages ($\mathbb{E}\xspace$) replaced with empirical sample averages.
However, (ref) gives a much broader set of equations. Leveraging this fact will be crucial to achieving efficiency. Indeed, it is well-known that even if a small number of moment equations are sufficient to identify a parameter (e.g., in the above, the $d$ equations identify $\theta^*$ via first-order optimality), taking into consideration additional moment equations that are known to hold can increase efficiency in semiparametric settings carrasco2014asymptotic.
In order to reason about efficiency, we need to reason about the model implied by (ref). To do so, we first establish the following lemma:
This model is generally a semiparametric model. That is, while (ref) is a parametric restriction on the function ${P(\psi > 0 \mid X=x)}/{\mathbb{E}\xspace \left[|\psi|\xspace \mid X=x \right]}$, the set of corresponding distributions on $(X,T,Y)$ that satisfy this restriction is still infinite-dimensional and non-parametric.
One question the reader might have at this point is why an approach different than empirical loss minimization is necessary for efficiency, given that the surrogate loss formulation seems mathematically identical to binary classification using logistic regression, which is known to be efficient.\footnote{This is because logistic regression performs maximum likelihood estimation (MLE), which is statistically efficient for well-specified parametric models.} The difference between the problems is that for actual classification we have that $\psi$ is a binary class label, i.e., $\psi \in \{-1, 1\}$. If we assume the policy class is well-specified and $\psi \in \{-1, 1\}$, the characterization of our semiparametric model from (ref) reduces to
which implies that our model is parametric, since the choice of $\theta^*$ now fully characterizes the distribution of the label $\psi$ given $X$. E.g., usually for logistic regression we let $g_\theta(x)=\theta^Tx$ so that the above says that the logit of $P(\psi = 1 \mid X)$ is linear. Therefore, performing logistic regression corresponds to MLE for this parametric model, which is efficient.
However in our general setting this is not the case and there is a non-trivial nuisance space, since an infinite-dimensional space of conditional distributions for $\psi$ given $X=x$ could result in the same function $P(\psi > 0 \mid X=x) / \mathbb{E}\xspace[|\psi|\xspace \mid X=x]$. This suggests that we may need to be more careful in order to obtain efficiency and that there may exist estimators that are more efficient than empirical loss minimization.
In this section we propose some efficient methods for policy learning based on the above conditional-moment formulation. In addition, we provide some analysis of these methods in terms of efficiency and regret.
We begin by proposing an approach based on using multi-step GMM to solve the conditional moment problem, which we will call FiniteGMM\xspace. This approach works by optimally enforcing for the moment conditions given by (ref) for a finite collection of critic functions $\mathcal F =\{f_1,\ldots,f_k\}$. Specifically, given some initial estimate $\tilde \theta_n$ of $\theta^*$, define:
We then estimate $\theta$ by $\hat \theta_n = \operatorname*{arg\,min}_{\theta} O(\theta;\tilde\theta_n)$. We can repeat this multiple times, plugging in $\hat\theta_n$ as $\tilde\theta_n$ and resolving.
An important issue with this estimator, however, is how to choose the critic functions. Standard GMM theory requires that the $k$ moment conditions are sufficient to identify $\theta^*$. And even then, the above is only the most efficient among estimators of the form $\operatorname*{arg\,min}_{\theta} \|(m_1(\theta),\ldots,m_k(\theta)\|\xspace$ for any norm $\|\cdot\|\xspace$, but there may still be more efficient choices of critic functions.
One nice result from the theory of conditional moment problems is the existence of a finite set of critic functions ensuring efficiency in the sense of (ref). Define:
We call $\mathcal F^* = \{f_1^*,\ldots,f_d^*\}$ the efficient instruments, and as long as the span of $\mathcal F$ contains these instruments then FiniteGMM\xspace is guaranteed efficiency newey1993efficient.
Given this, one approach would be to let $\mathcal F$ be flexible with the hope of approximately containing $\mathcal F^*$. Letting, for example, $\mathcal F$ be the first $k(n)$ functions in a basis for $L_2$ such as a polynomial basis and letting $k(n)\to\infty$ can be shown to be efficient under certain conditions newey1993efficient. This, however, can perform very badly in practice, especially with any reasonable amount of features. Ideally, we would instead be able to make use of modern machine learning methods and approximate $\mathcal F^*$ using some flexible function class such as neural networks rather than defining a finite set of basis functions.
Motivated by the above concerns, we now present our proposed approach: ESPRM\xspace (efficient surrogate policy risk minimization). This is based on the extension of bennett2019deep to our conditional moment problem. In the setting of instrumental variable regression, bennett2019deep proposes an adversarial reformulation of optimally-weighted GMM, which allows us to consider critic functions given by flexible classes such as neural networks. Then if this class provides a good approximation for the efficient instruments, this approach should be approximately efficient.
Specifically, we define:
where as above $\tilde\theta_n$ is some initial consistent estimate of $\theta^*$. Then following bennett2019deep, the ESPRM\xspace estimator is defined as
where $\mathcal F$ is our flexible function class (henceforth assumed to be a class of neural networks). It remains to describe how this adversarial game is to be solved, and how to define $\tilde\theta_n$. As in bennett2019deep we optimize the objective by performing alternating first-order optimization steps using the OAdam algorithm daskalakis2017training, which was designed for solving smooth game problems such as generative adversarial networks (GANs). In addition, we continuously update $\tilde \theta_n$ during optimization, where at each step of alternating first order optimization we set $\tilde \theta_n$ equal to the previous iterate of $\hat \theta_n$.
Finally we prove that efficiency not only ensures minimal MSE in estimating $\theta^*$ but also implies regret bounds. Let
This theorem implies that the regret of a policy is upper-bounded by the excess risk of the surrogate loss. Next, we make the following regularity assumption about the loss $L$:
Given this assumption, a Taylor's theorem expansion yields $\text{Regret}_L\xspace(\hat\theta_n) = (\hat\theta_n - \theta^*)^T H(\theta^*) (\hat\theta_n - \theta^*) + o(\|\hat\theta_n - \theta^*\|\xspace^2)$. For for any regular estimator $\hat\theta_n$, we can also define the asymptotic regret $\text{AR}_L\xspace$ as the limiting distribution:
which exists since regularity implies that $\sqrt{n}(\hat\theta_n - \theta^*)$ has a limiting distribution. Given this we can prove the following optimality result of our efficient estimators in terms of asymptotic regret:
Together with (ref), this means that both the actual regret ($\text{Regret}_J\xspace$) and the surrogate regret ($\text{Regret}_L\xspace$) of policies given by efficient estimators $\hat\theta$ are $O_p(1/n)$, and the surrogate regret has an optimal constant.
First we investigate the performance of our algorithms on a variety of synthetic scenarios. In all these scenarios $X$ is 2-dimensional, and $X$ and $Y(t) - \mu_t(X)$ are standard Gaussian distributed for each $t$; the scenarios only differ in the functions $\mu_t$ and $e_t$. In none of the scenarios is our policy class actually well-specified in the sense of (ref).
We consider the following kinds of synthetic scenarios:
In addition we experiment with the following policy classes: a linear policy class, where $g_\theta(x) = \theta^Tx + \theta_0$, and a flexible policy class where $g_\theta(x)$ is given by a fully-connected neural network with a single hidden layer of size 50, and leaky ReLU activations.
In all cases we use the surrogate loss method of jiang2019entropy described in (ref) as a benchmark, which we henceforth refer to as ERM\xspace. We note that although in the prior work they used $\hat \psi_{\text{IPS}}$, we instead use $\hat\psi_{\text{DR}}$, both because it is theoretically better grounded athey2017efficient,zhou2017residual and we found that it gives stronger results for all methods. For our ESPRM\xspace method we let $\mathcal F$ be the same neural network function class as for flexible policies, and perform alternating first-order optimization as described in (ref) for a fixed number of epochs.
For all methods, except where otherwise specified, we use the $\hat\psi_{DR}$ weights described in (ref), with nuisance functions fit using correctly specified linear regression or logistic regression algorithms on a separately sampled tuning dataset of the same size as the training dataset.\footnote{By correctly specified we mean that for Linear\xspace we fit using linear/logistic regression on $X$, whereas for Quadratic\xspace we fit on a quadratic feature expansion of $X$.} We provide some additional results in the appendix where nuisances were instead fit via flexible neural networks, which show that this has little effect on our results. In all cases except for ESPRM\xspace we perform optimization using LBFGS. Additional optimization details are given in the appendix.\footnote{Code for running all of our experiments is located at \url{https://github.com/CausalML/ESPRM}.}
For all configurations of scenario kind and policy we ran our experiments by sampling random scenarios of the respective kind, by setting all scenario parameters to be independent standard Gaussian variables. Specifically, for each $n \in \{100,200,500,1000,2000,5000,10000\}$ we sample 64 random scenarios of the respective kind, and for each random scenario we sample $n$ training data points and run all methods on this data. Results for FiniteGMM\xspace, which generally did badly as predicted, are given in the appendix.
Define Relative Mean Regret Reduction (RMRR), given by:
where each expectation in the fraction is taken over the joint distribution of randomly sampled scenario, and corresponding random estimate $\hat\theta$. Then for each scenario kind and policy class, we plot predicted RMRR against number of training data based on our ESPRM\xspace estimates in (ref). We see that ESPRM\xspace consistently obtains policies on average that are lower regret or on-par than those obtained by ERM\xspace (typically with around 10% to 20% RMRR), with 95% confidence intervals indicating clearly better performance in every case except for training flexible policies on random Quadratic\xspace scenarios (in which case performance seems roughly on par). It is notable that this even occurs in the Quadratic\xspace setting with the linear policy class, where our policy class is not even well specified for the loss, let alone the surrogate loss. We can also observe that the most significant regret benefits tend to occur with smaller training set sizes (since the same RMRR implies a larger absolute decrease in regret), indicating that the statistical efficiency of our method is leading to improved finite sample behavior.
In (ref) we plot the convergence in terms of the MSE of the estimated parameter from ESPRM\xspace and ERM\xspace, for the Linear\xspace setting and linear policy class (where parameters are low-dimensional and correctly specified). We plot both the MSE convergence, and the average difference in the squared error between the estimates, across the random scenarios.\footnote{All parameter vectors are normalized first given that the policy function is scale-invariant.} It is clear from these results that ESPRM\xspace consistently estimates optimal policy parameters with lower squared error on average compared to ERM\xspace across these random simulated scenarios. This provides strong evidence that the methodology indeed provides an improvement in statistical efficiency for solving the smooth surrogate loss problem.
We next consider an application to a dataset derived from a large scale experiment comparing different programs offered to unemployed individuals in France behaghel2014private. We focus our attention to two arms from the experiment: a treatment arm where individuals receive an intensive counseling program run by a public agency and a treatment arm with a similar program run by a private agency. The hypothetical application is learning a personalized allocation to counseling program, with the aim of maximizing the number of individuals who reenter employment within six months, minus costs. (The original study's focus was not personalization.) Our intervention is simply the offer of the counseling program; we therefore ignore the fact that some individuals offered one of the programs did not attend.
To make our policies focus on heterogeneous effects, we set the costs of each arm to be equal to their within-arm average outcome in the original data. That is, the outcome we consider is equal whether one reentered employment within 6 months, minus the average number of individuals who entered employment within 6 months in that arm. The covariates we consider personalizing on are: statistical risk of long-term unemployment, whether individual is seeking full-time employment, whether individual lives in sensitive suburban area, whether individual has a college education, the number of years of experience in the desired job, and the nature of the desired job (e.g., technician, skilled clerical worker, etc.).
We then consider 64 replications of the following procedure. Each time, we randomly split the data 40%/60% into train/test. We then introduce some confounding into the training dataset. We consider the following three binary variables: whether individual has 1--5 years experience in the desired job, whether they seek a skilled blue collar job, and whether their statistical risk of long-term unemployment is medium. After studentizing each variable, we segment the data by the tertiles of their sum. In the first tertile, we drop each unit with probability $7/8$. In the second tertile, we drop private-program units with probability $1/4$ and public-program units with probability $7/8$. In the third tertile, we drop public-program units with probability $1/4$ and private-program units with probability $7/8$. Given a policy learned on this training data, we evaluate it on the held-out test set using a Horvitz-Thompson estimator.
Of the training data, $20\%$ was set aside for training nuisances, and an additional $20\%$ as validation data for early stopping. We then trained both linear and flexible policies using ERM\xspace and ESPRM\xspace as in our simulation studies, with the exception that nuisances were fitted using neural networks (of the same architecture as the flexible policy class).
We summarize the mean estimated outcome for the policies from each method in (ref). We note from these values that on average ESPRM\xspace seems to be learning higher value job-assignment policies than ERM\xspace. Furthermore, performing paired two-sided $t$-tests on the two sets of repetitions for each policy to test for difference in mean policy value we obtained $p$-values of $.0429$ for the linear policy class and $.0007$ for the flexible policy class, clearly highlighting the benefit of our ESPRM\xspace method.
We considered a common reduction of learning individualized treatment rules from observational data to weighted surrogate risk minimization. We showed that, quite differently from actual classification problems, assuming correct specification in the policy learning case actually suggests more efficient solutions to this reduction. In particular, even if we use efficient policy evaluation, this may not necessarily lead to efficient policy learning. Specifically, under correct specification, the problem becomes a conditional moment problem in a semiparametric model and efficiency here translates to both better MSE in estimating optimal policy parameters and improved regret bounds.
Based on this observation, we proposed an algorithm, ESPRM\xspace, for efficiently solving the surrogate loss problem. We showed that our method consistently outperformed the standard method of empirical risk minimization on the surrogate loss, both over a wide variety of synthetic scenarios and in a case study based on a real job training experiment.
\onecolumn