Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
108,629 characters · 32 sections · 70 citation commands
Policy Learning With Rare Outcomes
\linespread{1.25}
In the health and social sciences, there is increasing interest in using observational data to learn optimal policy assignment rules that map an individual's covariate profile to a treatment decision murphy2003optimal,manski2004statistical,hirano2009asymptotics,kitagawa2018should. These rules can be optimal in the sense that they maximise expected outcomes, such as health gains. The problem of learning optimal policy assignment rules is linked to the concept of heterogeneous treatment effects: an individual’s expected benefit from receiving treatment depending on their characteristics, formalised as the conditional average treatment effect (CATE) function. Proposed approaches for learning optimal treatment assignment rules either try to learn the CATE function in order to assign the policy to those individuals where the estimated CATE shows a benefit of being treated (e.g. luedtke2016super), or they aim to directly estimate the population average benefit resulting from a given policy assignment rule, and search over a class of permissible policies to find the optimal one (e.g. athey2021policy). This latter approach may be preferable on the grounds of statistical theory – average treatment effects are easier to learn than the multi-dimensional CATE function – and can lead to more interpretable rules if the class of permissible policies is restricted to simple structures, such as the decision trees proposed by athey2021policy.
Underlying both the CATE-based and the direct approaches there is a crucial task of estimating so-called nuisance functions: the propensity score, defined as the conditional probability of receiving treatment given covariates, and the outcome regression, defined as the conditional expectation of the outcome given treatment and covariates. These nuisance components need to be estimated well for two reasons: first, to eliminate bias due to confounding, and second, to capture heterogeneity in treatment response. The use of flexible machine learning algorithms to estimate nuisance functions has been increasingly recommended, originally for settings where the interest is in average treatment effects van2011targeted,chernozhukov2017double, and more recently for settings that study the CATE kunzel2019metalearners,athey2019generalized. However, the implications of the choice of specific ML models, in terms of their practical performance when solving optimal policy learning problems, has received little attention so far. One particularly challenging setting common in evaluating health policies is that of rare outcomes.
In settings with rare outcomes - e.g. infant mortality, or incidence of side effects from a new medication - little information is contributed by the majority of observations, resulting in high variance of the estimators of treatment effects franklin2017comparing.\footnote{ Franklin et al. (2017) compared propensity score methods for estimating treatment effects on rare outcomes and find regression adjustment and inverse propensity weighting (IPW) produce lower bias and MSE for binary outcomes.} For the purpose of optimal policy estimation, the concern is that this lack of information may prevent good estimation of nuisance components and the CATE function. It is therefore crucial to understand the effect of this estimation challenge on optimal policy learning.
Policy learning from observational data has a long history in the economics and statistics literature manski2004statistical, hirano2009asymptotics,qian2011performance,kitagawa2018should. We build on insights from a recent strand of the policy learning literature luedtke2016statistical,bertsimas2020predictive, kallus2021minimax, athey2021policy, which relies on ML to estimate optimal policy allocation rules.
To concretely motivate this work, we first examine a commonly studied rare outcome for evaluation of health policies - infant mortality - which has been shown to exhibit significant heterogeneity in Low and Middle Income Countries (LMICs). In particular, kreif2021estimating show evidence of a reduction in infant mortality resulting from the expansion of social health insurance in Indonesia; however, this reduction was only statistically significant among those who were beneficiaries of a contributory health insurance scheme, and not among those who were recipients of subsidised insurance for the poor. For the contributory insurance group, a Causal Forests approach uncovered significant heterogeneity in the CATEs for the main mechanism behind infant mortality reduction, birth attended by a health professional. Due to the rare nature of the infant mortality outcome, estimation of CATEs was not attempted in the previous analysis. In this study, we aim to estimate CATEs of subsidised health insurance on infant mortality, and apply policy learning to come up with hypothetical allocation rules for subsidised health insurance, evaluate the value of these rules, and compare them to the one-size-fits-all policy of enrolling everyone.
Our objectives are three-fold. First, we wish to scrutinise the importance of the choice of ML method underlying the policy learning process. This includes learners for the nuisance components, and learners for the CATE function. Second, we aim to investigate the trade-off between choosing simple, depth-2 policy trees or “black box" plug-in rules. As part of this investigation, we propose a small modification to the policy tree learning approach proposed by Athey and Wager, which has some potential advantages in finite samples. Third, we want to assess the performance of measures of the estimated policy value suggested in the literature and to propose alernatives.
Specifically, we consider two main policy classes: tree-based approaches and so-called plug in rules. We build on the work by Athey and Wager (2021) that suggests minimising a minimax regret in a class of tree-based policy rules, relying on ML estimates of double-robust (DR) scores. Within the tree-based class, we also explore the possibility of learning policies from the estimated CATEs. We compare three state of the art ML approaches to estimate the DR scores and CATEs: Causal Forests athey2019generalized, which are designed to capture heterogeneity in CATEs, a normalised version of the double-robust learner (NDR learner) knaus2022double,kennedy2020optimal, which relies on the double machine learning framework of chernozhukov2017double, and Bayesian Additive Regression Trees (BART) hill2020bayesian, expected to perform well in settings with rare outcomes hu2021estimation. We also include a modified version of the Causal Forest which incorporates sample splitting (see Section 3). We then consider so-called plug-in rules hirano2009asymptotics that directly use the ML estimates of the conditional average treatment effect (CATE) function luedtke2016statistical, using its sign to assign treatment to those expected to benefit from the policy. Each of the ML algorithms we examine can produce estimated CATEs: the Causal Forest directly outputs CATEs, the NDR learner obtains them as predictions from a flexible regression on the double-robust scores, while the BART can estimate CATEs using the S-learner approach kunzel2019metalearners. Causal Forests and the NDR learner also need to specify ML models for the underlying nuisance functions, here we follow Athey and Wager and choose random regression forests athey2021policy.
We apply these policy learning approaches in the case study, and evaluate their finite sample performance using simulations. First, we evaluate a learned policy by reporting the true utility of potentially implementing it, and and comparing this to the utility of the best possible ”oracle" policy. We do this by reporting how large a percentage of the oracle policy advantage athey2021policy each learner can capture, and also express this as the root mean squared error (RMSE) of the advantage of the learned policy compared to the advantage of the oracle policy. Second, through simulations we are able to scrutinise how established performance measures of optimal policy learning algorithms (e.g. the policy advantage, estimated via double-robust scores, see section 2) perform in finite samples, when we compare the estimated measures to their true counterparts. Our simulations cover various settings, differing in whether the outcome is rare or not, different sample sizes, varying degrees of heterogeneity in the treatment effects, and varying degrees of confounding. We also vary sample sizes, and consider an alternative scenario with non-binary, continuous outcomes.
When contrasting across the methods, we find the NDR learner most often achieves the largest percentage of the oracle policy in settings with rare outcomes and effect heterogeneity, with the Causal Forest also performing well according to this metric. In rare outcome settings where heterogeneity is mildly present, the modified Causal Forest outperforms other methods. Importantly, the plugin (treat if $\hat{\tau} < 0 $) treatment assignment rule outperforms the learned policy trees when no heterogneiety is preesent, but our modified (CATE-based) tree often outperforms the plugin rule as heterogneity increases, and always does better than the DR score-based tree. Given this finding, we propose an alternative way of measuring the policy advantage, using the estimated CATEs, and find that a CATE-based metric gets closer to the true value of the learned policy in most settings and especially for the tree-based policy class.
These results have clear implications for researchers aiming to use the new toolkit of optimal policy learning to generate evidence that can inform policy. In our study we find that while CATE-based plug-in rules always outperform shallow (level 2) decision tree based rules learned from DR scores, there is a practical use for learning a decision tree-based policy from estimated CATEs to regain some of this lost performance - and in some settings, the CATE-based tree may even be preferable to the plugin rule. This improves the trade off between optimality and interpretability cockx2019priority in real world settings where policy makers may be reluctant to implement “black-box" decision rules lechner2007value.
We build on the framework proposed by athey2021policy. We have access to observational data, in the form of independent and identically distributed samples $(X_i,W_i,Y_i)$, where $X_i$ is a vector of individual covariates, $W_i \in \{0,1\}$ is a binary treatment and $Y_i \in \{0,1\}$ is the (binary) outcome of interest. Individual level causal effects are defined as $\tau_i=Y_i(1)-Y_i(0)$, where $Y_i(w)$ denotes the potential outcome if the treatment had been set to $W_i=w$ rubin1974estimating. We want to use this data to learn a policy allocation rule $\pi$ which maps $X_i \in \chi$ into a binary treatment decision, ie. $\pi : \chi \rightarrow \{0, 1\}$, for policies in a pre-specified policy class $\Pi $.
The traditional causal target parameter when evaluating a binary treatment is the difference between expected potential outcomes when everyone in the population is treated compared to when no one is treated, $\Theta= E[Y_i(1) - Y_i(0)]$ - in other words, the average treatment effect (ATE). We can think of the ATE as the expected benefit of deploying a very simple policy, where irrespective of covariates, everyone is assigned to treatment, and this policy is compared to when no one receives treatment. Generalising the ATE for an arbitary policy $\pi$, we define the utility of deploying a policy $\pi$ relative to treating no one as
Intuitively, the optimal policy $\pi_*$ is the one (within the policy class\footnote{A policy class can take various forms. In this paper, we focus on tree-based policies, and the plug-in policy.}) that maximizes the quantity $V(\pi)$.
We also define the utilitarian regret manski2009identification from deploying a policy $\pi$ relative to the best policy in the class $\Pi$ as
The formal goal of policy learning is to derive a policy $\hat{\pi} \in \Pi$ from observed (randomised or non-randomised) data, with the guarantee that the regret $R(\hat{\pi})=O_P(1/n)$. To achieve this, further assumptions need to be made on the data generating process, and the complexity of the policy class needs to be controlled.
Here, we make the assumption of no unobserved confounding, such as $\{Y_i(0), Y_i(1)\} \perp \!\!\! \perp W_i | X_i$, and overlap, $0<e(X_i)<1$, where $e(X_i)=Pr(W_i=1|X_i)$ is the probability of receiving the treatment, given observed covariates (the propensity score).
Another building block of the optimal policy estimation problem is the conditional average treatment effect function (CATE), defined as the expected difference between the potential outcomes as a function of the covariate profile:
Note that the ATE is the expectation of the CATE function over the empirical distribution of the covariates, $\Theta=E[\tau(x)]$.
It can be shown athey2021policy that the value of a policy $V(\pi)$ can be written in terms of the CATE as $V(\pi) = E[\pi(X_i)\tau(X_i)]$, and from this the so-called policy advantage, $A(\pi)$\footnote{$A(\pi)$ can be interpreted as an improvement achieved by the policy $\pi$ compared to applying the opposite action dictated by the policy, due to the following equivalence:
} can be defined as:
This quantity, as described in the next section, can be estimated from the observed data, hence becoming a good target of the optimisation.
The main assumption made by athey2021policy is that we have access to a double-robust estimator for the ATE, a so-called double robust score, where the estimator is formed by taking the average: $\hat{\Theta}=\frac{1}{n}\sum_{i = 1}^n\hat{\Gamma}_i$. One version of a double-robust score is equivalent to the score used in the double-machine learning estimator chernozhukov2017double, which is based on the the augmented inverse probability of treatment weighted estimator for the ATE robins1995semiparametric defined as
where $m(x,w)$ is the counterfactual response surface defined as $E[Y_i(w)|X_i=x]$ which under unconfoundedness can be identified as $E[Y_i|X_i,W_i]$, and its estimate we denote as $\hat{m}(X_i,W_i)$.\footnote{This score is used across all ML methods we consider except the Causal Forests, which constructs a modified version; see below.} Construction of the DR score requires three nuisance parameter estimates: the expected potential outcome under control $\hat{m}(X_i,0)$, the expected potential outcome under treatment $\hat{m}(X_i,1)$, and the expected outcome under the treatment actually received $\hat{m}(X_i,W_i)$. Note that the first part of the formula, $\hat{\tau}_m=\hat{m}(X_i,1)-\hat{m}(X_i,0)$ is a non-double robust estimate of the CATE, while the second part is a double-robust adjustment term, where the prediction error of the observed outcome is inverse weighted with the estimated propensity score $\hat{e}(X_i)$.
athey2021policy show that to achieve the regret guarantees, the components of $\hat{\Gamma}_i$ - the so-called nuisance functions - need to be estimated via ML. In this paper we consider several ways to estimate the nuisance functions in this double robust score, motivated by our setting of rare outcomes. We detail the estimators considered in Section 3. athey2021policy consider a somewhat modified double-robust score that directly uses estimates of the CATE obtained from the Causal Forests method, and plugs this into a double-robust score, using a different nuisance function $\hat{m}(X_i)$: the expectation of the outcome conditional on covariates but marginalised over the treatment groups. We also consider this approach, and describe the modified score in Section 3.
Regardless of the way $\hat{\Gamma}_i$ is estimated, in the following step it is used to construct the final target of optimisation $\hat{A}(\pi)$, as
Intuitively, we now require an algorithm that can search the space of permissible policy rules and find the rule $\pi$ that achieves the largest value of $\hat{A}(\pi)$. In the case where the outcome is harmful - in our setting, the outcome of interest is infant mortality - we simply change the sign of $\hat{A}(\pi)$.\footnote{Practically, this is done by multiplying the $\hat{\Gamma}_i$ by $-1$.}
athey2021policy propose using a depth-$k$ decision tree, found via exhaustive tree search, to find the optimal policy within the class of depth $k$ trees.\footnote{We use the {\tt policytree} package in {\tt R} to obtain the policies.} We also consider a “plug in" policy allocation rule based on the estimated CATE, where all individuals with a negative estimated CATE ($\hat{\tau}<0$) receive treatment. In this case, there is no restriction on the policy class, but if the $\frac{1}{\sqrt{n}}$-rate estimation of the CATE is not possible, treatment assignment rules derived from a simple sign rule may not be asymptotically minimax-optimal hirano2009asymptotics.
To summarise, the policy learning process in this paper consists of four main steps.
1. We estimate the nuisance parameters: the outcome regression functions (stratified by treatment: $\hat{m}(X_i,W_i=w)$, and pooled: $\hat{m}(X_i)$) and the propensity score model, using off-the shelf ML algorithms.
2. We estimate DR scores ($\hat{\Gamma}$) and CATEs ($\hat{\tau}$). Specifically, for the NDR learner, we construct the DR scores $\hat{\Gamma}_i$ for each method using the nuisance parameters. We then take these DR scores and in a second ML regression step, estimate CATEs. For BART and the Causal Forests, we first estimate CATEs, then use these as an input in the DR score. (Depending on the method, (1) and (2) involve cross-fitting as suggested by the developer of the methods, and we also propose our own cross-fitting scheme. See details in the next section.)
3. Using the estimated CATEs and DR scores, we estimate policy allocation rules. We estimate the optimal policy assignment using two policy classes:
4. For each method and policy class we report the estimated value of the learned policy, by constructing an estimated counterpart of the policy value (Equation (4)). We calculate this measure two ways: first using the estimated DR scores (as suggested by athey2021policy):
and second, using the estimated CATEs:
The motivation for the metric $\hat{A}(\pi)_{CATE}$ is that for binary, rare outcomes we expect better performance of the estimated CATE than the estimated DR scores in finite samples.
For the tree-based policies, we estimate these metrics as part of our cross-validation procedure, and for each cross-validation step where the policy tree has been trained using all data except for the data in fold $k$, we apply the learned policy on fold $k$, and construct the $\hat{A}(\pi)$ metrics for this fold, then take the average of them across the K folds.
Throughout, we multiply the policy advantages $\hat{A}(\pi)$ with $-1$ to reflect that we are minimising a harmful outcome in our setting.
In this section we review the approaches we selected to estimate the DR scores used for optimal policy estimation. Each ML algorithm will yield nuisance functions $\hat{m}(W_i, X_i)$ and $\hat{e}(X_i)$, while the Causal Forests approach requires a further nuisance function $\hat{m}(X_i)$.
Bayesian Additive Regression Trees (BART) is a nonparametric Bayesian regression approach chipman2010bart,hill2020bayesian. It uses two main components: a “sum-of-trees" model and a prior on the model parameters to approximate an unknown function. The imposed prior regularises the fit of each regression tree by keeping individual effects small chipman2010bart, with each tree explaining only a portion of the response surface. The BART approach has been adapted to the causal inference setting by hill2011bayesian, specifically to flexibly model the outcome regression, due to its ability to handle non-linearities and multi-way interactions between covariates without researcher input tan2019bayesian.
BARTs have been found to exhibit good performance in estimating treatment effect parameters carnegie2019examining, including in settings with rare outcomes and heterogeneous treatment effects hu2021estimation, as well as small effect sizes hahn2020bayesian \footnote{Hu and Gu (2021) study the causal effects of multiple treatments on rare outcomes, comparing Bayesian Additive Regression Trees (BART), regression adjustment on multivariate spline of generalized propensity scores (RAMS), and IPW, using simulations and a case study hu2021estimation.}. However, to date, BARTs have not been studied in the double-robust policy learning setting.
In our study we follow chipman2010bart and directly model the conditional expectation of a binary outcome in a probit framework:
where $\phi$ is the c.d.f. of the standard normal distribution, $T_j$ denotes a single regression tree and $A_j$ is the set of its associated parameters, and $g_j(w, X_i; T_j, A_j)$ represents the conditional mean assigned to the particular node associated with covariate profile $X_i$ and treatment $w$ in the $j$th regression tree \footnote{We include the estimated propensity score $\hat{e}(X_i)$ (also obtained using BART) as a covariate in the outcome model, as this has been shown to improve performance hahn2020bayesian.}. We use iterations of a Bayesian backfitted MCMC construct and fit separate residuals, which are effectively $S$ MCMC samples from an induced posterior distribution. We perform 2500 MCMC draws, with the first 500 treated as burn-in (i.e., the first 500 draws are discarded).
To estimate CATEs, we predict both potential outcomes for each observation using imputed c.d.fs, and treatment status $w$ using each sample draw $S$ hill2011bayesian, hill2020bayesian. We then obtain CATEs from $\sum_{s=1}^{S}m_s(1, X_i) - m_s(0, X_i)$. We augment this CATE estimate using BART estimates of $\hat{e}(X_i)$ to create DR scores as in Equation 5.
We next consider a double-robust approach that has been developed in parallel in the statistics robins1995semiparametric,van2006targeted,kennedy2020optimal and econometrics chernozhukov2017double,chernozhukov2017double,chernozhukov2018generic literature. The general idea is that estimators of ATEs can be constructed with double-robust scores formed from nuisance components - propensity scores and outcome regression functions - estimated by machine learning. The DR score (see Equation 5) is derived using semiparametric theory, ensuring that the resulting estimator for the ATE is asymptotically linear and consistent. When cross-fitting is used - see description below - a wide variety of ML methods can be used to estimate the nuisance models, making this approach model agnostic.
DR scores that can provide a valid estimator for the ATE can be further extended to estimate the CATEs.\footnote{Note that chernozhukov2018double do not advocate directly estimating CATEs from the scores obtained via DML, but instead, suggest obtaining so-called best best linear predictors (BLP) of CATEs, by regressing the obtained scores on covariates expected to drive treatment effects.} Here we follow the approach by kennedy2020optimal who propose a further step of (machine learning) regression to estimate the CATE function, using the DR scores as the dependent variable. This "double-robust (DR) learner" approach allows for valid estimation of the CATE under fairly mild regularity conditions.\footnote{That the second-stage regression satisfies mild stability assumptions and its squared-error loss functions are known.}
To address the practical challenge of potentially unstable inverse probability of treatment weights leading to instability of the estimator, we follow knaus2022double who extends the DR-learner described above. Their proposed normalised DR (NDR) learner normalises the inverse probability weights used in the DR score, ensuring that weights on the individual outcome residual portion of the $\hat{\Gamma}$s, $Y_i - \mu(w, X_i)$, sum to one. \footnote{Knaus observes that point estimates in the DR-learner can be expressed as $\hat{\tau}_{dr}(X_i) = \sum_{i=1}^N\alpha_i(X_i)\hat{\Gamma}_{i}$, when $\alpha_i$ can be calculated and is the weight that each observation receives in the second-stage regression of the DR scores on covariates $X_i$. This method restricts the underlying ML methods to linear smoothers only. A linear smoother is an operation by which transformed variables can be expressed as a linear transformation of observed values. That is, $\hat{X} = (\hat{x_1}...\hat{x_n})^t$ can be written in the form $\hat{X} = SX$, where the smoother matrix S does not depend on the original X buja1989linear. A simple example of a linear smoother is a moving average. ML examples include tree-based algorithms or ridge regression.}
The procedure of the DR learner is as follows. First, the data is randomly split into four parts. Next, $\hat{e}(w,X_i)$ and $\hat{m}_w(X_i)$ are each obtained from the first two separate training samples using the chosen ML algorithm. These are then used to construct the DR scores (as in Equation 5) in the third sample. These DR scores are regressed on the covariates $X$ using supervised ML to obtain an estimator for the CATE; i.e., $\hat{\tau}_{dr}(Y_i, W_i, X_i) = E_n[\hat{\Gamma}(Y_i, W_i, X_i) | X = x]$. The normalisation step described earlier is performed at this stage. The forth sample is then used to obtain predicted CATEs. The first two steps are repeated twice, so that each of the first three subsamples is used once to obtain $\hat{e}(w,X_i)$, $\hat{m}_w(X_i)$, and the DR score. The fourth sample will have been used three times to obtain predicted CATEs, and the final estimate of the CATEs for this fourth sample is taken as the average of the three predictions. This entire procedure is repeated four times, where each sample is used once as the hold-out sample for estimating CATEs. For our purposes of optimal policy estimation, we construct the DR scores as averages across all repetitions used in out-of-sample prediction of the CATEs. Finally, we opt to estimate both the two nuisance functions and the second stage CATE regression using random forests, to be consistent with the Causal Forests approach, presented next.
Finally, we turn our attention to the method of generalised random forests wager2018estimation,athey2019generalized, and its implementation for heterogeneous treatment effects estimation, Causal Forests athey2019estimating. Causal Forests are not model agnostic learners, i.e. they rely on a specific machine learning method - a version of random forests - to both estimate CATEs and obtain the double-robust scores. We consider two implementations of the Causal Forests approach to obtaining nuisance functions and CATEs used in constructing the double-robust scores for optimal policy estimation: the so-called Honest Causal Forests, and a novel extension we propose, cross-fitted Causal Forest.
In brief, the Causal Forests estimator relies on modified regression forests - generalised random forests - to find small neighbourhoods (leaves of a tree) where the CATEs are constant, by regressing the residualised outcomes on the residualised treatment variable,\footnote{Residualisation is performed by subtracting the predicted outcome $\hat{m}(X_i,W_i)$ and and the estimated propensity score $\hat{e}(X_i)$ from the outcome and treatment indicators, respectively.} and partitioning the data into leaves to maximise the between-leaf heterogeneity in the estimated treatment effects, defining so-called Causal Trees. To reduce noise stemming from using individual trees, this procedure is done many times on bootstrap samples, forming a Causal Forest. The Causal Forests are then used to calculate $\alpha_i(x)$ weights for each observation, based on how frequently an observation was used to estimate the treatment effect at $x$. The estimator for the CATE is then constructed using these weights and the nuisance components as
wager2018estimation call their forests “honest" in that each observation is only used to estimate within-leaf treatment effects $\hat{\tau}_i$, or to decide where to place splits within a tree, but not both.
Although the double-robust score resulting from a Causal Forest is conceptually identical to Equation 5, the ability of the Causal Forest to directly estimate CATEs allows for the direct use of $\hat{\tau}(X_i)$ in the score:
where $\hat{f}(X_i) = E[Y_i|X_i = x]$.
We use the {\tt grf R} package to estimate and fit the Causal Forests, tuning all parameters and growing 2,000 trees.\footnote{See tibshirani2021package for further details on Causal Forest tuning parameters. We grow 100 trees to tune parameters, and repeat the tuning process 500 times.} As suggested by athey2019generalized, we use random regression forests to estimate all the nuisance models.
Although the Honest Causal Forest uses out-of-bag sampling when estimating CATEs (i.e., an observation may not be used to estimate $\tau$ if it has been used to determine a split in that particular regression tree), over the space of the entire Causal Forest, it could still be the case that overfitting is occurring if the same observations are being used for obtaining both nuisance functions and the CATE prediction across many trees. To mitigate this potential limitation, we propose small modification to the honest Causal Forest algorithm by adding a cross-fitting component.
In our Cross-fitted Causal Forest algorithm, we first fit the relevant nuisance models and a Causal Forest based on $K - 1$ folds. We then use these fitted objects to generate new predictions of the nuisance functions and the CATEs on the $k$th holdout fold. We construct DR scores from these estimates as in Equation 9. The procedure is performed $K$ times to obtain predictions for each observation, where each $k$th fold acts as the holdout fold once. The entire process is then repeated $t=4$ times (i.e., the sample is re-split into $K'$ new folds and forests are predicted for each), and results across each repetition are averaged to obtain final estimates.\footnote{ By restricting/splitting the sample before the tree fitting occurs, there is a reduced likelihood for the same observations being used to determine splits (and therefore not used in the within-leaf predictions) across trees. Our cross-fitting procedure using testing and training (hold-out) data is standard in many ML applications and modifies the original Honest Forest algorithm only slightly. Within the individual fold fitting and prediction, we maintain tree honesty (so the same observations within the training fold that are used to obtain tree splits are not used to obtain CATEs). As before, we tune all parameters and fit 2,000 trees for each fold. Code is available at {\tt github.com/jhatamyar/OptPolicyRare}}.
Building on kreif2021estimating we aim to explore heterogeneity in the effect of subsidised health insurance on the reduction in infant mortality, but instead of focussing on the estimation of the individual level CATEs, we explore heterogeneity in the treatment effect across certain groups. Later in Section 7 we turn to the estimation of optimal policy assignment rules, using each of the methods described in Section 3.
The dataset consists of births between 2002 and 2014, extracted from the Indonesian Family Life Survey (IFLS), a longitudinal household survey strauss2016fifth. The unit of observation is a birth, while the treatment is defined as for a given birth, whether a mother was covered by subsidised health insurance in the year of the birth. The control group consists of those births where no insurance was reported in the year of the birth. The outcome of interest is infant mortality, measured as child death before the first year of life.
To deal with the challenge of self-selection into health insurance, we follow the approach taken by kreif2021estimating, and exploit variation in the expansion of subsidized health insurance schemes, across provinces and over time. The observed confounders take into account information on known predictors of infant mortality, as well as the eligibility criteria of subsidised health insurance. They include the mother's characteristics (age, education, wealth in quintiles) and household characteristics (social assistance, experienced a natural disaster, rurality, availability of health services: a village midwife, birth clinic, hospital). We also control for region effects that capture unobserved confounding factors that are common within regions and time-invariant. Births under subsidised insurance were more likely to be from a rural household and from mothers who are older at birth, less likely to have studied at university and more likely to have only elementary school education, belong to lower wealth quintiles, and receive social assistance programmes, compared to those with no insurance.
As demonstrated by kreif2021estimating, inverse probability weighting using a logistic regression based propensity score was effective in creating balance across the observed covariates. In this study, we apply the three DR score based ATE estimation methods - BARTs, NDR Learner and Causal Forests - to estimate average effects \footnote{Balance checks using the ML based propensity scores are available upon request.}, with nuisance functions obtained as described in the previous sections.
We also examine drivers of heterogeneity. Following chernozhukov2018generic, we estimate the so-called Best Linear Predictors of treatment effect heterogeneity, by regressing the DR scores on a selected group of covariates in a linear model. The coefficients from this linear model can be interpreted as the impact of a given covariate on the expected CATE in a ceteris-paribus way, and the standard errors of these coefficients can be used for testing the hypotheses of whether each covariate has a significant impact on the variation in the treatment effect. We include variables in the BLP that can have potential relevance in a policy makers targeting criteria, and exclude variables that were necessary for confounding adjustment but could not enter any meaningful targeting criteria (e.g. year of birth, gender of child, missingness indicators for variables). Later we will also investigate whether these covariates are chosen by the policy tree algorithm as important determinants of treatment assignment.
We report estimates of the average treatment effect of subsidised health insurance on infant mortality, derived from the double robust scores obtained using each method, in Table (ref). There is slight disagreement across the methods in terms of ATE estimates - Causal Forests are found to estimate the largest (most negative) average treatment effect, while the NDR-learner and BART result in smaller estimates. None of the methods report ATE that are statistically significantly different from zero at conventional levels.
Figure (ref) depicts the Best Linear Predictors for a selected group of covariates, using scores from an NDR-Learner, BART, and Honest Causal Forest (CF).
We find that those who became mothers at relatively later age (23-27 and over 31) have a significantly larger (more negative) than average benefit from subsidised health insurance. We also find some evidence that those who participate in cash transfer programs benefit more, while surprisingly, we find the opposite effect for those in possession of the so-called "poor card" - membership in this group is estimated to have a significantly harmful effect on the expected CATE.
We now turn to our simulation study in order to inform and motivate the optimal policy estimation for the case study.
In all settings, we consider three sample sizes of $n = 500$, $n=1000$ and $n=5000$. We generate two variations of the binary outcome prevalence: 40%-55% (common), and 2%-6% (rare). All settings use an approximate 25% treatment prevalence.\footnote{The goal is to maintain a consistent treatment prevalence across all settings, and comparable outcome prevalence by treatment group across the sub-settings within the rare and common outcome prevalence settings. Details of actual treatment and outcome prevalence are shown in the Appendix Table (ref).} As in hu2021estimating, we simulate 10 covariates with continuous $X_1, ... , X_5$ drawn from the standard common distribution and categorical $X_6, ... , X_{10}$ from Bernoulli(0.5).
As a baseline, we allow no confounding and set a constant propensity score $e(x) = 0.2$. We then induce confounding by allowing the treatment assignment and outcome response surfaces to both be functions of the covariates. The true propensity score $e(x)$ is given by:
where parameter values in the linear predictor are chosen to ensure the proportion of treated individuals is consistent across settings and there is a good overlap between the treated and control propensity score distributions.
We generate response surfaces for the binary outcome drawing from Bernoulli($\rho$), where $\rho$ is given by $m(x,\epsilon,w)$ as below, where $\epsilon$ is a variable that is not observable for the researcher, and is not a confounder (it is not present in the propensity score). We induce different levels of treatment effect heterogeneity by constructing three settings. Our primary interest lies in evaluating the performance of our methods for obtaining non-trivial treatment assignment rules. In our setting where we don't consider resource constraints, this corresponds to the scenario where some subgroups, defined according to observed covariates, do not benefit or are harmed by the treatment, while others have a benefit\footnote{Under resource constraints, it is sufficient that treatment effects are heterogeneous. While considering resource constraints is beyond the scope of this paper, we expect our results to be general for those settings.}. However, we also consider a setting where no treatment effect heterogeneity is present according to observed covariates, in order to assess the potential improvement in performance as explainable heterogeneity increases. The response surfaces are generated as follows:
where $\epsilon_i = \nu(\epsilon \sim \mathcal{N}(0,1))$ and $\nu = (0.1, 0.5, 0.8)$
Each scenario corresponds to a known CATE function - that helps evaluating the estimator's performance according to the true CATE - and an oracle policy assignment rule that treats those where the true CATE is negative. We note that this known CATE function is also a function of $\epsilon$, which we can access in the simulation, hence modelling the realistic setting when researchers don't have access to all effect modifiers contributing to the CATE function.
We use the known CATE to define an oracle policy for each scenario: rule which allocates treatment to all individuals with a true negative CATE.
For each DGP:
We use the following performance metrics to compare the methods.
One crucial objective of the simulations is to compare two policy classes: tree-based policies versus plug-in policies. We expect that some of the performance disparities between these two classes may stem from the inherent complexity difference. Shallow trees, as a less complex class, might capture less information compared to the fully non-parametric plugin class, which utilises estimated CATEs. To distinguish these performance differences from those that might arise due to estimation errors, we introduce a concept called the “oracle tree" in our analysis, where the true CATE is used to learn a depth-2 policy tree\footnote{We are limited to this depth due to computational power constraints.} for which the advantage $A_i$ is calculated and then compared to the oracle plug-in policy. We present the results in Table 2. (ref), reporting the proportion of the oracle plug-in policy advantage that the optimal tree-based policy is able to obtain.
Note that in settings with no heterogeneity, the oracle trees and the oracle plug-in perform equally. In settings with low heterogeneity, the oracle tree-based policy obtains 99% of the oracle value in settings with common outcomes, while this falls to around 80 % with rare outcomes. When there is complex heterogeneity (Setting 3) the oracle tree obtains around 60% to 80% of the advantage of the oracle plug-in.
Unless otherwise noted, we discuss results for the setting with mild confounding, with results for random treatment assignment (no confounding) reported in the Appendix.\footnote{Preliminary simulations also included variations in the level of confounding and overlap quality, which had little-to-no impact on results.}
\\
RMSEs of the CATE estimates are depicted in Figure (ref). We observe that there is more variation across methods in all settings with rare outcomes (bottom panel) compared to the common outcome prevalence setting (top panel). In low heterogeneity settings, the BART obtains the lowest RMSE in smaller sample sizes and across both common and rare outcome prevalence. In settings with more effect heterogeneity, however, the NDR learner achieves the lowest RMSE of the estimated CATEs. This difference is especially pronounced in the rare outcomes setting, with the RMSE of the NDR-learner CATEs being less than half of the other methods.\footnote{With no confounding present, the methods perform similarly across all settings (see Appendix Table (ref)). RMSE of CATEs are also numerically reported in the Appendix (Table (ref)) for both rare and common outcomes.}
Using the DR scores, for each method we also estimate the ATEs, for which wereport numeric and graphical results in the Appendix (Figure (ref) and Table (ref)). Unlike the CATE results, the NDR-learner obtains the lowest RMSE of estimated ATE in all sample sizes and settings, and the BART performs the worst by this metric.
In the the following two sections we present the main results of this paper. First, we compare the true values of the learned policies to the oracle advantage. Second, we investigate how accurately proposed estimators of the policy advantage capture the true policy advantage.
Here we first examine the trade offs between using the plug-in policy versus a tree-based policy class. We also look at whether using the modified trees changes our conclusions on how much we give up (or gain) by restricting the complexity of the policy. We focus on a selected setting (Setting 3, Rare outcomes, mild confounding), but at times refer to other scenarios.
Table (ref) contrasts the tree-based and plug-in policy assignment rules, in terms of percentage of the oracle advantage achieved by the learned policies. First we evaluate the performance of the learned policies according to policy class. Across all settings, for both common and rare outcome prevalence, the plug-in policy outperforms the tree-based policy learned from DR scores. However, it does not always outperform our proposed modified tree. When the degree of heterogeneity is low to none (Settings 1 and 2), the modified tree always outperforms the plug-in policy. As heterogeneity and sample size increase, the plug-in policy recovers its advantage and outperforms both types of trees - but in the case of rare outcomes, not by much (only a 1% improvement for N = 5000). When outcomes are rare (Table (ref), Panel B) and heterogeneity is complex, the performance of the tree-based and plug-in policies are similar, recovering around 70% of the oracle advantage.
Next we examine the performance of the specific ML methods within the policy classes, according to the percentage of oracle advantage achieved (Table (ref)) and according to the the RMSE of the true policy advantage. For the latter, Figure (ref) aims to compare the ML methods according to this metric, while Appendix Figure (ref) presents the same information by policy class.\footnote{Numeric values for all settings are reported in Appendix Tables (ref) and (ref).} For ease of comparison we also present numeric RMSE values for our setting of interest for each policy method in Table (ref).
In the common outcome settings (Panel A of Table (ref) and Figure (ref)), the ML methods perform similarly, with CF and CFTT being the best in Setting 3 (complex heterogeneity). In the rare outcomes case (Panel B of Table (ref) and Figure (ref)) we find that, among the plug in policies, the NDR-learner does the best across sample sizes. For the tree-based policy class, CFTT is best with the NDR-learner performing very similarly, and for the largest sample size (n=5000) NDR learner does equivalently to the CF and CFTT. When using the modified trees, NDR-learner again does the best across all sample sizes in this setting. The BART performs poorly in smaller samples, for example for $n=500$, it only recovers 8% of the oracle advantage (with the modified trees and plug-in policy class).
We now employ our simulations to evaluate the accuracy of the estimated policy advantage in capturing the true policy advantage of a learned policy. This evaluation is necessary since, in real-world scenarios, we only have access to the estimated policy advantages. Our objective is to offer guidance regarding the effectiveness of these measures in informing the selection of a policy learning method.
Due to its good performance in our preferred setting, here we are focussing on the NDR-learner (Figure (ref) Panel B).\footnote{All ML methods are depicted in Figure (ref).} We find that when using the tree-based policy class, calculating the policy advantage using the estimated CATEs is preferable to using estimated scores; that is, it results in an estimate that is closer to the true value of the learned policy (lower RMSE). For the plug-in based policy class, the error in this estimator is generally lower than the error observed for the trees, regardless of the version of the estimated advantage used. Here, we find that using the estimated DR score to report the policy advantage is in fact slightly better. When using the modified tree, once again the CATE-based estimator of the advantage is closer to the true value of the learned policy than the score based estimator. Importantly, these differences diminish with sample size.
We now proceed with a robustness check. Our objective is to determine whether our results are influenced by the binary outcome setting. Specifically, we aim to assess whether our modified tree-based policy still maintains its advantage compared to the standard trees learned from DR scores, and whether the CATE-based estimated advantages continue to be more accurate than the DR score based advantages, in a setting with continuous outcomes. To replicate conditions akin to our original specifications while simulating continuous outcomes, we have adjusted the response surfaces accordingly. The oracle policy remains unchanged: it assigns treatment all individuals with a negative CATE. Furthermore, we have made slight modifications to the BART algorithm to accommodate continuous outcomes. Details of the DGP alterations, as well as the BART modifications, can be found in the Appendix.
In setting 3 for rare continuous outcomes,\footnote{A ”rare continuous outcome" may correspond to a rate of infant mortality, for example, instead of an occurrence of infant mortality.} the BART does the best for plugin policy rule, while the Causal Forests do the best for trees (Table (ref)), in terms of the percentage of the oracle policy achieved. When looking at continuous outcomes with common outcome prevalence, the NDR learner performs the best again. In terms of the estimated values of the learned policies compared to their true performance, we observe similar patterns as in the binary outcome setting. For example, in Setting 3, the advantage obtained using the estimated CATEs captures the true policy advantage much better than the estimated advantage obtained from the DR scores (Table (ref)). This is true for both common and rare outcomes. We depict these results for ease of comparison again focusing the NDR-learner (Figure (ref)). Both panels show that as sample size increases, our suggested modifications continue to perform well. The modified tree performs best in terms of true value of the learned policy, and calculating the estimated advantage using $\hat{\tau}$ gets closest to the the true value of the learned policy. We note that in this setting, the error in the DR-score based metric doesn't diminish with sample size, for the sample sizes we consider.
One major finding of this paper is that using the estimated CATEs instead of the estimated double-robust scores as inputs to a tree-based policy learning method may be preferable, if the goal of the allocation is to get as close as possible to an “oracle" policy where everyone who benefits is treated, and still maintain a high level of interpretability. This modification to the policytree algorithm may also be useful in settings where DR-scores are not possible to obtain, or where practitioners are interested in working solely with individual treatment effect estimates (as in a49248a2417f4b649a92b0e2191af2e4). We show that not only is there no loss of performance should this modification be necessary or desirable, but also that the modification can perform better than the more complex plug-in rule.
Nonetheless, it's important to note that using estimated CATEs does not come with statistical guarantees in terms of minimising the regret athey2021policy. Therefore, practitioners must consider the balance between the practical advantages of the modified CATE-based tree and the loss of well-established properties.
We use these observations in the following section where we apply the methods to the case study.
We now return to our case study examining the impact of the Indonesian National Health Insurance Programme on infant mortality. This setting allows us to compare estimates across methods for a rare outcome, as mortality rates during the sample period were 2.6%. The objective of this portion of the case study is to learn rules assigning subsidised health insurance to mothers based on their observed characteristics, using the steps outlined in Section 2.3. An important difference from the simulation approach is that we restrict the covariates used in the policy learning step to a smaller subset (those used in the BLP analysis of Section 4). This reflects a real-world policy impact evaluation scenario where some variables may be sensitive or unavailable to policymakers.
We first discuss the findings of the case study, reflecting on the lessons learned from the simulations study. We think that the case study best reflects Setting 3 in the simulations. Then, we discuss the results obtained using the NDR learner in greater detail.
Table (ref) shows the number of individuals assigned to treatment according to each learned policy, alongside the estimated advantage of the learned policies, for all ML methods and policy classes considered. The policies treat 64-82 % of the sample (as a reference, in our data only 14% were treated). Overall (across ML methods, reporting metrics and policy classes) we find that there is an advantage from individualisation - the estimated policy advantage is around 50-100 % larger than the policy advantage of the “treat all" policy (Table (ref)). The estimated advantages reported by the tree-based policy learning methods tend to be higher than those reported from the plug-in policies.
We note, however, that depending on the ML method, the distribution of the CATE estimates vary substantially (seen Figure (ref) for a comparison of the Causal Forest and BART based CATEs). Therefore, we expect for some of the ML approaches, the CATE based plug-in-rules may be biased compared to a possible (unknown) oracle policy, and the CATE-based advantages may also be biased metrics of the true policy value.
Our modified tree learned from estimated CATEs yields larger estimated reductions in infant mortality than the score-based version across all ML methods. According to the lower reporting bias observed in this metric in our simulations, we think that the modified trees may be the better performing tree in this setting. We find that using the estimated DR scores to report the estimated policy advantage, reports larger estimated reductions in infant mortality.
In the remainder of this section, we zoom in to the results obtained using the NDR learner, our best performing method in the simulations. We find that the policy advantages range from -0.006 to -0.020 , compared to the corresponding ATE estimate of -0.003. Looking at the trees based on the NDR learner, the estimated advantage is -0.013 when calculated using estimated DR scores versus -0.006 if calculated using estimated CATEs (the corresponding figures are -0.020 and -0.008 for the modified trees). As our simulations showed that using the estimated CATEs to calculate the advantage was a more accurate reflection of the true value of the learned policy, we conclude that these larger DR score-based reductions would be overly optimistic in terms of the true reduction of infant mortality when implementing either version of the depth-2 tree-based policy.
Table (ref) compares mean characteristics of those who were actually insured in the sample, those who would be newly insured under NDR-learner depth-two tree-based and plug-in policies, and the full sample.\footnote{Recall that, although the algorithm is trained on the full set of covariates, we restrict the policy class to only use the set of covariates explored in the GATE analysis.} We note that the plug-in rule treats fewer individuals (N = 6,174) than the tree-based rules (N = 6,788 and N = 6,822 for the standard and modified trees, respectively), treats more with senior education, and fewer in the lower wealth quintiles, than the tree-based rule. The plug-in rule treats more with secondary education compared to the modified tree, which treats much fewer individuals in this subgroup than both of the other policies. We can see that all learned policies treat fewer individuals with the Poor Card (recall that in our heterogeneity analysis of BLP of GATEs, this subgroup had an undesirable positive signed treatment effect), and treat a larger portion of individuals ages 23-27, the subgroup exhibiting a significant portion of the negative treatment effect in our earlier analysis. This result demonstrates the potential for the learned policy in either class to point to drivers of treatment effect heterogeneity and account for them in the treatment assignment rule.
Figure (ref) depicts the treatment assignment rule from a depth-two tree learned from the NDR double-robust scores. This tree is restricted to the subset of covariates used in the BLP of GATE analysis. It is clear that the exhaustive tree-search algorithm (and the scores which this algorithm utilises) picks up on the undesirable positive effect of the Poor Card, as it yields a treatment assignment rule which does not treat individuals with a Poor Card if they have secondary education.\footnote{It is likely the case that a policy maker would not wish to exclude individuals from treatment should they be in possession of a card indicating poverty status. The learned policy would be easily modified by dropping this covariate from the list of acceptable decision criteria in the tree-search algorithm. In this paper, we allow the Poor Card variable to be a treatment decision criteria in order to demonstrate the ability of the policy tree class to pick up on potential drivers of heterogeneity.} The policy tree also indicates the potential importance of secondary education and cash transfers as drivers of treatment effect heterogeneity.
In this paper, we aim to answer three key questions: 1. Does the choice of machine learning method for obtaining double-robust scores and CATEs matter in policy learning settings with rare outcomes; 2. Under what circumstances does one class of policy (tree-based or plugin) do better than another, and by how much; and 3. how can we accurately estimate the advantage of a learned policy to closely approximate its true value?
In terms of the relative performance of the ML methods, we found that the with increasing sample size, the performance of all methods dramatically improved. While we haven't found radical differences in performance across the methods, we found some divergence in settings with rare outcomes and complex treatment effect heterogeneity. Here, we find that the NDR-learner performed best overall, closely followed by the the Causal Forests. In some settings, our adapted Causal Forest algorithm using testing and training (cross-fitting) samples yielded estimates which outperformed the standard “Honest" Causal Forest method.\footnote{The mechanism behind the better performance of our version in certain settings will need to be determined.}
Overall, we found that, as expected, plug-in rules generally outperformed shallow tree-based rules. While some of this performance difference can be explained by the shallow trees being able to capture less of the treatment heterogeneity than CATE-based plug in rules, we also investigated whether learning tree using double-robust scores (the standard method) as opposed to CATEs may contribute to this difference in performance.
Indeed, we found that in settings with rare outcomes and complex heterogeneity, using CATEs as opposed to DR scores to learn trees led to a treatment allocation that was closer to the oracle policy. This finding adds nuance to the theoretical proposition of athey2021policy in that the tree-based policies achieve asymptotic guarantees on the regret when double-robust scores are used to learn the policy. Our findings indicate that this phenomena is not an artifact of bias due to confounding or the binary outcome setting. Relatedly, we also find that reporting the estimated policy advantage in terms of the estimated CATEs, instead of the estimated DR-scores, may yield results which are closer to the true advantage of the learned policy - especially for depth-2 tree-based policies.
We note, however, that our simulation design has several limitations. First, it only has 10 covariates: as the size of the sample increases, the ratio of covariates to observations becomes very small, which may not capture the structure of observational datasets used in empirical research. For example, our health insurance case study data contains 64 covariates and 10,622 observations, resulting in a ratio of 0.006 (x to n). The simulation setting with 1,000 observations has a ratio of 0.01, and for 25,000 observations a ratio of 0.0004, implying that the larger simulation sample may be too low-dimensional to extrapolate to our setting. We recommend interpreting our findings for real-world data applications by first considering the dimensions of the observational dataset in question.
We acknowledge that our simulation design is stylised, and the parametric data generating process used may not pose a sufficient challenge for the ML estimators. Future work could utilise more complex data generation methods such as Generalised Adversarial Networks athey2019generalized. All the methods we consider assume no unmeasured confounding, which is also an important limitation of the case study presented. Future work may consider using methods to estimate CATEs for settings with unobserved confounding kallus2019interval. We may also see improved results by using ensembling such as the Super Learner for estimation of optimal policy rules luedtke2016super.
Our case study has highlighted a potential benefit of using a tree-based policy class, however, in that the learned decision tree is capable of pointing to potential drivers of treatment effect heterogeneity. In fact, tree-based rules make potential inequity on the resulting treatment assignment rule immediately obvious. Future work should examine the equity and interpretability concerns regarding various policy classes. Finally, in our application evaluating a health insurance program, it would be important to look at the optimal allocation of health insurance, when - as in the real world - providing subsidised health insurance is costly, and only a certain fraction of population can receive subsidies. It is straightforward to extend the methods we consider to account for resource constraints\footnote{Recently, alternative inputs have been proposed as inputs for the policy tree algorithm with the purpose of learning optimal subgroups under constraints cai2022capital} and costs of estimated policies, and we plan to make these extensions in subsequent work.