Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
45,513 characters · 16 sections · 17 citation commands
Policy-Oriented Binary Classification: Improving (KD-)CART Final Splits for Subpopulation Targeting
Policymakers and researchers often use recursive binary split rules to partition a population based on a binary outcome $Y\in\{0,1\}$ and target subpopulations with a probability of $Y = 1$ greater than 50% when implementing a policy. We call such policy targeting problems Latent Probability Classification (LPC). Practically, CART is often employed for LPC. For example, andini2018targeting uses CART to find subpopulations with a higher than 50% probability of being financially constrained and recommends targeting these households with a tax credit program. In Appendix (ref), we cite 23 empirical cases of using CART for various LPC problems to demonstrate the broad applicability of the setup that this paper investigates. In these studies, researchers typically use CART to divide the samples into many nodes, estimate the probability of $Y = 1$ for each node, and target those nodes with estimated probabilities higher than a threshold of 50%, denoted as $c = 0.5$. This approach, though intuitive, is not optimal for LPC. Here, we provide a toy example in Figure (ref) to illustrate the limitation of using CART for an LPC problem.
A toy example: Suppose the latent probability of a binary event $Y = 1$ is a sinusoidal function of an observable variable $X$, $\mathbb{P}(Y=1|X) = \frac{\sin (2 \pi X)+1}{3} \text{ where $ X \sim {\rm Unif}[0,1]$}$. Figure (ref) shows the function, where the green segment represents the subpopulation that should be targeted, and the orange segments represent the subpopulation that should not be targeted. In this example, we impose that the policymaker can only split the population once based on the value of $X$. CART splits at $X = 0.5$, denoted as $s^{CART}$ in Figure (ref). The left node has $\mathbb{P}(Y=1 | X > 0.5) = \frac{2}{3}(0.5 + \frac{1}{\pi}) > 0.5$, whereas the right node has $\mathbb{P}(Y=1| X \leq 0.5) = \frac{2}{3}(0.5 - \frac{1}{\pi}) \leq 0.5$. Consequently, we target the subpopulation by $X \leq 0.5$, i.e., left node. To demonstrate why $s^{CART}$ is suboptimal for the LPC problem, consider an alternative split at $s^* = \frac{5}{12}$. The left node still has $\mathbb{P}(Y=1|X \leq \frac{5}{12}) \approx 0.571 > 0.5$, right node has $\mathbb{P}(Y=1|X >\frac{5}{12}) \approx 0.164 < 0.5$. Only the left node ($X \leq \frac{5}{12}$) is targeted.
All subgroups that are correctly targeted/not targeted by $s^{CART}$ are also correctly targeted/not targeted by $s^*$. Moreover, $s^*$ excludes the group with $\frac{5}{12} < X < 0.5$ whose $\mathbb{P}(Y=1|X) < 0.5$ from being targeted whereas $s^{CART}$ incorrectly targets this subgroup. We say that $s^*$ strictly dominates $s^{CART}$. Section (ref) formally defines strict domination.
Proposed methods: This paper proposes methods that generate improved split rules. First, we replace the CART impurity function with a weighted sum of the distances between node means and the threshold $c$. This method is called Maximizing Distance Final Split (MDFS). Assuming the existence of a unique intersection between $\mathbb{P}(Y=1|X)$ and $c$, MDFS identifies the unique best split (hence, strictly dominating CART). The second method, Penalized Final Split (PFS), is a generalization of MDFS. It relaxes the unique intersect assumption and still strictly dominates CART. The third method, weighted Empirical risk Final Split (wEFS), adapts the weighted loss function from the cost-sensitive binary classification literature to our setup.
Generalizing the threshold from $c = 0.5$ to $c \in (0,1)$: Though most of the LPC studies use 50% as the threshold ($c=0.5$ corresponds to the classic 0-1 loss), there are more general choices. For example, sarkar2024ensembling uses CART to determine which forest zone has a higher than $c=61\%$ probability of forest fire to implement early warning systems. In some other cases, a policymaker may face budget constraints and hence, decides to adopt a threshold that is close to 1 hassanzadeh2021tradeoffs. In the rest of this paper, we use $c \in (0,1)$ to denote the threshold. We assume that $c$ is fixed before implementing our methods. In particular, policymakers can first tune $c$ with CART and then implement our methods with the chosen $c$. In this scenario, our methods still improve over CART because our theoretical results apply broadly to any fixed $c \in (0,1)$.
Extending to Knowledge distillation: Knowledge distillation (KD) refers to a two-step learning algorithm. The first step trains a teacher model with a higher learning capacity, e.g., a neural network or a random forest, to learn $\mathbb{P}(Y=1|X)$. The second step uses the learned $\mathbb{P}(Y=1|X)$ as the response variable to train a simpler student model. The goal of the student model is to output a simple and interpretable representation of the teacher model's knowledge. In our case, the student model is a CART. We refer the existing KD method with a CART student model as KD-CART. We apply the MDFS method to improve KD-CART and refer our proposed new method as KD-MDFS. This generalizes our contribution to a larger class of advanced tree-based methods.
Summary of contributions: Section (ref) formulates the LPC problem from a wide range of empirical works and shows that split rule generated by CART/KD-CART is strictly dominated. Section (ref) proposes MDFS, which point-identifies the unique best split rule assuming a unique intersection between $\mathbb{P}(Y=1|X)$ and $c$. In addition, we propose a consistent estimator for the MDFS split rule. Section (ref) shows that MDFS generates policies that target more vulnerable subpopulations. To relax the unique intersection assumption, we further propose PFS and wEFS in Section (ref). Lastly, in Section (ref), we demonstrate that the proposed methods outperform their respective baselines and target more vulnerable subpopulations, using simulations with synthetic and real-world datasets.
The novelty of our problem setup can be best established by comparing our work with various strands of existing frontier literature.
Nonparametric binary classification: Nonparametric binary classifiers are often not designed for transparent policymaking. babii2024binary's theorems apply to uninterpretable deep learning. singh2022optimal outputs stochastic decision rules, which raise fairness concerns. Our output policy is nonparametric, interpretable, and deterministic.
Policy targeting: Most of the existing policy targeting literature has been developed within the causal inference framework kitagawa2018should, athey2021policy, mbakop2021model. Such causal inference methods are incompatible with the LPC setup, where the treatment might not have been tested in real life. A more detailed comparison between LPC and policy learning literature is provided in Appendix (ref).
Tree-based methods' consistency: The tree-based method literature has a vested interest in consistency. wager2018estimation,zheng2023consistency show that different tree-based methods are consistent for estimating $\mathbb{P}(Y=1 | X)$. This work is interested in a different type of consistency. We show that our estimator consistently estimates the MDFS split rule.
Knowledge distillation with CART as student: liu2018improving,dao2021knowledge use CART as the student model in knowledge distillation, termed as KD-CART in our paper. We show that the theoretical results we develop for MDFS apply to KD-CART.
We first restrict the theoretical discussion to the following one-split, one-feature (OSF) LPC problem characterized as follows. Let $\mathcal{X} = [0,1]$ and $\mathcal{Y} = \{0,1\}$ be the univariate feature space and label space, respectively. Let $f: \mathcal{X}\xrightarrow[]{}\mathbb{R}^{+}$ be a probability density function that is continuous on $\mathcal{X}$, $F$ be its corresponding cumulative distribution function, and $\eta(x) := \mathbb{P}(Y=1|X=x)$ be continuous. Consider a dataset comprising $n$ i.i.d. samples, $(X_i, Y_i),\,i=1,\dots,n$, where $X_i\sim f$ and $Y_i|X_i\sim \text{Bernoulli}(\eta(X_i))$. A policymaker is allowed to split the feature space one time (i.e. one-split) using the univariate feature $X$ (i.e. one-feature). After the split, policymakers target those node(s) whose node mean is greater than $c$.
The most common criterion function optimized by CART to determine the split $s^{CART}$ is the weighted sum of variances of the two child nodes, i.e., $s^{CART} = \operatorname*{arg\,min}_{s \in (0,1)} \mathcal{G}^{CART}(s)$ where $\mathcal{G}^{CART}(s) = F(s)(\mu_L(s) - \mu^2_L(s)) + (1-F(s))(\mu_R(s) - \mu^2_R(s))$, $\mu_L(s) =~ \int_0^s \eta(x) dF(x) / F(s)$ and $\mu_R(s) =~ \int_s^1 \eta(x) dF(x) / (1 - F(s))$ are the left and right node mean, respectively.
To illustrate that $s^{CART}$ is strictly dominated, we first introduce the following definition of strict dominance for comparing two splitting rules $s$ and $s'$. The definition is adapted from dominating decision rule in classic decision theory. To guide our readers through the dense notation in Definition (ref), we provide an intuitive explanation after introducing the definition.
Despite its dense notation, Definition (ref) has a straightforward interpretation: split rule $s'$ strictly dominates $s$ if it performs no worse than $s$ for all $x \in [0,1]$ and strictly better than $s$ for all $x \in {\cal A} \subseteq [0,1]$, where $\cal A$ has nonzero measure.\footnote{“No worse” means that if $s$ targets/not target correctly at a point, then $s'$ must also target/not target correctly at that point, “strictly better” means that at some points where $s$ target/not target incorrectly, $s'$ targets/not target correctly at those points.}
We show that under very general conditions, there exist some rules that strictly dominate CART's split rule in Theorem (ref). Proofs of all lemmas and theorems are collected in Appendix (ref).
The key challenge in understanding Theorem (ref) is the interpretation of $s^*$. Figure (ref) provides a graphical illustration for interpreting $s^*$. Given that $\eta(s^{CART}) < c = 0.5$ and $\mu_L(s^{CART}) > \mu_R(s^{CART})$, Figure (ref) corresponds to the first minimization problem in the definition of $s^*$. The minimization problem searches for $s^*$ over the intersection of $s \in (0,s^{CART})$ and $s \in \{s: \eta(s) = c\}$, where $s^{CART} = 0.5$. In Figure (ref), there are two candidate values, $s = \frac{1}{12}$ and $s = \frac{5}{12}$, between which $s=\frac{5}{12}$ is closer to $s^{CART}$, and hence $s^* = \frac{5}{12}$. The graphical illustration is generalizable: $s^{CART}$ determines which of the two minimization problems is used to determine $s^*$, then $\eta(s) = c$ pins down a set of candidate values of $s^*$, and lastly, $s^*$ is set to be the one that is closest to $s^{CART}$ among all candidate values.
In the LPC setup, a teacher model learns $\eta(x), x\in[0,1]$. Prediction based on the teacher model is denoted as $\hat{\eta}(x)$. In the LPC setup, the student model is a CART that takes in $\hat{\eta}(x)$ as the response and learns to partition the population based on $\hat{\eta}(x)$. One can show that Theorem (ref) also applies to KD-CART, meaning there exist split rules that strictly dominates the split rule generated by KD-CART. Details are provided in Lemma (ref) in Appendix (ref).
The suboptimality of CART for solving OSF LPC problem motivates our proposed method MDFS, which point-identifies $s^*$. Since our theoretical results apply to OSF LPC, we advocate using our methods at the final splits with features identified by CART.\footnote{Though we focus on explaining MDFS in this section, all the theorems in this section easily extend to KD-MDFS, i.e., replacing the final splits' split criterion function of KD-CART with MDFS.} Restricting modifications to the last splits may appear trivial at first, however, note that as the tree grows deeper, the number of final splits increases exponentially, leading to non-trivial modifications to the policy designs. We substantiate this claim with empirical applications in Section (ref).
We argue for the plausibility of Assumption (ref) in Section (ref).
The proof of Theorem (ref) can be largely decomposed into two steps: first, we show that there exists an interval containing $s^*$ in which first-order condition guarantees that $s^*$ is the local minimum; second, we show that for all $s$ outside of the interval, $\mathcal{G}^*(s,c) < \mathcal{G}^*(s^*,c)$.
Theorem (ref) states that $s^*$ maximizes the population objective function ${\cal G}^*(s,c)$. In the finite sample regime (denote sample size as $n$), we can estimate the sample version of $s^*$ using $\{(X_i,Y_i)\}_{i=1}^{n}$ by maximizing the sample analogue of ${\cal G}^*(s,c)$. Following from Assumption (ref), we have $s^{*} \in (\epsilon,1-\epsilon)$. Define the MDFS final split estimator $\hat{s}$ as
where $\widehat{{\cal G}}^*(s,c) = s ~ \Bigg| \frac {\sum_{i=1}^{n} Y_i\mathbbm{1}\{X_i \leq s\}} {\sum_{i=1}^{n}\mathbbm{1}\{X_i \leq s\}} -c \Bigg|+ (1-s) \Bigg| \frac {\sum_{i=1}^{n} Y_i \mathbbm{1}\{X_i > s\}} {\sum_{i=1}^{n}\mathbbm{1}\{X_i > s\}} -c \Bigg|$ is the estimator of ${\cal G}^*(s,c)$. The following theorem states that $\hat{s}$ is a consistent nonparametric estimator for $s^*$.
Uniform $X$. The set of quantile statistics for any continuous feature follows a uniform distribution, so we can convert a continuous variable to its quantile statistics. For a discrete $X$, the LPC problem is simpler. We defer the explanation to Appendix (ref).
The unique intersection condition is weaker than monotonicity, which is assumed by some theoretical works on CART blanc2020provable. Moreover, in many real-life applications, the monotonicity of $\eta(X)$ is a reasonable assumption. For example, in our real-world dataset application in Section (ref), a final node is split based on blood glucose level, and the binary outcome $Y$ is diabetic status. It is reasonable to assume that the probability of diabetes increases with blood glucose level. Nevertheless, monotonicity is not necessary, see Figure (ref) as a non-monotonic example that satisfies Assumption (ref).
Further, when we apply MDFS to the final nodes, the splitting feature is determined by CART. CART selects the splitting feature that gives the greatest reduction in variances (i.e., the greatest increase in purity in binary classification). Hence, when many features are available, CART is likely to pick some features that exhibit a salient trend (e.g., monotonicity), ruling out features that are more likely to violate the unique intersect assumption. Also, as we go down the tree, the domain of all nodes becomes smaller and smaller, making Assumption (ref) easier to satisfy. For example, take Figure (ref) as an example. It violates the unique intersect assumption. However, if we split the population at $X = 0.3$, then both child nodes will satisfy the unique intersection assumption.
This section relaxes the OSF LPC setup and consider the policy relevance of MDFS under many-node many-feature setup. We demonstrate two advantages of MDFS policies: (i) targeting more vulnerable subpopulations than CART by using the same amount of resources and (ii) uncovering subgroups with a higher-than-threshold probability of event $Y=1$ that CART ignores.
Suppose there are two nodes $\{t_1,t_2\}$ with the same amount of population and corresponding features $X_1,X_2 \sim \rm{Unif}(0,1)$. We depict $\eta_1(X_1)$ for node $t_1$ in Figure (ref) and $\eta_2(X_2)$ for node $t_2$ in Figure (ref). The analytical forms of $\eta_1(X_1)$ and $\eta_2(X_2)$ are provided in Appendix (ref). Here we compare the targeted population using MDFS policy versus CART policy. Splitting nodes $t_1$ and $t_2$ individually at $s^{CART}$ versus $s^*$ results in different target subpopulations: $s^{CART}$: Target $\{X_1<\frac{1}{2}\}$ in $t_1$. $s^*$: Target $\{X_1<\frac{5}{12}\}$ in $t_1$ and $\{\frac{11}{12}<X_2<1\}$ in $t_2$. The two sets of policies target the same proportion of the population, but $\eta_1(x_1) < 0.75$ for $x_1 \in \{\frac{5}{12}<X_1<\frac{1}{2}\}$, which is targeted by $s^{CART}$, whereas $\eta_2(x_2) > 0.75$ for $x_2 \in \{\frac{11}{12}<X_2<1\}$, which is targeted by $s^*$. Therefore, policies based on LPC target a more vulnerable subpopulation than CART/KD-CART policy.
Admittedly, the fact that LPC and CART policies target the same proportion of the population in the previous example is by construction. It is also possible that the proportion of the population targeted by the MDFS policy is bigger than that by CART or KD-CART. In this scenario, comparing the effectiveness of the two sets of policies is not straightforward. Nonetheless, LPC still has policy significance. It discovers new latent groups with a higher-than-$c$ probability of an adversarial event happening to them (i.e., vulnerable subpopulations), e.g., $\eta_2(x_2) > 0.75$ for $\frac{11}{12} < x_2 < 1$.
We make two remarks to formalize the two advantages that the MDFS policy offers. Assuming a homogeneous targeting cost per unit of population, the targeting cost of a policy is the percentage of the subpopulation targeted. Denote all $M$ final splitting nodes as $\{t_1,t_2,\dots,t_M\}$ and $\mathcal{M} = \{1,2,\dots,M\}$. For node $m$, we denote the feature selected by CART as $X_{(m)}$ and the CDF of $X_{(m)}$ as $F_m$ and let $\eta_m(x) = \mathbb{P}(Y=1|X_{(m)} = x)$. The cost of CART and MDFS policies are
Assume that for some $~ m \in \mathcal{M}, ~s^{CART}_m \neq s^{*}_m$.
To relax Assumption (ref), we propose two additional methods: PFS and wEFS.
Intuitively, if $\mu_L$ is close to $c$ and it is slightly higher than $c$, by the continuity assumption on $\eta(x)$, it's likely that there is a considerable amount of subpopulation from the left node with $\eta(x) < c$, see $0<x<\frac{1}{12}$ and $\frac{5}{12}<x<\frac{1}{2}$ in Figure (ref) as examples. This can substantially increase misclassification cases because $\mu_L$ and $\eta(x)$ are on different sides of $c$ for all these $x$ values. Following this intuition, we consider adding a penalty to $\mathcal{G}^{CART}$ that pushes $\mu_L$ and $\mu_R$ away from $c$. Let
where $J = F(s) W(|\mu_L - c|) + (1-F(s)) W(|\mu_R - c|)$ and $W: \mathbb{R}^{+} \cup 0 \to \mathbb{R}^{+}\cup 0$ is a decreasing function that penalizes small distances between the node means (i.e., $\mu_L$ and $\mu_R$) and $c$, and $\lambda\ge0$ controls the weight of the penalty term, which we denoted as $J$. The following theorem states that under some regularity conditions of the function $W$ and weight $\lambda$, the split rule that minimizes ${\cal G}^{PFS}$, denoted as $s^{PFS}$, strictly dominates $s^{CART}$.
We provide an intuitive explanation for Theorem (ref) in Appendix (ref). Note that MDFS is a special case of PFS whose $\lambda = \infty$ and $W$ is the identity function.
We adapt cost-sensitive classifier from koyejo2014consistent to our setup and name it weighted Empirical Final Split (wEFS). Due to the lack of theoretical backing, we do not elaborate on wEFS. We defer the explanation for the lack of the theoretical results to Section (ref). Details about wEFS can be found in Algorithm (ref) in Appendix (ref).
In this section, we conduct comprehensive numerical experiments to compare the performance of our proposed methods and the classic CART under both regular and KD frameworks. Under regular framework, we compare the classic CART with MDFS, PFS, and wEFS. Under KD framework, we compare RF-CART with RF-MDFS. We replace KD with RF because the teacher model is a Random Forest.
Settings: We simulate synthetic datasets using 8 different data generation processes (DGP), e.g., the Friedman synthetic datasets friedman1991multivariate, breiman1996bagging. These setups are designed to capture various aspects of real-world data complexities and challenges commonly encountered when using tree-based methods, such as nonlinear relationships, feature interactions, collinearity, and noises (subtitles in Figure (ref)). Detailed descriptions of the DGP are provided in Appendix (ref). For each DGP, we consider a threshold of interest $c\in\{0.5, 0.6, 0.7, 0.8\}$, resulting in 32 unique tasks.
We set two stopping rules for growing the tree: (i) a max depth of $m\in\{4, 5, 6, 7\}$, (ii) a minimal leaf node size of $\rho n$, where $\rho \in \{1\%, 2\%, 3\%\}$ and $n$ is the sample size. We set $n = 5000$. The fitting procedures stop once either of the rules is met. These give us 12 configurations for each tasks. For each of the $32\times 12 = 384$ settings, we do 50 replicates of experiments.
Evaluation metrics: We define false positives (FP) and false negatives (FN) before defining the two performance metrics that we use in the synthetic data simulations.
where $\mathbf{X}_i$ is the multivariate feature vector which includes all features in the simulation and $\widehat{T}(\mathbf{X}_i)$ is the tree-estimate of $\eta(\mathbf{X}_i)$.
We use two metrics to evaluate the performance and robustness of all methods: misclassification rate (MR) and F1 score (F1). MR and F1 are defined as
Given that our theorems state that split rules generated by (RF-)MDFS and PFS strictly dominate (RF-)CART, we expect that our proposed methods outperform their counterpart in both metrics.
Model fitting procedures: As we mentioned in Section (ref), our methods and their counterparts only differ in the splits at the final level (final splits). We first implement CART until the final split is reached based on the stopping rules. Once the final split is identified, we select the splitting feature using CART, then implement MDFS, PFS and wEFS with the selected splitting feature. MDFS and PFS decide the split by optimizing $\mathcal{G}^*$ and $\mathcal{G}^{PFS}$, respectively, while wEFS selects the split by identifying $s$ such that the minimum weighted empirical risk is obtained.
For $W$ in $\mathcal{G}^{PFS}$ from (ref), we choose $W(d) = 1 - d$. This choice of $W$ satisfies conditions specified in Theorem (ref). We also experiment with alternative choices, such as $W(d)=(1-d)^2$ and $W(d)=\exp(-d)$, but observed similar performance across these choices. Based on Theorem (ref), we set $\lambda=0.1$, a sufficiently small value that presumably satisfies the conditions of the theorem while maintaining practical effectiveness. We also propose a standard $\lambda$ selection procedure inspired by cross-validation and the honest approach athey2016recursive (see details in Appendix (ref)).
Comparison against respective baselines: Figure (ref) provides a pairwise evaluation of MR differences between CART and its three refinements—MDFS, PFS and wEFS, and between RF-CART and RF-MDFS. Each boxplot depicts the distribution of the MR difference for a single DGP across 50 replicates, with $m=7, \rho = 2\%, c=0.5$. Tables beneath the boxplots report one-sided p-values from paired t-tests (mean $<$ 0) and Wilcoxon signed-rank tests (median $<$ 0). Except for a few cases in Ball, Friedman \#1, and Ring, MDFS and RF-MDFS reduce both the mean and the median MR relative to their counterparts, CART and RF-CART, at the 5$\%$ significance level. PFS and wEFS outperform CART in most cases, but have less stability than MDFS, as evidenced by their large p-values in a few cases.
The comparison in terms of F1 is similar. All proposed methods largely outperform their counterparts at the 5$\%$ significance level. We defer this result to Appendix (ref).
RF-MDFS has the strongest performance among six methods: Our methods have a clear advantage over their respective baselines, as shown by Figure (ref). In addition, we compare all six methods together, instead of pair-wise comparison. RF-MDFS performs the best out of all six methods on 26 out of the 32 tasks in terms of MR, 27 out of 32 tasks in terms of F1, as shown by Table (ref) and Table (ref) in Appendix (ref). These results highlight the strength of combining MDFS with KD-CART, a frontier tree-based algorithm.
We implement CART, MDFS, RF-CART and RF-MDFS with the Pima Indians Diabetes dataset smith1988using to demonstrate the policy significance of our proposed methods.\footnote{We supplement an additional forest fire empirical study in Appendix (ref) to showcase the wide applicability of our paper.} The Pima Indians Diabetes dataset measures health factors among Pima Indian women with the response variable being a binary indicator of diabetes status: 34.9% of the sample is diabetic. It contains 768 observations and 8 features: number of pregnancies, glucose level, blood pressure, skin thickness, insulin level, BMI, family diabetes history index, and age. We use these eight features to search for subpopulations whose probability of having diabetes is above 60% with depth of trees fixed at $m=3$. We pick these hyperparameter values so that one set of the empirical results match Remark (ref). We also experimented with other hyperparameter values. The results from these additional experiments align with Remark (ref).
CART vs MDFS: As shown in Figure (ref), CART and MDFS commonly target those with ${\rm Glucose} > 129.5$ and ${\rm BMI} > 29.95$, which consists of 188 observations. The two methods' targeting policies differ in two ways: CART additionally targets $127.5 < {\rm Glucose} \leq 129.5$ and ${\rm BMI} > 29.95$. This subgroup consists of 19 observations. MDFS additionally targets ${\rm Glucose} > 166.5$ and ${\rm BMI} \leq 29.95$. This subgroup consists of 12 observations. The difference between the sizes of these two subgroups is 7, which is small relative to 188, i.e., the size of the subgroup commonly targeted by both policies. Assuming that the sample is representative of the population of all Pima Indian women, then the two sets of policies will incur a similar amount of targeting resources. The additional group targeted by CART has a 57.9% probability of having diabetes, whereas the additional group targeted by MDFS has a 66.7% probability of having diabetes. MDFS targets a subpopulation that is more prone to diabetes. This corresponds to Remark (ref).
RF-CART vs RF-MDFS: As shown in Figure (ref), both RF-CART and RF-MDFS commonly target ${\rm Glucose} > 157.5$ and ${\rm BMI} > 29.95$, a subgroup consisting of 92 observations. RF-MDFS additionally targets two subgroups: ${\rm Glucose} > 166.5$ and ${\rm BMI} \leq 29.95$ consisted of 12 observations and $129.5 < {\rm Glucose} \leq 157.5$ and ${\rm BMI} > 29.95$, which consisted of 96 observations. RF-MDFS targets a much greater number of observations than RF-CART. If the sample is representative of the population, then RF-MDFS would use much more targeting resources than RF-CART. Nevertheless, RF-MDFS is useful in uncovering groups with higher than 60% probability of having diabetes that RF-CART is unable to find. For example, the first group that RF-MDFS additionally targets has a 63.6% probability of having diabetes. This aligns with Remark (ref).
Cost-sensitive binary classification: LPC can be related to the cost-sensitive binary classification problem, further highlighting the policy significance of the LPC problem. nan2012optimizing,menon2013statistical,koyejo2014consistent show that the optimal classifier of a cost-sensitive binary classification problem is determined by whether the latent probability is above some performance-metric-dependent threshold. Whether the theoretical results in the cost-sensitive classification literature carry over to our setup is unclear. For example, nan2012optimizing shows risk consistency with uniform convergence of the empirical risk. In our setup, $\{\mu_L(s),\mu_R(s)\}$ appear inside the indicator functions, and uniform convergence of empirical risk is not guaranteed even with a large sample. In another related work, koyejo2014consistent proves risk-consistency of weighted ERM assuming that the policymakers search through all real-valued functions. By definition, LPC searches binary-split-type policies only. The performance of wEFS is strong in our simulations, so it would be interesting to look into the theoretical property of wEFS in future research.
Subgroup discovery: lavravc2004subgroup,herrera2011overview,atzmueller2015subgroup,helal2016subgroup discuss a widely-used quality measure, called Weighted Relative Accuracy (WRAcc), in the subgroup discovery literature. Under the assumption that $ X \sim {\rm Unif}[0,1]$,
where the subscript $L$ indicates it is the WRAcc measure associated with the left node. $\mathcal{G}^*$ is a generalization of $\text{WRAcc}_L + \text{WRAcc}_R$. Hence, Theorem (ref) applies to a combination of CART and WRAcc.
Conclusion: Our paper points out that classic CART/KD-CART is suboptimal for LPC in each split. Based on different assumptions, we propose three alternative methods: MDFS, PFS and wEFS. MDFS and PFS generate policy rules that strictly dominate CART. Our proposed methods predominantly outperform their counterparts in our simulation and provide more policy insights in real-world applications.