Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
29,084 characters · 16 sections · 20 citation commands
Learning from Double Positive and Unlabeled Data for Potential-Customer Identification
Data-driven, effective target marketing has attracted significant attention in business. To promote product sales, we focus on the set of potential customers defined as:
This idea is based on the premise that loyal customers, who already feel an attachment to the company, are likely to purchase new products without additional advertising. On the other hand, those lacking strong company loyalty may fail to notice the product or choose similar products from competitors, even if they are interested. By identifying and targeting such potential customers, we aim to conduct marketing more efficiently.
In this work, we propose a framework for classifying individuals in a broader population to find those who meet the above definition of potential customers. Let each person be characterized by a feature vector $X$, and suppose they have two binary labels: \[ Y \in \{-1,+1\}\quad\text{and}\quad Z \in \{-1,+1\}. \] Here, $Y = +1$ indicates the person is interested in the product, while $Z = +1$ indicates the person has loyalty to the company. Conversely, $Y = -1$ and $Z = -1$ each signify the opposite of those attributes. We define a new binary label $W \in \{-1, +1\}$ such that
Thus, $W = +1$ identifies people who are potential customers (interested in the product but lacking loyalty to the company), whereas $W = -1$ indicates they do not meet those conditions.
Our goal is to learn a classifier that predicts $W$ using only positive data and unlabeled data. More concretely, we assume availability of three datasets:
These datasets need not be strictly separate; some overlap among them is permissible. Figure (ref) illustrates how the interest label $Y$, the loyalty label $Z$, and the newly defined label $W$ for potential customers correspond.
In many real-world scenarios, it is difficult or expensive to obtain reliable negative labels: a lack of purchase does not necessarily prove no interest in the product, and the absence of a subscription or account does not directly show no loyalty to the company. Consequently, we lack explicit negative labels. However, standard supervised learning algorithms typically require both positive and negative examples. Learning from positive and unlabeled data (PU learning) is a form of weak supervision that addresses this issue and has been employed in domains such as information retrieval, anomaly detection, and beyond elkan2008learning,ward2009presence,pmlr-v5-scott09a,blanchard2010semi,li2009positive,nguyen2011positive.
Existing PU learning algorithms enable binary classification when only positive and unlabeled data are available. However, our setting involves two sorts of positive labels---one indicating product interest ($Y=+1$) and another indicating both product interest and company loyalty ($Y=+1, Z=+1$)---and we aim to distinguish $W=+1$ (potential customers) from $W=-1$. Hence, a direct application of standard PU learning methods is not sufficient.
We resolve this by extending existing PU learning techniques to incorporate the positive loyalty data. Concretely, we first define a loss function for classifying $Y=+1$ versus $Y=-1$. Since $W=+1$ corresponds to $Y=+1$ and $Z=-1$, we then adjust this loss function accordingly to construct a single objective function for predicting $W=+1$ versus $W=-1$. Despite being optimized in one stage, our approach effectively contains two PU learning losses: one for identifying those with $Y=+1$ and another for adjusting loyalty status ($Z=+1$ versus $Z=-1$). Hence, we refer to our approach as double PU learning. Details of the proposed method are presented in Section (ref).
In Section (ref), we report simulation results to verify the validity of our algorithm. In Section (ref), we conduct an experiment using real-world marketing data. These experiments show that the proposed double PU learning classifier works as intended for the outlined task.
Let $X\in\mathcal{X}\subset \mathbb{R}^d$ be a feature random variable that characterizes an individual. Each individual with $X$ potentially has two labels $Y\in\{-1, +1\}$ and $Z\in\{-1, +1\}$. In our setting, we can only observe labeled individuals with $Y = +1$ or $Z = +1$ or unlabeled individuals. Given the positive and unlabeled data, our interest lies in classifying $\{Y = +1\} \land \{Z = -1\}$ or not given $X$. We introduce more detailed setting below.
To deal with missing random variables, we introduce potential outcomes, which are used in the literature of missing outcomes Vaart1998 and causal inference Neyman1923,rubin1974estimating.
Let $Y, Z \in \{-1, +1\}$ be binary labels, and $X\in\mathcal{X}\subset \mathbb{R}^d$ be a $d$-dimensional feature vector with a feature space $\mathcal{X}$. We suppose that the triple $(Y, Z, X)$ is generated as follows:
In this study, we consider a situation where there are two different labels $Y$ and $Z$ in each object whose feature is represented by $X$. Specifically, in our application, we consider the following situation.
\paragraph{Example:} Each object is a customer whose feature is summarized as $X$, such as its age. For the customer, $Y$ corresponds to an actual outcome of advertisement; that is, the customer bought a product or not after an advertisement is treated. Here, $Z$ corresponds to a loyalty of the customer, which represents that the customer likes the brand or not. If the labels are $Y = +1$ and $Z = +1$, the customer likes the brand and bought its product as a result of advertisement. If the labels are $Y = +1$ and $Z = -1$, the customer is not interested in the brand but bought its product.
This study assumes that we can observe the following three datasets, which are generated from $p\big(x\mid y = +1\big)$, $p\big(x\big)$, and $p\big(x\mid (y, z)=(+1, +1)\big)$, respectively:
Here, we defined $\mathcal{D}_{(y, z) = (+1, +1)}$ as a dataset independent of $\mathcal{D}_{y=+1}$. However, $\mathcal{D}_{(y, z) = (+1, +1)}$ can be observed as a subset of $\mathcal{D}_{y=+1}$; that is, $\mathcal{D}_{(y, z) = (+1, +1)}\subset \mathcal{D}_{y=+1}$.
This setting is a variant of case-control PU learning Gang2016, which has been adopted by duPlessis2015, Kiryo2017, and kato2018pubp. Another framework, known as censoring PU learning, has been used in earlier studies such as elkan2008learning and Bekker2018. Although case-control PU learning accommodates a more general setting than censoring PU learning, it requires additional information to train a classifier.
Our interest lies in classifying $(Y, Z) = (+1, -1)$ and not $(Y, Z) = (+1, -1)$ given $X$ (Figure (ref)); that is, we define positive label $W = + 1$ as $(Y,Z)=(+1, -1)$ and negative label $W = - 1$ as $\big\{(Y,Z)=(+1, +1)\big\}\cup \big\{(Y,Z)=(-1, +1)\big\}\cup \big\{(Y,Z)=(-1, -1)\big\}$:
Here, we note that because we cannot observe which observations are labeled $Y = -1$ or $Z = -1$. Figure (ref) shows the quadrants divided by these labels. The upper left quadrant represents $W=+1$, and the other quadrants are $W=-1$.
\paragraph{Classification risk.} Let $g:\mathcal{X} \to \mathbb{R}$ be a classifier and $\mathcal{G}$ be some set of $g$. Let $\mathbb{E}_{W = +1}$ and $\mathbb{E}_{W = -1}$ be expectations over $p\big(x\mid w = +1\big) = p\big(x\mid (y, z) = (+1, -1)\big)$ (density of $X$ conditioned on a positive label) and $p\big(x\mid w = -1\big) = p\big(x\mid (y, z) = (+1, +1) \lor (-1, +1) \lor (-1, -1)\big)$ (density of $X$ conditioned on a negative label). Then, we define an optimal classifier as \[g^* = \operatorname*{arg\,min}_{g\in\mathcal{G}}R_{0\mathchar`-1}(g),\] where $R_{0\mathchar`-1}(g)$ is the expected misclassification rate when the classifier $g(X)$ is applied to marginal (unlabeled) feature distribution $p(x)$, defined as
where $\ell_{0\mathchar`-1}(z) = \frac{1}{2}\mathrm{sgn(-z)} + \frac{1}{2}$ is the zero-one loss and $\alpha = p(w = +1) = p\big(y = +1, z = -1\big)$ is called the class-prior. We also define $\beta = p(y = +1)$ and $\gamma = p(y = +1, z= +1)$ and assume that $\beta$ and $\gamma$ are known.
This section provides our algorithm. First, we discuss an unbiased risk for $R_{0\mathchar`-1}(g)$ in Section (ref). Next, we define empirical risk minimization in Section (ref).
If we have data sampled from $p(x\mid w = +1)$ and $p(x\mid w = -1)$, we can directly construct $R_{0\mathchar`-1}(g) = \alpha\mathbb{E}_{W=+1}\big[\ell_{0\mathchar`-1}(g(X))\big] + (1-\alpha)\mathbb{E}_{W=-1}\big[\ell_{0\mathchar`-1}(-g(X))\big]$. However, since we do not assume the access to the data, we consider another approach.
Here, we discuss how we identify $R_{0\mathchar`-1}(g)$ from observations. For a function $h:\mathcal{X}\to \mathbb{R}$, we define the expected value of $h(X)$ over $p(x\mid y = +1)$, $p(x)$, and $p(x\mid y = +1, z = +1)$ as $\mathbb{E}_{y=+1}[h(X)]$, $\mathbb{E}_{U}[h(X)]$, and $\mathbb{E}_{y=+1, z = +1}[h(X)]$, respectively. Then, we write $R_{0\mathchar`-1}(g)$ using $\mathbb{E}_{y=+1}[h(X)]$, $\mathbb{E}_{U}[h(X)]$, and $\mathbb{E}_{y=+1, z = +1}[h(X)]$ as follows:
Thus, we obtain an equivalent formulation of $R_{0\mathchar`-1}(g)$ using $\mathbb{E}_{Y=+1}[h(X)]$, $\mathbb{E}_{U}[h(X)]$, and $\mathbb{E}_{(Y, Z) = (+1, +1)}[h(X)]$. This section presents an unbiased risk for $R_{0\mathchar`-1}(g)$ with replacing its zero-one loss with a surrogate loss.
\paragraph{Sample approximation} For $h:\mathcal{X}\to \mathbb{R}$, we define sample approximations of $\mathbb{E}_{Y=+1}[h(X)]$, $\mathbb{E}_{U}[h(X)]$, and $\mathbb{E}_{(Y, Z) = (+1, +1)}[h(X)]$ as
\paragraph{Unbiased empirical risk.} Then, an unbiased risk is given as
Here, it holds that $\mathbb{E}\big[\widehat{R}_{0\mathchar`-1}(g)\big] = R_{0\mathchar`-1}(g)$.
We replace the zero-one loss $\ell_{0\mathchar`-1}(z)$ with a surrogate loss $\ell:\mathcal{R} \to \mathbb{R}$.
\paragraph{Surrogate loss.} Since it is not easy to optimize the zero-one loss $\ell_{0\mathchar`-1}$ directly, we replace it with a surrogate loss, including the log-loss and hinge loss. Let us denote a surrogate loss by $\ell:\mathcal{R} \to \mathbb{R}$. We raise candidates of a surrogate loss in Table (ref). From the computational perspective, the use of surrogate losses with convexity is preferable. However, there are applications where non-convex surrogate losses perform well. Using a surrogate loss $\ell$, we define the population risk as
\paragraph{Empirical risk minimization.}
We train $g$ as
where
We refer to an algorithm that trains a binary classifier by ((ref)) as double PU learning, since this objective function essentially consists of two PU learning risks as explained in the following section.
We can decompose $\widehat{R}(g) $ as
where $\widehat{R}^{\mathrm{PU}}(g)$ is the standard PU learning risk defined as
\paragraph{Unbiased PU risk.} The part $\widehat{R}^{\mathrm{PU}}(g)$ corresponds to an unbiased PU risk for classifying $Y = +1$ and $Y = -1$. By minimizing $\widehat{R}^{\mathrm{PU}}(g)$, we can train a classifier that classifies individuals into $Y = +1$ and $Y = -1$. Figure (ref) illustrates this classification, where the blue part corresponds to $Y = +1$ and the remaining part corresponds to $Y = -1$.
\paragraph{Unbiased double PU risk.} The blue region in Figure (ref) includes $(Y, Z) = (+1, +1)$ in addition to $(Y, Z) = (+1, -1)$ ($W = + 1$). We aim to remove this region. This step can be interpreted as the second PU classification, since we minimize another empirical risk by regarding $Y = +1$ as unlabeled data and $(Y, Z) = (+1, +1)$ as labeled data. As a result, we obtain a classifier that classifies individuals into $W = +1$ and $W = -1$. In Figure (ref), the blue part corresponds to $W = +1$ and the remaining part corresponds to $W = -1$.
Thus, our double PU learning algorithm trains a classifier for identifying potential customers by minimizing the unbiased empirical risk. This unbiasedness is important because it guarantees finite-sample stability of the learning process. In our setting, obtaining an unbiased risk estimator is not obvious, as the sampling scheme is non-standard in binary classification.
When the hypothesis class $\mathcal{G}$ is highly complex—such as when it includes neural networks—empirical risk minimization in PU learning often breaks down Kiryo2017. The problem stems from the negative term in the risk expression, which can diverge to negative infinity during training, a phenomenon referred to as train-loss hacking in Kato2021bregman. To address this issue, Kiryo2017 introduced the non-negative risk correction, and the same correction can be applied to double PU learning . This adjustment has been shown to enhance training stability. One way of the correction is
In many real applications, dataset is imbalanced in terms of class distribution, and cost-sensitive learning is one way to cope with this problem. Double-PU learning can be transformed to cost-sensitive setting. Slightly modifying the derivation of Lemma (ref), we suggest
where $c_{FN}$ and $c_{FP}$ are the costs of false negative and false positive respectively.
To verify the proposed algorithm we generated 500 positive samples with label $(Y,Z)=(+1,-1)$ and 1000 negative samples with labels $(Y,Z)=(+1,+1)$ and $(Y,Z)=(-1,-1)$, respectively, from bivariate Gaussian distributions (Figure (ref)). Using 70% of the positive samples as labeled data, we randomized 30% of both $(Y,Z)=(+1,-1)$ and $(Y,Z)=(+1,+1)$ to unlabeled for $Y$, and further randomized 50% of $(Y,Z)=(+1,+1)$ to unlabeled for $Z$. Samples with $Y=-1$ were fully unlabeled for both $Y$ and $Z$. With $(x_1,x_2)$ as features we trained a logistic‑loss classifier. The learned decision boundary $p(w=+1\mid x)=0.5$ is shown in Figure (ref), demonstrating that double PU learning successfully separates potential customers despite partial labeling.
We applied our method to bank marketing dataset \footnote{https://www.kaggle.com/datasets/janiobachmann/bank-marketing-dataset/data} originally from UCI machine learning repository. There are 11,162 samples total with 15 features. We want to predict customers who are likely to accept a marketing offer but not likely to default. In order to do so, we defined $Y=+1$ if a customer accepted a marketing offer and $Z=+1$ if the customer is in default. We want to find customers of $W=+1$ ($Y=+1$ but $Z=-1$). Note that this situation is different from the main motivation of this paper (finding a customer who have shown interest and low loyalty), but this setting also aligns with double-PU learning, because a bank often does not know a customer defaults unless the bank itself offers a loan to the customer. We first divided the original dataset into training and test dataset by 80% and 20%. Next, we further divided the training dataset into three datasets where the first one (10% of total) used only samples with $Y=+1$ (corresponding to $\mathcal{D}_{y=+1}$), the second one (10% of total) used all samples (corresponding to $\mathcal{D}_{U}$) and the third one (80% of total) used only samples with $Y=+1$ and $Z=+1$ (corresponding to $\mathcal{D}_{(y,z)=(+1,+1)}$). We applied double-PU learning to these three datasets. We set $\beta=0.4738$ and $\gamma=0.0046$ and employed a linear classifier for $g$. Also, since a customer default can result in a significant cost, we assigned a cost to false positives that is $100$ times higher than the cost of false negatives for predicting $W$ using the cost-sensitive learning explained above. As a result of the experiment, ROC-AUC of the learned classifier on the test dataset is 0.6013, implying that it can predict better than the random classifier.
PU learning has also been investigated in statistics and economics Lancaster1996casecontrol. In the machine-learning literature, elkan2008learning proposed a practical algorithm for PU learning, and duPlessis2015 later introduced another practical approach, known as unbiased PU learning.
We assumed the class priors $\beta$ and $\gamma$ are known, but this is often not the case in real applications. While the class prior can be estimated under certain conditions Kato2018alternateestimationclassifierclassprior, doing so introduces extra assumptions.
Two main data-generation mechanisms are considered in PU learning: censoring PU learning and case-control PU learning (see Section (ref)). The setting analyzed by elkan2008learning falls under censoring PU learning, whereas those in Lancaster1996casecontrol and duPlessis2015 correspond to case-control PU learning. In case-control PU learning, positive instances are gathered independently of the unlabeled sample. In contrast, in censoring PU learning, a single sample is drawn first, after which only a subset of its positive instances is revealed. Due to this property, censoring PU learning is also called the one-sample scenario, while case-control PU learning is also called the two-sample scenario.
Our problem setting is a variant of case-control PU learning, but it can also be formulated within the censoring PU learning framework. Although the case-control setting is slightly more general, it requires knowledge of the class prior to train the classifier. As a result, censoring PU learning algorithms are sometimes viewed as more practical. Note that our algorithm is directly applicable to the censoring PU setting\footnote{As Gang2016 discusses, most case-control PU algorithm can be applied to censoring PU learning setting when the class prior is known}.
In this study, we proposed the double PU learning approach for targeted marketing. Our focus was on identifying potential customers who are interested in a product but do not have loyalty to the company. We proposed an algorithm for training a binary classifier in this task, despite limited data availability, referred to as the double PU learning. We confirmed the soundness of the proposed algorithm via simulated studies.