Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
287,443 characters · 25 sections · 17 citation commands
Feature quantization for parsimonious and interpretable predictive models
\twocolumn[ \icmltitle{Feature quantization for parsimonious and interpretable predictive models}
\icmlaffiliation{cacf}{Cr\'edit Agricole Consumer Finance, Roubaix, France} \icmlaffiliation{inria}{Inria, Villeneuve d'Ascq, France} \icmlaffiliation{soins}{EA2694 Sant\'e publique: \'epid\'emiologie et qualit\'e des soins, Univ. Lille, Lille, France} \icmlaffiliation{painleve}{UMR 8524 Laboratoire Paul Painlev\'e, Univ. Lille, Lille, France}
\icmlcorrespondingauthor{Adrien Ehrhardt}{[email removed]} \icmlkeywords{discretization, quantization, factor, grouping, credit scoring, Machine Learning, ICML}
\vskip 0.3in ]
\printAffiliationsAndNotice
As stated by Hosmer et al. \yrcite{hosmer2013applied}, in many application contexts (credit scoring, biostatistics, {\it etc.}), logistic regression is widely used for its simplicity, decent performance and interpretability in predicting a binary outcome given predictors of different types (categorical, continuous). However, to achieve higher interpretability, continuous predictors are sometimes discretized so as to produce a “scorecard”, i.e.\ a table assigning a grade to an applicant in credit scoring (or a patient in biostatistics, {\it etc.}) depending on its predictors being in a given interval. Discretization is also an opportunity for reducing the (possibly large) modeling bias which can appear in logistic regression as a result of the linearity assumption on the continuous predictors in the model. Indeed, this restriction can be overcome by approximating the true predictive mapping with a step function where the tuning of the steps and of their sizes allows more flexibility. However, the resulting increase of the number of parameters can lead to an increase of their variance (overfitting) as shown by Yang & Webb \yrcite{yang2009discretization}. Thus, a precise tuning of the discretization procedure is required. Likewise when dealing with categorical features which take numerous levels, their respective regression coefficients suffer from this high variance phenomenon. A straightforward solution formalized by Maj-Ka\'nska et al. \yrcite{maj2015delete} is to merge their factor levels which leads to less coefficients and therefore less variance.
From now on, the generic term quantization will stand for both discretization of continuous features and level grouping of categorical ones. Its aim is to improve the prediction accuracy but it suffers from yielding a highly combinatorial optimization problem whatever the predictive criterion used to select the best quantization. The present work proposes a strategy to overcome these combinatorial issues by invoking a relaxed alternative of the initial quantization problem leading to a simpler estimation problem since it can be easily optimized by a specific neural network. This relaxed version serves as a plausible quantization provider related to the initial criterion after a classical thresholding (maximum a posteriori) procedure.
The outline of this work is the following. In the next section, we formalize both continuous and categorical quantization. Selecting the best quantization in a predictive setting is reformulated as a model selection problem on a huge discrete space. In Section (ref), a particular neural network architecture is used to optimize a relaxed version of this criterion and propose good quantization candidates. Section (ref) is dedicated to numerical experiments on both simulated and real data from the field of Credit Scoring, highlightening the good results offered by the use of this new method without any human intervention. A final section concludes the work by stating also new challenges.
The quantization procedure consists in turning a $d$-dimensional raw vector of continuous and/or categorical features $\bm{x} = (x_1, \ldots, x_d)$ into a $d$-dimensional categorical vector via a component wise mapping ${\bm{q}}=({\bm{q}}_j)_1^d$: \[{\bm{q}}(\bm{x})=({\bm{q}}_1(x_1),\ldots,{\bm{q}}_d(x_d)),\] where each of the ${\bm{q}}_j$'s is a vector of $m_j$ dummies:
where $m_j$ is an integer and the sets $C_{j,h}$ are defined with respect to each feature type as we describe just below.
If $x_j$ is a continuous component of $\bm{x}$, quantization ${\bm{q}}_j$ has to perform a discretization of $x_j$ and the $C_{j,h}$s, $1\le h\le m_j$, are contiguous intervals
where $c_{j,1},\ldots,c_{j,m_j-1}$ are increasing numbers called cutpoints, $c_{j,0}=-\infty$ and $c_{j,m_j}=+\infty$.
For example, the quantization of the unit segment in thirds would be defined as $m_j=3$, $c_{j,1} = 1/3$, $c_{j,2} = 2/3$ and subsequently ${\bm{q}}_j(0.1) = (1,0,0)$.
If $x_j$ is a categorical component of $\bm{x}$, quantization ${\bm{q}}_j$ consists in grouping levels of $x_j$ and the $C_{j,h}$s form a partition of the set, say $\{1,\ldots,l_j\}$, of levels of $x_j$:
For example, the grouping of levels encoded as “1” and “2” would yield $C_{j,1} = \{1,2\}$ such that ${\bm{q}}_j(1) = {\bm{q}}_j(2) = (1,0,\ldots,0)$.
\textcolor{black}{In both continuous and categorical cases, keep in mind that $m_j$ is the dimension of ${\bm{q}}_j$. For notational convenience, the (global) order of the quantization ${\bm{q}}$ is set as \[|{\bm{q}}|=\sum_{j=1}^d m_j.\] The space where quantizations ${\bm{q}}$ live (resp. ${\bm{q}}_j$) will be denoted by $\mathcal{Q}_{\bm{m}}$ in the sequel (resp. $\mathcal{Q}_{j,m_j}$), when the number of levels $\bm{m} = (m_j)_1^d$ is fixed. Since it is not known, the full model space is $\mathcal{Q} = \cup_{m \in \mathbb{N}_\star^{d}} \mathcal{Q}_{\bm{m}}$.}
The current practice of quantization is prior to any predictive task, thus ignoring its consequences on the final predictive ability. It consists in optimizing a heuristic criterion, often either totally unrelated (unsupervised methods) or partially related (supervised methods) to the predictive task, and mostly univariate (each feature is quantized irrespective of other features' values). The cardinality of the quantization space $\mathcal{Q}$ can be calculated explicitely w.r.t.\ $d$, $(m_j)_1^d$ and, for categorical features, $l_j$. It is huge (see a more precise illustration of this combinatorial challenge in Section (ref)), so that a greedy approach is intractable and such heuristics are needed. Many algorithms have thus been designed and a review of approximatively 200 discretization strategies, gathering both criteria and related algorithms, can be found in ramirez2016data. For factor levels grouping, we found no such taxonomy, but some discretization methods, e.g.\ $\chi^2$ independence test-based methods can be naturally extended to this type of quantization, which is for example what the CHAID algorithm, proposed by Kass \yrcite{kass1980exploratory} and applied to each categorical feature, relies on.
Quantization is a widespread preprocessing step to perform a learning task consisting in predicting, say, a binary variable $y\in\{0,1\}$, from a quantized predictor ${\bm{q}}(\bm{x})$, through, say, a parametric conditional distribution $p_{\boldsymbol{\theta}}(y|{\bm{q}}(\bm{x}))$ like logistic regression. Considering quantized data instead of raw data has a double benefit. First, the quantization order $|{\bm{q}}|$ acts as a tuning parameter for controlling the model's flexibility and thus the bias/variance trade-off of the estimate of the parameter $\boldsymbol{\theta}$ (or of its predictive accuracy) for a given dataset. This claim becomes clearer with the example of logistic regression we focus on, as a still very popular model for many practitioners. It is classically described by
where $\boldsymbol{\theta} = (\theta_{0},(\boldsymbol{\theta}_j)_1^d) \in \mathbb{R}^{|{\bm{q}}|+1}$ and $\boldsymbol{\theta}_j = (\theta_{j}^{1},\dots,\theta_{j}^{m_j})$ with $\theta_{j}^{m_j} = 0$, $j=1 \ldots d$, for identifiability reasons. Second, at the practitioner level, the previous tuning of $|{\bm{q}}|$ through each feature's quantization order $m_j$, especially when it is quite low, allows an easier interpretation of the most important predictor values involved in the predictive process. Denoting the dataset by $(\textbf{x},\textbf{y})$, with $\textbf{x}=(\bm{x}_1,\ldots,\bm{x}_n)$, $\textbf{y}=(y_1,\ldots,y_n)$ and $n$ the sample size, the log-likelihood
provides a maximum likelihood estimator $\hat{\boldsymbol{\theta}}_{\bm{q}}$ of $\boldsymbol{\theta}$ for a given quantization ${\bm{q}}$. For the rest of the paper, the approach is exemplified with logistic regression as $p_{\boldsymbol{\theta}}$ but it can be applied to any other predictive model, as will be recalled in the concluding Section ((ref)).
As dicussed in the previous section, and emphasized in the literature review, quantization is often a preprocessing step; however, quantization can be embedded directly in the predictive model. Continuing our logistic example, a standard information criterion such as the BIC BIC can be used to select the best quantization ${\bm{q}}$:
where $\nu_{\bm{q}}$ is the number of continuous parameters to be estimated in the $\boldsymbol{\theta}$-parameter space. \textcolor{black}{We shall insist here on the fact that choosing the BIC as our model selection tool is unrelated to the proposed algorithm. The practitioner can swap this criterion with any other information criterion on training data such as AIC akaike1973information or, as Credit Scoring people like, the Gini index on a test set.} Note however that, regardless of the criterion used, an exhaustive search of $\hat{{\bm{q}}}\in\mathcal{Q}$ is an intractable task due to its highly combinatorial nature. For example, with $d=10$ categorical features with $l_j = 4$ levels each, $|\mathcal{Q}|$ is given by the sum of the Stirling numbers of the second kind over $m_j=1 \dots l_j$ to the power $d$, which is approximately \ $6 \cdot 10^{11}$. Anyway, the optimization in ((ref)) requires a new specific strategy, which is the main contribution of the present work, and that we describe in the next section.
The shifting of cutpoints ((ref)) anywhere strictly between two successive raw values of a given continuous feature induce the same quantization. Thus, the identifiability of such quantizations is obtained from the dataset $\textbf{x}$ by fixing arbitrary cutpoints between successive data values, feature by feature.
In this section, we propose to relax the constraints on ${\bm{q}}_j$ to simplify the search of $\hat{{\bm{q}}}$. Indeed, the derivatives of ${\bm{q}}_j$ are zero almost everywhere and consequently a gradient descent cannot be directly applied to find an optimal quantization.
A classical approach is to replace the binary functions $q_{j,h}$ (see Equation ((ref))) by smooth parametric ones with a simplex condition, namely with $\bm{\alpha}_j=(\bm{\alpha}_{j,1},\ldots, \bm{\alpha}_{j,m_j})$:
where functions $q_{\bm{\alpha}_{j,h}}(\cdot)$, properly defined hereafter for both continuous and categorical features, represent a fuzzy quantization in that, here, each level $h$ is weighted by $q_{\bm{\alpha}_{j,h}}(\cdot)$ instead of being selected once and for all as in ((ref)). The resulting fuzzy quantization for all components depends on the global parameter $\bm{\alpha} = (\bm{\alpha}_1, \ldots, \bm{\alpha}_d)$ and is denoted by ${\bm{q}}_{\bm{\alpha}}(\boldsymbol{x})=\left({\bm{q}}_{\bm{\alpha}_j}(x_j)\right)_{j=1}^d \in \widetilde{Q}$. This approximation will be justified in Section (ref).
{\bf For continuous features}, we set for $\bm{\alpha}_{j,h} = (\alpha^0_{j,h},\alpha^1_{j,h}) \in \mathbb{R}^2$ \[q_{\bm{\alpha}_{j,h}}(\cdot) = \frac{\exp(\alpha^0_{j,h} + \alpha^1_{j,h} \cdot)}{\sum_{g=1}^{m_j} \exp(\alpha^0_{j,g} + \alpha^1_{j,g} \cdot)}.\]
{\bf For categorical features}, we set for $\bm{\alpha}_{j,h}=\left(\alpha_{j,h}(1),\ldots, \alpha_{j,h}(l_j)\right) \in \mathbb{R}^{l_j}$ \[q_{\bm{\alpha}_{j,h}}(\cdot) = \frac{\exp\left(\alpha_{j,h}(\cdot)\right)}{\sum_{g=1}^{m_j} \exp\left(\alpha_{j,g}(\cdot)\right)}.\]
With this new fuzzy quantization, the logistic regression for the predictive task is then expressed as
where ${\bm{q}}$ has been replaced by ${\bm{q}}_{\bm{\alpha}}$ from Equation ((ref)). Note that as ${\bm{q}}_{\bm{\alpha}}$ is a sound approximation of ${\bm{q}}$ (see Section (ref)), this logistic regression in ${\bm{q}}_{\bm{\alpha}}$ is consequently a good approximation of the logistic regression in ${\bm{q}}$ from Equation ((ref)). The relevant log-likelihood is here
and can be used as a tractable substitute for ((ref)) to solve the original optimization problem ((ref)), where now both $\bm{\alpha}$ and $\boldsymbol{\theta}$ have to be estimated, which is discussed in the next section. We wish to maximize the log-likelihood ((ref)) which would yield parameters $(\hat{\bm{\alpha}},\hat{\boldsymbol{\theta}})$; To “push” $\widetilde{\mathcal{Q}}$ further into $\mathcal{Q}$, we deduce ${{\bm{q}}}^{\text{MAP}}$ from a maximum a posteriori procedure applied to ${\bm{q}}_{\hat{\bm{\alpha}}}$:
If there are two levels $h$ that satisfy ((ref)), we simply take the level that corresponds to smaller values of $x_j$ to be in accordance with the definition of $C_{j,h}$ in Equation ((ref)). This {\it maximum a posteriori} principle are exemplified in Figure (ref) on simulated data by the plain vertical lines (see Section (ref)).
From a deterministic point of view, we have $\mathcal{Q} \subset \widetilde{\mathcal{Q}}$: First, the maximum a posteriori step ((ref)) produces contiguous intervals (i.e.\ there exists $C_{j,h}$; $1 \leq j \leq d$, $1 \leq h \leq m_j$, s.t.\ ${{\bm{q}}}^{\text{MAP}}$ can be written as in (ref)) same2011model. Second, in the continuous case, the higher $\alpha_{j,h}^1$, the less smooth the transition from one quantization $h$ to its “neighbor”\footnotemark[1] $h+1$, whereas $\dfrac{\alpha_{j,h}^0}{\alpha_{j,h}^1}$ controls the point in $\mathbb{R}$ where the transition occurs chamroukhi2009regression. Concerning the categorical case, the rationale is even simpler as $q_{\lambda \alpha_{j,h}}(x_j) \to 1 \text{ if } h = \operatorname*{arg\!max}_{h'} q_{\alpha_{j,h'}}(x_j), 0 \text{ otherwise}$ as $\lambda \to +\infty$ reverdy2016parameter.
From a statistical point of view, under standard regularity conditions and with a suitable estimation procedure (see later for the proposed estimation procedure), the maximum likelihood framework ensures the consistency of $({\bm{q}}_{\hat{\bm{\alpha}}}, \hat{\boldsymbol{\theta}})$ towards $({\bm{q}},\boldsymbol{\theta})$. This is further ensured by the maximum a posteriori step ((ref)).
However, and as is usual, the log-likelihood $\ell_{{\bm{q}}_{\bm{\alpha}}}(\boldsymbol{\theta},(\textbf{x},\textbf{y}))$ cannot be directly maximized w.r.t.\ $(\bm{\alpha},\boldsymbol{\theta})$, so that we need an iterative procedure. To this end, the next section introduces a neural network of particular architecture.
From an empirical point of view, we will see in Section (ref) and in particular in Figure (ref), that the smooth approximation ${\bm{q}}_{\bm{\alpha}}$ converges towards “hard” quantizations\footnotemark[1] ${\bm{q}}$.
\footnotetext[1]{Up to a permutation on the labels $h=1 \ldots m_j$ to recover the ordering in $C_{j,h}$ (see Equation ((ref))).}
To estimate parameters $\bm{\alpha}$ and $\boldsymbol{\theta}$ in model ((ref)), a particular neural network architecture can be used. \textcolor{black}{We shall insist that this network is only a way to use common deep learning frameworks, namely Tensorflow tensorflow2015-whitepaper through the high-level API Keras chollet2015keras instead of building a gradient ascent algorithm from scratch to optimize (ref).} The most obvious part is the output layer that must produce $p_{\boldsymbol{\theta}}(1|{\bm{q}}_{\bm{\alpha}}(\bm{x}))$ which is equivalent to a densely connected layer with a sigmoid activation $\sigma (\cdot)$.
For a continuous feature $x_j$ of $\bm{x}$, the combined use of $m_j$ neurons including affine transformations and softmax activation obviously yields ${\bm{q}}_{\bm{\alpha}_{j}}(x_j)$. Similarly, an input categorical feature $x_j$ with $l_j$ levels is equivalent to $l_j$ binary input neurons (presence or absence of the factor level). These $l_j$ neurons are densely connected to $m_j$ neurons without any bias term and a softmax activation. The softmax outputs are next aggregated via the summation in model ((ref)), say $\Sigma_{\boldsymbol{\theta}}$ for short, and then the sigmoid function $\sigma$ gives the final output. All in all, the proposed model is straightforward to optimize with a simple neural network, as shown in Figure (ref).
By relying on stochastic gradient ascent, the smoothed likelihood ((ref)) can be maximized over $\left(\bm{\alpha}, \boldsymbol{\theta} \right)$. Due to its convergence properties bottou2010large, the results should be close to the maximizers of the original likelihood ((ref)) if the model is well-specified, when there is a true underlying quantization. However, in the mis-specified model case, there is no such guarantee. Therefore, to be more conservative, we evaluate at each training epoch $(t)$ the quantization ${{\bm{q}}}^{\text{MAP}(t)}$ resulting from the maximum a posteriori procedure explicited in Equation ((ref)), then classically estimate the logistic regression parameter via maximum likelihood, as done in Equation ((ref)): \[\hat{\boldsymbol{\theta}}^{(t)} = \operatorname*{arg\!max}_{\boldsymbol{\theta}} \ell_{{{\bm{q}}}^{\text{MAP}(t)}}(\boldsymbol{\theta}; (\textbf{x},\textbf{y}))\] and the resulting $\mbox{BIC}(\hat{\boldsymbol{\theta}}^{(t)})$ as in ((ref)). If $T$ is a given maximum number of iterations of the stochastic gradient ascent algorithm, the quantization retained at the end is then determined by the optimal epoch
\textcolor{black}{The number of iterations $T$ can be seen as a computational budget: contrary to classical early stopping rules (e.g.\ based on validation loss) used in neural network fitting, this network only acts as a stochastic quantization provider for (ref) which will naturally prevent overfitting. We reiterate that, in (ref), the BIC can be swapped for the user's favourite model choice criterion. Lots of optimization algorithms for neural networks have been proposed, which all come with their hyperparameters. We chose the “RMSProp” method, which showed good results, is one of the standard methods, and tuned only its learning rate.}
The number of intervals or factor levels $\boldsymbol{m} = (m_j)_1^d$ were supposed up to now known but in practice also have to be estimated. In fact, they play an overriding role in the bias-variance “tuning” effect which motivated this work in Section (ref). By relying on the maximum a posteriori procedure developed in Equation ((ref)) \textcolor{black}{parallel to the neural network candidate generator}, we might drop a lot of unseen factor levels, e.g.\ if $q_{\bm{\alpha}_{j,h}}(x_{i,j}) \ll 1$ for all training observations $x_{i,j}$, the level $h$ “vanishes”, i.e.\ $\hat{q}_{j,h} = 0$. Thus, it is not necessary to go through such a loop and in practice, we recommend to start with a user-chosen $\bm{m}=\boldsymbol{m}_{\max}$ and we will see in the experiments of Section (ref) that the proposed approach is able to explore small values of $\boldsymbol{m}$ and to select a value $\hat{\boldsymbol{m}}$ drastically smaller than $\boldsymbol{m}_{\max}$. This phenomenon, which reduces the computational burden of the quantization task, is also illustrated in the next section. \textcolor{black}{The hyper-parameter $\boldsymbol{m}_{\max}$ is problem-dependent and should be adjusted by the practitioner to meet his/her interpretability requirements.}
This section is divided into three complementary parts to assess the validity of our proposal, that we call hereafter glmdisc. First, simulated data are used to evaluate its ability to recover the true data generating mechanism. Second, the predictive quality of the new learned representation approach is illustrated on several classical benchmark datasets from the UCI library. Third, we use it on Credit Scoring datasets provided by Cr\'edit Agricole Consumer Finance (CACF), a major European company in the consumer credit market. The Python notebooks of all experiments, excluding the confidential real data of CACF, \textcolor{black}{are available online\footnote{\url{https://adimajo.github.io}}}.
We focus here on discretization of continuous features (similar experiments could be conducted on categorical ones). Two continuous features $x_1$ and $x_2$ are sampled from the uniform distribution on $[0,1]$ and discretized using \[{\bm{q}}_1(\cdot)={\bm{q}}_2(\cdot) = (\mathds{1}_{(-\infty,1/3]}(\cdot),\mathds{1}_{(1/3,2/3]}(\cdot),\mathds{1}_{(2/3,\infty]}(\cdot)).\] Here, following ((ref)), we have $d=2$ and $m_1=m_2=3$ and the cutpoints are $c_{j,1}=1/3$ and $c_{j,2}=2/3$ for $j=1,2$. Setting $\boldsymbol{\theta}=(0,-2,2,0,-2,2,0)$, the target feature $y$ is then sampled from $p_{\boldsymbol{\theta}}(\cdot | {\bm{q}}(\bm{x}))$ via the logistic model ((ref)).
From the glmdisc algorithm, we studied three cases:
From a statistical point of view, experiment (A) assesses the empirical consistency of the estimation of $C_{j,h}$ motivated in Section (ref), whereas experiments (B) and (C) focus on the consistency of the estimation of $m_j$ motivated in Section (ref). The results are summarized in Table (ref) where either 95% confidence intervals \textcolor{black}{(sun2014fast, hereafter CI)} or bar plots are given, with a varying sample size. Two iterations of experiment (A) are displayed on Figure (ref): at first (Figure (ref)), the proposed neural network fails to recover the true underlying discretization but after 300 iterations (Figure (ref)), the “smooth” discretization ${\bm{q}}_{\bm{\alpha}}$ and its maximum a posteriori $\hat{{\bm{q}}}$ get closer to the data generating mechanism, resulting in a very good estimation of $c_{j,2}$ (Column (A) of Table (ref)). Also note that the slight underestimation ($\hat{m}_1 = 2$ for 9 experiments out of 100) in (B) for $n=1{,}000$ is a classical consequence of the BIC criterion on small samples. As for (C) and as expected, spurious correlations with a small sample allow $x_3$ to enter the model with either $\hat{m}_3=2$ intervals (32 experiments out of 100) or $\hat{m}_3=3$ intervals (8 experiments out of 100). However, with a larger sample, feature $x_3$ is rightfully omitted from the final model, i.e.\ with $\hat{m}_3=1$ interval (88 experiments out of 100).
\newlength\figureheight \newlength\figurewidth \setlength\figureheight{4cm} \setlength\figurewidth{16cm}
To test further the effectiveness of glmdisc in a predictive setting, we gathered 6 datasets from the UCI library: the Adult dataset ($n=48{,}842$, $d=14$), the Australian dataset ($n=690$, $d=14$), the Bands dataset ($n=512$, $d=39$), the Credit-screening dataset ($n=690$, $d=15$), the German dataset ($n=1{,}000$, $d=20$) and the Heart dataset ($n=270$, $d=13$). Each of these datasets has mixed (continuous and categorical) features and a binary response to predict. To get more information about these datasets, their respective features, and the predictive task associated with them, the interested reader may refer to the UCI website\footnote{Dua:2017 : http://archive.ics.uci.edu/ml}.
Now that we made sure that our approach is empirically consistent, i.e.\ it is able to find the true quantization in a well-specified setting, we wish to verify now that embedding the learning of a good quantization in the predictive task via glmdisc is better than other methods that rely on ad hoc criteria. As we were primarily interested in logistic regression, we will compare our approach to a “na\"{\i}ve” additive linear logistic regression (\textcolor{black}{on non-quantized features} - hereafter ALLR), a logistic regression on continuous discretized data using the now standard MDLP algorithm from fayyad1993multi and categorical grouped data using $\chi^2$ tests of independence between each pair of factor levels and the target in the same fashion as the ChiMerge discretization algorithm proposed by Kerber \yrcite{kerber1992chimerge} (hereafter MDLP/$\chi^2$). As the original use case stems from Credit Scoring, we use the performance metric usually monitored by Credit Scoring practitioners, which is the Gini coefficient, directly related to the Area Under the ROC Curve (Gini $= 2 \times \text{AUC} -1$). \textcolor{black}{In this Section and the next, Gini indices are reported on a random 30 % test set.} Table (ref) shows our approach yields significantly better results on these rather small datasets where the added flexibility of quantization might help the predictive task.
Discretization and grouping are preprocessing steps relatively “manually” performed in the field of Credit Scoring, using $\chi^2$ tests for each feature or so-called Weights of Evidence zeng2014necessary. This back and forth process takes a lot of time and effort and provides no particular statistical guarantee.
Table (ref) shows Gini coefficients of several portfolios for which there are $n=50{,}000$, $n=30{,}000$, $n=50{,}000$, $n=100{,}000$, $n=235{,}000$ and $n=7{,}500$ clients respectively and $d=25$, $d=16$, $d=15$, $d=14$, $d=14$ and $d=16$ features respectively. Approximately half of these features were categorical, with a number of factor levels ranging from $2$ to $100$.
We compare the rather manual, in-house approach that yields the current performance, the na\"{\i}ve additive linear logistic regression (ALLR) and ad hoc methods (MDLP/$\chi^2$) introduced in the previous section to our glmdisc proposal. Beside the classification performance, interpretability is maintained and unsurprisingly, the learned representation comes often close to the “manual” approach: for example, the complicated in-house coding of job types is roughly grouped by glmdisc into e.g.\ “worker”, “technician”, etc. Our approach shows approximately similar results than MDLP/$\chi^2$, potentially due to the fact that contrary to the two previous experiments with simulated or UCI data, the classes are imbalanced ($< 3 \%$ defaulting loans), which would require special treatment while back-propagating the gradients anand1993improved. Note however that it is never significantly worse; for the Electronics dataset and as was the case for most UCI datasets, glmdisc is significantly superior, which in the \textit{Credit Scoring} business might end up saving millions to the financial institution.
Regarding complexity, there are at most $\mathcal{O}(m_j^2)$ $\chi^2$ tests performed in all benchmarks for categorical features as initially, all pairwise tests have to be computed. The MDLP algorithm has to first sort the training samples ($\mathcal{O}(n \ln{n})$ operations) and then recursively assess the entropy produced by cutting at each “boundary point”, i.e. where consecutive training points, say $x_{i,j},x_{i',j}$, have different targets ($y_i \neq y_{i'}$). There are $\mathcal{O}(b_j^2)$ such operations where $b_j$ is the number of these “boundary points” ramirez2016data. Our approach, the glmdisc algorithm, requires that we fit a softmax with $m_j$ output classes per feature and training epoch $(t)$ \textcolor{black}{which is quite low. About the length of the gradient ascent chain, there is no stopping rule except the time budget T. However, the required T value to obtain relevant candidates is low: approx.\ 20-40 iterations for the experiments of Section (ref). Figure (ref) uses a small learning rate to showcase both the empirical consistency of the relax and the effect of the MAP scheme in exploring a lower number of quantization levels $m_j$.} On Google Collaboratory, and relying on Keras chollet2015keras and Tensorflow tensorflow2015-whitepaper as a backend, it took less than an hour to perform discretization and grouping for each dataset of Table (ref), making it in this regard also comparable to MDLP/$\chi^2$ methods.
Feature quantization (discretization for continuous features, grouping of factor levels for categorical ones) in a supervised multivariate classification setting is a recurring problem in many industrial contexts. It was formalized as a highly combinatorial representation learning problem and a new algorithmic approach, named glmdisc, has been proposed as a sensible approximation of a classical statistical information criterion.
This algorithm relies on the use of a softmax approximation of each discretized or grouped feature. This proposal can alternatively be replaced by any other univariate multiclass predictive model, which makes it flexible and adaptable to other problems. Prediction of the target feature, given quantized features, was exemplified with logistic regression, although here as well, it can be swapped with any other supervised classification model, provided it is the same as the output layer of the proposed neural network. \textcolor{black}{Thus, the extension to penalized logistic regression or any Generalized Linear Model is straightforward.} Its good computational properties were put to use while maintaining the interpretability necessary to some fields of application.
The experiments showed that, as was sensed empirically by statisticians in the field of Credit Scoring, discretization and grouping can indeed provide better models than standard logistic regression. This novel approach allows practitioners to have a fully automated and statistically well-grounded tool that achieves better performance than ad hoc industrial practices at the price of decent computing time but much less of the practitioner's valuable time. As a rule of thumb, a month is generally allocated to data pre-processing for a single data scientist working on a single scorecard that can now be invested in tasks that add more value, e.g.\ more data, better data quality.
As described in the introduction, logistic regression is additive in its inputs which does not allow to take into account conditional dependency, as stated by Berry et al.\ \yrcite{berry2010testing}. This problem is often dealt with by sparsely introducing “interactions”, i.e.\ products of two (pairwise interactions) or more features. This leads again to a model selection challenge on a highly combinatorial discrete space that could be solved with a similar approach. In a broader context with no restriction on the predictive model, Tsang et al.\ \yrcite{tsang2018detecting} already made use of neural networks to estimate the presence or absence of statistical interactions. The parsimonious addition of pairwise interactions among quantized features, that might influence the quantization process introduced in this work, is a future area of research.