Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
85,566 characters · 16 sections · 38 citation commands
Measuring the Driving Forces of Predictive Performance: Application to Credit Scoring
Keywords: Explainability; Credit scoring; Performance metrics; Shapley values
\baselineskip=2\normalbaselineskip
Why is the AUC of a given machine learning model equal to 0.7? Which features mainly explain this performance? What are the contributions of the different features to the MSE of a regression model? To answer these questions, we develop and apply a general methodology, called eXplainable PERformance (XPER), which measures the marginal contribution of a particular feature to the predictive performance of a regression or classification model.
Being able to identify the driving forces of the performance of a predictive model is crucial in the context of credit scoring, where complex and opaque machine learning models are increasingly being used by banks and fintechs. Such decomposition is of primary importance both for banking supervisors and internal model validation teams as they need to understand why a given model is working or not, and for which types of borrowers. Furthermore, it also permits to address heterogeneity issues by identifying groups of individuals for which the features have similar effects on performance. One can then estimate group-specific models to improve overall performance and make sure the credit scoring model operates effectively for all borrowers. In this paper, we propose an application where we use XPER to decompose the AUC of a scoring model forecasting loan defaults and to identify the features with the strongest impact on model performance.
The XPER framework is based on Shapley values Shapley_1953. While Shapley values decompose a payoff among players in a game, XPER values decompose a performance metric among features in a model. More precisely, our method breaks down the difference between a performance metric and a benchmark value among the various features of the model. Formally, an XPER value is defined as the weighted average marginal contribution of a given feature to a performance metric, obtained in a set of coalitions of other features. For instance, evaluating the XPER value of $x_{1}$ in a three-feature model implies to assess the incremental performance due to $x_{1}$ by successively considering four subsets or coalitions of features: one coalition including no features, another coalition only $x_{2}$, another one only $x_{3}$, and a last coalition including both $x_{2}$ and $x_{3}$.
The application of the Shapley methodology in the context of explaining model performance is not trivial. Indeed, many Shapley values can be defined given the assumptions made on the model, the data, and the features excluded from the coalitions (see for instance strumbelj_efficient_2010, Redell2019, Kumar_2020, Sundararajan2020, and Aas2021). In this paper, we define XPER values by considering the expectation of the performance metric with respect to the joint distribution of the target variable and of the features which are excluded from the coalition. A key advantage of this definition is that the benchmark value has a meaningful interpretation: it corresponds to the performance metric that we would obtain on a hypothetical sample in which the target variable is independent from all the features included in the model, i.e., a fully misspecified model with irrelevant features. Moreover, our method does not require to re-estimate the model (\`a la gromping_estimators_2007) nor to pick ad-hoc values for features excluded from the coalition (\`a la Israeli2007).
XPER offers several other advantages. First, as the XPER decomposition is based on Shapley values, it is theoretically grounded and XPER values satisfy the axioms of efficiency, symmetry, linearity, and null effects. These axioms collectively ensure equitable and consistent allocation of model performance contributions among the involved features.\footnote{Efficiency dictates that the total contribution of all features equals the overall model performance metric, minus the benchmark value. Symmetry ensures that features making equivalent contributions receive identical XPER values. Linearity stipulates that if model performance can be decomposed into separate parts, the XPER value of the whole model equals the sum of its parts. The null effect axiom ensures features that do not influence model performance receive a XPER value of zero.} Second, XPER is model-agnostic as it permits to interpret the predictive performance of any econometric or machine learning model. Third, it is metric-agnostic as it can break down any performance metric: predictive accuracy (e.g., AUC, Gini, accuracy), goodness of fit (e.g., $R^2$), information criterion (e.g., AIC, BIC), loss function (e.g., MSE, MAE, Q-like), or economic performance (e.g., profit-and-loss or P&L). Fourth, XPER can be implemented either at the global level or at the local level. At the global level, the XPER value of a given feature measures its contribution to the performance of the model. At the local level, the XPER value of a given feature measures its contribution to the model performance that comes from a given individual. Finally, as the number of coalitions grows fast with the number of features, we propose two estimation procedures: an exact one when the number of features remains moderate and an approximated one, which is a modified version of the Kernel SHAP method of Lundberg2017. We developed a package in Python to compute the XPER values and made it available at \href{https://github.com/hi-paris/XPER}{https://github.com/hi-paris/XPER}.
We apply our methodology to a novel dataset of auto loans provided by an international bank. We demonstrate the usefulness of XPER for decomposing the AUC and other performance metrics of a credit scoring model. Our findings reveal that a small number of features can explain a surprisingly large part of the model performance. Furthermore, the empirical analysis confirms that XPER values differ from standard feature importance metrics. More importantly, we show how to use XPER to boost model performance. To do so, we build homogeneous groups of individuals by clustering them based on their individual XPER values. By construction, within a given group, the features tend to have similar effects on performance (same sign, same intensity). We then show that estimating group-specific models yields to a higher accuracy than with a one-fits-all model.
Our paper contributes to the burgeoning literature on eXplainable Artificial Intelligence (XAI) Murdoch2019, Gelman2021, Molnar2024. One well-known limitation of AI and machine learning methods comes from their opacity and lack of explainability. Most of these algorithms are considered as black boxes in the sense that the corresponding outcomes cannot be easily explained to final users nor related to the initial features Sun2021. Recently, many explainability methods have been designed to explain black-box models by measuring which features most affect its predictions (see Zhao2021,Horel2022,Liu2023,Molnar2024). One of the most impactful XAI methods is the Shapley additive explanation (SHAP) of Lundberg2017, which distributes the predictions of a model among its features (see also Sundararajan2017, Lundberg2018, SundararajanDhamdereAgarwal2020, Casalicchio_2019, Bowen2020, Aas2021). However, explaining the predictive performance of a model (XPER) is not equivalent to explaining its forecasts (SHAP). The latter approach identifies the features contributing the most to model predictions both when the model is correct and when the model is wrong. This implies that some features identified as contributors that lead the predictions may, in fact, mislead the model.
While several other studies also decompose model performance using Shapley values (see Appendix (ref) for a comparison table of the main studies), XPER distinguishes itself in three ways. First, XPER stands out for its versatility, enabling the analysis of the drivers of any performance metric for any estimated model. In the literature, most articles focus exclusively on the decomposition of a single performance metric owen_shapley_2017,Redell2019,sutera2021global,Verdinelli2023. Second, XPER sets itself apart from most existing methodologies, which only focus on providing a global analysis of the features influencing model performance Casalicchio_2019,ghorbani2019data,covert2020understanding,fryer2021shapley,moehle2021portfolio,Borup2022,Zhang2023, by also enabling a local analysis. This allows us to address heterogeneity issues by identifying groups of individuals for which features exhibit similar effects on model performance. Third, XPER meaningfully decomposes the performance metric without relying on strong assumptions or re-estimating the model. In particular, many alternative approaches require re-estimating the model with different subsets of features, which may lead to omitted variable bias Lipovetsky2001,Israeli2007,bell2024efficient.
In this section, we introduce XPER using a simple classification problem with $y\in \{0,1\}$ the target variable and three features $\{x_1, x_2, x_3\}$. We estimate a model $\hat{f}(x_1,x_2,x_3)$ on a training sample and evaluate its predictive performance by considering its AUC on a test sample. If none of the three features provides any useful information to predict the target variable, the AUC is 0.5, indicating random classification. We refer to this value as the benchmark AUC, denoted $\phi_0$. In contrast, when AUC $>0.5$, at least one feature contains some relevant information to predict the target variable.
What is the relative contribution of each feature to this predictive performance? We answer this question using XPER by breaking down the difference between the AUC obtained on the test sample and the benchmark value, among the features of the model. Let us denote by $\phi_j$ the XPER contribution of feature $x_j$ to the predictive performance of the model. For instance, the AUC of the model (e.g., predicting/detecting loan default, customer churn, fraud, money laundering) is 0.78 and its XPER decomposition is:
It shows that the feature $x_1$ is the main driver of the predictive performance of the model as it explains half of the difference between the AUC of the model and its benchmark ($=0.14/(0.78-0.50)$). The second most informative feature is $x_2$ as it explains another 35.7% ($=0.10/(0.78-0.50)$) of the difference. Finally, $x_3$ explains the remaining 14.3%.
Such performance decomposition can prove handy in several contexts. XPER can help to highlight a potential heterogeneity in the predictive performance of a model. Indeed, one can compute the XPER decomposition of a given performance measure on a subset of the test sample. For instance, let us consider two subsets of the test sample, denoted $A$ and $B$. The performance of the model can be evaluated and decomposed with XPER independently in each subset. The illustrative results displayed in the table below indicate that the model performs poorly in $A$. In this case, XPER values can be used to explain the sources of such underperformance of the model. For instance, if the XPER decomposition of the AUC over the subsets $A$ and $B$ is as follows, the weak performance observed in A is only due to feature $x_1$.
XPER can also help to understand the origin of overfitting. Overfitting occurs when a model performs significantly better on the training sample than it does on the test sample. XPER can identify some features which contribute more to the performance of the model in the training sample than in the test sample, and thus explains overfitting. In the table below, the drop in AUC between the training sample (0.90) and the test sample (0.78) illustrates a typical overfitting issue. In this case, the overfitting problem is entirely due to $x_2$.
A related application of XPER could prove useful during the variable selection phase of model development. Indeed, by contrasting XPER values estimated in the training and validation sets, one could only keep the variables maintaining a high-enough level of importance in the validation test.
XPER can be applied to any statistical performance metrics (e.g., $R^2$, Brier Score, Gini index), but also to any economic performance metrics (e.g., P&L, return-on-investment, performance of a financial portfolio). As an example, we can decompose the P&L of a credit scoring model:
where $profit$ is the money made on any reimbursed loan ($y_i=0$) and $loss$ is the money lost on any defaulted loan ($y_i=1$). In this case, XPER values would be expressed in dollar terms, and would sum to the total P&L.
We consider a classification or regression problem involving a target variable, denoted $y$, taking values in $\mathcal{Y}=\{0,1\}$ or $\mathcal{Y} \subseteq \mathbb{R}$ given the problem considered. The $q$-vector $\mathbf{x}\in \mathcal{X} \subseteq \mathbb{R}^q$ refers to the input features. We denote by $f:\mathbf{x}\rightarrow \hat{y}$ an econometric model or a machine learning algorithm, where $\hat{y}\in \mathcal{Y}$ is either a classification output or a regression output, such as $\hat{y}=f(\mathbf{x})$. We impose no constraint on the model: it may be parametric or not, linear or not, a weak learner or an ensemble method, etc. For simplicity, for parametric models we exclude the parameters from the notation, i.e., $f(\mathbf{x})\equiv f(\mathbf{x};\theta)$. The model is estimated (parametric model) or trained (machine learning algorithm) once for all on a training sample $ S_T=\{\mathbf{x}_j,y_j\}^T_{j=1}$. The sample size $T$ is considered as fixed. The trained model, denoted $\hat{f}(.)$, is evaluated on a test sample $S_n=\{\mathbf{x}_i,y_i,\hat{f}(\mathbf{x}_i)\}^n_{i=1}$ with a performance metric (PM). A PM is defined as a measure used to assess the predictive performance of a model. It quantifies the accuracy and effectiveness of a model’s predictions. Standard PM for classification models include accuracy, precision, recall, F1-score, and AUC, among others. Similarly, for regression models, commonly used PM include MSE, RMSE, MAE, and $R^2$. More generally, any information criteria, loss function, or economic performance indicator can be considered as PM. See Table (ref) in Appendix (ref) for examples of PM.
For instance, for a linear regression model $\hat{f}(\mathbf{x})=\hat{\beta}_0+\hat{\beta}_1x_1$ and the PM defined as the opposite of the MSE, we have $ G_n(\mathbf{y};\mathbf{X})=-\frac{1}{n}\sum_{i=1}^n(y_i-\hat{\beta}_0-\hat{\beta}_1x_{i,1})^2$. We introduce the following three assumptions on the PM.
Assumption 1 simplifies the interpretation of the PM by asserting that higher values indicate more accurate model predictions. For example, since a higher $R^2$ suggests smaller differences between model predictions and target values, $R^2$ can be directly used as a PM. Conversely, if we want to decompose the MSE, we recommend considering the opposite of the MSE and defining PM = $-$MSE. Hence, higher PM values signify smaller prediction-target differences. In this case, a positive XPER value $\phi_j$ always indicates that the variable $x_j$ enhances the model predictive performance.
For simplicity, we only consider models for which the outcome $y_{i}$ for instance $i$ only depends on features $\mathbf{x}_{i}=(x_{i,1},...,x_{i,q})^{'}$. For regression or classification models with cross-sectional interactions (e.g., spatial econometrics model) or time-series dependence, notations have to be adjusted such that $\hat{y}_{i}=\hat{f}\left( \mathbf{w}_{i}\right)$, where $\mathbf{x}_{i}\subseteq \mathbf{w}_{i}$, $\exists j\neq i:\mathbf{x}_{j}\subseteq \mathbf{w}_{i}$ and/or $y_{j}\subseteq \mathbf{w}_{i}$. Then, the additive assumption becomes $G_n(\mathbf{y};\mathbf{X})=\frac{1}{n}\sum_{i=1}^{n}G\left( y_{i};\mathbf{ w}_{i};\hat{\delta}_n\right)$.
These three assumptions are consistent with a wide range of PMs. In Appendix (ref), we detail $G\left( y;\mathbf{x};\delta _{0}\right)$ for standard PMs associated with regression or classification models. For instance, in the case of a linear regression model $\hat{f}(\mathbf{x})=\mathbf{x}\hat{\beta}$, when the $R^2$ is used as PM, we have:
with $G(y_i;\mathbf{x}_i;\hat{\delta}_n)=1-\frac{1}{\hat{\delta}_n}(y_i-\mathbf{x}_i\hat{\beta})^2$ and $\hat{\delta}_n=\frac{1}{n}\sum_{j=1}^{n}(y_j-\bar{y})^2$. The corresponding population $R^2$ is defined as $\mathbb{E}_{y,\mathbf{x}}(G(y;\mathbf{x};\delta_0))=1-\frac{1}{\sigma_y^2}\mathbb{E}_{y,\mathbf{x}}\left({(y-\mathbf{x}\hat{\beta})^2}\right)$, with $G(y;\mathbf{x};\delta_0)=1-\frac{1}{\hat{\delta}_0}(y-\mathbf{x}\hat{\beta})^2$ and $\delta_0=\sigma_y^2$ the variance of the target variable.
This set of assumptions imposes minimal constraints. In particular, Assumption (ref) is satisfied under three key conditions related to the performance metric, the sampling assumptions, and the consistency of the estimator $\hat{\delta}_n$. First, regarding the performance metric, the function $G(\cdot)$ needs to be integrable with respect to the joint distribution of $y$ and $\mathbf{x}$. This ensures that $\mathbb{E}_{y, \mathbf{x}}(G(y; \mathbf{x}; \delta_0))$ exists and is finite. Second, depending on the type of data (cross-sectional or time series), specific assumptions about the joint distribution of $y$ and $\mathbf{x}$ are necessary. For cross-sectional data, we assume that the elements $G(y_i; \mathbf{x}_i; \hat{\delta}_n)$ are independently distributed or weakly dependent. This assumption is typically mild since $G(y_i; \mathbf{x}_i; \hat{\delta}_n)$ generally represents a function of the error term. For example, for the (opposite) MSE, we have $G(y_i; \mathbf{x}_i; \hat{\delta}_n) = -(y_i - \hat{f}(\mathbf{x}_i))^2$. This ensures the conditions for applying the weak law of large numbers when $n$ is large. For time series data, we assume that the elements $G(y_i; \mathbf{x}_i; \hat{\delta}_n)$ are stationary and weakly dependent. Finally, the estimator $\hat{\delta}_n$ should be consistent for $\delta_0$. This means $\hat{\delta}_n$ converges in probability to $\delta_0$ as the sample size $n$ increases.
Our objective is to identify the contribution of the model's features to its predictive performance, as evaluated by a PM on a given sample. We measure this contribution through Shapley values Shapley_1953, a method used in game theory to fairly distribute a payoff $Val(x_1,...,x_q)$ among several players $x_1,...,x_q$. The Shapley value $\phi_j$ measures the marginal impact of a player $x_j$ on the payoff by assessing the changes in $Val(x_1,...,x_q)$ when this player is added or not to a coalition of players already in the game. Denote by $\mathbf{x}^S$ the vector of players included in a coalition $S$ and $\mathbf{x}^{\overline{S}}$ the vector of excluded players, such that $ \{\mathbf{x}\} = \{\mathbf{x}^{S}\} \cup \{\mathbf{x}^{\overline{S}}\} \cup \{x_j\}$ and $\mathbf{x}=(x_1,...,x_q)$. The Shapley value is then defined as the weighted average of the marginal contributions associated with all possible coalitions $S$.
By analogy, we decompose the performance metric $\mathbb{E}_{y,\mathbf{x}}(G(y;\mathbf{x};\delta_0))$ (the “payoff”) among the features $x_1,...,x_q$ (the “players”) of the model $\hat{f}(\mathbf{x})$ (the “game”). The main difference with the previous notations is that the payoff $Val(\mathbf{x};y)=\mathbb{E}_{y,\mathbf{x}}(G(y;\mathbf{x};\delta_0))$ depends on the joint distribution of the features $x_1,...,x_q$ and the target variable $y$. Formally, we define XPER as the Shapley value associated with the performance metric $\mathbb{E}_{y,\mathbf{x}}(G(y;\mathbf{x};\delta_0))$ and to the model $\hat{f}(\mathbf{x})$.
The XPER value $\phi_j$ measures the weighted average marginal contribution of the feature $x_j$ to the PM over all features coalitions. For each coalition, the marginal contribution of $x_j$ is defined as the difference between the expected values of (1) the PM obtained while including this feature in the coalition and (2) the PM obtained while excluding this feature from the coalition. In Definition (ref), the term $\mathbb{E}_{\mathbf{x}^{\overline{S}}}$ refers to the marginalization with respect to the joint distribution of the features which are excluded from the coalition $S$.\footnote{An alternative approach would be to compute the expectation over the conditional distribution of the excluded features, $\mathbf{x}^{\overline{S}}$, given the joint distribution of the target variable $y$, the feature of interest $x_j$, and the features included in the coalition $\mathbf{x}^{S}$, i.e., $\mathbb{E}_{\mathbf{x}^{\overline{S}}|y,x_{j},\mathbf{x}^{S}}$. However, the corresponding decomposition would no longer satisfy the four axioms mentioned in Section (ref). See the discussion about the Conditional Expectation Shapley (CES) methods in Sundararajan2020.} The second expectation $\mathbb{E}_{y,x_j,\mathbf{x}^{S}}\left(G\left(y;\mathbf{x};\delta_0\right)\right)$ refers to an averaging effect, i.e., expectation with respect to the joint distribution of the features $\mathbf{x}^{S}$ included in the coalition with $x_j$, and the target variable $y$.
As an illustration, we consider a three-feature model $\hat{f}(x_1,x_2,x_3)$ and pick $x_1$ as the feature of interest. In Table (ref), we report all the coalitions among the set $\{x_2,x_3\}$ (column 1), the associated weights computed according to Equation (ref) (column 2), and the marginal contributions (column 3) used to compute the XPER value $\phi_1$. For instance, when considering a coalition with only $x_1$, we first compute the expectation of the PM with respect to the joint distribution of excluded features $x_2$ and $x_3$, i.e., $\mathbb{E}_{x_2,x_3}(\tilde{G}(y,\hat{f}(x_1,x_2,x_3)))$. This corresponds to a marginalization of the PM with respect to the features which are excluded from the coalition. Then, we consider the expectation of the PM with respect to the joint distribution of the features included in the coalition and the target, i.e., $y$ and $x_1$ in our example. Thus, we get an expected value $\mathbb{E}_{y,x_1}\mathbb{E}_{x_2,x_3}(\tilde{G}(y,\hat{f}(x_1,x_2,x_3)))$, where the first expectation refers to the averaging effect, whereas the second one refers to the marginalization effect. The Shapley value is then computed by summing the weighted differences in the expected PM obtained with or without the feature of interest, for all the coalitions of other features.
One advantage of our marginalization-based approach is that there is no need to re-estimate any sub-model for each subset of features. We consider the expected value of the PM with respect to the features $\mathbf{x}^{\overline{S}}$ excluded from the coalitions, while leaving the model $\hat{f}(\mathbf{x})$ unchanged. An alternative solution would consist in estimating a submodel for each subset of features. For example, for a model with three features $x_1,x_2,x_3$, the computation of the Shapley value associated with $x_1$ would require to estimate four sub-models, namely without any feature, with $x_2$ only, with $x_3$ only, and with $x_2,x_3$, and then to re-estimate the same sub-models while including $x_1$ as an additional feature. This approach was adopted for instance by Israeli2007 to decompose the $R^2$ of a linear model. However, re-estimating a submodel with only a subset of the initial features can lead to model specification errors, e.g., omitted variable bias in the case of linear regression. This specification error necessarily distorts the Shapley value and thus may lead to an unreliable decomposition of the performance metric.
XPER is a weighted average of the expected performance improvement resulting from adding the feature of interest to a model (coalition) that “turns off” this feature. Therefore, the choice of the weighting scheme is crucial. To compute the XPER values, we utilize the coalition weights derived from Shapley's original definition Shapley_1953, as defined in Equation (ref). Given this definition, the weights of coalitions are typically uniform and based on the number of features. This choice offers a significant advantage: it guarantees compliance with the four axioms that define a Shapley value — efficiency, symmetry, linearity, and null effects. Therefore, most XAI methods relying on Shapley values adopt these coalition weights (e.g., Sundararajan2017, Lundberg2018, Casalicchio_2019, Bowen2020, SundararajanDhamdereAgarwal2020). For instance, Lundberg2017 follows this approach when introducing the Shapley value to justify their Shapley Additive Explanation (SHAP) method.
The XPER values satisfy different axioms associated with Shapley values. These axioms are particularly relevant in the context of statistical performance analysis.
One of the main advantages of the XPER decomposition is that the benchmark value $\phi_0$ has an insightful interpretation: it corresponds to the PM that we would obtain on a hypothetical sample in which the target variable $y$ is independent from all model features $\mathbf{x}$, i.e., in a case where the model $\hat{f}(\mathbf{x})$ is fully misspecified. As an example, for the AUC, the benchmark $\phi_0$ corresponds to the AUC associated with a random predictor and is equal to $0.5$. For the sensitivity (true positive rate), the benchmark corresponds to the probability $\Pr(\hat{y}=1)$, for the specificity (true negative rate) the benchmark is $\Pr(\hat{y}=0)$, etc. Thus, we can decompose any PM into two parts: (i) a base value $\phi_0$ obtained in a hypothetical case where $y$ and $\mathbf{x}$ would be independent, and (ii) a component determined by the XPER feature contributions, which depends on their dependence with the target, i.e., their relevance.
The XPER values also satisfy the other main axioms of the Shapley values:
To illustrate these axioms, consider a linear regression model $\hat{f}(\mathbf{x}_i)= \sum_{j=1}^{q}\hat{\beta}_jx_{i,j}$ estimated on a training set, and the $R^2$ as PM. For simplicity, we assume that the Data Generating Process (DGP) of the test sample $S_n=\{\mathbf{x}_i,y_i,\hat{f}(\mathbf{x} _i)\}^n_{i=1}$ satisfies $ \mathbb{E}\left(\mathbf{x}\right) = \mu_q \mbox{ and } \mathbb{V}(\mathbf{x}) = \Sigma$ a positive semi-definite matrix, with $\Sigma_{k,j} = \sigma_{x_k,x_j}$ the covariance between feature $x_k$ and $x_j$. Then, the XPER contribution $\phi_{j}$ of feature $x_j$ to the $R^2$ is:
with $\sigma_y^2$ the variance of the target variable and $\sigma_{y,x_j}$ its covariance with feature $x_j$. See Appendix (ref) for the proof. Formally, $\phi_j$ depends on the estimated parameter $\hat{\beta}_j$ (training set) and the covariance (test set) between $x_j$ and the target variable $y$, i.e., $\sigma_{y,x_j}$. If the DGP of the training and test samples are identical, model parameters $\hat{\beta}_j$ and covariance $\sigma_{y,x_j}$ have the same sign, which means $\phi_j > 0$.\footnote{ Note that the XPER values $\phi_j$ can be negative, even if the DGP of the training and test samples are identical. This occurs in a particular case where the true parameter $\beta_j$ of the feature of interest $x_j$ is null. In this case, the parameter estimate $\hat{\beta}_j$ obtained using the training sample can be either positive or negative. Additionally, the estimate of the covariance between the target variable $y$ and the feature $x_j$, denoted $\hat{\sigma}_{y,x_j}$, obtained using the test sample can also be either positive or negative, even if the true covariance $\sigma_{y,x_j}$ is null as $\beta_j=0$. Therefore, if the sign of the parameter estimate $\hat{\beta}_j$ and the covariance estimate $\hat{\sigma}_{y,x_j}$ differ, then the XPER value $\phi_j$ can be negative.} Otherwise, XPER values may be negative. Similarly, if $\hat{\beta}_j=0$ (i.e., the variable is useless in the model) or if the feature is uncorrelated with the target variable on the test sample ($\sigma_{y,x_j}=0$), then $\phi_j=0$. Finally, a feature $x_j$ has a larger contribution to the $R^2$ than a feature $x_s$ if $\hat{\beta}_j\sigma_{y,x_j} > \hat{\beta}_s\sigma_{y,x_s}$, meaning that $x_j$ is more related to the target variable than $x_s$ both in-sample (through $\hat{\beta}_j$) and out-of-sample (through $\sigma_{y,x_s}$).
In this simple framework, the XPER values correspond to the contribution of the features to the explained variance, as typically computed in econometrics. For ease of explanation, consider the following DGP:
In this framework, the variance explained by the model is typically decomposed as follows:
Thus, in our example, the $R^2$ is equal to 50%, with the contributions of features $x_1$ and $x_2$ to the $R^2$ amounting to 5% and 45%, respectively. Similarly, the XPER decomposition is:
where $\phi_j$ represents the XPER value associated with the feature $x_j$ and $\phi_0 = -R^2$ the benchmark value.\footnote{Appendix (ref) presents a detailed example illustrating the interpretation of the benchmark value $\phi_0$ in the context of a standard linear regression model and the decomposition of the $R^2$.} From Equations (ref) and (ref), we can see that the contribution $\tilde{\phi}_j$ of the feature $x_j$ to the explained variance can be derived from the XPER value by normalizing it with respect to the difference $R^2 - \phi_0$:
We generally report normalized XPER values, defined as $\phi_j / (\text{PM} - \phi_0)$, where $\text{PM}$ denotes the performance metric under consideration (e.g., $R^2$). Thus, in this particular setting, the XPER values exactly match the features’ contributions to the explained variance.
In many cases, analytical expressions for the XPER values $\phi_{j}$ are not available. Indeed, the function $G\left(y;\mathbf{x};\delta_0\right)$ is in general non-linear in $y$ and $\mathbf{x}$, as the model $\hat{f}(.)$ can be highly non-linear in $\mathbf{x}$ (e.g., machine learning model) and/or the performance metric can be non-linear in $y$ and $\mathbf{x}$. See Appendix (ref) for the XPER values decomposition of several regression and classification performance metrics.
The XPER framework can be used to conduct a global analysis of the model predictive performance through feature contributions $\phi_{j}$, or a local analysis at the individual level.
The individual XPER value can be interpreted as the contribution of individual $i$ and model feature $j$ to the performance metric. For a given realization ($y_i,\mathbf{x}_i$), the corresponding individual contribution to the performance metric can be broken down into:
where $\phi_{i,j}$ is the realization of $\phi _{i,j}(y_i;\mathbf{x}_i)$ and $\phi_{i,0}$ is the realization of $\phi_{i,0}(y_i)= \mathbb{E}_{\mathbf{x}}(G(y_i;\mathbf{x};\delta_0))$. The benchmark $\phi_{i,0}$ corresponds to the contribution of an individual to the performance metric when the target variable $y_i$ is independent from the features $\mathbf{x}_i$. Therefore, the difference between the individual contribution to the performance metric $G(y_i;\mathbf{x}_i;\delta_0)$ and the individual benchmark $\phi_{i,0}$ is explained by individual XPER values $\phi_{i,j}$.
As an illustration, consider the $R^2$ of a linear regression. Then, the individual XPER value $\phi_{i,j}$ can be expressed as:
with $A=\left( 2y_i -\sum_{\substack{k=1\\ k \neq j}}^q\hat{\beta}_k(x_{i,k} +\mathbb{E}(x_{k})) \right)$. See Appendix (ref) for the proof. The value $\phi_{i,j}$ measures the contribution of feature $x_j$ to $G(y_i;\mathbf{x}_i;\delta_0)=1-\frac{1}{\sigma_y^{2}}(y_i-\hat{f}(\mathbf{x}_i))^2$. A positive individual XPER value $\phi_{i,j}$ means that incorporating the information included in feature $x_j$ improves model prediction for individual $i$ compared to the benchmark, hence increasing $R^2$. This gives rise to several observations. First, if $\hat{\beta}_j=0$, i.e., the feature has no impact on the model outcome, the feature $x_j$ does not have any impact on the $R^2$, for all individuals. Second, the closer the realization of feature $x_{j}$ is to its expected value $\mathbb{E}(x_j)$, the lower is the contribution of this feature to the performance metric, for all individuals. A similar result occurs when $x_j^2$ is close to its expected value. Indeed, when the characteristics of an individual are close to the mean values in the population, his or her contribution to the predictive performance of the model is also close to the average contribution of other individuals. We include a numerical example in Appendix (ref) to provide further intuition.
Individual XPER values can be particularly useful to detect and understand sources of model heterogeneity. In machine learning and econometrics, model homogeneity refers to a consistent relationship between features and the target variable across the entire dataset, implying that a single model can adequately capture the underlying patterns and make accurate predictions for all data points. Conversely, model heterogeneity arises when this relationship varies across different parts of the dataset, indicating that a single model may fail to capture the diverse patterns within the data. In such cases, different models may be needed to achieve better predictive accuracy, indicating distinct feature-target relationships within subgroups of the data.
To identify these subgroups, we suggest relying on the individual XPER values. Indeed, by definition, individual XPER values measure the contribution of features on individual predictive performance, hence capturing the relationship between the features and the target variable. Intuitively, if a feature's impact on performance varies between two groups, it indicates that the feature does not exert the same influence on the accuracy of the classification or regression model.
In practice, model heterogeneity can be revealed by applying standard clustering techniques to individual XPER values, such as the K-Medoids algorithm park2009simple. Specifically, we propose the following three-step approach. First, using the original classification or regression model applied to the entire training sample, we calculate the individual XPER values based on the features $\mathbf{x}$ and the true target variable $y$. Second, we apply a clustering algorithm to the XPER values to identify $C$ clusters that reveal the heterogeneity in the model's performance. Third, we estimate a separate model for each of the $C$ subgroups derived from the initial sample. The empirical application presented in Section (ref) suggests that this approach could lead to an improvement in the model's accuracy.
In this section, we discuss the estimation procedure for the XPER values $\phi_{j}$ and $\phi_{i,j}$, based on a test sample $ S_n=\{\mathbf{x}_i,y_i,\hat{f}(\mathbf{x} _i)\}^n_{i=1}$. Below, we assume that the model has a small number of features, typically $q<10$.\footnote{The standard estimation framework is only feasible for a small number of features $q$. The computation of the XPER values $\phi_j$ according to Definition (ref) becomes cumbersome as the number of features increases. Indeed, it requires to consider $q\times 2^{q-1}$ coalitions of features, e.g., $5,120$ for $q = 10$ and $10,485,760$ for $q = 20$, etc.} When the number of features is larger, an approximation of the XPER values is required and we propose a modified version of the Kernel SHAP method of Lundberg2017 (see Appendix (ref) for more details).
Under Assumption (ref), the XPER value $\phi_{j}$ can be estimated by a weighted average of individual contributions differences such as:
with $S$ a coalition, i.e., a subset of features, excluding the feature of interest $x_{j}$, and $\mathcal{P}(\{\mathbf{x}\} \setminus \left\{x_{j}\right\})$ the powerset of the set $\{\mathbf{x}\} \setminus \left\{x_{j}\right\}$.
Table (ref) illustrates the computation of the estimated value $\hat{\phi}_1$ associated with feature $x_1$ in a model with three features $(x_1,x_2,x_3)$. For each coalition of features (column 1), we report the corresponding weight (column 2) along with the estimated marginal contribution of feature $x_1$ to the performance metric (column 3). The intuition is as follows: the sum over index $u$ refers to the marginalization effect, whereas the sum over index $v$ refers to the averaging effect. For a given instance $v$, we compute its average performance metric by replacing the features ${x}^{\overline{S}}$ which are not included in the coalition by the corresponding values observed for all the instances of the test sample (marginalization). Then, we compute the average performance metric for all the instances $v$ (averaging).
Similarly to the estimation of features $\phi_{j}$, we can estimate the individual XPER values $\phi_{i,j}$.\footnote{Performing inference on XPER values presents significant challenges. Williamson2020 proposed an approach for statistical inference on a variable importance measure called the “Shapley Population Variable Importance Measure” (SPVIM). However, this approach is incompatible with XPER values due to its reliance on sub-model re-estimation and its focus on global rather than individual analyses. Another potential approach involves using bootstrap methods to generate a block bootstrap distribution of XPER values, though the high computational cost remains an important limitation. Finally, conformal prediction Vovk2005, Romano2019 represents a promising avenue, as it could be adapted to construct prediction intervals for individual XPER values.} For any individual $i$ of the test sample of $S_n$ and any feature $x_j$, a consistent estimator of the individual XPER values $\phi_{i,j}$ is defined as:
By definition, these individual XPER values satisfy:
In this section, we conduct a Monte Carlo simulation experiment to illustrate the XPER methodology and to highlight some of its properties. In this experiment, XPER values are used to explain the predictive performance of a probit model, measured by the AUC computed on a test set $S_n$. To understand the mechanisms of the XPER decomposition, we consider a white-box model for which the predictions are explainable. The DGP is given by a latent variable model such that $y_i = \mathbf{1}(y^*_i>0)$, where $\mathbf{1}(.)$ is the indicator function, $y_i^{*} = \mathbf{\omega}_i\beta + \varepsilon_{i}$, $\mathbf{\omega}_i=(1:\mathbf{x}^{'}_i)$ and $\varepsilon_{i}$ an i.i.d. error term with $\varepsilon_{i}\sim \mathcal{N}(0,1)$. We consider three i.i.d. features such that $\mathbf{x}_i=(x_{i,1},x_{i,2},x_{i,3})^{'} \sim \mathcal{N}(\mathbf{0},\mathbf{\Sigma})$ with $diag(\mathbf{\Sigma})=(1.2,1,1)$. The true vector of parameters is $\beta=(\beta_0,\beta_1,\beta_2,\beta_3)^{'} = (0.05,0.5,0.5,0)^{'}$ with $\beta_0$ the intercept.
We simulate $K=5,000$ pseudo-samples $\lbrace y_i^k,\mathbf{x}_i^k \rbrace_{i=1}^{T+n}$ of size 1,000, for $k=1,\dots,K$. For each pseudo-sample, we use the first $T=700$ observations to estimate a probit model and the remaining ones as test set $S_n$ to compute the AUC and the corresponding XPER values according to Equation (ref). For instance, the estimated parameters obtained for the simulation $k=1$ are equal to $\{\hat{\beta}_0^k,\hat{\beta}_1^k,\hat{\beta}_2^k,\hat{\beta}_3^k\} = \{0.0109,0.4943, 0.5234, 0.0688\}$ and the $AUC^k$ is equal to $0.7775$. For this simulation, the average feature contributions, taken over the individuals of the test set, are the following:
As expected, the estimated benchmark $\hat{\phi}_{0}^k$ is close to 0.5. As a reminder, an AUC equal to 0.5 is associated with a random predictor. Hence, the benchmark $\hat{\phi}_{0}^k$ corresponds to the AUC of the model that we would obtain on a virtual test sample in which the target variable is independent from all features. The difference between the estimated AUC and this hypothetical benchmark is explained by the feature contributions. The feature with the largest variance ($x_1$) has also the largest contribution to the predictive ability of the model ($0.1716/(0.7775-0.4984) \simeq 62\%$). In contrast, the contribution of $x_3$, which is not used to generate the data ($\beta_3 = 0$), is very close to zero.
Figure (ref) displays the empirical distributions of the AUC, the benchmark values $\hat{\phi}_0$, and the XPER values associated with features $x_1$, $x_2$, and $x_3$, computed from the $K$ simulations. It confirms the robustness of our analysis, but also illustrates the possibility to make inference on XPER values by Bootstrap or other numerical methods.
In Table (ref), we report the values of the individual XPER values obtained for the first simulation $k=1$. The third column displays the individual benchmark. For a binary classification model, the benchmark $\hat{\phi}_{i,0}\equiv \hat{\phi}_{i,0}(y_i)$ only takes two values. In our simulations, for individuals $y_i=1$ (respectively $y_i=0$), this value is equal to $0.5001$ (respectively $0.4967$). Remind that the individual benchmark corresponds to the contribution to the AUC obtained from a random predictor for an individual with a target value $y_i=1$ or $y_i=0$. Consider the first instance $i=1$ with $y_i^k=1$, $G(y^k_1;\mathbf{x}^k_1;\hat{\delta}^k_n)=0.9$ and $\hat{\phi}_{1,0}=0.5001$. As $G(y^k_1;\mathbf{x}^k_1;\hat{\delta}^k_n) > \hat{\phi}_{1,0}^k$, it means that the features allow the probit model to better predict the event $y_1$ for this instance than a random predictor.
Finally, columns 4, 5, and 6 report the individual XPER values associated with features $x_1$, $x_2$, and $x_3$. First, we verify that feature $x_3$ has close to no impact on the AUC for all instances. Second, the heterogeneity of contributions to the AUC depends on individual characteristics. Remind that at the global level, the contribution of feature $x_1$ is higher than the one of feature $x_2$. However, at the local level, we observe for $i=300$ that the contribution of feature $x_1$ is smaller than the one of feature $x_2$, i.e., $0.1316 < 0.3069$. Third, some individual contributions are negative. For instance, the negative contribution of feature $x_2$ for individual $i=297$ implies that, for this instance, this feature hinders the model from predicting the true target value $y_i^k=0$.
To further illustrate the XPER methodology, we provide in Appendix (ref) two additional Monte Carlo simulation experiments illustrating how XPER values can be used to detect the origin of overfitting.
We implement our methodology on a proprietary database of auto loans provided by an international bank. For each borrower, we know whether he or she has eventually defaulted ($y=1$) or not ($y=0$) on the loan. Given the sensitive nature of the data, we had to randomly under-sample individuals to set the default rate to an arbitrary 20% level. Besides benefits in terms of confidentiality, setting a high arbitrary default rate also protects us against concerns arising from using an unbalanced database. After undersampling, our database includes 7,440 borrowers.\footnote {Besides undersampling, potential techniques to deal with imbalanced datasets include oversampling (e.g., ADASYN, ROSE, SMOTE) and cost-sensitive learning. For more details, see Jian2023 and Chen2024.} Besides the default target variable, we have access to ten features on the loan (funding amount, funded duration, vehicle price, down-payment) and on the borrower (job tenure, age, marital status, monthly payment in percentage of income, home ownership status, credit event). We divide the database into a stratified training (70%) and test (30%) samples to have the same default rate in both subsamples.
We provide in Table (ref) some summary statistics about the features and the target variable. In our dataset, a typical loan amounts to around 11,500 euros, finances a 13,000 euro car, and lasts for 56 months. A typical borrower is 45 years old, married, not owning his or her home, has spent nine years in the same job, experienced no credit event over the past six months, provides less than a 50% down payment, and allocates 10% of his or her monthly income to reimburse the car loan. To get a first sense of the role of each feature on default, we display in Figure (ref) their distributions separately for defaulting and non-defaulting borrowers. This preliminary test indicates that the list of discriminating feature includes age, credit event, down payment, marital and ownership status.
Using the training sample, we estimate an XGBoost model to predict default. We selected this model because it is recognized as one of the most powerful scoring engines gunnarsson_deep_2021. Another reason for using an XGBoost is its black-box nature. Indeed, while the XPER methodology is model-agnostic, we believe it is interesting to assess its usefulness when used with a particularly complex and opaque algorithm. We select the hyperparameters of the XGBoost using stratified five-fold cross-validation based on a balanced accuracy criterion bergstra_random_nodate.
Table (ref) displays the values of six performance metrics for the XGBoost model obtained on the training and test samples. Specifically, (1) the AUC measures the discriminatory ability of the model, (2) the Brier Score evaluates the accuracy of the probabilities, and (3) Accuracy, Balanced Accuracy (BA), Sensitivity, and Specificity assess the correctness of the categorical predictions. As shown in Table (ref), the XGBoost has an AUC of 0.7521, a Brier Score of 0.1433, and an accuracy of 79.53 on the test sample. Despite the cross-validation of the hyperparameters, we observe some over-fitting in the model as its performances drops slightly from the training sample to the test sample. Overall, the performance metrics of our model are comparable to those reported in gunnarsson_deep_2021.
We display in Figure (ref) the decomposition of the AUC among the ten features obtained with the estimation method detailed in Appendix (ref). For ease of presentation, we express the feature contributions in percentage of the spread between the AUC and its benchmark (see Equation (ref)). As shown in Table (ref), the AUC in the test sample is equal to 0.7521, which is significantly better than the benchmark value of 0.5 obtained for a random predictor.\footnote{In Appendix (ref), we vary the average default rate in the dataset from 5% to 30% by randomly removing some default observations or non-default observations from the sample, respectively, to assess the impact of imbalanced data on XPER values.} We see that around 40% of this over-performance is coming from funding amount. The second most contributing feature is job tenure, which accounts for another 18%. It is interesting to note that with only two features, we can explain more than half of the performance of the model. Next, we have five features which contribute each for another 8-10% to the performance. At the other side of the spectrum, down payment is the feature contributing the least to the AUC. On the test sample, this feature does not help the model to better predict default than a random predictor as its XPER value is close to 0. Note that in Appendix (ref), we also analyze the effect of the various features on the performance for each borrower individually.
We now compare the XPER performance decomposition to standard feature contribution methods commonly used in machine learning to assess the impact of a feature on performance or to explain the output of a black-box model, namely Permutation Importance (PI), feature importance, and SHAP values.
First, we distinguish between the XPER decomposition of the AUC and the PI method introduced by breiman_random_2001. The latter computes for a feature $x_j$, the decrease in the AUC of the model when the values of this feature are randomly reshuffled across instances. As the permutation breaks the dependency between the target variable and the feature, the resulting drop in model AUC indicates how much the model depends on the feature. In our analysis, the PI results indicate the average decrease in AUC when the values are reshuffled 200 times to obtain more robust results. As for the previous methods, we divide each feature contribution by the sum of all features contributions to compare them to the XPER values. We see in Figure (ref) that XPER and PI produce different results. For instance, the contribution of the car price is twice as important with PI than with XPER.\footnote{For a more detailed comparison between XPER and PI, see Appendix (ref). We show that PI is a special case of XPER and is only designed for a global analysis.}
Second, we contrast in Figure (ref) the XPER values and the XGBoost-based feature importance. The latter computes for a feature $x_j$, the average gain across all splits where the feature is utilized. For ease of comparison, we divide each feature contribution by the sum of the ten feature contributions. As shown in Figure (ref), the result is rather striking. Indeed, the two methodologies lead to very different contributions as some dominating features in a given methodology play a minor role in the other. For instance, credit event exhibits the highest feature importance but it is only the $9th$ most contributing feature according to XPER. Differently, funding amount plays a very important role to explain performance but does not contribute much in terms of feature importance.
Third, we compare in Figure (ref) the XPER decomposition of the AUC with the SHAP values introduced by Lundberg2017. In our context, the SHAP values assess the impact of the different features on the probabilities of default of each borrower. As it is the norm, we take the average absolute SHAP values for each feature to assess the feature contribution at the model level. As before, we divide each feature contribution by the sum of all features contributions to express them in percentages. In Figure (ref), we see that SHAP and XPER provide for some features very different information. For instance, the car price contribution is more than twice as important for SHAP than for XPER. Similarly, the funding amount contribution is around 40% for XPER whereas only 28% for SHAP. \footnote{For a more detailed comparison between XPER and SHAP, see Appendix (ref).}
We now show how XPER can be used to deal with heterogeneity issues and improve performance. In practice, it is often challenging to estimate a single model able to correctly estimate the relationship between the target variable and the features for all individuals in the sample. In this section, we propose an alternative two-step procedure. First, we build homogeneous groups of individuals displaying similar XPER values using a clustering algorithm. Second, within each group, we estimate a group-specific model. Given the definition of XPER, in each group, the features have a similar impact on the target variable. This alternative approach is likely to increase the model performance compared to the one-fits-all model, which has been used so far in the empirical application.
In the first step, we estimate the XPER values $\hat{\phi}_{i,j}$ on the training sample and apply the K-Medoids method park2009simple to create two clusters of individuals with similar XPER values.\footnote{XPER values are always continuous, even when they are associated with categorical variables. To determine the optimal number of clusters, we used the “silhouette score” criterion. We report this score in Table (ref) in Appendix (ref) for a number of clusters ranging from 2 to 10.} We see in Figure (ref) that individuals in group 1 tend to be older, married, home-owners and ask for a mid-range loan to finance a mid-priced car. One striking difference is that in group 2, individuals tend to finance cars with extremely low prices or high prices. Overall, group 1 gathers individuals displaying a lower risk profile compared to those in group 2. This is confirmed by the fact that the default rate is much higher in group 2 (52%) than in group 1 (13%). Moreover, as shown in Figures (ref) and (ref) in Appendix (ref), the feature distributions are very different for defaulters and non-defaulters in each group. We also see in Figure (ref) that in group 2, using the car price deteriorates the performance of the model as its XPER value is close to -20% vs. 20% in group 1. Therefore, these results suggest that estimating group-specific models is likely to boost predictive performance.
In the second step, we estimate using the training dataset one XGBoost model per group to predict default. Then, we aim to compare the resulting out-of-sample performance with the one of the one-fits-all model. To do so, we first assign each individual of the test sample to either group 1 or group 2 using the clustering rule built on the training sample. We see in Table (ref) that the performance of our strategy (column 2) is significantly higher than the performance of the initial model (column 1). For instance, the AUC of the model on the test sample increases by 16 percentage points, from 0.752 to 0.912. Another notable result is that by using two distinct models, the bank would be able to correctly detect 64% of the defaulting borrowers against only 24% with the initial model. Yet, it would still be able to correctly detect more than 95% of the non-defaulting borrowers.
Finally, we benchmark our strategy with a standard clustering approach based on the features themselves. Specifically, we use the K-prototype algorithm huang1997clustering to create two clusters of individuals with comparable feature values.\footnote{We utilize the K-Prototype algorithm instead of the K-Medoids algorithm as the former is specifically designed to handle both categorical and continuous features. More precisely, when handling continuous features, the algorithm calculates the squared Euclidean distance between the features and a representative vector (or prototype) of clusters. For categorical features, it determines the number of mismatches between the features and the cluster’s prototypes.} As shown in column 3, the performance of the standard approach is close to the one of the one-fits-all model (AUC = 0.744 vs. 0.752), and is much lower than the performance of the XPER-based models.
We have introduced XPER, a methodology designed to measure the feature contributions to the performance of any regression or classification model. XPER is built on Shapley values and interpretability tools developed in machine learning but with the distinct objective of focusing on model performance, such as AUC or $R^2$, and not on model predictions, $\hat{y}$.
Specifically, XPER breaks down the difference between a performance metric of the model and a benchmark value. The latter corresponds to the performance metric that we would obtain on a hypothetical sample in which the target variable is independent from all the features included in the model. As such, the XPER decomposition only focuses on the part of the performance that directly originates from the features. Other advantages of the method include being implementable either at the model level or at the individual level, and not being plagued by omitted variable biases, as it does not require re-estimating the model with different features coalitions.
We show that identifying the driving forces of the performance of a predictive model is very useful in practice. In a loan default forecasting application, XPER appears to be able to efficiently deal with heterogeneity issues and to boost out-of-sample performance. To do so, we create homogeneous groups of borrowers by clustering them based on their individual XPER values. We find that estimating group-specific models yields to a higher predictive accuracy than with a one-fits-all model.
XPER could go beyond performance. Indeed, as XPER can decompose any function of both $\hat{y}$ and $y$, one could use it to identify the features responsible for the lack of algorithmic fairness of a given machine learning model hurlin2024fairness. Another interesting avenue for future research relies on developing inference methods for XPER values, enabling the assessment of their statistical significance.
\baselineskip=1\normalbaselineskip
\baselineskip=1.5\normalbaselineskip