Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
69,740 characters · 26 sections · 44 citation commands
Functional effects models: Accounting for preference heterogeneity in panel data with machine learning
\sloppy
Human choices are immensely complex and are inherently linked to observed variables, such as the cost of an alternative, and unobserved variables, such as life experiences and preferences. Consecutive choices made by the same individual share these unobserved variables and will be inevitably correlated. This violates the assumption of independence between observations, a common assumption in Machine Learning (ML) models and basic statistical models such as the Multinomial Logit (MNL) model. Therefore, these types of choice experiments, referred to as panel or longitudinal studies, require specific methodologies to address the correlation between observations. Common problems dealing with panel data include, for example, longitudinal health studies, household income variation over time studies, stated preference experiments where respondents are being asked to make several choices consecutively, or even time series analysis.
Traditionally, practitioners model preference parameters with two types of choice models based on the random utility theory:
For a detailed discussion, see 10.1093/oxfordhb/9780199940042.013.0006. These models differ in how they account for the serial correlation between the error terms of the same individual. The first type models preference parameters with either fixed effects, i.e., individual-specific constants (or intercepts), estimated from maximum likelihood of observed choices, or random, or mixed, effects, i.e., random parameters distributed across the population\footnote{Note, that the mixed effects model refers to each parameter being the combination of a fixed "population-level" parameter with an individual-specific random intercept (i.e., $\beta_m + u_{mn}$ where $u_{mn}$ is a random intercept), whereas the fixed effects model refers to individual-specific preference parameters (i.e., $\alpha_{in}$ which is learnt for each individual using maximum likelihood estimation). While mixed effects is a general term, in the context of panel data we distinguish between the random intercepts and random slopes models. }. However, the fixed effects model suffer from the incidental parameters problem, where the parameters are inconsistent when the number of observations per individual is small, and is typically only used in regression\slash ordinal tasks 10.1093/oxfordhb/9780199940042.013.0006. Therefore, in this work, we focus on the mixed effects model. Two special cases of mixed effects models are the random intercept model, with the intercept, or constant, assumed to be a random variable, and the random slopes model, where the coefficients related to a variable are assumed to be randomly distributed. This method has been first developed to identify the true values of estimated parameters, removing bias from not accounting for individual heterogeneity. However, in recent years, with the ever-increasing use of ML models, predictive tasks have become predominantly important, and the mixed effects model is limited when predicting preferences of unknown individuals. More specifically, the random effects fitted during training have to be averaged (i.e., use the population-level mean with the mean of the random effects distribution) krueger2021evaluating, reducing predictive power. In addition, the random effects model rely on strong distributional assumptions that need to be specified by the modeller.
The second type of models is Markov chain models, where the last observed choice made by an individual is used to account for panel effects. When handling the first observed choice of an individual, dynamic models can be thought of as static models, since there are no previous choices to use as exogenous variables. This is known in the literature as the initial conditions problem of dynamic models train2009discrete. Therefore, dynamic models are also not suitable to predict preferences of unknown individuals, and there is a crucial need to develop specific methodologies to make static models suitable for forecasting. Note that we emphasise the parallels between dynamic models and recommender systems, especially in the cold start problem panda2022approaches, where static models could be used to generate recommendations for individuals without prior choice knowledge. Hereafter, since we are interested in panel data predictions for unobserved individuals, we focus on static models for panel data. Table (ref) provides a glossary of all the terminology used in this paper.
There have been several attempts to adapt static models, such as the mixed effects model, with Machine Learning (ML) methodologies. The main idea is to learn the population-level mean with an out-of-the-box ML regressor and estimate random effects as in a linear mixed effects model. For example, this has been done for regression trees hajjem2017generalized, sela2012re, linear trees fokkema2018detecting, Random Forests (RF) hajjem2014mixed, Gradient Boosting Decision Trees (GBDT) 9759834, Neural Networks (NN) mandel2023neural, and Convolutional Neural Networks (CNN) xiong2019mixed. All these papers use some sort of Expectation-Maximisation (EM) algorithm to iteratively estimate the population-level mean and random effects. This framework has also been generalised by ngufor2019mixed and kilian2023mixed. However, these models face two main problems when applied in choice modelling with panel data:
Note that in xiong2019mixed, the authors decompose the output of the model as fixed and random effects, where both effects depend on input features. However, by doing so, the random effects of their model are effectively fixed effects and do not have any random component, falling back to an out-of-the-box CNN. Finally, it is worth mentioning that there is very little existing research into dynamic ML models in a choice modelling context.
In this paper, we use existing gradient-based ML methodologies to learn individual-specific parameters as a function of socio-demographic characteristics. We learn two types of functional effects:
At a high level, we are using unrestricted ML regressors to impute an individual-specific constant from the socio-demographic characteristics. This constant mimics the random effect in traditional mixed effects models, but since it is a function of the socio-demographic characteristics, it can be easily used for inference. We do so with the two most popular ML regressors for tabular data, Gradient Boosting Decision Trees (GBDTs) and Deep Neural Networks (DNNs). It is worth noting that, whilst the primary contribution of the paper is a general framework for capturing individual effects for panel data, this framework incorporates and builds on several existing works in the literature, specifically: L-MNL sifringer2020enhancing, TasteNet-MNL han2022neural, and RUMBoost salvade2025rumboost. We further note that all of the models presented in this paper could similarly be used to account for individual heterogeneity in cross-sectional data, though this is not the focus of this paper. We systematically evaluate our methodology on a synthetic experiment and three real-world panel case studies.
The main strengths of our proposed methodology are:
All the code used in this paper, including the implementation of a generalised functional effects model learnt with GBDTs and DNNs for different datasets and choice situations, is open source and freely available on Github\footnote{https://github.com/big-ucl/functional-effects-model}.
The rest of the paper is structured as follows: Section (ref) provides the theoretical background, and Section (ref) introduces the methods used in the paper. We validate the methodology with a synthetic experiment in Section (ref) and provide a thorough benchmark with three real-world case studies in Section (ref). Finally, Section (ref) concludes the paper.
Choice models based on the random utility theory assume that choice makers are rational and will choose the alternative that maximises their utility, a latent representation of their preferences. The utility function is typically composed of a deterministic part and a random part, i.e.:
where:
The deterministic utility is widely represented as a linear-in-parameter function of the variables, i.e.;
where:
Interpretable non-linear ML utility models (e.g. RUMBoost salvade2025rumboost) extends the deterministic utility function specification of Eq. (ref) by replacing $\beta_{im}x_{inm}$ with the output of a gradient-based ML regressor:
where:
Note that some researchers have attempted to interpret the output of an ML regressor as the deterministic utility:
where:
However, $f_i$ is usually not consistent with the random utility theory and is not interpretable.
In this work, we assume the error term to be independent and identically distributed (i.i.d.), such that the probability $P_{in}$ of an individual $n$ to choose alternative $i$ with $J$ alternatives can be derived as in an MNL model, that is:
The parameters are chosen to minimise the negative Cross-Entropy Loss (CEL) function, akin to Maximum Log-Likelihood Estimation (MLE):
where:
Panel data have more than one observation per individual. Therefore, we extend Equation (ref) as follows:
where $t$ represents the $t^{th}$ choice of an individual $n$. Inevitably, the error term $\epsilon_{int}$ is not i.i.d. across $t$ and specific methodologies are required to deal with this intrinsic correlation, i.e., dynamic models. However, we omit the dynamic correlations here, as we do not address these in the current methodology, instead leaving this to further work. In order to reduce the parameter bias by accounting for inter-individual heterogeneity, the fixed effects model extends Eq. (ref) as follows:
where the model parameters $\alpha_{in}$ and $\beta_{inm}$ are now individual-specific. A fixed intercept model is a model with only $\alpha_{in}$ being individual-specific, and a fixed slopes model is a model with only the slopes, or coefficients, $\beta_{inm}$ being individual-specific.
On the other hand, the mixed\slash random effects model incorporates inter-individual heterogeneity with random variables, i.e.:
where $\alpha_{i}$ and $\beta_{im}$ are population-level parameters and $u_{inm}$ is a random variable (typically normally distributed) with 0 mean and standard deviation to be estimated from the data. A mixed\slash random intercept model is a model with only $u_{in0}$ being randomly distributed, and a mixed\slash random slopes model is a model with only the slopes, or coefficients, $u_{inm}$ being individual-specific.
In this study, we assume that the correlation of the error term across $t$ can be captured by individual-specific parameters learnt from the socio-demographic characteristics, such that the error term $\epsilon_{int}$ can be assumed to be i.i.d. and the methodology described in Section (ref) is still applicable.
More formally, given a set of $M$ variables $\mathbf{x}_{int} \in \mathbb{R}^{M}$ and a set of $Q$ socio-demographic characteristics $\mathbf{s}_n \in \mathbb{R}^Q$ for individual $n$ and alternative $i$, we define the deterministic utility function as follows:
where:
We define more precisely the Functional Intercept (FI) model, where only $g_{i0}(\mathbf{s}_n)$ is the output of a gradient-based ML regressor, and the Functional Slopes (FS) model, where only $g_{im}(\mathbf{s}_n), \forall m, i$ are the outputs of a gradient-based ML regressor. Finally, the Functional Intercept and Slopes (FIS) model combines both functional intercept and slopes. If not learnt from the socio-demographic characteristics, the parameters can be either estimated linearly as in Eq. (ref) or non-linearly imputed from a gradient-based ML regressor as in (ref). Figure (ref) provides an overview of all possible functional effects models. We note that keeping the same linear-in-parameter utility function as in a traditional MNL model allows for the functional effects model to maintain significant interpretability.
The functional intercept or slopes are the output of a gradient-based ML regressor. Note that this approach requires the models to be fit to the gradients of the loss function with respect to the individual-specific parameters, and so can be adapted for any gradient-based ML algorithm.
In this paper, we present a comparison between the two most popular ML regressors for tabular data: GBDTs and DNNs.
The first ML regressor used in this paper is GBDT friedman2001greedy, chen2016xgboost, which uses ensembles of regression trees to learn functional effects. salvade2025rumboost extend this concept to a parametric utility context with RUMBoost, where, for a model with $R$ iterations, the GBDT predictive function is an additive function of the form:
where $w_{inmr}$ is the leaf value (a constant) of a tree $r$, where the individual $n$ has been partitioned to. At each iteration, the leaf values of a new tree are chosen to directly minimise the second-order Taylor expansion of the loss function, such that:
where:
In this paper, we extend this concept to functional effects being a function of socio-demographic characteristics $\mathbf{s}_n$. We have:
where the leaf values are computed as in Eq. (ref), but in the parameter space, that is $g_{inm} = \partial\mathcal{L}/\partial\beta_{inm} = \partial\mathcal{L}/\partial V_{in} \cdot \partial V_{in}/\partial\beta_{inm} $ the first derivative of the loss function with respect to $\beta_{inm}$ and $h_{inm} = \partial^2\mathcal{L}/\partial^2\beta_{inm} = \partial^2\mathcal{L}/\partial^2 V_{in} \cdot (\partial V_{in}/\partial\beta_{inm})^2$ is the second derivative of the loss function with respect to $\beta_{inm}$. Since the utility function is linear-in-parameter, the first-order derivatives are scaled by $x_{inmt}$ and the second-order derivatives are scaled by $x_{inmt}^2$ compared to a RUMBoost model.
Note that, for the FI model, we combine RUMBoost to find generic non-linear coefficients (i.e., $f_{im}(x_{inm})$) with individual-specific intercept (i.e., $g_{i0}(\mathbf{s}_n)$) to capture preferences.
The second ML regressor that we consider to learn functional effects is feed-forward DNNs goodfellow2016deep. More formally, the feed-forward DNN predictive function takes the following form:
where:
For training stability, it is common to split the datasets into batches of data. After each batch, the parameters of the models are updated with some variant of the gradient descent with first-order derivatives of the loss function (see e.g. kingma2014adam), with the gradients being efficiently computed through back-propagation (an algorithm efficiently applying the chain rule).
Note that the FI model falls back to the L-MNL proposed in sifringer2020enhancing and the FIS model falls back to the TasteNet-MNL proposed in han2022neural. We refer the reader to the original papers for complete explanations of the underlying methodology.
For both models, we can constrain the sign of the functional effects with a Rectified Linear Unit (ReLU) activation function:
where $c = 1$ for a positive monotonic constraint and $c=-1$ for a negative monotonic constraint.
Table (ref) enumerates all 11 potential functional effects models, with the corresponding model numbers from Figure (ref).
Note that, whilst this paper presents a unified modelling framework for capturing functional effects in panel data, all models in Table (ref) can also be applied to cross-sectional data. The proposed framework incorporates several existing models in the literature:
The FS-GBDT, FIS-GBDT, and FS-DNN models are all new to this paper.
Note we use the names FI-DNN and FIS-DNN as:
Note further that our naming convention distinguishes between FI-RUMBoost and FS-\slash FIS-GBDT, where the former uses the functional non linear coefficients $f_m(x_m)$ from salvade2025rumboost whilst the latter use linear coefficients. Both FI-RUMBoost and FIS-GBDT use a single GBDT ensemble for the functional intercept.
We further highlight that there are three model combinations in the framework that are not evaluated in this paper. To the best of our knowledge, there is no existing methodology that enables the estimation of interpretable non-linear functional coefficients $f_m(x_m)$ with neural networks, hence we not present a DNN equivalent to the RUMBoost and FI-RUMBoost models. Similarly, due to the complexity of estimating generic parameters within gradient boosting, we do not evaluate the FI-GBDT model with linear coefficients.
Table (ref) provides an overview of the datasets used in the experiments. For all experiments, we tune the hyperparameters with the Python library Optuna akiba2019optuna, using the Tree-structured Parzen Estimator (TPE) algorithm with 100 trials. Table (ref) summarises the hyperparameters tuned and the search space. We keep the hyperparameters that have optimal values on the validation set. Note that for all functional effects models with GBDTs, the learning rate is fixed and calculated as $\min(0.1, 1/\min(\mathbf{M}))$ where $\min(\mathbf{M})$ is the smallest number of variables in a utility function.
The only true way of assessing the behavioural quality of the functional effects models is through a synthetic experiment. Indeed, in a control setting, we can generate known true functional effects and verify the ability of the functional effects models to recover them. We do so for FI-RUMBoost, FI-DNN and a Randnom Intercept model (i.e. a mixed logit model with distributed alternative specific constants) on a fully synthetic dataset of 100000 observations from 10000 individuals. We simulate a multinomial discrete choice problem with 4 alternatives. We generate 4 socio-demographic characteristics ($x_1$-$x_4$) and 4 alternative-specific variables ($x_5$-$x_8$), drawn from a continuous uniform distribution between 0 and 1 such that $x_k \sim \mathcal{U}_{[0, 1]}, \forall k = 1,\cdots ,8$. We draw 10000 unique individuals as independent draws of $x_1$-$x_4$. For each individual, we then draw 10 different choice scenarios, as separate independent draws of $x_5$-$x_8$, repeating the socio-demographic characteristics for each scenario. The complete model specification is the following:
We assume a Type-1 Extreme Value i.i.d. error term, and so obtain choice probabilities from the logit\slash softmax function. Discrete choices are then sampled from the probabilities using a Monte-Carlo simulation. The functional effects are normalised to 0 in the third utility function, as the functional effects would not be recoverable because of the overspecification of the logit\slash softmax function. We repeat the process to obtain a test set of 20000 observations from 2000 previously unobserved individuals. Table (ref) of (ref) summarises the optimal hyperparameter values from the hyperparameter search. We implement the Random Intercept model with Biogeme bierlaire2023short, making the assumption that the random intercepts are normally distributed, and using the mean (i.e. without Monte-Carlo Simulation) for forecasting on the test set. We follow krueger2021evaluating to simulate the maximum likelihood estimation with 500 draws generated from the Modified Latin Hypercube sampling approach hess2006use.
Table (ref) shows the Mean Absolute Error (MAE) between recovered and true functional intercepts, the negative Cross-Entropy Loss (CEL) on both train and test set and the computational time to train the models. We observe that FI-DNN marginally outperforms FI-RUMBoost on the MAE and CEL, while FI-RUMBoost is about 4 times faster. The good performance of the two models with functional effects contrasts with the MAE of the Random Intercept model, which is bound to a normal distribution on the train set, and can only use the mean of this distribution on the test set.
Figure (ref) shows the recovered functional effects and parameters on the train set. The distributions of the functional effects are overall well-recovered by FI-RUMBoost and FI-DNN. We note that the distributions of the functional effects learnt with DNN are more concentrated around the mean for alternative 1 and 2. However, the functional effects learnt with GBDT seem to match more closely the true distribution. The Random Intercept model, on the other hand, fails to recover the true distribution, because of the a priori assumption of normality.
Figure (ref) displays the recovered functional intercepts on the holdout test set. FI-RUMBoost and FI-DNN are able to recover the functional effects even on unseen data. These results indicate that the functional effects models are suitable for inference, and contrast with the limitations of the Random Intercept model, which can only output the mean of the random effects estimated during training.
Finally, we also observe in Figure (ref) that all models are able to recover the true linear parameters for all variables. This special linear case highlights the proficiency of linear-in-parameters models to recover linear functions, but it is reassuring to notice that FI-RUMBoost, developed to discover non-linear coefficients, is still able to recover the linear functions.
We use the open-source Swissmetro bierlaire2001acceptance and London Passenger Mode Choice (LPMC) hillel2018recreating datasets for the benchmarks. The first dataset is a stated preference datasets where respondents are asked to choose between Swissmetro (a new and hypothetical underground mag-lev transportation mode in Switzerland), car, or train. We follow the same data preprocessing and model specification as in han2022neural. We keep observations where the choice is known. After this preprocessing, we have $10692$ observations from $1192$ individuals (about 9 observations per individual) that we split into 70 % for training, 15 % for validation, and 15 % for testing.
The second dataset is a revealed preference dataset obtained from the London Travel Demand Survey (LTDS) travel diary dataset, enriched with alternative-specific travel times obtained from the Google Directions API and a bespoke cost model. Possible alternatives are walking, cycling, public transport, and driving. We follow the same data preprocessing and model specification as in salvade2025rumboost. The dataset contains 81086 trips made by 31954 individuals (about 2.5 trips per individual). The first two years of observation are used as a training set that we split at the household level into 80% for training and 20% for validation.
Table (ref) summarises which variables are included in the models for the Swissmetro and LPMC datasets, as well as monotonicity constraints. Note that we assume that all alternatives are available for every individual. Table (ref) and (ref) in (ref) summarise optimal hyperparameter values obtained from the hyperparameter search for all models.
We compare the different models with their negative cross-entropy loss on a holdout test set. We include in the benchmarks two out-of-the-box ML classifiers where the utility is computed as in Eq. (ref): GBDT, using the LightGBM python library ke2017lightgbm; and DNN, using the PyTorch python library 10.1145/3620665.3640366. These models provide state-of-the-art predictive performance without accounting for inter-heterogeneity, therefore being useful to measure the impact of accounting for preference heterogeneity. The results are shown in Table (ref). Overall, all functional effects models learnt with GBDTs exhibit better predictive performance than the functional effects models learnt with DNNs on the Swissmetro dataset. FIS-GBDT performs the best, while the interpretable baselines perform the least well. We note that GBDT, a blackbox baseline, is the second best performing model, but the model has full trip feature interaction, as opposed to the functional effects models and interpretable baselines, which have no trip feature interaction. Looking at the computational time, FS-DNN and FIS-DNN are faster than the FS-GBDT and FIS-GBDT, whereas FI-RUMBoost and RUMBoost are slightly faster than FI-DNN and MNL. On the LPMC dataset, FI-RUMBoost performs best. On this dataset, FS-DNN and FIS-DNN perform better than their GBDT counterpart, whereas the baseline models perform the least well. We deduce from these results that non-linear coefficients and inter-individual heterogeneity are of greater importance in this dataset. Finally, the functional effects models learnt with GBDTs are faster than the ones learnt with DNNs, and all models are faster proportionally to the number of functional effects in the model.
Due to the large number of parameters and model combinations across multiple alternatives for the Swissmetro and LPMC, we do not present a structured overview of the model parameters for each model in this paper. We direct the reader to salvade2025rumboost for an in-depth discussion of the FI-RUMBoost parameters on the LPMC data. For the remaining models and DNN-based models, all results are available on GitHub\footnote{https://github.com/big-ucl/functional-effects-model}.
We summarise key findings for the model coefficients and functional effects as follows:
For this case study, we use data from the Survey of Health, Ageing and Retirement in Europe (SHARE), a longitudinal study of older people's health across 28 European countries. We use the simplified easySHARE dataset easyshare, which has been preprocessed to combine the data from the 9 waves of the study in a long table format. For this study, we follow mendorf2023prospective where we model the EURO-D measure of depressive symptoms, a scale composed of 12 depressive symptoms. A measurement of 0 indicates that the patient has no depressive symptoms and a score of 12 means that the patient exhibits all 12 depressive symptoms. This is an ordinal target variable, meaning that we need to adapt the generic multiclass methodology described in (ref). This is done using the CORAL methodology shi2023deep, with complete derivations in (ref).
We preprocess the dataset to remove missing values for the target variable and drop variables with more than 10% missing values. We further remove observations that would still have missing values. We drop wave 3 and wave 7 observations as the survey differs from other waves. We encode all categorical variables with dummy variables, where we normalise one category to 0. After preprocessing, we have 281975 observations from 130620 individuals (about 2.15 observations per individual). We split the data at the individual level to avoid data leakage, keeping 20% for the hold-out test set and 80% for training. We further split (also at the individual level) the training set into 80% for training and 20% for validation, to perform a hyperparameter search for both functional effects models with GBDTs and DNNs. Table (ref) in (ref) summarises the optimal hyperparameter values from the hyperparameter search. Table (ref) summarises and provides a brief description of all variables used in the model.
We report the performance of the 8 different models described in (ref) with respect to the MAE, EMAE, and MCEL on the holdout test set (see (ref) for complete derivations of the metrics). Table (ref) shows all these results.
On average, all models predict the number of depressive symptoms between $1.368$ to $1.421$ away from the observed measurement. We observe that all models accounting for preferences perform better than models without any inter-individual heterogeneity. FI-RUMBoost has the best MAE, whereas FI-RUMBoost, FS-GBDT, FIS-GBDT, and FS-DNN exhibit the best MCEL and EMAE. We remark that the metric difference between models with functional effects is minimal. Since we perform only a single train--validate--test split for computational reasons, it is difficult to conclude that one model is better than another. In terms of computational time, learning functional effects with DNNs is about 3 to 16 times faster than with GBDTs. Finally, we also compare the estimation of the ordinal thresholds for all models in (ref). It is reassuring to observe that all values are in the same range, with minor differences.
The primary advantage of using ML regressors in a constrained setting, such as functional effects models, compared to out-of-the-box ML classification models, is that they are interpretable. When trained without functional slopes, we can observe the traditional parameters as in an Ordinal Logit from a functional effects model with linear coefficients and the non-linear utility function from functional effects models with non-linear coefficients. We can also observe the distribution of the individual-specific functional intercept and functional slope values as histograms.
Figure (ref) shows the linear and non-linear effects from the four models without functional slopes. Overall, we observe that the linear and non-linear utility output from the functional effects models exhibit the same trends. An increase in the number of conditions, number of doctor visits, fine motor difficulties, large muscle difficulties, mobility difficulties, daily activity difficulties, instrumental activity difficulties, if the respondent has been hospitalised or temporarily in a nursing home, and if the self-perceived health is not excellent (reference category) increases the likelihood of having more depressive symptoms. On the other hand, a higher BMI, better gross motor difficulties, being able to recall more words, having a better max grip strength, and living permanently in a nursing home decrease the likelihood of having more depressive symptoms. We also observe that the non-linearity provides benefits for: the number of doctor visits, which is logarithmic; the BMI, which exhibits a plateau; and the mobility difficulties, which is a parabola.
We can also visualise histograms of the functional intercept and slopes of the six models having functional effects. We show the functional intercept in Figure (ref), and we select the functional slopes of the max grip strength variable to analyse in Figure (ref). Other figures are in (ref). We observe that all functional intercepts learnt with GBDT are unimodal, mostly normally distributed, whereas the functional intercepts learnt with DNN seem bimodal. It is interesting to see how the functional intercept learnt with DNN has a slightly bigger standard deviation than the one learnt with GBDT when the model is also trained with functional slopes. We also observe that all distributions have a positive mean. For the max grip strength functional slopes distribution, we mostly observe an extreme value distribution with a negative mean for all models. For most individuals, an increase in max grip strength is negatively correlated with a higher number of depressive symptoms. It is similar to what was observed for the models without functional slopes, but with more nuances, since we can observe that some individuals have a positive slope, meaning that an increase in the max grip strength is positively correlated with more depressive symptoms.
In this paper, we have adapted and applied existing methodologies to panel data, effectively learning functional effects from socio-demographic characteristics. This allows for modelling inter-heterogeneity in an interpretable ML framework, outperforming out-of-the-box and interpretable ML choice models that would not account for preferences. In addition, we have extensively benchmarked 8 out of the 11 possible functional effects models, showing how they can account for preferences of unobserved individuals, a limitation of traditional choice models such as the fixed and mixed effects models. By doing so, we make the implicit assumption that the choices and tastes of individuals are inherently linked with their socio-demographic characteristics. We note that it is important to review the socio-demographic characteristics included in the model to stay on ethical grounds and be sure that no specific strata of the population are being discriminated against. Interestingly, our approach captures the relationship of individuals at a given point in time. This means that socio-demographic characteristics can change, also adapting the functional effects that these individuals are experiencing. This is likely to be the case for real-life situations, where, for example, having a higher monthly income can change the life experience and, therefore, unobserved factors influencing the decision process. In addition, we have applied the methodology to an ordinal choice problem and multiclass classification problems, but the methodology is not problem-specific and can easily be applied to regression tasks and more complex choice models. Further work includes applying eXplainable Artificial Intelligence (XAI) techniques to the functional effects distributions to map heterogeneity to socio-demographics, e.g., certain demographic groups having preferences to alternatives or being more time sensitive, and extending the parametric ML approach to dynamic effects for panel data.
Nicolas Salvadé: Writing – original draft, Visualization, Software, Methodology, Conceptualization, Formal analysis, Investigation. Tim Hillel: Writing – review & editing, Supervision, Methodology, Conceptualization, Project administration.
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
This research is supported by a Chadwick PhD Scholarship awarded to Nicolas Salvadé by the UCL Department of Civil, Environmental, and Geomatic Engineering.
This paper uses data from SHARE Waves 1, 2, 3, 4, 5, 6, 7, 8 and 9 (DOIs: 10.6103/SHARE.w1.900, 10.6103/SHARE.w2.900, 10.6103/SHARE.w3.900, 10.6103/SHARE.w4.900, 10.6103/SHARE.w5.900, 10.6103/SHARE.w6.900, 10.6103/SHARE.w6.DBS.100, 10.6103/SHARE.w7.900, 10.6103/SHARE.w8.900, 10.6103/SHARE.w8ca.900, 10.6103/SHARE.w9.900, 10.6103/SHARE.w9ca900, 10.6103/SHARE.HCAP1.100) see borsch2013data for methodological details.(1) The SHARE data collection has been funded by the European Commission, DG RTD through FP5 (QLK6-CT-2001-00360), FP6 (SHARE-I3: RII-CT-2006-062193, COMPARE: CIT5-CT-2005-028857, SHARELIFE: CIT4-CT-2006-028812), FP7 (SHARE-PREP: GA N°211909, SHARE-LEAP: GA N°227822, SHARE M4: GA N°261982, DASISH: GA N°283646) and Horizon 2020 (SHARE-DEV3: GA N°676536, SHARE-COHESION: GA N°870628, SERISS: GA N°654221, SSHOC: GA N°823782, SHARE-COVID19: GA N°101015924) and by DG Employment, Social Affairs & Inclusion through VS 2015/0195, VS 2016/0135, VS 2018/0285, VS 2019/0332, VS 2020/0313, SHARE-EUCOV: GA N°101052589 and EUCOVII: GA N°101102412. Additional funding from the German Federal Ministry of Education and Research (01UW1301, 01UW1801, 01UW2202), the Max Planck Society for the Advancement of Science, the U.S. National Institute on Aging (U01_AG09740-13S2, P01_AG005842, P01_AG08291, P30_AG12815, R21_AG025169, Y1-AG-4553-01, IAG_BSR06-11, OGHA_04-064, BSR12-04, R01_AG052527-02, R01_AG056329-02, R01_AG063944, HHSN271201300071C, RAG052527A) and from various national funding sources is gratefully acknowledged (see www.share-eric.eu). This paper uses data from the generated easySHARE data set (DOI: 10.6103/SHARE.easy.900), see gruber2014generating for methodological details. The easySHARE release 8.8.0 is based on SHARE Waves 1, 2, 3, 4, 5, 6, 7,8 and 9 (DOIs: 10.6103/SHARE.w1.900, 10.6103/SHARE.w2.900, 10.6103/SHARE.w3.900, 10.6103/SHARE.w4.900, 10.6103/SHARE.w5.900, 10.6103/SHARE.w6.900, 10.6103/SHARE.w7.900, 10.6103/SHARE.w8.900, 10.6103/SHARE.w9.900).