Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
89,805 characters · 27 sections · 57 citation commands
A Neural-embedded Discrete Choice Model: Learning Taste Representation with Strengthened Interpretability
Discrete choice model (DCM) is a robust econometric framework to understand and predict choice behaviors. With a solid foundation in random utility theory, DCM has the significant advantage of interpretability: it can explain how individuals choose among a set of alternatives and provide reliable answers to “what-if” scenario questions, which is crucial for making policy decisions.
A DCM requires making assumptions about a choice-maker's decision rules, information processing strategies, consideration sets and utility functions. Various approaches are developed to capture heterogeneity in these aspects, such as bacsar2004parameterized, hess2012allowing, vancrane2019, barseghyan2021discrete, and CRAWFORD20214. Under the most widely used random utility maximization paradigm, how to characterize taste heterogeneity in the utility specification has been a main research topic. The systematic part of a utility function describes how a choice-maker values an alternative attribute (“taste”), and how tastes vary across choice-makers (“taste heterogeneity"), usually through the interactions between alternative attributes and individual characteristics. Correctly specifying systematic taste heterogeneity can be challenging, especially when the choice problem is novel or complicated without a priori knowledge. Misspecified utility functions can lead to lower predictability, biased estimates and misinformed decisions bentz2000neural, torres2011wrong, VANDERPOL2014.
Nonlinear functions (e.g., higher-order polynomial, semi-log transform, piecewise linear) can characterize systematic taste heterogeneity in more flexible ways. However, it is unwieldy to specify such utility functions as there are many possible nonlinear transformations to be tested. Advanced DCMs, such as the Mixed Logit Model and the Latent Class Choice Model, are developed to model random/unobserved taste heterogeneity not captured by the systematic utility mcfadden2000mixed, hensher2003mixed, Gupta1994, gopinath1995. These models substantially enrich taste heterogeneity representation and improve predictability. However, they require both the systematic utility and the random error structure known as a priori. Capturing random heterogeneity alone does not resolve the bias induced by misspecified systematic utilities ben2002hybrid.
In the machine learning (ML) community, Neural Networks (NN) have become the state-of-art method in many computer vision and natural language processing applications due to superior predictability compared to theory-driven models. Neural networks are shown as a universal function approximator cybenko1989approximation, hornik1991approximation, having the ability to learn any arbitrary function from data. This empowers them to learn flexible representation from large datasets, the complexity of which can be difficult to achieve with hand-engineered features by domain experts.
Numerous studies have examined NN's predictability in the context of mobility choice and shown higher accuracy. However, a significant limitation of NN is the lack of interpretability. Interpretability is crucial for high-stakes decisions in transportation planning, such as infrastructure investment and congestion pricing. The model should represent the real relationships between explanatory variables and choice outcomes and provide reliable answers to “what-if” questions at the disaggregated level.
Recent studies utilize NN as a tool to learn flexible behavior representation within the DCM framework while keeping model interpretability vancrane2019, han2019, SIFRINGER2020236. The key idea is injecting a neural network into a DCM such that the neural network learns part of the model specification, while the base model is a DCM. We call this type of design the “Neural-embedded Discrete Choice Model" (NEDCM) han2019. SIFRINGER2020236 develop the first type of such models (L-MNL and L-NL), using neural networks to learn a data-driven part of the utility specification.
We extend SIFRINGER2020236's L-MNL to a structure called TasteNet-MNL. A neural network (TasteNet) is embedded in an MNL to learn representations of systematic taste heterogeneity. Specifically, a set of taste coefficients in a utility function are learned by a neural network as functions of individual characteristics. Taste coefficients predicted by the neural network are fed into an MNL with explicitly defined utilities. Unknown parameters of the TasteNet and the MNL are jointly estimated. We formalize and strengthen the model interpretability condition. We show that disaggregated interpretability can be achieved by a constrained network architecture and parameter transform that encodes expert knowledge. The source code is made publicly available\footnote{https://github.com/YafeiHan-MIT/TasteNet-MNL}.
We organize the rest of this paper as follows. Section 2 reviews previous NN applications to travel choice prediction, learning behavior representation, and the challenge of interpretability. Section 3 describes the Taste-MNL structure and implementation. Section 4 shows that TasteNet-MNL can recover the underlying taste heterogeneity and outperform MNLs with under-specified utilities. Section 5 applies TasteNet-MNL to the Swissmetro dataset and shows its higher predictability than MNLs and Mixed Logit benchmarks, and its interpretability at the disaggregated level. Lastly, we discuss our key contributions, limitations of the model, and future works.
Neural networks have been long treated as a black-box for travel choice prediction. The majority of these studies find the examined NN outperforms a certain type of DCM, such as MNL Kumar1995, agrawal1996market, west1997comparative, Carvalho1998forecasting, omrani2015predicting, Lee2018, LHERITIER2019198, nested logit Mohammadian2002, cantarella2005, wang2020multitask, mixed logit wang2021 and latent class logit model LHERITIER2019198. A few studies find similar prediction performance of NN in comparison to nested logit hensher2000comparison, MNL sayed2000comparison, zhao2018MLvsLogit and mixed logit zhao2018MLvsLogit. Neural network's superior predictability is attributed to its universal function approximation capacity hornik1991approximation: a NN is able to learn complex nonlinear functions, which can be under-represented in a parametric or semi-parametric DCM setting.
However, a deep neural network (DNN) with many hidden layers is prone to over-fitting and poor performance on test datasets. Several studies show that increasing the number of hidden layers does not improve predictability and can rather harm it. nam2017model shows similar log-likelihood of a DNN compared to the nested logit and cross-nested logit baselines. wong2019reslogit find a feed-forward DNN suffers from over-fitting as the number of hidden layers increases. With 4 or more hidden layers, log-likelihood on the validation dataset no longer improves and performs worse than the MNL benchmark. In addition to over-fitting, a DNN often experiences large variations across model runs glorot2011deep. In a recent study, wang2021 conduct a systematic comparison of 11 ML classifiers with 3 types of DCM (logit, nested logit and mixed logit) on 3 datasets with varying sample sizes. They find DNNs and ensemble methods achieve the highest predictability on average and outperform DCMs by 3-4%. This study shows that an overly complex DNN or DNNs with poorly tuned hyper-parameters can under-perform a DCM due to its large estimation error.
To prevent a NN from over-fitting and improve its predictability, regularization strategies such as l2 regularization and drop-out, are commonly employed for NN training. Alternatively, more constrained network architectures are proposed with domain knowledge. WANG2020234 use alternative-specific networks and find improved predictability compared to fully-connected DNNs. SIFRINGER2020236's L-MNL integrates a NN with a logit model, and uses NN to learn part of the utility. Though interpretability and representation learning is their main motivations, the more customized NN structure can be seen as a regularization strategy. Our study presents a new way to integrate NN and MNL, where NN is utilized to learn taste heterogeneity and constrained by the MNL. We also introduce parameter constraints to incorporate expert knowledge and improve predictability.
A major criticism against NN for travel demand modeling is its black-box nature and a lack of interpretability. However, the term “interpretability" has not been clearly defined or agreed upon. Interpretability has been a hot topic in the machine learning community, lipton2017mythos summarizes different motivations behind the quest for interpretability, such as trust, understandability, transparency, causality, transferability, informativeness, and fairness. He classifies the properties and techniques that render a model interpretable into two general categories: transparency - to what extent a model can be understood by human, and post-hoc interpretability - the ability to derive useful information from a model. We find this framework helpful to reflect on what kind of “interpretability" is needed for transportation applications. In the same spirit, alwosheel2020trustworthy categorizes the effort to make NN interpretable as: the ante-hoc approach - incorporating interpretability into the model structure; and the post-hoc approach - deriving useful information from models. As interpretability researches spread across many disciplines, and our study is not primarily dedicated to interpretability, our review focuses on interpretability of NNs for travel demand analysis.
The conventional view of NN as a “black-box" hensher2000comparison, karlaftis2011statistical follows the “transparency" type of interpretability: A NN offers no direct behavioral meaning from the parameters, and it is difficult for a human to understand how it makes certain predictions. In a similar vein, SIFRINGER2020236 considers only the parametric part of the utility as interpretable, and associate interpretability with obtaining unbiased estimates for the transparent part.
Perhaps a more popular view of interpretability is the “post-hoc" kind - having the ability to obtain useful information from a model zhao2018MLvsLogit, WANG2020234. Researches from early days have shown the analogy between a logit model and an FFW, and the ways to derive economic information from NNs bentz2000neural, hruschka2002flexible, hruschka2004empirical. CHIANG2006514, HAGENAUER2017273 and GOLSHANI201821 conduct sensitivity analysis to measure variable importance. WANG2020extra show how to extract a full list of economic indicators from NNs.
Though we have plenty of tools to interpret a NN, a major concern in applying NN for travel demand analysis is the credibility and consistency of its interpretations. WANG2020extra note that economic indicators aggregated over model runs make sense, but results from particular training runs or for individual observations can be unreasonable. For example, choice probability can be non-monotonically decreasing as cost increases and highly sensitive to a particular model run. The derivative of choice probabilities with respect to cost and time can be positive; and value-of-time can be negative, zero, arbitrarily large, or infinite.
Scenario analysis and policy decision in the field of transportation depends on answers to “what-if” questions at the disaggregated level, such as how individuals with specific characteristics would respond to a toll increase. Prediction performance alone does not guarantee correct answers to these questions, as a predictive NN does not necessarily learn the underlying relationships. With an emphasis on trustworthiness, alwosheel2020trustworthy frames interpretability as to what degree the NN learns intuitive and desirable relations and hence can be trusted. He adapts the method of prototypical examples used in computer vision to diagnose NNs' trustworthiness for discrete choice analysis.
We share a similar emphasis on trustworthiness in addition to post-hoc information extraction. We define interpretability for transportation applications as follows:
A model is interpretable if, at the disaggregated level, it can provide credible answers to “what will happen if” questions. We emphasize the credibility of the economic indicators and interpretability at the disaggregated (both model and choice-maker) level. By “credible”, we mean the answer should conform with a set of prior knowledge, for example, non-positive choice elasticity regarding cost and non-positive value-of-time. Prior knowledge may change over time and vary across application contexts.
The main interpretability difficulty for neural networks arises from the fact that many NNs can fit the data equally well, but not all can provide realistic behavior interpretations. To improve interpretability, we design a hybrid structure consisting of a neural network and an MNL, and add parameter constraints to reflect expert knowledge. We show that TasteNet-MNL can obtain realistic behavioral indicators at the disaggregated level, and that predictability does not necessarily come at the cost of interpretability.
Beyond the prediction focus in most NN applications, an interesting avenue of studies attempts NN to uncover the complex relationships in choice behaviors west1997comparative, Carvalho1998forecasting, bentz2000neural, Wong2018, vancrane2019. For example, Wong2018 use a restricted Boltzman Machine to capture latent behavior attributes. vancrane2019 develop a NN to learn decision rule heterogeneity.
There has been a continuing interest in learning flexible utility forms and behavior interpretations with NNs, which is the direct precedent of our study. Early studies conduct Monte-Carlo experiments to show a neural network can capture nonlinearity in utility functions west1997comparative, Carvalho1998forecasting, bentz2000neural. Non-linearity may reflect the saturation effect or threshold effect of attributes on utility, or non-compensatory decision rules. For example, west1997comparative find that NNs consistently outperform logit and discriminative analysis when predicting the outcome of non-compensatory choice rules. bentz2000neural show the analogy between NN and MNL, and NN with hidden layers as a more general version of MNL. With synthetic data and an empirical study, they show that a NN can detect interaction and threshold effects in utility, and therefore can be used as a diagnostic tool to improve MNL utility specification. This sequential approach requires manual analysis of NN results to identify the nonlinear effect, and thus applies only to simple problems. hruschka2002flexible compare a NN with an MNL and a Latent Class Logit (LCL) model in an empirical study of brand choice, and find the NN model can identify interaction effects, threshold effects, saturation effects and other nonlinear forms (like inverse S-shape) of attributes on brand utility. A follow-up study by hruschka2004empirical compares NN with two other MNLs with flexible systematic utility, and draws similar conclusions. These studies show that a NN can outperform an MNL, when the nonlinearity in attributes are neglected or mistaken. However, these studies have not addressed nonlinearity in taste. They consider NN as either an alternative to MNL; or a diagnostic tool to improve the utility specification of MNL, which works only for simple problems.
Inspired by the study by bentz2000neural, SIFRINGER2020236 propose to integrate NN and DCM, and learn utility representations with the NN. They divide the systematic utility into a “knowledge-driven” part and a “data-driven” part learned by a neural network. This structure reduces bias in estimating the knowledge-driven part of the utility, and enhances predictability while maintaining interpretability. Applying this idea to logit and nested-logit, they propose the L-MNL and L-NL structure.
We extend L-MNL to TasteNet-MNL, using a NN to learn flexible representation of taste functions. In L-MNL, the data-driven part and the knowledge-driven part of the utility are added. Variables in the two components neither overlap nor interact with each other. This additive structure can be restrictive since the unused features can interact with variables in the knowledge-driven part of the utility. A common scenario is taste heterogeneity - certain individual characteristics interacting with alternative attributes. L-MNL can be viewed as learning flexible representations of the Alternative Specific Constants (ASCs) as functions of the unused features (individual characteristics and/or attributes of alternatives), though the data-driven part of the utility it learns is no longer a “constant" per alternative, but an Individual Specific Constant per alternative.
TasteNet-MNL differs from L-MNL and traditional FFWs in three aspects. First, we allow all or a subset of taste parameters to be modeled by an FFW as a flexible function. Second, we impose constraints on taste parameters generated by the neural network according to prior knowledge, to regularize the network and obtain interpretable results. Third, we model taste parameters instead of the systematic utilities by a NN, different from the conventional application of an FFW. The key idea is to assign the more complicated or less known task (taste heterogeneity) to a NN while keeping the well-known part parametric.
Problem Setup Suppose for a choice task, each of N individuals makes a one-time choice from an individual-specific choice set $C_n$\footnote{This problem setup can be generalized to repeated choice. We choose one-time choice to simplify the notation}. Observed data for individual $n$ includes personal characteristics $\bm{z}_n$, attributes of each alternative $\bm{x_{in}} (\forall i\in C_n$), and the choice $y_n$. The conditional probability of person $n$ choosing alternative $i$ is $P(y_n=i|\bm{z_n}, \bm{x_{jn}} \forall j \in C_n)$.
If tastes are homogeneous across individuals, the systematic utility ($V_{in}$) of alternative $i$ to a person $n$ can be specified as a linear combination of $K_i$ attributes (Eqn. (ref)), where $\beta_{ki}$ represents homogeneous taste for attribute $x_{ki}$; and $\beta_{0i}$ is the alternative-specific constant for alternative $i$.
More realistically, tastes vary across individuals. A systematic utility typically includes interactions between alternative attributes and individual characteristics. Eqn. (ref) shows such an example, where $\gamma_{pqi}$ represents the interaction effect between the p-th attribute of alternative i ($x_{pin}$), and the q-th characteristics ($z_{qn}$) of individual $n$; and $I_i$ is a set of such pairwise interactions between alternative i's attributes and individual characteristics. Interaction effects are specified based on theory or hypothesis. Given numerous function forms and set of interactions, it can be challenging to test all possible scenarios.
We use a neural network to model the interactions between alternative attributes and individual characteristics. The overall TasetNet-MNL model consists of two modules: a neural network (TasteNet) and a choice module (MNL).
TasteNet models individual $n$'s taste parameters as flexible functions of this person's characteristics (Eqn. (ref)). The network takes person $n$'s characteristics ($\bm{z_n}$) as inputs, and outputs a vector of taste coefficients $\bm{\beta_n}^{TN}$. $\bm{w}$ is the weights of the network. $\bm{\beta_n}^{TN}$ correspond to all or a subset of taste coefficients. The meaning of each element of $\bm{\beta_n}^{TN}$ is determined by the utility specification in the MNL choice module. The elements of $\bm{\beta_n}^{TN}$ can be alternative-specific, such as mode-specific travel time coefficients in a travel mode choice problem.
We specify the choice module as a logit model, which can be extended to other model forms such as nested logit. The systematic utility of alternative $i$ is divided into two parts: 1) a flexible part, defined as a dot product of taste coefficients predicted by TasteNet and alternative $i$'s corresponding attributes with flexible coefficients ($\bm{x_{in}^{TN}}$, TN for TasteNet); and 2) a parametric part, a utility function $f$ with attributes $\bm{x_{in}^{MNL}}$ and characteristics $\bm{z_{n}^{MNL}}$ as inputs and $\bm{\beta_{i}^{MNL}}$ as parameters (Eqn. (ref)).
According to the condition to obtain unbiased estimates for the MNL part given by SIFRINGER2020236, inputs to the neural network should not overlap with variables entering the parametric part of the utility. TasteNet by definition only takes individual characteristics as inputs, and thus separates them from attributes entering the parametric part ($\bm{z_n^{TN}} \cap \bm{x_{in}^{MNL}} = \emptyset$). Additionally, TasteNet and MNL should not share individual characteristics ($\bm{z_n^{TN}} \cap \bm{z_{n}^{MNL}} = \emptyset$).
The extent to which we let a neural network learn the taste function is up to the modeler. In principle, when we lack confidence about how certain tastes vary among individuals, we let TasteNet learn. Each taste coefficient is either predicted by TasteNet or estimated in the parametric utility. This is not necessary but desirable for clarity. When a taste function is learned by both TasteNet and MNL, the TasteNet will learn the “residual" part of the function and should be combined with the parametric part to recover the full taste function.
Figure (ref) depicts the overall structure of the TasteNet-MNL. TasteNet learns a function mapping from individual characteristics to tastes. The MNL module combines the flexible and the parametric utility to compute the choice probability for each alternative (Eqn. (ref)) and choice likelihood. The unknown parameters to estimate are neural network weights ($\bm{w}$) and coefficients ($\bm{\beta}^{MNL}$) in the parametric utilities. They are learned jointly by maximizing the likelihood function.
TasteNet-MNL is distinguished from previous studies in several ways. First, it extends the L-MNL SIFRINGER2020236 by using a neural network to learn the interactions between characteristics and attributes. Second, different from the majority of neural network applications to discrete choice, TasteNet learns a representation of taste rather than utility. This gives us direct control over parameters that carry behavioral meanings, such as value-of-time (VOT), as they are no longer part of the parameters to estimate, but intermediate outputs to predict. Thirdly, we realize the necessity of incorporating domain knowledge to obtain interpretable results from neural networks. We introduce parameter constraints along with $l2$ regularization to combat over-parameterization, a common issue with insufficient data and a cause for large estimation variability. Avoiding over-fitting of TasteNet allows for correct learning of both the taste functions and parameters of the utility functions. Hyper-parameter search and model diagnostics should be carefully conducted to avoid over-fitting and ensure stable and interpretable results.
The rest of this section provides more details on the network architecture, parameter constraints, and estimation.
The simplest neural network structure is the feed-forward neural network (also called a multi-layer perceptron (MLP)) for TasteNet. An MLP consists of an input layer, one or more hidden layers and an output layer. Essentially, MLP is a composition of linear and nonlinear functions to map inputs to outputs.
In an MLP with 1 hidden layer of H hidden units, the k-th output of the network $\beta_k^{^{TN}}$ can be written as Eqn. (ref), where $D$ is the input dimension, H is the number of hidden units, $A^{(1)}$ is the first hidden layer's activation function, and T is the output layer's activation function. Neural network parameters $\bm{w}^{(1)}$ and $\bm{w}^{(2)}$ correspond to weights of connections from input layer to hidden layer 1, and weights of connections from hidden layer 1 to output layer.
A general MLP with $L$ hidden layers can be denoted as MLP$(L,[H_1,..,H_L],[A^{(1)}, ...,A^{(L)}],T)$. We need to specify the number of hidden layers $L$, the size of each hidden layer $H_l$, activation function for each hidden layer $A^{(l)}$, and output transform function $T$. These hyper-parameters are selected based on a model's prediction performance on a hold-out set for model development.
Over-parameterization is common for neural networks, especially when the sample size is relatively small compared to the model complexity. Adding constraint is a method to regularize a neural network. We impose constraints on taste parameters not only to improve model generalization ability; but also to ensure a reasonable range for taste parameters.
A typical constraint is on the signs of parameters. For example, the coefficient for travel time or waiting time is usually negative. We incorporate sign constraints through an output transform function $T$. For taste parameter $\beta$s with non-negative sign constraints, choices of $T$ can be the rectified linear function ($ReLU(\beta)$) or exponential function ($\exp(\beta)$). For $\beta$s with non-positive signs, choices of $T$ can be the rectified linear unit $-ReLU(-\beta)$ or $-\exp(-\beta)$. For $\beta$s without constraints, $T$ is the identity function. Such transformations redistribute the parameters to the desirable range through continuous differentiable functions, which resemble the exponential transform for scale or time coefficient commonly used in a DCM utility function.
An advantage of using a transform function for parameter sign constraints is that the constraints can be strictly kept. Other methods, such as adding a penalty for constraint violation to the learning objective, cannot enforce the constraint on unseen data.
The model is estimated by optimizing an objective function with stochastic gradient descent. The goal is to minimize a loss function, which is the average of negative log-likelihood plus a regularization term for the p-norm of neural network weights (Eqn. (ref)) to prevent the model from over-fitting.
TasteNet-MNL is trained in an integrated fashion through back-propagation using Adam, one of the most successful stochastic gradient descent algorithms for neural network training kingma2014adam. Parameters to estimate include neural network weights ($\bm{w}$) and unknown coefficients in the MNL module ($\bm{\beta}^{MNL}$).
We generate synthetic datasets with an underlying logit model, which contains higher-order interactions between characteristics and attributes in its utility functions. We compare TasteNet-MNL with benchmarking MNLs on the synthetic datasets, expecting that the TasteNet-MNL can improve predictability, reduce bias in parameter estimates, and provide more accurate behavioral interpretations, compared to under-specified MNLs.
The data generation model is a binary logit, with the systematic utility of alternative $i$ for person $n$ defined in Eqn. (ref). Explanatory variables include three characteristics: income ($inc$), full-time employment dummy ($full$) and flexible work schedule dummy ($flex$); and three alternative attributes: travel cost ($cost$) and travel time ($time$) and waiting time ($wait$) (see Table (ref) in Appendix (ref) for details). Values of the coefficients are chosen to be realistic: income has a positive effect on value-of-time (VOT) and value-of-waiting-time (VOWT), full-time workers have higher VOT and VOWT, and people with flexible schedules have lower VOT and VOWT. We intentionally include second-order interactions between individual characteristics and alternative attributes to test whether TasteNet can learn them. We assign -1 as the ground-truth cost coefficients for both alternatives, so that outputs of the TasteNet are in the willingness-to-pay space, corresponding to VOT and VOWT (Eqn. (ref)). Alternative specific constant (ASC) for alternative 0 is fixed to 0. The random component of each utility follows an Extreme Value distribution.
We create two synthetic datasets: TOY_UNCORREL with uncorrelated alternative attributes; and TOY_CORREL with a 0.6 correlation between $time$ and $wait$. Having the correlated dataset helps examine whether the neural network can learn multiple taste functions accurately and consistently when the corresponding attributes are correlated.
Each dataset contains 14,000 observations, randomly split into training (10,000), development (2000) and test (2000) sets. Details about the input data distribution and synthetic data generation are reported in Appendix (ref).
We specify three MNL benchmarks with increasing complexity in the utility specifications. MNL-I's utility functions only include first-order interactions between individual characteristics and attributes ($time$ and $wait$) (Eqn. (ref)). Compared to MNL-I, utilities of MNL-II have one additional interaction $inc*full*time$ (Eqn. (ref)). MNL-TRUE has the same utility function as the data generation model.
Figure (ref) shows the structure of the TasteNet-MNL for the synthetic data. Time coefficient ($\beta_{time}$) and waiting time coefficient ($\beta_{wait}$ is modeled by an MLP. Hyper-parameters include the number of hidden layers, the size(s) of hidden layers, type of regularizer, regularization strength, activation function for hidden layers, and output transform function.
We train a TasteNet-MNL model on the training dataset with different combinations of hyper-parameters. One hidden layer turns out to be sufficient to achieve the same accuracy as the data generation model. We vary the number of hidden units from 10 to 100. For each hidden layer size, we apply $l2$ penalty in [0, 0.0001, 0.001, 0.01]. For the hidden layer activation function, we experiment with the ReLU and Tanh function. For output transformation, we experiment with functions: $-ReLU(-\beta)$ and $-e^{-\beta}$, to impose non-positive constraint on the value of time coefficient $\beta_{time}$ and $\beta_{wait}$. For each combination of hyper-parameters, we train the model 100 times with different random initialization.
The optimal set of hyper-parameters is selected based on the lowest average negative log-likelihood ($NLL$) on the development dataset. On the TOY_UNCORREL dataset, the best TasteNet-MNL has 1 hidden layer with 60 hidden units, $ReLU$ for hidden layer activation, $-ReLU(-\beta)$ for output transformation, and $l2$ penalty 0.001. On TOY_CORREL dataset, the optimal TasteNet-MNL has 1 hidden layer with 100 hidden units, $ReLU$ for hidden layer activation, $-ReLU(-\beta)$ for output transformation, and $l2$ penalty 0.001. We choose the smallest layer-depth model with similar performances to larger models to reduce the chance of model overfitting to spurious examples. We compare MNLs and TasteNet-MNL with respect to their predictability, parameter bias, and interpretability.
Model predictability is measured by average negative log-likelihood (NLL) and prediction accuracy (ACC) on training, development and test data. NLL is the total negative log-likelihood in equation ((ref)) divided by the number of observations. The higher the NLL, the poorer the model fit and predicative power. Prediction accuracy is the percentage of correct predictions. Table (ref) and Table (ref) summarize the prediction performance of different models on the synthetic datasets.
On TOY_UNCORREL, MNL-I and MNL-II result in higher NLL (0.58, 0.5) than MNL-TRUE (0.45); and lower prediction accuracy (74%, 76%) than MNL-TRUE (78%) due to missing interaction terms. The performance gap is widened on the TOY_CORREL dataset: MNL-I and MNL-II achieve higher NLL (0.66, 0.53) compared to MNL-TRUE (0.45) and lower accuracy (71%, 75%) compared to the true model (78%). On both datasets, TasteNet-MNL's best estimation run, as well as the average of the 100 estimation runs reach the same level of predictability as MNL-TRUE. The small standard deviation suggests its prediction performance is stable and significantly higher than the under-specified MNLs.
Readers may wonder: does TasteNet-MNL discover the underlying utility function form? Is this predictability gain attributed to TasteNet's ability to learn the underlying taste functions or simply a result of over-fitting? The next section examines the taste functions learned by TasteNet and compare it with the underlying model.
Coefficients in the MNL utility functions describe the marginal effect of a unit change in the inputs. A neural network does not provide direct estimates of the coefficients. In this experiment, because we know the terms in the underlying utility functions, we can regress predicted $\beta_{time}$ and $\beta_{wait}$ for all individuals by TasteNet against these terms to compare the coefficient estimates with the ground-truth. This measures how accurately the TasteNet recovers the underlying taste functions. The alternative specific constant ($ASC_1$) is obtained from the MNL component as it is not included in the TasteNet part of the utility. Table (ref) and (ref) show the coefficient estimates and error metrics by different models for the two toy datasets.
On the TOY_UNCORREL dataset, TasteNet-MNL's parameter bias is slightly higher than MNL-TRUE, and much lower than the misspecified MNLs. The best run of TasteNet-MNL achieves slightly higher parameter errors (RMSE=0.03, MAE=0.02, MAPE=13.2%) than MNL-TRUE (RMSE=0.024, MAE=0.016, MAPE=8.1%). The average bias from 100 TasteNet-MNL runs is higher (RMSE=0.052, MAE=0.038, MAPE=23.6%) than the best run, but still lower than MNL-I (RMSE=0.09, MAE=0.05, MAPE=31.5%) and MNL-II (RMSE=0.28, MAE=0.16, MAPE=132%).
Though MNL-II's utility specification is more complete and closer to the ground-truth, its parameter bias is larger than the simple MNL-I. Adding the second-order interactions (time*inc*flex and wait*inc*flex) causes severe bias in related coefficients, such as b_time, b_time_flex, b_time_inc, b_wait, b_wait_flex and b_wait_inc. This bigger bias may not be detected by modelers as MNL-II's predictability is higher than MNL-I (Table (ref)). This suggests that unless the exact utility form is correctly specified, there is a risk of having large bias with good predictability. A neural network is able to learn taste functions from data and mitigate this risk.
With a 0.6 correlation between $time$ and $wait$, all MNLs' parameter bias increase (Table (ref)). Standard errors of the coefficients also increase. MNL-TRUE's parameter RMSE increases from 0.0238 to 0.0469, almost doubled. This is because when alternative attributes are highly correlated, the decision boundary becomes less certain.
With the correlated attributes (time and wait), readers may suspect TasteNet-MNL to suffer from less accurate and more unstable parameter estimates, because the two tastes are estimated by a shared neural network, which may interfere with each other. The best run of TasteNet-MNL's parameter bias stays almost at the same level, and the average bias increases by about 3 to 5%. There is no strong evidence of deteriorated accuracy or stability caused by interactions between the two taste functions. It seems that a linear model is more sensitive to the correlation in the input data.
Why is it? Our intuition is as follows. Although the two tastes are estimated through a shared neural network, they do not necessarily interact much with each other because 1) each taste output has its own set of weights for the connections between the hidden layer and the output node; and 2) the number of hidden nodes is not too small (restrictive). For example, when the hidden layer is fairly large, taste output 1 might be mainly connected with a subset of the hidden nodes, while taste output 2 can be mainly connected with another subset. With a wide hidden layer, there is enough flexibility to learn two taste parameters that do not interfere with each other and together optimize the objective function.The $l2$ regularization penalty, if properly tuned, can keep the most useful connections, while shrinking the rest towards zero. The weights of the network may vary much across runs (the well-known unidentifiable nature of a neural network), but the output of the network - the taste parameters are constrained by the downstream MNL utility, and therefore can remain relatively stable. Because MNLs are constrained by the overarching linear structure of the utility, there is less freedom to adapt to the changes in the shape of the input distribution. However, with the flexibility of the neural network also comes a weakness: a neural network does not perform well on out-of-distribution samples or samples with low occurrences in the training data. We will discuss this issue in the next section.
We estimate VOT and VOWT for each individual in the synthetic datasets, and compute the errors compared to the ground-truth (Table (ref)).
On the TOY_UNCORREL dataset, TasteNet-MNL obtains smaller errors in its predicted VOT and VOWT than MNL-I and MNL-II. TasteNet-MNL's mean absolute error (MAE) for VOT is 0.15\$/hr (0.8% of the true value), compared to MNL-I's 1.7\$/hr (11%) and MNL-II's 0.8\$/hr (4.8%). TasteNet-MNL's MAE for VOWT is 0.6\$/hr (2%) compared to MNL-I's 4.8\$/hr (17.7%) and MNL-II's 2.3\$/hr (7.7%).
On the TOY_CORREL dataset, both MNL-TRUE and TasteNet-MNL's VOT and VOWT's errors increase, but the error magnitudes relative to each other remain the same: TasteNet-MNL's taste error is about 3 times as big as MNL-TRUE's. TasteNet-MNL maintains its significant lead over MNL-I and MNL-II.
To further investigate the shape of the taste functions learned by the neural network compared with other models, we create a dataset with 200 individuals with uniformly distributed characteristics. Note that this input distribution is different from that of the training data. 50 individuals are in each of the four groups defined by all combinations of full-time (yes/no) and flexible schedule (yes/no). The income of individuals from each group is evenly distributed from 0 to 60\$ per hour with a step size 1.2. We plot the predicted VOTs and VOWTs against the continuous income for the four types of individuals. Figure (ref) shows the results from applying models for the TOY_UNCORREL dataset and Figure (ref) shows the results from applying models for the TOY_CORREL dataset. The ground-truth utility functions are the same between the two.
MNL-I cannot distinguish the differences in the shape of VOT and VOWT functions between the full_flex group and the nofull_noflex group, due to missing all 3 second-order interaction terms. Adding two second-order interactions to MNL-II causes larger bias, and significant changes in the slope of the VOT and VOWT functions of income for each group.
Unsurprisingly MNL-TRUE's estimated taste functions are the closest to the ground-truth, as its utility has the exact terms as the underlying utility. Comparing MNL-TRUE's taste functions between uncorrelated data (Figure (ref)) and correlated data (Figure (ref)), we notice the MNL-TRUE's VOWT function has an apparent deviation from the ground-truth due to the correlation in attributes.
TasteNet-MNL overall recovers the true shape of the taste functions for both the uncorrelated and the correlated case. However, the functions are not strictly linear. Near the limits of income, we observe larger bias and variance. This is because the majority of the training samples have income between 10 and 50 \$/hr (Figure (ref)), with only 208 (2.1%) of the 10,000 training samples beyond this range. With insufficient examples in a certain region of the input space, the neural network is unable to learn an accurate and precise taste function. This reveals a major weakness of TasteNet-MNL that it does not generalize well to out-of-distribution samples.
Despite this weakness, TasteNet-MNL has the ability to recover the underlying taste functions for the majority of the population, given that there is enough data to learn from. It can lower the risk of getting the taste functions systematically wrong without knowing the true utility form. Whether the data is sufficient depends on the complexity of the problem. An MNL model tends to generalize better to out-of-distribution samples because of the highly restricted global utility functions. However, this generalization ability is only possible if we have a correct specification of the utility functions. As the MNL-II example shows, a misspecified MNL can lead to greater bias for the out-of-distribution samples than a TasteNet-MNL does.
It is worth noting that in this synthetic data experiment, we examine a particular form of nonlinearity - higher-order interactions - for the convenience of illustration and understanding. Based on the universal function approximation property of NN, TasteNet can be employed to learn general forms of nonlinearity in taste functions.
We apply TasteNet-MNL to a publicly available dataset -- Swissmetro to model mode choice for inter-city travel. The purpose of this application is to 1) examine whether TasteNet-MNL is able to predict more accurately compared to a manually specified, relatively sophisticated MNL; and 2) whether TasteNet-MNL can draw reasonable behavioral interpretations and, if so, how its interpretations differ from those of the MNLs. To compare with TasteNet-MNL, we set up three benchmarking MNL models with increasing complexity in the utility function and random coefficient logit models.
The Swissmetro is a proposed revolutionary mag-lev underground system. To assess potential demand, the Swissmetro Stated Preference (SP) survey collected data from 1,192 respondents (441 rail-based travelers and 751 car users), with 9 choices from each respondent. Each respondent is asked to choose one mode out of a set of alternatives for inter-city travel given the attributes of each mode (e.g. travel time, headway and cost). The universal choice set includes train (TRAIN), Swissmetro (SM), and car (CAR). For individuals without a car, the choice set includes only TRAIN and SM. Table (ref) provides a description of the variables. For more information, readers can refer to bierlaire2018.
The original data has 10,728 observations, downloaded in Jan 2019\footnote{Data link: https://biogeme.epfl.ch/data.html}. After removing observations with unknown age, “other" trip purpose and unknown choice, we retain 10,692 observations. We randomly split the data into training (“train"), development(“dev") and test(“test") set with 7,484, 1,604 and 1,604 observations, respectively.
We set up four benchmarks: three MNL models and a Mixed Logit model. MNL-A is similar to Bierlaire et al.(2001)'s MNL specification but with some enhancements: 1) the value of travel time and value of headway is made mode-specific; 2) all levels of age and luggage categories are included; and 3) cost coefficients are fixed to -1.0 for directly reading VOT from time coefficients (Table (ref)). In the benchmark MNL-B, we add the interaction terms: time*age, time*income and time*purpose (Table (ref)). The third benchmark MNL-C is an MNL with all pairs of first-order interactions between characteristics and attributes (Table (ref)). This model is equivalent to a TasteNet-MNL with all taste coefficients modeled by a neural network with no hidden layer. The Mixed Logit model has a normally distributed travel time coefficient for each alternative, along with interactions between individual characteristics with travel time for each alternative. Intercepts and coefficients for heading are alternative specific and not randomly distributed. Estimated model coefficients for the benchmarking models are shown in Table (ref), (ref), (ref) and (ref).
L-MNL is not selected as a benchmark because it is a special case covered by TasteNet-MNL for the Swissmetro dataset: all unused variables in the data-driven part of the L-MNL utility are individual characteristics, so that the data-driven utility in L-MNL corresponds to the ASC taste functions estimated by TasteNet. Given a more complex dataset, we could have a unified version of L-MNL and TasteNet-MNL, whose utility consists of a data-driven component from L-MNL, a TasteNet component, and a knowledge-driven part.
The TasteNet-MNL structure for Swissmetro data is shown in Figure (ref). We specify the utility function of each alternative in the MNL module. Coefficient for cost is fixed to -1 so that the coefficient for time is the negative VOT. There are 7 coefficients in the MNL utilities, including two alternative specific constants. We assume all these coefficients (“tastes") are functions of individual characteristics and model them by TasteNet. This is a special case of the general structure: the set of $\bm{\beta}^{MNL}$ is empty and all taste parameters are modeled by TasteNet as $\bm{\beta}^{TN}$ (Eqn. (ref) and (ref)).
The TasteNet module consists of a linear layer from input $\bm{z}$ to hidden layer $\bm{h}^{(1)}$, a nonlinear activation $A^{(1)}$ for the hidden layer, followed by a linear layer from hidden layer to output layer and an output activation function $T$ for the output. We choose only 1 hidden layer, since the predicted log-likelihoods on hold-out datasets do not improve with more hidden layers. Input $\bm{z}$ includes all characteristics: age, gender, income, first-class, who pays for travel cost, trip purpose and luggage. We experiment with various sets of hyper-parameters and activation functions shown in Table (ref).
Among all TasteNet-MNL scenarios, the one with 110 hidden units, $relu$ for hidden layer activation, negative exponential for non-positive output activation, and no regularization achieves the best prediction performance on the development dataset.
TasteNet-MNL out-performs all benchmarking models' prediction accuracy. We use average negative log-likelihood (NLL) and prediction accuracy (ACC) to measure predictability. To avoid model over-fitting, we split data into training, development and test set, and present the performance on each subset. As test set is not used in model training or hyper-parameter selection, prediction performance on it provides a reliable comparison across models. The result is shown in Table (ref).
From MNL-A to MNL-C, as more interactions between attributes and individual characteristics are added, the predicted NLL is reduced from 0.755 (MNL-A) to 0.698 (MNL-C). Mixed-Logit's performance is similar to MNL-C (0.703). The best run of TasteNet-MNL (minimal NLL on development set) reduces test NLL from 0.698 to 0.645, and improves ACC from 0.678 to 0.703 and F1 score from 0.574 to 0.62. The mean and standard deviation of each predictability metric from 100 runs also shows consistent improvement: NLL reduced to 0.652, ACC improved to 0.7 and F1 score improved to 0.62. The improved predictability is attributed to the fact that TasteNet learns a more complex representation of systematic taste heterogeneity.
Since interpretability is a primary concern for transportation application, we need to investigate whether the interpretations make sense. We show several ways to diagnose. As TasteNet-MNL are direct extensions to MNLs, and Mixed Logit does not show superior predictability, we focus on the comparisons between MNLs and TasteNet-MNL's interpretations here and in the following sections.
First, we compare the average taste estimations for observations in the Swissmetro dataset by different models. We apply each model to obtain individual estimates, and compute the averages (Table (ref)). Since the cost coefficients are fixed to -1.0, all taste coefficients are in the willingness-to-pay (WTP) space measured by Swiss Franc (CHF).
We find that from MNL-A to MNL-C, average values of travel time (VOT) and values of headway (VOHE) increase with more interaction terms being added to the utility function. For example, train VOT increases from 1.34 to 1.85 CHF per minute. Swissmetro VOT increases from 1.4 to 1.51 CHF per minute, and car VOT rises from 1 to 1.35 CHF per minute. Both MNL-B and MNL-C suggest that VOT of train is higher than that of Swissmetro or car. MNL-C also gives higher average VOHE for both train and Swissmetro than MNL-B or MNL-A.
TasteNet-MNL gives the largest average VOT and VOHE among all models (Table (ref)). Its average VOT estimates for train, swissmetro and car are 26%, 17%, and 24% higher, respectively than those predicted by MNL-C. Its average VOHE estimates for train and swissmetro are 25% and 67% higher than those estimated by MNL-C.
We further investigate where the higher average VOTs come from. We plot histograms for each taste parameter and for each model (Figure (ref)). As interactions are incrementally added from MNL-A to MNL-C, the model captures more taste variations. Compared to MNLs with linear utilities, TasteNet-MNL discovers a wider range of taste variations. In particular, the VOTs and VOHEs for all travel modes have longer tails on the high end of WTP. The last row of histograms (Figure (ref)) shows the mean and standard deviation from 100 runs. Overall, we see a consistent pattern in the distributions, with different amount of variability across estimation runs. There is greater uncertainty in Train's VOT estimates, and Train and Swissmetro's ASC estimates. The bigger uncertainty in Train's VOT (compared to Car or Swissmetro's) coincides with the larger standard deviation of the randomly distributed travel time coefficient for Train (1.33), compared to the standard deviation of 0.62 for Swissmetro and 0.35 for Car (Table (ref)).
Based on the synthetic data experiments, we have confidence to believe that TasteNet-MNL's better predictability is due to more accurately capturing the underlying taste functions. However, readers should be aware of the risk of over-fitting TasteNet to outliers, especially when dealing with a noisier dataset or using more complex neural networks. We recommend conducting a thorough interpretability check at both the aggregate and the disaggregate level, and comparing the results against the benchmarking models to spot any spurious behaviors. The next two sections provide additional ways to check.
An interpretable model should provide a reasonable function relationship between choice outcome and input at the disaggregate level. We propose a diagnostic tool for model interpretability: visualizing the taste function.
Each model provides a taste function that maps individual characteristics to a type of taste value (e.g. VOT, VOHE). We want to check whether TasteNet-MNL learns sensible taste functions at the individual level, in comparison to benchmarking MNLs.
Since function input $\bm{z}$ is multi-dimensional, we cannot directly visualize the functions. Instead, we pick an individual with characteristics $\bm{z}$. We vary one dimension of $\bm{z}$: $z_i$, while keeping other dimensions fixed $z_{j\neq i}$. We plot a particular taste parameter as a function of $z_i$: $\beta_k = f_{model}(z_i; z_{j\neq i})$.
For example, we pick a person with characteristics shown in Table (ref). We vary this person's income and ask each model a question: what are the VOTs for such a person as his income varies? We compare the answers given by different models. Figure (ref) shows the VOTs and VOHEs estimated by different models versus income with other characteristics fixed.
Compared with the benchmarking MNLs, VOT and VOHE estimated by TasteNet-MNL all fall within credible ranges. TasteNet-MNL gives more or less different estimates. Swissmetro VOT estimates are not very different between TasteNet-MNL and MNL-C. Regarding train VOT, TasteNet-MNL gives smaller estimates for all three income groups than MNL-C. Car VOT estimated by TasteNet-MNL is higher for higher-income groups and lower for the lowest income group. With respect to VOHEs, TasteNet-MNL gives higher estimates for train and lower estimates for Swissmetro for all income levels. MNL-C shows a monotonic relationship between VOT and income only for train VOT, while TasteNet-MNL identifies the monotonicity for swissemtro VOT and car VOT.
As we do not know the ground truth, the interpretability and credibility of the model inevitably depend on expert knowledge and judgment. We draw many individual cases, visualize the taste functions, and compare across models. Overall, taste parameters by TasteNet-MNL fall within similar ranges as the MNLs do. Yet a particular taste of a specific individual given by TasteNet-MNL can agree with or differ from the MNLs. Based on TasteNet-MNL's better predictability, we trust that it obtains more accurate taste estimates for individuals.
Elasticity is another aspect to check model interpretability and catch abnormal neural network behaviors. We first apply each model to calculate disaggregate point elasticity of Swissmetro mode choice with respect to Swissmetro travel time for each observation. The point elasticity equation for MNL models is shown in Eqn. (ref) and for TasteNet_MNL in Eqn. (ref). TasteNet-MNL's individual elasticity estimates differ from MNL-C by 0.2287 on average.
With individual elasticities, we then compute aggregate elasticity, which measures a group of decision-makers' response to an incremental change in a variable. This is defined in Eqn (ref) as the percentage change in the expected share of the group choosing alternative $i$ ($W_i$) with respect to one percentage change in variable $x_{ki}$. It is equivalent to a weighted average of the individual elasticities using the choice probabilities as weights.
The aggregate elasticities of Swissmetro mode share with respect to Swissmetro travel time are -0.43, -0.45, and -0.41 for MNL-A, MNL-B and MNL-C, compared to -0.437 for TasteNet-MNL. We further compare aggregate elasticity by group, such as income (Table (ref)). TasteNet-MNL suggests higher elasticities for low income and high income groups than MNL-C. But overall, TasteNet-MNL gives choice elasticities close to MNLs and within reasonable range. We recommend performing this experiment systematically against a dummy model (e.g. MNL-C) as a sanity check.
In this paper, we utilize a neural network to learn taste representation while keeping model interpretability. Departing from a traditional either-or approach, we integrate neural networks and DCMs to take advantage of both.
With a synthetic dataset, we show that TasteNet-MNL can learn the underlying taste function. It achieves the same level of accuracy as the true model in predicting choice and deriving economic indicators. Exemplary MNLs with misspecified utility result in parameter bias and lower prediction accuracy. On the Swissmetro dataset, TasteNet-MNL not only surpasses MNLs and Mixed Logit model benchmarks in predictability, but also achieves interpretable economic indicators: individual-level VOTs and elasticities from TasteNet-MNL are comparable to the results of the benchmarking MNLs. Moreover, TasteNet-MNL discovers a greater range of taste variations in the population than the benchmarking MNLs. The average VOT estimates by TasteNet-MNL are higher than the MNLs'.
We demonstrate the benefit of integrating neural networks and DCM. Neural networks can learn complex functions from data and reduce bias in manual specifications. The theory behind a DCM can guide the neural networks to output meaningful results. A high-level idea behind TasteNet-MNL is to assign the more complex or unknown part of the model (e.g. taste heterogeneity) to a neural network, and keep the well-understood part (e.g. the trade-offs between alternative attributes) parametric. This idea can be applied to other settings. For example, a neural network can be used to model class membership in a latent class choice model.
TasteNet-MNL is most suitable for a fairly large dataset (thousands of examples or more) with a rich set of individual characteristics. Modelers can compare TasteNet-MNL against MNL benchmarks provided by human experts to understand the predictability gap and existence of misspecified taste heterogeneity. Modelers should apply proper regularization strategies in order not to over-fit the model. Based on our findings from the synthetic data experiments, TasteNet-MNL yields large estimation errors on out-of-distribution samples. Users should be aware of this risk and careful when applying the model to samples near or outside the training data's input distribution. We recommend estimating the models for multiple times (e.g. 100) with different initialization to measure the uncertainty in estimates; and conducting a systematic sanity check of the behavioral indicators (e.g. value of time, elasticity) against baseline models to spot spurious behaviors.
There are limitations with our approach and open questions for future research. First, the TasteNet-MNL model only accommodates systematic taste variations. Random taste heterogeneity is an important source of heterogeneity. How to model distributions of taste parameters with neural networks is an intriguing question for future research. han2019 proposes a neural network embedded latent class choice model, as one way to represent random heterogeneity. Future work can develop a neural embedded continuous mixed logit model.
Second, TasteNet-MNL focuses on modeling taste heterogeneity. Non-linear effects of attributes have been observed empirically Monroe1973, gupta1992discounting, kalyanaram1994. Nonlinear effects, such as the saturation effect and threshold effect, are explained by prospect theory and assimilation-contrast theory kahnemann1979prospect, winer1986, winer1988. Future work may extend TasteNet-MNL model to reflect nonlinearity in attributes.
Lastly, we suggest comparing TasteNet-MNL with DCMs under various empirical settings, in terms of prediction performance and behavioral interpretations. We find in the Swissmetro case study that TasteNet-MNL yields higher average VOTs and a greater variety of tastes than MNLs. Future research may test if this finding holds for other datasets, and understand the practical implications for demand forecast and scenario analysis.
Acknowledgements This work was supported by the New England University Transportation Center at MIT [grant number: DTRT13-G-UTC31; funding agency: USDOT/RITA.]