Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
93,250 characters · 21 sections · 64 citation commands
The Transfer Performance of Economic Models
\pagenumbering{gobble}
\thispagestyle{empty}
\pagenumbering{arabic} \setcounter{page}{1}
Economists routinely make predictions in environments where data is unavailable, relying on evidence from related but distinct contexts. For example, an economist at a development agency may need to predict diffusion of microfinance takeup in one Indian village given data on diffusion in others. Or an economist at an insurance company may need to predict willingness-to-pay for certain insurance plans given data on willingness-to-pay for others.
Traditionally, economists address this challenge by estimating economic models on existing data and using the estimated parameters to make predictions in new domains. But the predictive success of machine learning algorithms in several economics applications HartfordWrightLeytonBrown,Hofmanetal,banerjee2023selecting raises the question of whether the economic model is essential in this process, or whether predictions would be improved by training and porting a flexible machine learning algorithm instead. This is an empirical question: On the one hand, machine learning methods are capable of uncovering novel patterns that existing models miss FudenbergLiang,Petersonetal,LudwigMullainathan; on the other, many researchers believe that structured economic models capture fundamental regularities that generalize more reliably across domains CoveneyDoughertyHighfield,Athey2017,Beery_2018_ECCV,Manski2021. Understanding whether economic models do in fact transfer better across domains is important to understanding their future role within economic analysis and policymaking.
Our paper provides a conceptual framework that formalizes the “out-of-domain” transfer problem, and allows systematic comparison between economic modeling and machine learning algorithms. Relative to the large body of work on external validity described in Section (ref), our out-of-domain test considers performance within a class of prediction problems rather than conditioning on a particular realization of the data available to the economist. For example, instead of assessing whether a structural model of information diffusion will transfer well from a specific Indian village to another, we ask whether the structural model transfers well “in general” across Indian villages. Formally, we consider an ex-ante perspective in which the economist is aware of a set of domains that are relevant (such as Indian villages) but does not know which specific transfer problem will realize from this class of possible transfer problems.
Our main theoretical contribution is the construction of finite-sample forecast intervals that characterize how well a model or algorithm performs in new domains.\footnote{ We use the term “forecast interval,” rather than “confidence interval,” to reflect the random nature of the target, namely the realized (rather than expected, median, etc.) transfer error, but they can also be viewed as confidence intervals for these random targets.} In the main text, we provide intervals that are valid for a benchmark setting where all domains are equally likely to realize for training and testing (corresponding to an assumption that domains are exchangeable) as well as for a setting in which the testing domain is qualitatively different from the training domains. (Appendix (ref) generalizes our approach and results even further.) The definition of “performance” is left to the user, and our results apply for a large class of measures that includes our motivating example of predictive accuracy, as well as others, such as how well a qualitative finding based on parameter estimates generalizes across domains.
Finally, we use our framework to compare the generalizability of economic models and black box machine learning methods when predicting certainty equivalents for lotteries. While economic models of risk preference have been studied extensively from the perspectives of how well they fit economic data HarlessCamerer1994,HeyOrme1994,bruhin2010risk,bernheim2020empirical and how their predictive performance compares to that of machine learning algorithms PeysakhovichNaecker2017,Plonskyetal,FKLM, the existing tests have primarily been within the context of a single domain.\footnote{An important exception is Einavetal2012, which examines how general risk preferences are across different domains of choice under uncertainty.} We offer a new perspective on how well these models transfer across domains, finding that although machine learning algorithms outperform the economic models when trained and tested on (disjoint) data generated under the same conditions, the economic models generalize better. Our analysis suggests the primary reason for this is not that the machine learning algorithms overfit, but rather that economic models of risk preference are more effective at extrapolating from observed cases to new ones.
We now describe the main parts of the paper in more detail. Section (ref) describes our conceptual framework, which extends the familiar notion of out-of-sample evaluation to out-of-domain evaluation. In the standard out-of-sample test, a model's free parameters are estimated on a training sample, and the predictions of the estimated model are evaluated on a test sample, where the observations in the training and test samples are disjoint but drawn from the same distribution. We depart from this framework by supposing that the distribution of the data varies across a set of “domains,” e.g. different subject pools or choice frames.
We adopt the perspective of an external analyst who wants to assess the efficacy of a procedure that a researcher uses to make predictions in a new domain. For example, the researcher's procedure might be to estimate an economic model on data from one domain and use the estimated model to make predictions in another. Alternatively, the researcher's procedure might involve pooling data across domains for estimation, or instead training and porting a black box algorithm. The procedure's exact performance depends on the domains that are used for estimation---the training domains---and the domain that is realized for prediction---the target domain. The analyst's goal is to develop a forecast interval for the random performance of this procedure across the many possible realizations of those domains. This forecast interval can then be used to compare procedures for generalization.
Section (ref) constructs forecast intervals for the baseline setting where the distributions governing different domains are exchangeable. This means that while there may be ex-post differences between the domains on which the model is estimated and on which it is applied, these differences are not ex-ante known to the analyst. We propose the following protocol: The analyst first collects data across as many domains as possible, henceforth the analyst's meta-data set. The analyst then repeatedly splits these into training and test domains, and runs the researcher's procedure for each of these domain realizations. For example, if the researcher's procedure involves transferring an economic model, then the analyst would estimate the researcher's model on the training domains and use it to make predictions in the test domain. Intuitively, the transfer performance between domains in the analyst's meta-data set can serve as a proxy for the transfer performance to the new (unobserved) target domain. Pooling these transfer performances across different choices of training and test domains yields an empirical distribution of transfer errors. We show that the quantiles of this distribution can be used to compute valid finite-sample forecast intervals for the transfer error on the target domain. (Because in most applications, models are estimated on data from a small number of domains, we exclusively consider finite-sample results in this paper.)
Section (ref) extends our methods for the scenario where the training domains are known to be systematically different from the target domain. Specifically, we suppose that the training domains are exchangeable, but allow the target domain to be governed by a qualitatively different distribution. We fully generalize our procedure and results when the analyst either knows or can bound a likelihood ratio relating the target and training distributions. We further develop two partial orders for comparing how well models generalize in environments where the analyst lacks any knowledge at all about the likelihood ratio. Though this ordering is inherently incomplete, we demonstrate that it can be used to compare economic models and black box algorithms in our subsequent application. Each of these procedures (and all other methods described in this paper) are implemented in an R package (transferUQ), available on Github.\footnote{\href{https://github.com/lihualei71/transferUQ}{https://github.com/lihualei71/transferUQ}}
Our statistical framework and results extend the recent literature on conformal inference vovk2005algorithmic, tibshirani2019conformal, lei2021conformal and randomization inference ritzwoller2024randomization by replacing the assumption of (weighted) exchangeable observations with that of (weighted) exchangeable domains. Importantly, while conformal inference aims to predict a single data point from other data points, our goal is to generate forecast intervals on a function of multiple data points, involving both observed and unobserved data. Our problem thus involves a multi-dimensional structure that falls outside of the standard framework and requires substantively new arguments.
Section (ref) applies our procedures to compare the transferability of economic models and black box algorithms for predicting certainty equivalents for binary lotteries. In this application, we evaluate how well a model (or algorithm) predicts certainty equivalents reported by one subject pool when estimated on data from another. We evaluate two models of risk preferences (expected utility and cumulative prospect theory) and two popular black box machine learning algorithms (random forest and kernel regression). We find that although the black box algorithms outperform the economic models out-of-sample when trained and tested on data from the same domain, the economic models generalize more reliably across domains. Specifically, while the forecast intervals for the black box algorithms and economic models overlap, the forecast intervals for the black box methods are wider, and their upper bounds are substantially higher.
Why do the black boxes perform worse at transfer prediction? A natural explanation, based on intuition from conventional out-of-sample testing, is that black boxes are very flexible and hence learn idiosyncratic details that do not generalize across subject pools. But when we restrict the analysis to a subset of samples involving the same set of lotteries, the economic models and black box algorithms have nearly indistinguishable forecast intervals. Alternatively, when we define the domains so that they involve disjoint lotteries, the improvement of the transfer performance of economic models over black box algorithms is even larger than in our main analysis. This tells us that black box algorithms are not universally worse at transfer, but rather their relative performance depends on specific characteristics of the transfer problem. In particular, black boxes seem to transfer worse when the primary source of variation across samples is a shift in the marginal distribution over features (i.e., which lotteries appear in the sample), rather than a shift in the distribution of outcomes conditional on features (i.e., the distribution of certainty equivalents given fixed lotteries).
Our results apply for a broad class of definitions of transfer error, but our primary motivation is evaluating how well an economic model estimated on data from one domain predicts in another. Hofmanetal gives an in-depth argument for why this important, calling for more work on the question “how well does a predictive model fit to one data distribution generalize to another?" for social science models. This is exactly what we consider.
Predictive accuracy has long had a central role in experimental economics.\footnote{As discussed by HarlessCamerer1994, the poor predictive performance of expected utility theory was a primary motivation for the development of alternative models in behavioral economics. Both HarlessCamerer1994 and HeyOrme1994 provide early assessments of alternative theories on the basis of predictive performance.} While models have often been assessed based on how well they fit data from a given domain, recent papers also examine how well predictions transfer across domains. For example, kuzmics estimates various game-theoretic models on $2\times 2$ normal-form games and evaluates their predictive performance on $3 \times 3$ normal-form games; Natenzon2019 estimates discrete choice models on data for four choice menus and evaluates their predictive performance on a fifth menu; and fudenberg2024predicting evaluates the cross-domain predictive accuracy of models of how players learn in the infinitely-repeated prisoner's dilemma, where the domains are various sets of payoff matrices for the stage game. This paper provides a general framework that nests these transfer problems, and develops formal statistical results for assessing transfer performance.
Transfer prediction is also an important consideration when comparing economic models with black box algorithms. Several papers have compared the predictive performance of machine learning algorithms with that of more structured economic models in out-of-sample tests PeysakhovichNaecker2017,Notietal,Plonskyetal,CamererNaveSmith,FudenbergLiang,Kandori,Ke, concluding that machine learning algorithms are more predictive. In the application we consider, black box methods turn out to be less effective at transferring predictions across domains. This result joins gechter2019evaluating's finding that structural models deliver better policy recommendations for conditional cash transfer policies in new contexts than black box methods do. We hope that our methods will be used to assess the transferability of economic models in other problems as well, contributing to a more comprehensive picture of how well economic models generalize.
Finally, our theoretical framework and results lie at the intersection of several literatures in economics, computer science, and statistics. These literatures consider several related but distinct objectives: synthesizing evidence across different domains, improving the quality of extrapolation from one domain to another, and quantifying the extent to which insights from one domain generalize to another.
The first objective, synthesizing results across different domains, is a particular focus of the literature on meta-analysis CardKrueger,dellavignapope,BandieraFischerPratYtsma,ImaiRutterCamerer,Vivalt. Our goal is instead to assess the cross-domain forecast accuracy of a model. These problems are related, and Meager2019 and Meager2022 in particular provide posterior predictive intervals for new domains in the context they consider. Unlike our approach, the predictive intervals reported in those papers are valid only under a parametric model for the distribution of effects across domains and a distributional assumption on the domain-specific effect estimates.
The second objective is to extrapolate results from one domain to another. Within computer science, the literature on domain generalization (DG1 and DG2) develops models that generalize well to new unseen domains DGSurvey. Similarly, several papers within economics (e.g., Hotzetal2005 and DehejiaPopElechesSamii) use knowledge about the distribution of covariates to extrapolate out-of-domain. Our focus is not on developing new models or algorithms with good out-of-domain guarantees, but rather on developing forecast intervals for the out-of-domain performance of models and algorithms that are used in practice.
The third related objective in the literature is to understand the extent to which results obtained in one domain hold more generally, i.e. their external validity. Unlike papers that assess the generalizability of insights from randomized control trials Deaton,Imbens2010,dellavignalinos or laboratory experiments LevittList,AlUbaydliList, here we consider generalizability across random domains.\footnote{Another set of papers study the generalizability of instrumental variables estimates AngristFernandezVal,BertanhaImbens and causal effects PearlBareinboim2014,Tafti.} Section (ref) and Appendix (ref) relaxes the exchangeability assumption to allow bounded deviations from exchangeability; our results there connect to the literature on sensitivity analysis rosenbaum2005sensitivity, aronow2013interval, AndrewsOster, sahoo2022learning.
Consider a fixed procedure for extrapolating predictions across domains, e.g., estimating a structural economic model on data from one domain and using the estimated model to make predictions in another. We adopt the perspective of an external analyst who wants to evaluate the effectiveness of this procedure. The analyst is not focused on extrapolation from one specific domain to another (e.g., from an American dataset to a German dataset), but would rather like to understand whether the procedure generally performs well across a class of transfer tasks (e.g., extrapolating across countries). To this end, the analyst evaluates transfer error from an ex-ante perspective without knowing which domains are used to estimate and evaluate the model. The analyst seeks to construct forecast intervals for the procedure's error when transferring from a (random) set of training domains to a new (random) target domain. This speaks to the question of whether one procedure for extrapolation (such as transferring an estimated economic model) generally performs better than another (such as transferring a trained black box algorithm).
This section proceeds as follows: Section (ref) describes the extrapolation procedure that the analyst would like to evaluate, and a large class of measures for the procedure's transfer error. Section (ref) formalizes the analyst's problem.
Let $\mathcal{X}$ be a set of covariate vectors and $\mathcal{Y}$ be a set of outcomes. An observation is a pair $(x,y) \in \mathcal{X} \times \mathcal{Y}$, and a sample is a set of observations $S = \{(x_i,y_i)\}_{i=1}^m$. We consider samples $S_d$ indexed to domains $d=1,2,\dots$, such as in the following examples:
For now we take these samples as given; Section (ref) lays out the underlying statistical model for how these samples are generated, which we will use to prove our results.
A researcher observes samples from some set of training domains $d \in \mathcal{T}$, and uses these samples $S_{\mathcal{T}} \equiv (S_d)_{ d\in \mathcal{T}}$ to make predictions in a new (yet unseen) target domain $d^*$. We will refer to $S_{\mathcal{T}}$ as the training samples and to $S_{d^*}$ as the target sample.
The number of training domains $r\equiv \vert \mathcal{T}\vert$ is a parameter of the research procedure, and should reflect what is done in practice. In economics, it is common to transfer quantitative conclusions from a single domain to another, e.g., for parameter calibration in structural models Greenwoodetal1997, McKayetal2016, Oswald2019 and extrapolation of treatment effect estimates beyond the experimental population mogstad2018identification, tipton2018review, cattaneo2021extrapolating, maeba2022extrapolation. In this case $r=1$, and the relevant question is whether extrapolating from one sample leads to good predictions in the new domain. If instead data is gathered from $r>1$ different domains and the observations are aggregated and used to estimate a model (as in the meta-analyses of Meager2019,Meager2022), the relevant question may be how well the estimated model on the aggregated data generalizes to a new domain, and $r>1$ is appropriate.
Our results apply to the following class of measures for transfer performance.
Our leading examples are transfer errors that measure how well a fixed model or algorithm transfers across domains. That is, suppose the training samples $S_{\mathcal{T}}$ are used to select a prediction rule $f_{S_{\mathcal{T}}}: \mathcal{X} \rightarrow \mathcal{Y}$, e.g., by estimating a parametric model or by fitting a black box algorithm.\footnote{That is, let $\mathcal{S}$ denote the set of all finite sets of finite samples, and let $\mathcal{Y}^{\mathcal{X}}$ be the set of all prediction rules. Then a “model” is a mapping $\rho: \mathcal{S} \rightarrow \Delta(\mathcal{Y}^{\mathcal{X}})$ and we write $f_{S_{\mathcal{T}}} = \rho(S_{\mathcal{T}})$ for the realized prediction rule.} The accuracy of the prediction rule is evaluated using a loss function $\ell: \mathcal{Y} \times \mathcal{Y} \rightarrow \mathbb{R}_+$, where \[e(f,S) = \frac{1}{\# S} \sum_{(x,y) \in S} \ell(f(x),y)\] denotes the average loss when using $f$ to predict $y$ given $x$ for observations $(x,y) \in S$. One specification of a transfer error is then
i.e., the raw error of the model when it is estimated on the training samples and used to predict outcomes in the target sample.
Any normalization of equation ((ref)) with respect to a function of the target sample is also a transfer error. For example, we might normalize ((ref)) with respect to the in-sample error of the model when trained on the target sample,
This quantity reveals how much less accurate the model is than if it had been directly trained on the target sample.
Although we focus on the transfer errors defined in ((ref)) and ((ref)), Definition (ref) is substantially broader. Appendix (ref) describes several other specifications of transfer errors, including the stability of parameters and errors in counterfactual predictions.
We now consider the perspective of an external analyst, who would like to evaluate the transfer guarantees of the procedure described above. Rather than assessing the transfer error $e_{\mathcal{T},d^*}$ for a specific set of training domains $\mathcal{T}$ and target domain $d^*$, the analyst considers a random version of this quantity, where the samples used for training and evaluation of the model are not yet known. In Example (ref), this corresponds to an analyst who is interested in how well the CRRA model transfers across arbitrary locations, as opposed to from one specific location to another.
Formally, the analyst has access to metadata consisting of $n$ samples \[\bold{M} = \{S_1, \dots, S_{d}, \dots S_n\}.\] We assume that $n>r$; that is, the analyst can collect a larger number of samples than were used by the researcher. The analyst models the researcher's set $\mathcal{T}$ of $r$ training domains as drawn uniformly over all subsets of $\{1, \dots,n\}$ of size $r$. We use $\bold{T}$ to mean the random variable whose realization is $\mathcal{T}$, so that $S_{\bold{T}}=(S_d)_{d\in \bold{T}}$ is the researcher's (random) vector of training samples. The target domain (on which predictions will be made) is a final sample $S_{n+1}$, which, unlike the metadata, is not observed by the analyst. The quantity of interest is $e_{\bold{T},n+1}$, i.e., the random transfer error when the researcher extrapolates predictions from $S_{\bold{T}}$ to $S_{n+1}$. Figure (ref) depicts this transfer error for the model transfer specification of ((ref)).
The analyst's goal is to develop forecast intervals for the transfer error $e_{\bold{T},n+1}$, i.e., interval-valued functions of the meta-data $\bold{M}$ which cover $e_{\bold{T},n+1}$ with the prescribed probability, regardless of the distribution $\mu$ that governs samples across domains. Section (ref) provides these intervals for a baseline setting in which domains are exchangeable, and Section (ref) generalizes them to a setting in which the observed domains are systematically different from the target domain.
We begin with a baseline setting in which the different domains are governed by exchangeable distributions. Specifically, we suppose that in the analyst's statistical model, the samples $S_1,S_2, \dots$ are generated in the following way:
In Examples (ref)-(ref), this assumption implies that (from the analyst's perspective) the subject pools, choice frames, or choice menus differentiating the samples are themselves drawn i.i.d. from a fixed distribution. Assumption (ref) is standard in conformal inference vovk2005algorithmic, permutation testing romano1990behavior, and randomization inference ritzwoller2024randomization.\footnote{It can also be understood as a Bayesian hierarchical model Meager2019, Meager2022 or a version of cluster sampling liang1986longitudinal,bugni2023inference.If framed in this way, the analyst's goal is to do predictive inference for new clusters. When $\mu$ assigns probability $1$ to a single distribution in $p\in\mathcal{P}$ or when $\mu$ assigns probability $1$ to $m=1$, this reduces to i.i.d.\ sampling of observations from a fixed joint distribution, but our focus is on settings where neither of these is the case. } In contrast, the literature on external validity (see Section (ref)) typically assumes that the distributions governing behavior in different domains are close in some distance metric adjaho2023externally, share a common support over $\mathcal{X}$ or $\mathcal{Y}$ sahoo2022learning,lei2023policy, or can be estimated using background covariates tipton2018review. Relative to these assumptions, our approach has the advantage of allowing for arbitrary and unknown relationships between the realized distributions governing domains, but it rules out ex-ante predictable patterns in how the joint distribution varies across samples (such as time trends).
Section (ref) presents our forecast intervals for this setting, and Section (ref) proves the validity and tightness of these intervals.
Thanks to Assumption (ref), the observed samples in the metadata, $\{S_1, \dots, S_n\}$, can act as surrogates for the unseen target sample, $S_{n+1}$. As before, let $e_{\mathcal{T},d}^\mathbf{M}$ denote the (observed) transfer error from any selection of training samples $\mathcal{T} \subseteq \{1,\dots,n\}$ to any surrogate target sample $d \in \{1,\dots,n\}\backslash \mathcal{T}$ from the metadata (where we now make the dependence of this quantity on $\mathbf{M}$ explicit). We use $\mathbb{T}_{r+1,n}$ to denote the set of $\frac{n!}{(n-r-1)!}$ unique pairs $(\mathcal{T},d)$ that can be constructed in this way. Then
is the empirical distribution of transfer errors in the pooled sample $\left\{ e^{\mathbf{M}}_{\mathcal{T},d} : (\mathcal{T},d) \in \mathbb{T}_{r+1,n}\right\}$ as we vary which samples in the metadata are used for training and testing. (Throughout $\delta$ denotes the Dirac measure). In the case where $r=1$, so that a single sample is used for training, the observed transfer errors can be represented as a matrix as depicted in Figure (ref), and $F_{\bold{M}}$ is their empirical distribution.
These quantiles coincide for continuously distributed variables with connected support.
Our proposed forecast interval for the transfer error on the target sample is $[\underline{e}_\tau^\mathbf{M},\overline{e}_\tau^\mathbf{M}]$.
We first prove that $[\underline{e}_\tau^\mathbf{M},\overline{e}_\tau^\mathbf{M}]$ is indeed a valid forecast interval.
Thus $\left(-\infty, \overline{e}_\tau^\mathbf{M}\right]$ is a level-$\left(\frac{\tau(n-r)}{n+1}\right)$ one-sided forecast interval for $e_{\mathbf{T},n+1}$, and $\left[\underline{e}_{\tau}^\mathbf{M}, \bar{e}_{\tau}^\mathbf{M}\right]$ is a level-$\left((2\tau - 1)\left(\frac{n-r}{n+1}\right)\right)$ forecast interval for $e_{\mathbf{T},n+1}$.
Parameter $\tau$ influences the width of the forecast interval, where larger choices of $\tau$ lead to wider forecast intervals with higher confidence guarantees. Parameter $r$ determines how many samples in the meta-data are used for training versus testing. As discussed in Section (ref), $r$ is determined by the the research procedure under evaluation.\footnote{We expect that in general, larger choices of $r$ will lead to lower but wider forecast intervals, since the model is estimated on a larger quantity of data, but there are fewer samples with which to evaluate the performance of the estimated model.}
The number of samples $n$ and the sizes of these samples $(m_d)_{d=1}^n$ enter into our result in different ways: Increasing the number of observed domains $n$, holding fixed the distribution over sample sizes within each domain, does not change the distribution of $e_{\mathbf{T}, n+1}$ but instead allows this distribution to be estimated more precisely. In contrast, increasing the number of observations per domain changes the distribution of $e_{\mathbf{T}, n+1}$ and corresponds to the measurement of a different quantity. For example, in the limit of infinitely many observations per sample, the error $e_{\mathbf{T},n+1}$ measures how well the best predictor from the model class in the training domains transfers across domains, while if the number of observations is small, $e_{\mathbf{T},n+1}$ measures how well an imperfectly estimated model transfers.
The next result shows that the guarantees in Proposition (ref) are tight to $O(1/n)$. We use $\mathbb{T}_{s,t}$ to denote the set of all vectors of length $s$ that consist of distinct elements from $\{1,\dots,t\}$.
To gain intuition for the intervals in Proposition (ref), fix a realization of the unordered set $\{S_1, \dots, S_n, S_{n+1}\}$. Because all samples are exchangeable by assumption, the realization of $e_{\mathbf{T},n+1}$ (conditional on $\{S_d\}_{d=1}^{n+1}$) is a uniform draw from
If we let $e^*_\tau$ denote the upper $\tau$-th quantile of this empirical distribution, then by definition
In the case $r=1$ where precisely one sample is used for training, the set of pooled errors ((ref)) is the shaded cells in Figure (ref) (either yellow or blue), and the inequality in ((ref)) says that the probability that the value of a randomly drawn cell falls below the $\tau$th upper quantile of cells is at least $\tau$.
The analyst does not observe the target sample $S_{n+1}$, and so does not know $e^*_\tau$. We instead use $\overline{e}_\tau^\mathbf{M}$, the $\tau$th upper quantile of the pooled sample of errors when transferring across samples in $\mathbf{M}$, to construct the forecast intervals. In Figure (ref), the probability that $e_{\mathbf{T},n+1} \leq \overline{e}_\tau^\mathbf{M}$ is the probability that the value of a randomly drawn shaded cell (yellow or blue) falls below the $\tau$th quantile of the yellow cells. By a straightforward counting argument, \[\mathbb{P}\left(e_{\mathbf{T},n+1} \leq \overline{e}_\tau^\mathbf{M} \mid \{S_i\}_{i=1}^{n+1}\right) \geq \tau {n \choose r+1}/{n+1 \choose r+1} = \tau \left(\frac{n-r}{n+1}\right).\] Applying the law of iterated expectations (with respect to the sample $\{S_i\}_{i=1}^{n+1}$) yields the one-sided forecast interval in ((ref)). The proof for the two-sided forecast interval follows a similar logic but is more involved, see Appendix (ref) for details.
Our results so far assume that the distributions governing the different samples $S_d$ are themselves independent and identically distributed. This assumption is not always appropriate. For example, suppose variation in domains corresponds to variation over locations, and the samples in the metadata (but not the target sample) are from experiments run at locations chosen by experimenters. If there is selection bias over where experiments are run---for example, if the observed sites are chosen based on characteristics correlated with effect sizes (as Allcott found in the Opower energy conservation experiments)---then it may be that the the target sample has fundamentally different properties from anything that is observed in the metadata. We thus now relax Assumption (ref) to allow the distribution governing the training samples and the distribution governing the target sample to be drawn from different meta-distributions. Specifically, suppose that the analyst's metadata consists of samples $S_1, \dots, S_n \sim_{iid} \mu$ as in our main model, but $S_{n+1}$ is independently drawn from some other density $\nu$. Let \[\omega(S) = \frac{\nu(S)}{\mu(S)} \] denote their likelihood ratio. We initially assume this likelihood ratio is known by the analyst (although $\nu$ and $\mu$ need not be), and subsequently consider weakenings of this assumption. As before, $e_{\mathbf{T},n+1}$ is the transfer error when training on $r$ samples drawn uniformly at random from $\{S_1,\dots,S_n\}$, and testing on $S_{n+1}$.
The following subsections consider successively weaker assumptions about what the analyst knows about the likelihood ratio $\omega$. Sections (ref) and (ref) respectively generalize our previous results when the analyst either knows $\omega$ or can bound it. Section (ref) provides two ways of comparing the generalizability of models in environments where the analyst knows nothing about $\omega$.
We again construct a forecast interval for $e_{\mathbf{T},n+1}$ using the pooled sample of transfer errors across samples in the metadata, that is, $\left\{ e^{\mathbf{M}}_{\mathcal{T},d} : (\mathcal{T},d) \in \mathbb{T}_{r+1,n}\right\}$. Different from the previous section, we no longer assign uniform weights to each $e^{\mathbf{M}}_{T,d}$. Intuitively, under our previous i.i.d.\ assumption, each sample in the metadata was equally representative of the training and target distributions, but in this relaxed model, whether a sample $S_d$ is more representative of the training or testing distribution depends on its relative likelihood under $\nu$ and $\mu$.
A crucial quantity is the following:
To interpret this quantity, consider an alternative data-generating process for the metadata where for some permutation $\pi:\{1, \dots, n\}\rightarrow \{1,\dots,n\}$, the samples $S_{\pi(1)},\dots,S_{\pi(n-1)} \sim_{iid} \mu$ while $S_{\pi(n)} \sim \nu$. Fix a realization of the metadata $(S_1,\dots,S_n)$, and suppose the analyst does not observe the permutation $\pi$. Let $\Pi$ denote the set of all permutations on $\{1, \dots, n\}$, and for any vector of sample indices $(t_1,\dots,t_r,d)$ let \[\Pi_{(t_1,\dots,t_r,d)} = \{\pi \in \Pi : (\pi(1), \dots, \pi(r)) = (t_1,\dots, t_r) \mbox{ and } \pi(n) = d\}\] denote the permutations that specify $(t_1,\dots,t_r)$ for training and $d$ as the target. Then conditional on a realization of the metadata $(S_1,\dots,S_n)$, the probability that $(S_{t_i})_{i=1}^r$ are the training samples and $S_d$ is the test sample is\footnote{This is a special case of weighted exchangeability; see tibshirani2019conformal. The results in this subsection continues to hold if the domains are not independent but satisfy the weighted exchangeability condition, which is more general but harder to interpret.}
This quantity depends only on the identity of the target sample $d$, and not on the identity of the training samples $t_1, \dots, t_r$. Finally, let \[F^\omega_{\mathbf{M}} = \sum_{(\mathcal{T},d)\in \mathbb{T}_{r+1,n}}W_d\cdot \delta_{e^{\mathbf{M}}_{\mathcal{T},d}}\] be the weighted empirical distribution of transfer errors, where each sample $d$ is weighted according to $W_d$. When the two meta-distributions $\mu$ and $\nu$ are identical as in our main model, then $W_d \equiv (n-r-1)! / n!$ for every domain $d$, so the distribution $F^\omega_{\mathbf{M}}$ is simply $F_{\mathbf{M}}$ as defined in ((ref)).
This result strictly generalizes Proposition (ref) and Claim (ref), since when $w(\cdot)$ is the identity then $\overline{e}_\tau^{\mathbf{M},\omega} = \overline{e}_\tau^\mathbf{M}$ and $\underline{e}_\tau^{\mathbf{M},\omega} = \underline{e}_\tau^\mathbf{M}$, and the bounds in this theorem reduce to those given in Proposition (ref).
We next extend our results when the analyst does not know the likelihood ratio function $\omega$ precisely, but---as in the literature on sensitivity analysis rosenbaum2005sensitivity---knows that it admits an upper and lower bound.
Define the following worst case bounds for $\underline{e}_{\tau}^{\mathbf{M},\omega}$ and $\overline{e}_{\tau}^{\mathbf{M},\omega}$:
As shown in Appendix (ref), these quantities can be computed from data in $O(n^{r+1})$ time.
Finally, we provide two ways for comparing the transferability of two models $i=1,2$ when the analyst cannot bound $\omega$. We do not provide formal results about these orders, but show that they have bite in our subsequent application (see Section (ref)).
Let $\overline{e}_{i,\tau}^{\mathbf{M}}(\Gamma)$ and $\underline{e}_{i,\tau}^{\mathbf{M}}(\Gamma)$ again denote the worst case bounds for model $i$, as defined in ((ref)).
That is, model 1 worst-case-upper-dominates model 2 at the $\tau$-th quantile if for every $\Gamma$, the worst-case upper bound for model 1 exceeds the worst-case for upper bound for model 2.
We can strengthen this comparison by requiring the upper bound of the forecast interval for model 1 to be smaller than the upper bound of the forecast interval for model 2 pointwise for each $\omega \in \mathcal{W}(\Gamma)$, rather than simply comparing worst-case upper bounds.
Many decision rules will not be comparable under either of these definitions, but we show they are empirically relevant in our application. The even stronger requirement that $\overline{e}_{1,\tau}^{\mathbf{M},\omega} \leq \underline{e}_{2,\tau}^{\mathbf{M},\omega}$, i.e., that the upper bound of model 1's forecast interval is smaller than the lower bound of model 2's forecast interval, is likely too stringent to be useful in practice.\footnote{This stronger order has bite only when the transfer error for model 1 across “the most dissimilar" training and testing domains is lower than the transfer error for model 2 for “the most similar" training and testing domains.}
To illustrate our methods, we evaluate the transferability of predictions of certainty equivalents for binary lotteries across domains that have different subject pools and also differ in other ways. We focus on this application for several reasons: First, there are many public data sources that we can use to construct our metadata. Second, the associated economic models have been extensively examined from the perspective of predictive performance HarlessCamerer1994,HeyOrme1994,bruhin2010risk,bernheim2020empirical, and recent work evaluates how well these models predict relative to black box algorithms PeysakhovichNaecker2017,Plonskyetal,FKLM. Finally, as Einavetal2012 points out, given how models of risk preferences are often used in practice, it is also important to evaluate how well they transfer across domains. We thus view this application as a natural setting to illustrate our methods.
Section (ref) describes our metadata, and Section (ref) describes the decision rules we consider. Section (ref) conducts “within-domain" out-of-sample tests, where the training and test data are drawn from the same domain. Section (ref) compares transfer performance across domains by constructing forecast intervals for three different definitions of transfer error.
Our metadata consists of samples of certainty equivalents from 44 subject pools, which we treat as the domains. These data are drawn from 14 papers in experimental economics, with twelve papers contributing one sample each, one paper contributing two, and a final paper (a study of risk preferences across countries) contributing 30 samples. Our samples range in size from 72 observations to 8906 observations, with an average of 2752.7 observations per sample.\footnote{Online Appendix (ref) describes our data sources in more detail.} Besides the difference in subject pools, these samples may differ in other details, such as whether the lotteries were restricted to the gain domain. We convert all prizes to dollars using purchasing power parity exchange rates (from OECD) in the year of the paper's publication
Within each sample, observations take the form $(z_1,z_2,p;y)$, where $z_1$ and $z_2$ denote the possible prizes of the lottery (and we adopt the convention that $\vert z_1 \vert > \vert z_2 \vert$), $p$ is the probability of $z_1$, and $y$ is the reported certainty equivalent by a given subject. Thus our feature space is $\mathcal{X} = \mathbb{R}\times \mathbb{R} \times [0,1]$, the outcome space is $\mathcal{Y} =\mathbb{R}$, and a prediction rule is any mapping from binary lotteries into predictions of the reported certainty equivalent. We use squared-error loss $\ell(y,y') = (y-y')^2$ to evaluate the error of the prediction, but for ease of interpretation we report results in terms of root-mean-squared error, which puts the errors in the same units as the prizes.\footnote{This transformation is possible because none of the results in this paper change if we redefine $e(\sigma,S) = g\left(\frac{1}{\# S} \sum_{(x,y) \in S} \ell(\sigma(x),y)\right)$ for any function $g$. Root-mean-squared error corresponds to setting $g(x) = \sqrt{x}$.} Since different subjects report different certainty equivalents for the same lottery, the best achievable error is generally bounded away from zero.
We consider two parametric economic models of certainty equivalents and two off-the-shelf black box algorithms.
Economic models. First we consider an expected utility agent with a CRRA utility function parameterized by $\eta \geq 0$ (henceforth EU). For $\eta \neq 1$, define \[v_\eta(z) = \left\{
\right.\] and for $\eta=1$, set $v_\eta(z) = \ln(z)$ for positive prizes and $v_\eta(z) = -\ln(-z)$ for negative prizes. For each $\eta \geq 0$, define the prediction rule $\sigma_{\eta}$ to be \[\sigma_{\eta}(z_1,z_2,p) = v_\eta^{-1}\big( p\cdot v_\eta(z_1) + (1-p) \cdot v_\eta(z_2)\big).\] That is, the prediction rule $\sigma_\eta$ maps each lottery into the predicted certainty equivalent for an EU agent with utility function $v_\eta$.
Next we consider the set of prediction rules $\Sigma_{CPT}$ derived from the parametric form of Cumulative Prospect Theory (CPT) first proposed by goldstein1987expression and lattimore1992influence. Fixing values for the model's parameters $(\alpha,\beta,\delta,\gamma)$, each lottery $(z_1,z_2,p)$ is assigned a utility
where
is a value function for money, and
is a probability weighting function.
For each $\alpha,\beta,\gamma,\delta$, the prediction rule $\sigma_{(\alpha,\beta,\gamma,\delta)}$ is defined as \[\sigma_{(\alpha,\beta,\gamma,\delta)}(z_1,z_2,p) = v^{-1}\big( w(p)v(z_1) + (1-w(p)) v(z_2)\big).\] That is, the prediction rule maps each lottery into the predicted certainty equivalent under CPT with parameters $(\alpha,\beta,\gamma,\delta)$. Following the literature, we impose the restriction that the parameters belong to the set $\Theta = \{ (\alpha, \beta,\gamma,\delta) : \alpha,\beta,\gamma \in [0,1], \delta \geq 0\}$.
We also evaluate restricted specifications of CPT that have appeared elsewhere in the literature: CPT with free parameters $\alpha$ and $\beta$ (setting $\delta=\gamma=1$) describes an expected utility decision-maker whose utility function is as given in ((ref)); CPT with free parameters $\alpha$, $\beta$ and $\gamma$ (setting $\delta=1$) is the specification used in karmarkar1979; and CPT with free parameters $\delta$ and $\gamma$ (setting $\alpha=\beta=1$) describes a risk-neutral CPT agent whose utility function over money is $u(z)=z$ but who exhibits nonlinear probability weighting. Additionally, we include CPT with the single free parameter $\gamma$ (setting $\alpha=\beta=\delta=1$), which FudenbergGaoLiang found to be an especially effective one-parameter specification.
Black Box Algorithms. We consider two popular machine learning algorithms. First, we train a random forest (RF), which is an ensemble learning method consisting of a collection of decision trees.\footnote{A decision tree recursively partitions the input space, and learns a constant prediction for each partition element. The random forest algorithm collects the output of the individual decision trees, and returns their average as the prediction. Each decision tree is trained with a sample (of equal size to our training data) drawn with replacement from the actual training data. At each decision node, the tree splits the training samples into two groups using a True/False question about the value of some feature, where the split is chosen to greedily minimize mean squared error.} Second, we train a kernelized ridge regression model (KR), which modifies OLS to weight observations at nearby covariate vectors more heavily, and additionally places a penalty term on the size of the coefficients. Specifically, we use the radial basis function kernel $\kappa(x, \tilde{x}) = e^{-\gamma \|x-\tilde{x}\|_2^2}$ to assess the similarity between covariate vectors $x$ and $\tilde{x}$. Given training data $\{(x_i,y_i)\}_{i=1}^N$, the estimated weight vector is $\vec{w} = (\mathbb{K} + \lambda I_N)^{-1} \vec{y}$, where $\mathbb{K}$ is the $N\times N$ matrix whose $(i,j)$-th entry is $\kappa(x_i,x_j)$, $I_N$ is the $N \times N$ identity matrix, and $\vec{y}=(y_1, \dots, y_N)'$ is the vector of observed outcomes in the training data. The estimated prediction rule is $\sigma(x) = \sum_{i=1}^N w_i \kappa(x,x_i)$.
There are at least two approaches for cross-validating hyper-parameters such as the size of the trees in the random forest algorithm. First, when there are multiple training domains one can cross-validate across them; we use this in Appendix (ref). Second, one can cross-validate across observations within the training domains. Since we are interested in cross-domain performance, rather than within-domain performance, it is not guaranteed that this will improve performance, and indeed we find that choosing the hyper-parameters via within-domain cross-validation leads to worse transfer performance than using default values. Thus in our main analysis with a single training domain, we set all hyper-parameters to default values.\footnote{Specifically, we set $\lambda=1$ and $\gamma = 1/(\#\mbox{covariates})=1/3$ in the kernel regression algorithm. See scikit-learn and Chapter 14 of Murphy for further reference. For the random forest model, we set the maximum depth to none, so the tree is extended until outcomes are homogeneous within each leaf.}
Discussion. There is no established definition of what constitutes an economic model versus a black box algorithm, but one way of distinguishing between the two approaches is whether the prediction method is tailored to a general application or a general-purpose method of prediction. EU and CPT model the risk preferences of economic agents; we would not expect these models to predict well if we changed our problem to image classification. In contrast, random forest algorithms and kernel regression have been successfully applied across a wide array of prediction problems. In this sense, EU and CPT are economic models, while RF and KR are not. Our approach and results can, however, equally be applied to evaluate prediction methods that are a hybrid of the two approaches. For example, Plonskyetal and Ke consider black box algorithms whose inputs are based on prior economic theory. We leave investigation of the transfer performance of such methods to future work.
We note finally that although black box algorithms are traditionally perceived as more flexible than economic models, whether this is in fact the case is something that has to be determined case-by-case. In particular, FudenbergGaoLiang shows that although CPT uses only four parameters, it imposes very few restrictions on mappings from binary lotteries to certainty equivalents.
We first evaluate how these models perform when trained and evaluated on data from the same subject pool. We compute the tenfold cross-validated out-of-sample error for each decision rule in each of the 44 domains.\footnote{We split the sample into ten subsets at random, choose nine of the ten subsets for training, and evaluate the estimated model's error on the final subset. The tenfold cross-validated error is the average of the out-of-sample errors on the ten possible choices of test set.} The two black box methods (random forest and kernel regression) each achieve lower cross-validated error than EU and CPT in 38 of the 44 domains, although the improvement is not large. To obtain a simple summary statistic for the comparison between the economic models and black boxes, we normalize each economic model's error (in each domain) by the random forest error. Table (ref) averages this ratio across domains and shows that on average, the cross-validated errors of the economic models are slightly larger than the random forest error. That is, the CPT error is on average 1.06 times the random forest error, and the EU error is on average 1.21 times the random forest error.\footnote{The numbers in Table (ref) are very similar if we normalized by the kernel regression error instead.}
These results suggest that the different prediction methods we consider are comparable for within-domain prediction, with the black boxes performing slightly better. But the results do not distinguish whether the economic models and black boxes achieve similar out-of-sample errors by selecting approximately the same prediction rules, or if the rules they select lead to substantially different predictions out-of-domain. We also cannot determine whether the slightly better within-domain performance of the black box algorithms is achieved by learning generalizable structure that the economic models miss, or if the gains of the black boxes are confined to the domains on which they were trained. We next separate these explanations by evaluating the transfer performance of the models.
We use the results in Section (ref) to construct forecast intervals for the two specifications of transfer error defined in ((ref)) and((ref)), which we will subsequently call raw transfer error and transfer shortfall respectively. We also consider another normalization of the raw transfer error with respect to a proxy for the best achievable error on the target sample. Let $m \in \mathcal{M}$ index a set of models that each prescribe rules $f^m$ for mapping data to prediction rules. Then transfer shortfall
reveals how much lower the accuracy of the transferred model $f_{S_\bold{T}}$ is compared to the best in-sample accuracy using a model from $\mathcal{M}$.\footnote{This quantity (subtracted from 1) is similar to the “completeness" measure introduced in FKLM, without the use of a baseline model to set a maximal reasonable error, and adapted for the transfer setting by training and testing on samples drawn from different domains.} One advantage of this specification relative to ((ref)) is that the raw error is very sensitive to the predictability of $y$ given $x$ in the target sample, which may differ across domains but is not directly related to the model's transferability.
In our meta-data there are $n=44$ domains, and we choose $r=1$ of these to use as the training domain which corresponds to the question, “If the researcher draws one domain at random, and then tries to generalize to another domain, how well will they do?" Figure (ref) displays two-sided forecast intervals for transfer performance, transfer deterioration, and transfer shortfall (where $R$ includes all decision rules shown in the figure). These forecast intervals use $\tau=0.95$, so the upper bound of the forecast interval is the 95th percentile of the pooled transfer errors (across choices of the training and test domains), and the lower bound of the forecast interval is the 5th percentile of the pooled transfer errors. (See Table (ref) in Appendix (ref) for the exact numbers.) Applying Proposition (ref), these are 86% forecast intervals. Choosing larger $\tau$ results in wider forecast intervals that have higher coverage levels, and we report some of these alternative forecast intervals in Online Appendix (ref), including a 96% forecast interval.
Our main takeaway from Figure (ref) is that although the prediction methods we consider are very similar from the perspective of within-domain prediction, they have very different out-of-domain implications. Panel (a) of Figure (ref) shows that the black box forecast intervals for raw transfer error have upper bounds that are roughly twice those of the economic models. Panel (b) shows that the contrast between the economic models and the black boxes is even larger for transfer shortfall, which removes the common variation across models that emerges from variation in the predictability of the different target samples. Thus, although the economic models and the black box models select prediction rules that are close for the purposes of prediction in the training domain, they sometimes have very different performances in the test domain, and the prediction rules selected by the economic models generalize substantially better. Panel (c) of Figure (ref), which reports transfer deterioration, shows that it is less important to re-estimate the economic models on new target domains than to retrain the black-box algorithms.
All of the forecast intervals overlap for each of the three measures. This is not surprising, as variation in the transfer errors due to the random selection of training and target domains cannot be eliminated even with data from many domains. We expect the black box intervals and the economic model intervals to overlap so long as the economic model errors on “upper tail" training and target domain pairs exceed the black box errors on “lower tail" training and target domain pairs. Section (ref) provides confidence intervals for different population quantities, including quantiles of the transfer error distribution and the expected transfer error, whose width we do expect to vanish as the number of domains grow large. There, we find similar conclusions with regards to the relative performance of the black box algorithms and economic models.
The appendix provides several robustness checks and complementary analyses. Online Appendix (ref) plots the $\tau$-th percentile of pooled transfer errors as $\tau$ varies, demonstrating that forecast intervals constructed using other choices of $\tau$ (besides $\tau=0.95$) would look similar to those shown in the main text. Online Appendix (ref) provides 86% forecast intervals for the ratio of the raw CPT transfer error to the raw random forest transfer error, and finds that the random forest error is sometimes much higher than the CPT error, but is rarely much lower. Online Appendix (ref) considers an alternative choice for the number of training domains, setting $r=3$ instead of $r=1$. While the results are similar, the contrast between the economic models and black boxes is not as large, suggesting that the relative performance of the black boxes improves given a larger number of training domains. Online Appendix (ref) provides forecast intervals when each of the 14 papers is treated as a different domain; once again the black box methods transfer worse than the economic models do.
We next use our theoretical results from Section (ref) to study the consequences of relaxing the i.i.d.\ assumption in our comparison of CPT$(\alpha,\beta,\delta,\gamma)$ and RF. Since the main differences observed above concerned the upper bounds of our forecast intervals, we limit attention to $\tau \ge 0.5,$ and compare the methods in terms of worst-case and everywhere upper-dominance with respect to all three measures of transfer performance. These results are summarized in Table (ref).
Table (ref) shows that CPT worst-case-upper-dominates RF at all quantiles $\tau\ge 0.5$ and for all three transfer error measures. Hence, our finding that the upper tail of transfer errors is larger for RF than for CPT is robust to relaxing the assumption that the training and test domains are drawn from the same distribution, provided that we are comfortable comparing the upper bound for one method to the upper bound for the other. In Appendix (ref), we provide a more detailed view of worst-case-upper-dominance by plotting $\bar{e}_{\tau}^{\mathbf{M}}(\Gamma)$ as functions of $\tau$ and $\Gamma$, respectively.
We can also consider the more demanding everywhere-upper-dominance criterion, which asks what happens if we relax our i.i.d.\ sampling assumption in a way which is as favorable to RF (and as unfavorable to CPT) as possible. We find a substantial degree of robustness even under this highly demanding criterion: CPT everywhere-upper-dominates RF in raw transfer error for all $\tau\ge0.954,$ everywhere-dominates in transfer shortfall for $\tau\ge0.866,$ and everywhere dominates in transfer deterioration for $\tau\ge0.647.$
One tempting explanation of why the black box algorithms transfer less well is that they may overfit to idiosyncratic details of the training samples that do not generalize across subject pools. For example, suppose some subject pools tend to value lotteries depending on the specific digits they contain.\footnote{For example, Fortin find that in neighborhoods with a higher than average percentage of Chinese residents, homes with address numbers ending in “4” are sold at a 2.2% discount and those ending in “8” are sold at a 2.5% premium. } This regularity could not be captured by the economic models, because they do not include parameters for individual digits, but could be learned by a random forest algorithm. If so, the random forest would have better within-domain prediction for those subject pools, but worse transfer performance (assuming this regularity does not generalize across subject pools).
While the flexibility of black box algorithms is likely an important determinant of their transfer performance, a second analysis shows that this cannot be a complete explanation of our result. One of the papers we use is based on samples of certainty equivalents from 30 countries l2019all. Of the 30 samples from this paper, 29 samples report certainty equivalents for the same 28 lotteries, and the remaining sample reports certainty equivalents for 24 of those lotteries. We repeat our analysis using these 30 samples as the metadata, and find that the forecast intervals for raw transfer error are indistinguishable across the prediction methods (Panel (a) of Figure (ref)). There is some separation between the forecast intervals for the remaining two measures, but in both cases the CPT and random forest forecast intervals are substantially more similar than in the original data. If overfitting were the main explanation of our previous results, we would expect the black box algorithms to overfit here as well.
In contrast, we find that the economic models again outperform the black box algorithms when we consider a different definition of domains for the l2019all data. Specifically, we aggregate all observations for the 24 lotteries that are shared in all 30 samples, and split these observations into 24 samples, where each sample includes all reported certainty equivalents (across subject pools) for a given lottery. For this new definition of domains, our transfer measures evaluate how well a model estimated on data from certain lotteries predict certainty equivalents for other lotteries. Figure (ref) reports 83-level confidence intervals, and we find that the economic models transfer substantially better than the black box algorithms. In fact, isolating the difference across domains to be differences across lotteries exaggerates the relative value of economic models even relative to our original Figure (ref) (which uses a definition of domains that combines several sources of variation). For consistency, Figure (ref) reports confidence intervals for $r=1$ (corresponding to training on one lottery and predicting on another), but we show in Appendix (ref) that the qualitative features of this figure hold also for $r=3$ and $r=5$.
Taken together, our empirical results suggest that the crucial difference between economic models and black box algorithms isn't that one is more flexible and hence more inclined to overfit, but rather that economic models perform better in certain kinds of transfer tasks.\footnote{In fact, the flexibility gap between the black boxes and economic models is not large: many conditional mean functions (for binary lotteries) can be well approximated by CPT for some choice of parameters values $\alpha,\beta,\delta,\gamma$ FudenbergGaoLiang.} The next section discusses more formally one potential explanation for the difference in the relative performance of economic models and black box algorithms in these two transfer tasks.
Our framework allows the distribution $P$ governing the training sample and the distribution $P'$ governing the test sample to differ. At one extreme, $P$ and $P'$ may share a common marginal distribution on the feature space $\mathcal{X}$, but have very different conditional distributions $P_{Y\mid X}$ and $P'_{Y \mid X}$ (known as model shift). In our application, this would mean that the distribution over lotteries is the same, but the conditional distribution of reported certainty equivalents is different across domains. At another extreme, the conditional distributions $P_{Y \mid X}$ and $P'_{Y \mid X}$ might be the same, but the marginal distributions over the feature space could differ across domains, e.g., if different kinds of lotteries are used in different domains (known as covariate shift).
Our findings in Figure (ref) suggest that black boxes do as well as economic models at transfer prediction when the marginal distribution over features $P_X$ is held constant across samples. Intuitively, when the relevant feature vectors are the same in every sample, a black box algorithm can perform well by simply memorizing a prediction for each of these feature vectors. In contrast, when the set of lotteries varies across samples, then good transfer prediction necessarily involves extrapolation, and an algorithm that hasn't identified the right structure for relating behavior across lotteries will fail to generalize. Since economic models of risk preferences are intended to relate an individual's preferences across lotteries that permits extrapolation of this form, our empirical results suggest that they do so effectively.
For a simple, stylized, example of this contrast, consider three domains with degenerate distributions over observations. In domain 1, the distribution is degenerate at the lottery $(z_1,z_2,p)=(10,0,1/2)$ and certainty equivalent $y=3$. In domain 2, the distribution is degenerate at the lottery $(z_1,z_2,p)=(10,0,1/2)$ and certainty equivalent $y=4$. In domain 3, the distribution is degenerate at a new lottery $(z_1,z_2,p)=(20,10,1/10)$ and certainty equivalent $y=11$. Suppose EU and a decision tree are both trained on a sample from domain 1. The CRRA parameter $\eta \approx 0.64$ perfectly fits the observation $(10,0,1/2;3)$, as does the trivial decision tree that predicts $y=3$ for all lotteries. The estimated EU model and decision tree are equivalent for predicting observations in domain 2: both predict $y=3$ and achieve a mean-squared error of $1$. But their errors are very different on domain 3: the EU prediction for the new lottery is approximately 10.8 with a mean-squared error of approximately 0.05, while the decision tree's prediction is 3 with a mean-squared error of 64.
The preceding sections suggest that the relative transfer performance of black boxes and economic models is determined primarily by shifts in which lotteries are sampled, rather than shifts in behavior conditional on those lotteries. To further test this conjecture, we examine how well we can predict the ratio of the raw random forest transfer error to the raw CPT transfer error given information about the training and test lotteries but not about the distribution of certainty equivalents in either sample. If the relative performance of these methods depended importantly on behavioral shifts in the two domains---i.e., a change in the distribution of certainty equivalents for the same lotteries---then we would expect prediction of the relative performance based on lottery information alone to be poor. We find instead that lottery information has substantial predictive power for this ratio.
For each sample $S = \{(z_{1,i},z_{2,i},p_i;y_i)\}_{i=1}^m$, we consider the following features: the mean, standard deviation, max, and min value of $z_1$ among the lotteries in $S$; the mean, standard deviation, max, and min value of $z_2$ among the lotteries in $S$; the mean, standard deviation, max, and min value of $p$ among the lotteries in $S$; the mean, standard deviation, max, and min value of $1-p$ among the lotteries in $S$; the mean, standard deviation, max, and min of $pz_1 + (1-p) z_2$ among the lotteries in $S$; the size of $S$; and an indicator variable for whether $z_1,z_2 \geq 0$ for all lotteries in $S$.
We consider three possible feature sets: (a) Training Only, which includes all features derived from the training sample $\mathbf{M}_\mathcal{T}$; (b) Test Only, which includes all features derived from the test sample $S_d$, (c) Both, which includes all features derived from the training sample $\mathbf{M}_\mathcal{T}$ and the test sample $S_d$. We evaluate two prediction methods: OLS and a random forest algorithm. Table (ref) reports tenfold cross-validated errors for each of these feature sets and prediction methods. As a benchmark, we also consider the best possible constant prediction.
The best constant prediction achieves a mean-squared error of 2.57, which can be more than halved using features of the training set alone. Using features of both the training and test sets, the random forest algorithm reduces error to 30% of the constant model. Crucially, the random forest algorithm is permitted to learn nonlinear combinations of the input features, and thus discover relationships between the training and test lotteries that are relevant to the relative performance of the black box and the economic model.
The random forest algorithm is too opaque to deliver insight into how it achieves these better predictions, but we can obtain some understanding by examining the best 1-split decision tree, shown in Figure (ref) below. This decision tree achieves a cross-validated MSE of 1.75, reducing the error of the constant model by 32%. It partitions the set of (train,test) domain pairs into two groups depending on whether the standard deviation of $z_1$ (the larger prize) in the training set of lotteries exceeds zero. There are three domains in which the prizes $(z_1,z_2)$ are held constant across all training lotteries (although the probabilities vary). In the 129 transfer prediction tasks where one of these three domains is used for training, the decision tree predicts the ratio of the random forest error to the CPT error to be 5.47. For all other transfer prediction tasks, the decision tree predicts 1.43.
This finding reinforces our intuition that the relative performance of the black boxes and economic models is driven in part by whether the training sample covers the relevant part of the feature space. When the training observations concentrate on an unrepresentative part of the feature space (such as all lotteries that share a common pair of prize outcomes), then the black boxes transfer much more poorly than economic models.
Our results also clarify a contrast between transfer performance and classical out-of-sample performance. In out-of-sample testing, the marginal distribution on $\mathcal{X}$ is the same for the training and test samples, so the set of training lotteries is likely to be representative of the set of test lotteries as long as the training sample is sufficiently large. When test and training samples are governed by distributions with different marginals on $\mathcal{X}$, the set of training lotteries can be unrepresentative of the set of test lotteries regardless of the number of training observations. Training on observations pooled across many domains alleviates the potential unrepresentativeness of the training data, but the number of domains needed will depend on properties of the distribution: An environment where each domain puts weight on exactly one lottery that is itself sampled i.i.d.\ may be difficult for black-box algorithms,\footnote{In this case, the number of domains black boxes need to achieve good transfer performance is likely comparable to the number of observations they need for good out-of-sample performance, which can be quite large.} while an environment where the marginal distribution is degenerate on the same lottery in all domains may be easier. There is no analog in out-of-sample testing for the role played by variation in the marginal distribution on $\mathcal{X}$ across domains. Moving beyond our specific application, we expect this variation to be an important determinant of the relative transfer performance of black box algorithms and economic models in general.
Our measures of transfer error quantify how well a model's performance on one domain extrapolates to other domains. We applied these measures to show that the predictions of expected utility theory and cumulative prospect theory outperform those of black box models on out-of-domain tests, even though the black boxes generally have lower out-of-sample prediction errors within a given domain. The relatively worse transfer performance of the black boxes seems to be because the black box algorithms have not identified structure that is commonly shared across domains, and thus cannot effectively extrapolate behavior from one set of features to another. Our finding that the economic models transfer better supports the intuition that economic models capture regularities that are general across a variety of domains.
One may view our theoretical results, and the statistical assumptions which justify them, from two perspectives. Taken literally, we consider a researcher who has access to data from a small set of domains, and an analyst who is interested in how well the researcher's procedure extrapolates. To provide guarantees we restrict the ways domains relate to each other; the simplest and strongest of these is that these is that the domains are exchangeable. We also assume that the analyst has access to data from a larger set of domains than the researcher does. One might wonder why the researcher uses a small set of domains for training, when the larger meta dataset is available to the analyst. In particular, the researcher might prefer use the larger cross-domain dataset for training, leaving no “fresh" domains for performance evaluation. Although one could potentially weaken our data requirements to cover such cases by restricting the researcher's estimator, this would rule out the black box prediction methods that are a focus of our analysis.
Alternatively, one can view our results as a tool for evaluating and comparing the performance of different methods for cross-domain prediction. The starting point for our analysis of a given method is the matrix collecting that method's cross-domain prediction errors for a set of observed domains. These matrices are rich objects, and it is not obvious how to compare them, but our theoretical analysis points to a natural set of summary statistics: quantiles of the observed error distribution. Focusing on these quantiles, or equivalently on our forecast intervals, provides a theoretically-motivated way to make comparisons across methods.