Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
115,664 characters · 19 sections · 52 citation commands
Decoding Consumer Preferences Using Attention-Based Language Models
Identifying the determinants of consumer value is a fundamental challenge of understanding market demand. For many products, commercial success hinges on how well their features reflect the underlying consumer preferences in the market, and thus firms have a strong economic incentive to determine how various combinations of features produce value. One common approach to estimating the strength of consumer preferences is to convert raw, unstructured feature data into a structured dataset that could plausibly delineate the sources of all value. For instance, in a used car market one might collect data on vehicles' make, model, year, and miles, while for theater shows one might collect data on the type of show, cast, venue seat, and day-of-week. Using these data, prices can be modeled as a function of the overall value, as estimated by summing the {\it hedonic} values of the features in the product rosen1974hedonic.
As a generic illustration, consider a standard approach presented in Figure (ref). First, a set of features $X$ is defined from an unstructured data source (e.g., vehicle description in a used car market) to serve as inputs into a structural estimation model $\mathcal{F}$. The structural model produces estimates $\hat{Y}$ on targets $Y$ (e.g., sales price of used cars), and an error metric $\mathcal{L}$ is used to optimize the parameters of the structural model (e.g., beta-coefficients of OLS regression).
Although simple and intuitive, one drawback to this approach is that it often involves making {\it a priori} assumptions on the sources and structure of consumer valuations. Without sufficient knowledge of how to code the true sources of value, one must simply rely (perhaps unscientifically) on intuition for this task ludwig2024machine. And, even with perfect knowledge of the feature set, feature-based preferences can be diverse, dynamic, and interdependent -- potentially requiring complex identification methods in order to estimate the true hedonic values of consumers cropper1988choice. Furthermore, any resulting insights will be limited to the defined attributes and specific market dynamics, as it is not immediate how, for instance, a standard regression-based inference might apply to an out-of-sample good, or a set of previously unobserved attributes, or under different market mechanisms.
Motivated by the above limitations and the recent development and applications of natural language processing, we offer a new approach to modeling the sources of consumer valuations. Specifically, we feed a detailed text description of a good within a specific market environment to an attention-based language model, which converts the text into {\it embedding} vectors of a high-dimensional, real-valued space that can numerically represent the good's underlying demand and market information. We label this space as the {\it Value Embedding Space}, and each specific embedding vector, corresponding to the description of a particular good, as a {\it Demand Embedding Vector} (DEV). The benefit of this language-model-based approach is that it can process and interpret intricate language patterns, contextual information, and nuanced connotations of product features, and thus generate more complex representations of consumer preferences than traditional methods.
The proposed approach employs a two-stage estimation process, as illustrated in Figure (ref). In Stage 1, a language model $\mathcal{M}$ is trained to generate a DEV that {\it encodes} point predictions $\hat{Z}$ for the target outcomes $Z$ using the textual description of a product as input. To achieve this, a weight vector $\mathcal{H}_1$ is simultaneously optimized to project the DEV into the prediction space corresponding to these outcomes. For the empirical illustration of online car auction data, the target outcomes include metrics on bid values, the number of unique bidders, auction views, watchers, and whether the reserve price was met. Both $\mathcal{M}$ and $\mathcal{H}_1$ are trained with an error metric $\mathcal{L}_1$, ensuring that the DEVs effectively capture the necessary market information to serve as the foundation for subsequent econometric analysis.
In Stage 2, a fully connected feedforward network, $\mathcal{H}_2$, {\it decodes} the market information embedded in the DEVs from Stage 1 into the parameter space of the structural model $\mathcal{F}$, which produces estimates $\hat{Y}$ for the target market outcomes $Y$, which may be a subset of the target market outcomes $Z$ from Stage 1. This second stage is crucial for enabling robust counterfactual analysis, as it explicitly constrains the model to use the assumed data generating process to predict the observed target outcomes. In the empirical illustration, these estimates are obtained using semi-nonparametric methods to model probability density functions over consumer valuations and market size for each auction vehicle's description. The network $\mathcal{H}_2$ is trained with an error metric $\mathcal{L}_2$, optimizing over the predicted outcomes from $\mathcal{F}$ while ensuring adherence to the specified structural relationships.
This {\it encoder-decoder} approach compresses relevant market information into DEVs during Stage 1 and then transforms it into optimized estimates within the econometric framework in Stage 2. By fine-tuning the language model in Stage 1, the model learns subtle patterns in product descriptions that reflect consumer preferences and the value generated by these products, establishing a sophisticated mapping between unstructured textual features and market outcomes. The structural constraints imposed in Stage 2 then ensure that counterfactual analyses remain grounded in economic theory, making the predictions more reliable when simulating market interventions or policy changes. This approach overcomes key limitations of traditional hedonic models by eliminating the need for a priori assumptions, enhancing flexibility and scalability across diverse markets and product categories, and improving the granularity of analysis to detect nuances in consumer preferences and market dynamics.
To illustrate the proposed model, we collected transactional data from a large US online auction house for used vehicles. Using time-stamped and user-identifiable (yet anonymous) bids on approximately 75,000 car auctions from 2014-2023, we train both stages of this model and test its ability to generalize to previously unseen descriptions on a holdout sample of 5,000 auction observations, including an ablation study, or “zero shot” test, with an unusual make and model of vehicle -- the DMC DeLorean.
To benchmark the model against familiar econometric baseline, we also estimate a conventional pooled-OLS hedonic specification and show the two-stage attention-based language model's superior prediction performance. Additionally, we validate the need for a two-stage model framework by training an alternate “direct” specification where we ignore the first stage $\mathcal{H}_1$ estimation and simply train $\mathcal{M}$ and $\mathcal{H}_2$ simultaneously. Although this direct estimation procedure predicts the training data as well as the proposed two-stage approach, we demonstrate it severely underperforms over the unseen validation set (i.e. worse predictions than taking a static average).
Our ability to offer these results is based on two recent developments. First, the scaling of machine learning algorithms has generated powerful emergent abilities to parse semantic meaning from natural language. In the context of natural language processing, an attention-based language model's architecture allows it to establish nonlinear relationships between various components of a text sequence mikolov2013linguistic. After being trained on large swaths of the Internet to predict missing or “masked” text elements, these models can later be fine-tuned for alternative natural language applications devlin2018bert. If tasked with predicting market outcomes such as prices from the natural language descriptions of products, as done here, training the language model will convert its initial semantic representations of goods to demand representations that are suitable for downstream econometric estimation. Second, the benefit of the abundance of naturally occurring market data, such as online auctions, that can offer strong signals of individual consumer valuations while satisfying the training requirements of a language model.
The managerial implications of integrating machine learning with economic modeling extend beyond improved prediction; they redefine how firms generate causal insights for strategic decision making athey2018impact. Language models unlock vast new data sources, such as product descriptions and written consumer feedback. However, their outputs must be structured within econometric models that enable reliable counterfactual analysis. The contribution of this paper is that the approach bridges the aforementioned gap by embedding language-model-driven demand signals into a structural model, and thus allowing managers to move beyond correlation-based forecasting to derive actionable, policy-relevant insights. This is particularly critical in digital marketplaces, where firms must anticipate demand shifts, optimize platform design, and refine dynamic pricing strategies in response to evolving market conditions mehta2021sell. By integrating machine learning representations with economic inference techniques, the model ensures that machine-driven recommendations align with underlying economic forces, enhancing their applicability for real-world inferences. However, realizing this potential requires overcoming key challenges, including bias mitigation, interpretability, and model robustness across different market contexts van2022overcoming. As firms increasingly rely on machine learning for decision support, the approach offers a framework for merging algorithmic flexibility with economic inference, establishing a foundation for machine-powered business strategy. Thus, our ambition is for this paper and its proposed model to serve the calls of abbasi2024pathways and sarker2025advancing.
The rest of the paper proceeds as follows. Following an overview of the existing literature in Section (ref), we provide a more detailed description of the model framework in Section (ref), and the technical model training in Section (ref). In Section (ref) we provide an overview and summary statistics of the online auction data, and then in Section (ref) provide an empirical illustration of the proposed model framework, and discuss the numerical results and counter-factual analysis from the online auction data. Finally, in Section (ref), we conclude with some remarks and ideas for future extensions.
Throughout this paper, we adhere to a notational convention in which uppercase letters indicate random variables, lowercase letters represent their realizations, Greek letters are used to denote model parameters, and cursive letters denote a model-specific function. As the ensuing empirical illustration is based on vehicles, we use the term “product” instead of “good,” but stress that the presented framework equally applies to “services”.
With the creation of the transformer architecture vaswani2017attention, attention-based language models have achieved unprecedented capabilities in representing the semantic meaning of natural language. Initially developed for natural language translations, these models have since demonstrated broad applicability across diverse domains, including text generation radford2019language, protein structure prediction jumper2021highly, medical image processing chen2021transunet, writing advertising copy Chen2024, privacy policy changes lin2024automated, qualitative research zhou2024can, and prediction on course and instructor evaluation wang2025predicting. More recently, these advancements have led to a surge in the use of language models for empirical research in economics, business, and management, where textual data is increasingly leveraged to make economic insights on consumer preferences, firm strategies, and market and business dynamics gentzkow2019text, Berger2020, ash2023text, deKok2025.
Although language models have proven effective in areas such as sentiment prediction in customer service Puranam2021, personality detection yang2023getting, product reviews ma2023beyond, aggregate consumer decisions qiu2023consumer, and financial disclosures mccarthy2024fin, their potential for demand estimation remains underexplored. Recent advances suggest that language models may provide a useful tool for extracting latent demand signals from natural language descriptions of goods and services, offering new possibilities for new approaches to demand estimation in a variety of market settings Timoshenko2019,Liu2023,wang2024large, chen2025conversation.
The integration of machine learning into econometric methods has led to advances in predictive accuracy and modeling flexibility, especially in settings where traditional methods face limitations athey2019machine. Recent research has leveraged deep learning techniques to capture complex relationships between product attributes and consumer preferences, incorporating high-dimensional representations derived from structured and unstructured data sources adam2024machine, aceves2024mobilizing. Language models, in particular, have been used to explore individual preferences and risk tolerances in a variety of settings dillion2023can, wu2023large, goli2024frontiers, zhu2024language.
Although these methods improve predictive performance, a key challenge remains their lack of structural coherence and interpretability.\footnote{See olah2020zoom, elhage2022toy, templeton2024scaling for language model interpretability methods.} Models that directly estimate economic primitives within a structural framework are typically better suited for counterfactual analysis verma2024counterfactual, and improve performance on “zero-shot” (i.e. novel) problems wei2021finetuned, yet many machine learning approaches lack this foundation. Recognizing this, recent work has explored hybrid approaches that integrate machine learning representations into structural models, using deep embeddings as inputs to traditional econometric frameworks vafa2024estimating. These approaches suggest a path forward for demand estimation methods that combine the flexibility of machine-driven representations with the interpretability and counterfactual validity of structural econometric models.
The framework for the proposed model begins with a set of assumptions over the data generating process on market demand, outlined in Figure (ref). Within a given market environment, the introduction of a product with description $d$ generates, to an exposed population, two stochastic demand primitives that together produce an aggregate demand schedule. The first primitive of interest is $V(d)$, a continuous random variable that produces {\it i.i.d.} realizations on the {\it willingness to pay} that consumers have for $d$. The corresponding distribution function is given by $F_{V}(v)=\text{Pr}\{V(d)\le v\}$, with support on $\mathbb{R}$. The second primitive of interest is $N(d)$, a discrete random variable that represents the {\it number of potential consumers} a description $d$ attracts (i.e., market size). Likewise, this is defined by the distribution function $F_{N}(n)=\text{Pr}\{N(d)\le n\}$, with support on $\mathbb{N}$. To generate a demand schedule, we assume that the realization in market size $n$ determines the set of realizations over valuations $\{v_1,v_2,\dots,v_n\}$. Without loss of generality, we assume that the valuations are ordered, i.e. $v_n\leq \cdots \leq v_2 \leq v_1$. For simplicity, we assume that all consumers have unitary demand, and that $V(d)$ and $N(d)$ are independent for all $d$, i.e. $f_{V,N}(v,n)=f_{V}(v)f_{N}(n)$.
Once valuations are realized, consumers engage with the market's pricing mechanism, which can be a function of the market structure and available information, to produce observable prices. In the context of the empirical application of online vehicle auctions, the structural model $\mathcal{F}$ follows the English auction format, which implies that demand for a vehicle is stochastically determined when it is up for auction. We do not predetermine the dynamics of $V(d)$ and $N(d)$ or the conditions for their realization. Instead, we leverage the language model to uncover these dynamics based on the vehicle's description, imposed structural model, and available market data. Next, we discuss the two-stage estimation method for the distributions of $V$ and $N$, as presented in Figure (ref).
The proposed estimation approach uses an encoder-only language model to estimate the demand characteristics of a market object from the features represented in its description $d$. For this task, $d$ is broken down into a vector of the language model's most basic text units, called tokens, which we denote as $\ensuremath{\textsf{d}_{}}$. Tokens are integer indices of words, subwords (words broken into smaller parts), or characters. To convert text to tokens, the language model is accompanied by a designated tokenizer, which is a function $\mathcal{D}$ that converts a text string to a vector of integer values within the language model's vocabulary set, $\mathcal{D}: d \rightarrow \vec{\ensuremath{\textsf{d}_{}}}$. We denote the length of the token vector by $\ensuremath{\textsf{d}_{\max}}$. A pedagogical example of tokenization is provided in Table (ref) using the tokenizer we employ in the empirical section of this study, which has a vocabulary set of approximately 30,000 distinct tokens li2023gte.
Attention-based language models process tokens through multiple layers of self-attention mechanisms, allowing each token to encode contextual information from the entire text sequence. This mechanism enables the model to evaluate the relevance of each token by considering its relationship with all other tokens turian2010word,mikolov2013linguistic,vylomova2015take. To illustrate, in the ensuing online vehicle auction application, the model specifically encodes relevant market demand information associated with each object, e.g., the encoding for the tokens for “manual transmission” depends on the context provided by surrounding tokens, such as whether tokens representing “Honda” or “Lamborghini” are present.
Mathematically, each token $\ensuremath{\textsf{d}_{j}}$ in a textual description $d$ is individually mapped to a $q$-dimensional real-valued vector $\vec{e_j} \in \mathbb{R}^{q}$, labeled Demand Embedding Vectors (DEVs), $j=0,\dots,\ensuremath{\textsf{d}_{\max}}$,
where the initial token $\ensuremath{\textsf{d}_{0}}$ is a special classification token [CLS] placed at the beginning of every token sequence. After processing through the self-attention layers, the [CLS] token aggregates information across all tokens, acting as an information-dense fingerprint that captures the overall context of the input description li2020sentence. Thus, the language model $\mathcal{M}$ produces both token-specific DEVs and a comprehensive DEV via the [CLS] token, summarizing the collective market-demand context embedded in the description, see Table (ref).
The goal of the first stage estimation is to, for a given product description $d$, train $\mathcal{M}$ to encode the relevant market information of the product into its [CLS] token's DEV $\vec{e}_0$. To do this, we append a fully connected network head $\mathcal{H}_1$ to $\mathcal{M}$ so that it projects this DEV to $m$ real-valued target outcomes of interest, $\hat{y}_{j} \in \mathbb{R}^{m}$. For example, in the case of vehicle auctions, the outcomes could include the submitted bid values, the number of active bidders, and how many times the auction was viewed. This approach leads to the following proposition, which reformulates a well-established result from the computer science literature in the context of the economic application.
\noindentProposition 1. Let $D=\{(d,\vec{y})\}$ be a set of text descriptions $d$ and associated economic information vectors $\vec{y}\in\mathbb{R}^m$. For any error tolerance $\epsilon > 0$, there exists a language model $\mathcal{M}:D\rightarrow\mathbb{R}^{q}$, and a decoding function $\mathcal{H}_1:\mathbb{R}^{q}\rightarrow \mathbb{R}^m$, such that
\noindentProof. See Appendix (ref)
Drawing on established universal approximation theorems in neural networks, this result simply restates that any information purposefully embedded in the language model's representation space is recoverable with arbitrary precision through an appropriate decoding function. Therefore, we apply this approach to generating predictions $\hat{y}_{i}$ on target outcome $i$ with the [CLS] token's DEV, producing a set of $m$ observed market outcomes $\{y_1(d), y_2(d), \dots, y_m(d)\}$ with the estimated decoder,
That is, the purpose of the $\hat{\mathcal{H}}_1$ head is to decode the high-dimensional DEV representation of the [CLS] token into predictions on the target outcomes.
We optimize the estimated parameters in $\hat{\mathcal{M}}$ and $\hat{\mathcal{H}}_1$ with a loss function that minimizes the mean squared error (MSE) between the estimated market outcomes $\{\hat{y}_1(d), \hat{y}_2(d), \dots, \hat{y}_m(d)\}$ and the true observed market outcomes $\{y_1(d), y_2(d), \dots, y_m(d)\}$,
where $D$ represents the set of training samples. The loss function $\mathcal{L}_1$ ensures that the model learns to produce accurate point predictions by reducing the squared deviations between the predicted and actual market outcomes. We label the ensuing models, with the optimized parameters minimizing $\mathcal{L}_1$, by $\hat{\mathcal{M}}^{*}$ and $\hat{\mathcal{H}}_{1}^{*}$.
While Stage 1 produces a representation of each description through the information-dense embedding of the [CLS] token, it does not by itself yield direct inference on the structural parameters governing market outcomes. To address this, the second-stage estimation process projects the [CLS] embedding—summarizing the full input description—into the parameter space of an econometric model. This enables a more interpretable and structured analysis of demand primitives.
The objective of Stage 2 is to estimate two latent demand primitives of our structural model: the valuation distribution $F_V(v)$ and the market size distribution $F_N(n)$. While the machine learning approach in Stage 1 successfully encodes market outcomes into DEVs derived from product descriptions' [CLS] tokens, these predictions alone do not identify the structural economic forces driving demand; for further discussion, see mullainathan2017machine, iskhakov2020machine. By estimating the respective demand primitives via a structural model, we constrain the prediction process to follow the assumed data generating process, which is essential for performing robust counterfactual analyses. This structural approach enables us not only to infer the underlying relationships generating consumer behavior but also to make reliable predictions for unseen applications, such as new product descriptions or market structures, by ensuring that all counterfactual scenarios remain consistent with the economic theory embedded in our model.
To enable a general and flexible representation of the valuation distribution, we employ a semi-nonparametric approach developed by gallant1987semi, fenton1996convergence, and fenton1996qualitative. Specifically, the density function $f_{V}(v)$ is approximated by,
where $\varphi(v\mid \mu,\sigma)$ is the Gaussian density component that imparts location ($\mu$) and scale ($\sigma$) properties, and, for a given level $\kappa\in \mathbb{N}$, a Hermite polynomial expansion capturing the deviations from normality through the weighted terms $\alpha_{k}$. The squaring of the expansion guarantees non-negativity, and the normalizing constant,
ensures that $\hat{f}_V(v)$ integrates to one, and thus a well-defined probability density function. This specification yields a parameter space for the valuation distribution of $\vec{\lambda}_V=\{\alpha_1,\dots,\alpha_\kappa,\mu,\sigma\}$.
For the market size distribution, we adopt a semi-nonparametric approach that uses the softmax function to construct a probability mass function. Specifically, the parameter space for the market size distribution $\vec{\lambda}_N=\{\rho_{\underline{n}},\dots,\rho_{\overline{n}}\}$, consists of likelihood values $\rho_r$ on the discretized grid of potential bidder counts $B=\{\underline{n},\dots,\overline{n}\}$, $B\subseteq \mathbb{N}$. The likelihood values are then transformed via the softmax function, which exponentiates and normalizes them so that they are nonnegative and sum to one. This transformation yields a probability mass function for each $n$ in the defined grid. That is, the probability mass function $f_{N}(n)$ is approximated by, for $n\in\{\underline{n},\dots,\overline{n}\}$,
This formulation not only captures the inherently discrete nature of market participation but also allows for a flexible, data-driven estimation of the underlying bidder distribution.
Following farrell2020deep on deep network architecture, our Stage 2 estimation approach integrates a neural network with semi-nonparametric density estimation techniques. Specifically, we append a fully connected network head, $\mathcal{H}_2$, to the fine-tuned language model $\mathcal{M}^*$ from Stage 1. For each product description $d$, $\mathcal{M}^*$ produced the [CLS] token's DEV, $\vec{e}_0$, which is then decoded by $\mathcal{H}_2$. This network head maps the classification DEV into the parameter spaces of both distributions,
The dual-head architecture ensures that although both distributions are derived from the same DEV, each branch is tailored to its respective estimation task. By mapping the DEVs into these conditional density spaces, $\mathcal{H}_2$ effectively decodes the market-relevant signals encoded in Stage 1 into coherent, interpretable demand estimates well-defined for structural analysis.
Finally, we include a note on how one should seek to train $\mathcal{H}_2$ (i.e., estimate the model parameters). It is necessary to specify how the latent demand primitives generate the observed target outcomes through the market-specific pricing mechanism, as depicted in Figure (ref). That is, the calibration of $\mathcal{H}_2$ is context-dependent on the data generating process. Motivated by the ensuing empirical analyses, we illustrate by considering the English auction format.
In an English auction, participating bidders sequentially submit incremental bids until no bidder is willing to exceed the current highest bid and the auction concludes. It should be fairly intuitive that, a rational bidder $j$, with private valuation $v_j$, should continue to incrementally bid $b_j^+$, as long as their surplus $v_j - b_j^+$ is non-negative. Hence, bidders' weakly dominant strategy is to bid up to their valuation, $b_j^+\leq v_j$, and, conditional on the seller's reserve price $p_R$ being met, the final transaction price is determined by the second highest valuation plus the bid increment submitted by the winning bidder. To simplify notation, denote bidder $j$'s last bid as $b_j$, $j=1,\dots, n$. Given ordered valuations, it follows that $b_j=v_j$, for $j=2, \dots, n$, while $b_1=v_2+b_{inc}$, where $b_{inc}$ is the minimum bid increment. That is, for a subset of bidders, it is assumed the last (maximum) bid submitted represents their valuation. Note, in the English auction the winning bidder's valuation is censored and only known to exceed the second highest valuation with the minimum bid increment. We apply this revelation of valuations through the equilibrium bid strategy to the auction data.
While the equilibrium bidding strategy for the English auction is robust to several complicating factors, including risk tolerance, the number of bidders wolfstetter1996auctions, and other bidders strategies lu2019information, there are a couple of potential shortcomings. First, it is unable to point identify the necessary order statistics in the presence of jump bidding, i.e. bidding much more than the minimum bid increment; see avery1998strategic,easley2004jump. If there are large bid gaps due to jump bidding then assuming $b_j=v_j$ will produce biased estimates. Second, this is also a concern if participation is temporally intermittent, or sparse, or subject to rapid counter-bidding. For instance, if two bidders aggressively counter-bid each other, then the price trajectory might escalate quickly such that other bidders' last bids are far below their respective valuations. Finally, if there are multiple ongoing, or sequential, auctions of identical or similar objects, then bidders may strategically not bid up to their valuation in a given auction, in anticipation of maximizing their expected surplus over a set of auctions. These shortcomings may thus introduce estimation bias on the order statistics of the $2^\text{nd}, 3^\text{rd}, 4^\text{th}, \dots$ highest valuations. We address these concerns in our inclusion criteria for the auction data discussed in the empirical section of the paper.
Given that the equilibrium bid strategy can serve as reasonable proxies for at least some bidders' private valuation, we use the approximated density and probability mass functions, $\hat{f}_V(v)$ and $\hat{f}_N(n)$, to predict the order statistics of the observed bids in the auctions. Specifically, recall that the number of bidders $n$ is drawn from $F_{N}(n)$, and bidders' valuation are drawn i.i.d. from $F_{V}(v)$. This implies that conditional on a realized market size $n$, the estimated density function for the order statistics of the top $j \leq n$ valuations, is given by, for $j^\prime=j,\dots,2,1$,
Using Equation (ref), the structural model $\mathcal{F}$ numerically computes, conditional on market size $n$, the expectation of the top $j$ order statistic by,
To compute the unconditional expectation, we average over the distribution of $n$, for $j^\prime=j,\dots,2,1$,
Finally, we collect these expectations into a vector representing the predictions of the $j$ largest bids,
To train $\mathcal{H}_2$, we optimize its parameters to ensure that the estimated valuation and market size distributions generate expected bid predictions via $\mathcal{F}$ that align with the observed bid results. As previously discussed, the largest bid $b_1$ is biased downward due to the bidding strategies induced by the English auction. Therefore, $\mathcal{H}_2$ is trained to estimate valuations $\{V_2, \dots, V_j\}$, using the bids $\{b_2, \dots, b_j\}$, according to their computed order statistics from $\mathcal{F}$. We do this by minimizing the mean squared error (MSE) between the predicted bids $\hat{b}$ and the observed bids $b$ for the bids that can proxy for bidder valuations. The loss function is given by,
where $D$ represents the set of training auctions. This objective ensures that the estimated distributions $\hat{F}_V(v)$ and $\hat{F}_N(n)$ generate bid predictions that match empirical observations. Similar to above in Stage 1, we denote $\mathcal{H}_2^*$ as the model with the optimized parameters that minimize $\mathcal{L}_2$.
Although the estimation framework for $\mathcal{F}$ is tailored for English auctions, our broader approach can accommodate alternative market settings. For example, in posted-price markets, firms set prices based on expected residual demand rather than competitive bidding. Similarly, in bargaining environments, price formation is influenced by strategic negotiation between buyers and sellers. To extend our methodology to such settings, the structure of $\mathcal{F}$ must be adjusted to reflect the equilibrium conditions that govern the formation of prices in these alternative markets.
The foundation of our training framework is the base mGTE encoder-only language model introduced by li2023gte. mGTE is an advanced variant of the {\it Bidirectional Encoder Representations from Transformers} (BERT) model that demonstrated how a transformer-based language model could encode information from unstructured text devlin2018bert. In our application to online auction data, we use the base mGTE model to initialize our language model $\mathcal{M}$, which facilitates a two-stage training process to extract structured demand representations and estimate the economic primitives assumed to be governing the observed auction outcomes. Table (ref) summarizes training setup and the hyperparameter settings used in both stages.
For the empirical analysis, only auctions that met the time-based inclusion criteria, which required that the last (largest) bids from five unique bidders be submitted within 3 hours of the auction's end time, were used in the training and validation sets. This resulted in a total of 74,605 auctions for training, 5,000 auctions withheld (i.e. not seen by the model during training), and 93 “zero-shot” auctions for the DMC DeLorean for validation analysis. More details on the data are provided below in Section (ref).
The objective of Stage 1 is to train $\mathcal{M}$ to encode demand-relevant outcomes into the [CLS] embeddings generated from the auction items' descriptions. To project this token's demand embedding vector $\vec{e}_0$ from $\mathcal{M}$ into the auction outcome space, we append a fully connected neural network layer, $\mathcal{H}_1$. Specifically, $\mathcal{H}_1$ produces estimates for the log value of the second through fifth highest bid values ($b_2, b_3, b_4, b_5$), as well as the auction-level engagement metrics total views, watchers, and unique bidders. During training, a small amount of Gaussian noise was added to the true target values of these variables to prevent over-fitting of the model. Additionally, $\mathcal{H}_1$ predicts whether the auction's reserve price $p_R$ was met, denoted by the binary indicator $\mathbbm{1}[b_1 \geq p_R]$. Given the binary nature of this target outcome, we apply a sigmoid transformation to the prediction made by $\mathcal{H}_1$ to ensure its values are in the unit interval, $[0,1]$. Stage 1 is trained for 5 epochs using the AdamW optimizer with a maximum learning rate of $3 \times 10^{-5}$ and a linear warmup decay schedule. We employ a batch size of 64 and monitor the mean squared error (MSE) loss in equation (ref) across all predicted outcomes in the training dataset.
The second stage of training refines the structured representations of demand learned in Stage 1 by estimating the latent distributions of consumer valuations and market size in our structural model $\mathcal{F}$. While Stage 1 provides direct predictions of auction outcomes, Stage 2 introduces a structural estimation framework that parameterizes the probability density functions of the demand primitives as defined in equations (ref) and (ref), allowing for a deeper economic interpretation of how textual descriptions influence bidder behavior.
In Stage 2, the demand embedding vector $\vec{e}_0$ from $\mathcal{M}^*$ is frozen and passed through a nonlinear transformation layer in $\mathcal{H}_2$. This transformation extracts a lower-dimensional representation optimized for structural inference. The transformation consists of two fully connected layers with hidden dimensions of 32 and 16, each followed by LayerNorm regularization and a SiLU activation function, which introduces smooth non-linearity to enhance feature extraction while maintaining numerical stability. Separate output heads within $\mathcal{H}_2$ then estimate the parameters of the valuation distribution $F_V(v)$ and market size distribution $F_N(n)$. The optimization objective stated in equation (ref) for Stage 2 minimizes the discrepancy between the $2^\text{nd}-5^\text{th}$-largest bids submitted by unique bidders and those generated from the estimated joint order statistic distributions in equation (ref). As in the Stage 1 estimation, a small amount of Gaussian noise was added to these variables during training.
Similar to liu2020preserving, we use Sobol sequences, which are low-discrepancy quasi-Monte Carlo (QMC) sequences, to efficiently approximate the expectations of these order statistics. This approach provides more uniform coverage of the probability space compared to standard Monte Carlo sampling morokoff1995quasi,jackel2002monte, and also allows for a more stable and sample-efficient numerical integration when computing bid expectations over the estimated valuation distribution. Stage 2 is trained using the AdamW optimizer with a learning rate of $3 \times 10^{-5}$ and a batch size of 32. The training process spans 5 epochs and incorporates a linear warmup schedule with 1,000 warmup steps.
Once trained, the full model consists of $\mathcal{M}^*$ and $\mathcal{H}_2^*$, which can be applied to any vehicle description. In the empirical section that follows, we validate the predictive accuracy of the model and perform counterfactual simulations to examine how different vehicle descriptions influence estimated demand.
For the empirical application of our estimation method, we collected data from \href{bringatrailer.com}{bringatrailer.com} (BaT), one of the largest online vehicle auction marketplaces in the USA. BaT utilizes an ascending English auction mechanism with a secret reserve price $p_R$ for their vehicle auctions. The description of each auction vehicle is written by BaT staff following a stylized template that reviews the history of the vehicle, its current condition, any modifications or restorations, and its provenance.
Each auction begins at \$1 (all prices in USD), typically with a seven day clock that eventually ends the auction. To minimize {\it snipe bidding}, BaT automatically extends the auction's clock by two minutes whenever a new bid is submitted within this threshold. When the auction concludes, the highest bid $b_1$ is privately compared to the vehicle's secret reserve price $p_R$, and automatically announced as to whether the reserve was met. If the seller's reserve price is met ($b_1\geq p_R$), then the winning bidder pays the seller the amount $b_1$. In addition, the winning bidder is charged a BaT transaction fee of 5% of $b_1$ or \$250, whichever was greater, but no more than \$5,000, i.e. $\min(\max(.05\times b_1, \$250), \$5,000)$. If the seller's reserve price is not met ($b_1<p_R$), then BaT connects the highest bidder with the seller to attempt a privately negotiated a sale price. However, regardless of the outcome, the seller is charged an unrecoverable listing fee of \$99.
Important to this study, all text-based communications and bids submitted during a given auction were publicly recorded in a chat section at the bottom of the auction's webpage. While presumably designed to enhance the social aspects of the auction site, and reduce some information asymmetries, our estimation methods benefit from having every submitted bid timestamped and user-specified (yet anonymous) by BaT. We sourced all market data from the information in these chat sections.
From July 2014 through December 2023, BaT held a total of 89,080 auctions, however only 79,698 of them met our inclusion criteria that the final bids from five unique bidders be submitted within 3 hours of the auction's end time. This decreased the likelihood that some high value bidders were not present toward the end of the auction. Additionally, recall that to mitigate snipe biding, BaT extended the end time by two minutes whenever a bid was submitted within the last two minutes. We summarize the characteristics of the auctions meeting the inclusion criteria in Table (ref).
Approximately 31.6% of the auctions were managed by dealers, and the reserve prices were met in 77.2% of cases. Bidder participation averaged 12.007 active bidders per auction, with a minimum of 1 and maximum 39 unique bidders. For bid values, the average winning bid across the auctions was \$44,452, with the highest recorded bid reaching \$5,360,000 and the lowest at \$105. When normalized for analysis with the highest bid set to 100%, the second and third highest bids averaging 96.9% and 94.1%, respectively, of the winning bid. Notably, the third highest bids submitted by non-winning participants (i.e. those who did not ultimately submit the single largest bid) averaged slightly lower, at 87.8% of the highest bid.
The timing of bids demonstrates regular late bidding, with a significant portion of activity occurring close to the auction’s conclusion. Specifically, 16% of all bids were submitted in the final five minutes, 34.2% in the last 15 minutes, 47% within the last hour, and 63.3% on the final day. While partly a consequence due to BaT default listing of auctions in ascending order of remaining time, i.e. auctions closest to end are listed first, it also reflects that it is only the most committed bidders that actively participate. Thus providing further support to that bidders are bidding up to their valuation.
The auction platform engaged a total 199,468 unique bidders during our data collection period, each participating in an average of 4.797 auctions. Despite this active engagement, 74.9% of bidders never won an auction, and only 25.1% won at least once. Furthermore, only a small fraction of the participants achieved multiple wins: 5.9% won at least twice, and a mere 0.9% won five or more auctions.
We begin our analysis of the trained model by evaluating its predictive accuracy on held-out auction data and assessing its ability to capture key economic relationships in the online vehicle auction market. Specifically, we examine the structural estimates produced from the full model $\mathcal{M}^*$ and $\mathcal{H}_2^*$ by validating the recovered valuation and market size distributions against empirical bid distributions. To illustrate the superior performance of our estimation procedure, we provide comparative results with a standard OLS regression and a “direct” (joint) estimation of $\mathcal{M}^*$ and $\mathcal{H}_2^*$ (i.e. not training $\mathcal{H}_1$). Finally, we conduct counterfactual simulations to analyze how variations in vehicle descriptions influence estimated demand primitives, allowing us to quantify the economic impact of textual attributes on bidding behavior and price formation.
Figure (ref) presents the nominal prediction performance from the structural two-stage model, evaluated on a validation dataset of 5,000 auctions. Each panel reports the log prediction value for the $2^{nd}$ through $5^{th}$ highest bids against their respective log target values. The predictions were estimated using the inferred valuation and market size distributions. In each panel, observations above and below the 45 degree line indicate instances of over- and under-estimating the (true) bid values, respectively. Overall, we observe the model producing consistent predictions with the observations clustered along the 45 degree line.
Model fit statistics are presented in Table (ref), which also documents the performance of our two alternative benchmark specifications. This analysis includes a standard OLS estimation of the winning bid, or $2$nd highest bid, and a direct estimation procedure of the attention-based language model where the first stage embedding training of $\mathcal{H}_1$ is ignored, and instead $\mathcal{M}$ and $\mathcal{H}_2$ are jointly estimated. Details of the OLS analysis are presented in Appendix (ref).
Table (ref) presents out-of-sample predictive accuracy for the second through fifth-highest bids using the three estimation procedures. Across all metrics and bid ranks, the two-stage estimator (TS) exhibits consistently superior performance. On the second-highest bid, which serves as the price-setting bid in an English auction, TS achieves a 25.6% reduction in root mean squared error (RMSE) relative to OLS (.4418 vs. .5940) and a corresponding increase in explanatory power (\(R^2 = 0.779\) for TS vs.\ 0.597 for OLS). The improvements over direct estiamation (DE) are more substantial, with TS reducing RMSE by 54.2% and increasing \(R^2\) by more than 83 percentage points (from –0.055 to 0.779). These performance differentials persist across higher-order bids, indicating the robustness of the proposed framework.
Importantly, the two-stage procedure eliminates the substantial upward bias exhibited by the direct estimator. While DE systematically overpredicts all bids, with multiplicative bias ranging from 24.26% to 33.91%, the TS model produces mean log prediction errors that correspond to bias levels below 5% in magnitude and are consistently negative, reflecting mild underprediction. The TS model also outperforms both alternatives on hit rate, defined as the percentage of predictions with less than 10% absolute percentage error. It achieves hit rates above 20% across all bid ranks, more than doubling those of DE and modestly outperforming OLS on the second bid (20.18% vs.\ 17.88%). In terms of robustness to outliers, the median absolute percentage error (MdAPE) for TS is roughly 26%, compared to 34% for OLS and over 60% for DE.
To leverage the unique economic insights provided by the language model in our estimation procedure, we also report on how each token in a vehcile's description contribute to the value of the 2$^\text{nd}$-largest (i.e. price-setting) bid. To extract the ways textual descriptions influence bid predictions we apply Integrated Gradients (IG; sundararajan2017axiomatic), which is an interpretability method that decomposes continuous prediction values into token-level contributions. IG quantifies each token's incremental contribution to the model's prediction relative to a neutral baseline embedding (we employ the [PAD] token as this baseline). Formally, IG integrates the gradient of the model's bid prediction function along a straight-line interpolation between the baseline and actual input embeddings, using, in our case, $K = 256$ discrete intervals. The resulting attribution scores fulfill a completeness criterion such that their sum closely approximates the overall prediction difference within a controlled tolerance of $\max\left(10^{-3},\, 0.02\cdot |f(x)-f(x')|\right)$. Token-level attributions are further aggregated to meaningful word-level insights, excluding special tokens that lack interpretative significance.
Figure (ref) provides a visualization of normalized token attributions aggregated at the word-level for an illustrative auction listing. Positive and negative influences on predicted log-bid values are color-coded in shades of blue and red, respectively, enabling straightforward identification of description elements driving bidding outcomes. This granular interpretability highlights specific textual attributes, such as vehicle condition, rarity, or desirable features, that critically influence the marginal bid value. Insights derived from this stage of analysis not only enrich theoretical understanding of hedonic pricing mechanisms but also offer practical guidance for sellers aiming to optimize listing descriptions for enhanced market outcomes. Figure (ref) in Appendix (ref) illustrates an example of the words driving the number of potential bidders to the auction.
A natural test of the Stage 2 model's ability to generalize beyond its training distribution is to evaluate its predictive performance on vehicles that were entirely absent from the training set. To assess this capability, we deliberately withheld a specific and unusual make and model from both the training and validation datasets: the DMC DeLorean. The vehicle, famous from the Back to the Future movie franchise, possesses a distinct combination of design elements, historical significance, and cultural associations that are unlikely to be collectively shared by any other vehicles in the dataset. Consequently, accurate predictions on this category of vehicles would provide strong evidence that the model has successfully learned generalizable demand patterns within the target market.
In total, the dataset contained 278 bid observations from 93 auctions for DMC DeLoreans. Figure (ref) presents the nominal prediction errors on the second-to-fifth largest bids observed in these zero-shot auctions. The prediction errors for the zero-shot evaluation of the DMC DeLorean exhibit distinct characteristics relative to the broader validation set. The model maintains relatively stable predictive accuracy in terms of log-scale error dispersion, with RMSLE values ranging from 0.44 to 0.50 across the second- through fifth-highest bids. Nevertheless, we do observe a systematic bias across all bid ranks where the model consistently underestimates realized prices. This persistent undervaluation likely arises because the model does not, and indeed, given no relevant training data, cannot, fully capture the substantial and idiosyncratic premium placed on the DeLorean, which partly may be due to its iconic association with Back to the Future. The vehicle's cultural significance, rather than conventional automotive attributes captured by structured auction data, appears to drive much of its market value.
Our fine-tuned model's ability to make robust out-of-sample predictions on vehicles it has never encountered directly stems from the deep representations learned during pretraining wei2021finetuned. Large-scale language models, such as mGTE, are trained on extensive corpora that include information about a wide range of consumer goods, historical events, and market trends. It is likely that the foundational model trained in li2023gte was exposed to content about the DeLorean, including design and production history, collector status, and pop culture significance. During the fine-tuning process in Stage 1, some of these latent representations were likely preserved and refined within the context of auction markets, allowing the model to infer a reasonable valuation for the DeLorean by positioning it relative to other vehicles with similar attributes in the dataset. However, this explanation remains speculative, as testing whether and how these representations were preserved is beyond the scope of this paper. Future research could more directly investigate the extent to which pretrained language models retain and adapt knowledge representations during fine-tuning, particularly in economic applications templeton2024scaling.
To assess the economic consistency of our trained model, we conduct a series of counterfactual simulations that systematically vary a key determinant of vehicle value: mileage. Vehicle mileage is one of the most robust indicators of depreciation, influencing resale values across all makes and models. Therefore, as a robustness check, we test whether the model's price-setting bid predictions exhibit an economically reasonable response to changes in reported mileage. Due to the computational cost of this analysis, we randomly select a subset of 1,000 vehicles from the validation dataset and use a large language model (LLAMA 3.1 70B) to generate modified versions of each vehicle's description while holding all other details constant. Specifically, we replace the mileage value in the original description with one of six predetermined values: 25k, 50k, 75k, 100k, 125k, and 150k miles. Each edited description is then processed by the trained Stage 2 model to generate a new set of predicted price-setting bids.
Figure (ref) presents the results of this experiment, where the price-setting bid predictions are normalized to 1 for the 25k-mile counterfactual and 0 for the 150k-mile counterfactual. Across the vehicles included in the analysis, the model predicts a strong monotonic decline in the price-setting bid as mileage increases. This finding aligns with standard economic expectations regarding vehicle depreciation, and reinforcing the model's ability to appropriately internalize the role of accumulated mileage in determining vehicle value.
These results also highlight the utility of our framework for evaluating the marginal effects of key vehicle attributes on expected auction outcomes. By systematically modifying vehicle descriptions and observing the model's predictions, researchers and practitioners can conduct detailed counterfactual analyses that would be infeasible to observe in real-world auction data. Future work could extend this approach by examining other vehicle characteristics, such as accident history, number of previous owners, or specific aftermarket modifications, to further explore the model’s sensitivity to economically relevant factors.
This paper introduces a novel approach to demand estimation that integrates language models with structural econometric techniques to infer primitives of demand from unstructured textual descriptions. By training an encoder-only mGTE model in a two-stage process, we demonstrate how natural language descriptions of auctioned vehicles can be systematically transformed into structured economic representations that capture latent demand primitives. Our empirical validation using online vehicle auction data confirms that the proposed model effectively encodes market-relevant information, yielding competitive predictive accuracy on auction outcomes while preserving interpretability through its structural estimation framework.
A key strength of our approach is its ability to generalize beyond the training distribution, as evidenced by the model’s performance on the validation dataset and the zero-shot predictions on the DMC DeLorean. This result highlights the potential of language models to leverage their pre-trained knowledge when making economic inferences in unfamiliar contexts, a property that could be further explored in other market settings. Additionally, our counterfactual simulations on vehicle mileage confirm the model’s ability to produce economically coherent predictions, reinforcing its capacity to extract meaningful demand signals from textual descriptions.
These findings have several important implications for both researchers and practitioners. For firms and digital platforms, the ability to extract consumer demand signals directly from product descriptions offers a new avenue for designing “smart markets” via strategic pricing and inventory management bichler2010research. Rather than relying solely on historical sales data, firms can leverage language models to predict demand for new products, optimize product descriptions for maximum market appeal, and conduct counterfactual analysis to explore pricing and positioning strategies. From a methodological perspective, our study contributes to the growing literature on machine learning applications in economics by demonstrating how deep learning models can be integrated with structural estimation techniques to improve demand inference.
Despite these contributions, important avenues for future research remain. First, while our approach successfully captures demand primitives from textual data, further work is needed to understand how fine-tuned language models retain and adapt pre-trained knowledge for economic inference. The extent to which language models internalize and preserve economic reasoning during domain-specific training is an open question that warrants further exploration using interpretability methods, such as feature attribution and causal probing techniques. Second, while our empirical application focuses on English auctions, the underlying framework could be extended to other market settings, including posted-price markets, bargaining environments, and two-sided platforms where product descriptions and buyer preferences interact dynamically. Lastly, given the rapid evolution of language model architectures, future studies could investigate how different model families (e.g., multimodal models with vision capabilities) compare in their ability to encode and generalize economic information.
None.