Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
155,845 characters · 38 sections · 82 citation commands
LABOR-LLM: Language-Based Occupational Representations with Large Language Models
This paper introduces a new approach to predicting the evolution of worker careers, specifically the problem of predicting a worker's next occupation as a function of the worker's prior history of occupations and covariates. The problem is challenging because of the high-dimensional feature space: When there are 335 possible occupations, there are $335^t$ possible sequences of occupations in $t$ periods of observation. In addition, the prediction space is large: a predictive model produces 335 probabilities corresponding to the possible next occupations. Historically, economists have often dealt with this complexity by using heuristics to summarize occupational histories with a small number of summary statistics, e.g. most recent occupational category and years of experience (e.g., blau_gender_2017), and by focusing on relatively coarse outcomes, such as employment status, (boskin_conditional_1974) or characteristics of occupations (cortes2016have).
In this paper, we build a model that predicts the next occupation in a sequence, given a history of preceding occupations, while addressing the challenge of the high-dimensional feature space. Our model delivers substantially better predictive performance for the granular task of predicting the next occupation, as well as for auxiliary tasks commonly considered in labor economics, such as predicting whether a worker becomes unemployed or leaves the labor market. Better predictive performance translates into better estimators of causal effects, decompositions, and structural models; further, a model with granular predictions allows researchers to answer questions about a richer set of outcomes.
We construct our occupation transition model using an approach popularized in the generative artificial intelligence (AI) literature (bommasani_opportunities_2022), where we start with a “foundation model” (e.g., a large language model (LLM)) with parameters estimated on a large, unrepresentative dataset, and then move to “fine-tuning,” where the parameter estimation process continues on a smaller dataset, so that the final parameter estimates incorporate “foundational” knowledge from the larger dataset but are tailored to make accurate predictions consistent with the conditional distributions in the smaller dataset. Most LLMs as of 2025 are built using flexible parametric models (transformer neural networks) with billions to trillions of parameters. The parameters are estimated by maximizing the conditional log-likelihood of the next word in a text, given the history of preceding words. The estimation uses substantial computing resources on a very large dataset of text, e.g., text scraped from the open Internet. Using these parameters as starting values for fine-tuning retains information about the general meaning of text while also specializing the predictions to match the joint distribution of text in the fine-tuning dataset.
Recently, a series of papers (vafa_career_2024, vafa2025gendergapPNAS) took language modeling as an inspiration and built a custom foundation model (referred to as “CAREER”) for occupation transitions using the transformer neural network functional form popularized by LLMs, showing that CAREER improved accuracy substantially relative to existing econometric models. However, that work treated occupation as a categorical variable and did not make use of the textual content of job titles. In this paper, we introduce an approach that directly uses the next-word probability models from open-weight Large Language Models (LLMs), thus allowing us to effectively make use of a much larger model (more parameters) while incorporating the information encoded in the text, which may help the model understand which occupation histories are similar to one another in terms of likely future occupation transitions. We refer to our approach as the LAnguage-Based Occupational Representations with Large Language \textbf{M}odels (LABOR-LLM) framework.
In contrast to traditional methods that rely on tabular categorical data (e.g., vafa_career_2024), our approach leverages the insight that a sequence of occupations can be represented as a sequence of words via a “text template.” In our preferred approach, Fine-Tuned LABOR-LLM (FT-LABOR-LLM), we fine-tune an open-source foundation model (Llama-2, by Meta) on textual representations of worker careers constructed from publicly available U.S. survey datasets, the National Longitudinal Survey of Youth (NLSY) and Panel Study of Income Dynamics (PSID), using a next-word prediction objective.
The FT-LABOR-LLM approach involves using estimates of the probability of the next word (conditional on that word being preceded by a particular sequence of words) from the fine-tuned language model to calculate the probability that a given occupation comes next, given career history. Specifically, we use the probability model associated with the fine-tuned LLM to evaluate the probability that the next text in our text template is the text corresponding to a particular occupation, conditional on the preceding text being equal to the text of the text template truncated at the year of interest.
We show that the performance of FT-LABOR-LLM is better than that of CAREER, despite CAREER being custom-designed for the problem and pre-trained on a very relevant corpus of documents, resumes of U.S. workers. Recalling that CAREER in turn substantially outperformed alternatives from the literature, our results show that FT-LABOR-LLM is state-of-the-art in terms of predictive performance. We further show that not only is FT-LABOR-LLM better at the granular next-occupation prediction problem, but it also performs better when applied to predict specific, commonly studied outcomes such as changing occupations, becoming unemployed, or leaving paid employment. We highlight the importance of the fine-tuning step by showing that, without fine-tuning, off-the-shelf Llama-2 makes plausible-sounding predictions of occupations, but it is not as accurate in terms of the next occupation probability distributions conditional on history, and it “hallucinates” invalid job titles because it is not fine-tuned exclusively on labor sequence data. Similar problems arose even with the commercial LLMs that were state-of-the-art at the time our analysis was conducted in 2025.
In the remainder of the paper, we assess the sources of the performance benefits. We begin by assessing the role of model size (number of parameters) and the volume of data. We show that using a larger LLM as the foundation model, in particular the version of Llama-2 with 13 billion parameters rather than the version with 7 billion parameters, improves predictive performance. However, we show that adding in data from different government surveys (even though they are drawn from different time periods) quickly improves the performance of the smaller model, matching and then surpassing the performance of the larger model. Thus, data (even if non-representative) is a substitute for model size.\footnote{Other papers have shown that more data improves model performance for both pre-training (vafa_career_2024, kaplan2020scaling) and fine-tuning (dong2023abilities, bucher2024fine) data.} Since smaller models are less expensive to estimate, and especially cheaper to make predictions from, working with a smaller model has distinct advantages. More broadly, an open question about the fine-tuning approach is whether the fact that the pre-training dataset is not representative of the target implies that the final estimated model will exhibit bias relative to the true conditional transition probabilities in the population of interest. There may be a tradeoff between using a large, non-representative dataset to better learn underlying structure (e.g., meaning of language, or general information about occupation transitions), and getting a model that makes conditional predictions that are representative of a target dataset of interest. In this paper, we show that if such biases are important, the advantages of the foundation model approach outweigh them in our application, and further, adding non-representative data about occupation transitions in the fine-tuning step improves performance further. The best approach to combining representative and non-representative data in fine-tuning provides an interesting opportunity for future research.
We next assess whether FT-LABOR-LLM is making use of information embedded in the text of the job title. To do so, we replace the job titles with numeric codes in the training data and show that this approach degrades predictive performance substantially. We further establish that demographics, most notably gender, but also the interaction of gender, ethnicity, and region, play an important role in predicting occupation transitions. Finally, we show that the model's performance is highest when workers' full history is included; predictive performance is degraded unless at least 10 periods of worker history are included.
We also compare our approach to alternative ways we could use a fine-tuned model to make predictions, showing that the alternatives do not perform as well as FT-LABOR-LLM. One alternative approach involves using it to create data-driven low-dimensional embeddings of history (latent factors), and then using those embeddings as if they were observed covariates in a multinomial logistic regression where occupations are the alternatives.
Overall, the success of FT-LABOR-LLM provides an example of how LLMs can be used as foundation models for an economic problem that was traditionally studied using categorical, discrete-choice prediction models. In addition to providing superior predictive performance, the LABOR-LLM approach has some advantages because the pre-training step does not have to be carried out by the individual researcher; rather, open, general-purpose LLMs can be used (or closed models can be used through paid API access, although with less control on the part of the analyst). Since the data used for fine-tuning is textual and does not require a fixed format for either estimation or prediction, the approach allows the researcher to add in additional textual information for some or all observations in either the training set or the test set without changing any code. The approach handles missing data, unbalanced panels, irregular sampling intervals for individuals, and textual information about workers or their jobs, where the only required changes to the pipeline involve modifying the text included in the training or testing data.
How can FT-LABOR-LLM be used? We highlight three categories of economic applications that benefit from a high-quality occupation transition model: estimation of causal effects, decompositions, and structural estimation. Consider first causal effects. Under the assumptions required for identification of average treatment effects with many commonly considered statistical designs (e.g., unconfoundedness, instrumental variables, or panel data), estimating causal effects for a treated group requires the prediction of counterfactual outcomes in the absence of the treatment. The theory of semi-parametric efficiency in estimation (e.g. chernozhukov2018double) shows that the performance of efficient treatment effect estimators is determined by the performance of estimators for auxiliary prediction problems, notably estimators of outcome models and treatment assignment (propensity scores) as functions of covariates. Better predictive models lead to more precise estimates and avoid omitted variable bias, and the convergence rates of these estimates must be sufficiently fast to guarantee asymptotic normality.\footnote{In practice, it is common to use cross-validation based on prediction quality to select outcome and propensity models. A large literature compares the performance of estimators using different approaches to outcome modeling and propensity weighting; see, e.g., athey2018residual and athey2024gan and references therein.} vafa2025gendergapPNAS refines the standard results for the case where embedding functions are used to reduce dimensionality while avoiding omitted variable bias induced by the dimension reduction.
In our context, predicting the next occupation may be useful for estimating causal effects in several scenarios: (i) The occupation (or a categorization of the occupation) may be the outcome, e.g. taking a desirable occupation, becoming unemployed (del2021occupational, dabed2023resilience), entering poverty (stevens1994dynamics), entering education (frederiksen2007did), or transitioning industries (schmidt_prediction_1975, dauth2018adjusting) Future work may use the more granular predictions enabled by FT-LABOR-LLM to ask questions such as, which types of occupational transitions become more likely as a result of an intervention, and which become less likely? (ii) The occupation (or a categorization of the occupation) may be the “treatment,” e.g., researchers may study the effect of being laid off (jacobson_earnings_1993, athey2023heterogeneous), out of the labor force (mincer_and_ofek_1982_interrupted_work_careers), in education (card1999causal_effect_of_education_on_earnings), in a union occupation (oaxaca1975estimation,lewis1986union), or in an AI-impacted industry or occupation (webb2019impact). (iii) As shown in vafa_career_2024, if the outcome of interest is wages or another continuous outcome, next-occupation prediction models can still play a supporting role.
Second, accurate next-occupation prediction models are central to a variety of decomposition problems. The decomposition literature (see, e.g., blau_gender_2017 for a review) asks, how much of the difference in outcomes between two groups can be explained by differences in observable characteristics? Although the economic interpretation is distinct, the theory of estimation of decomposition terms is isomorphic to estimating causal effects, and so the theory of semi-parametric estimators also implies that high-quality prediction models play a central role in decompositions. Further, omitted variable bias is a major concern in decompositions; vafa2025gendergapPNAS demonstrates that traditional models based on heuristics leave out elements of occupation history and only adjust for current occupation, omitting factors that are correlated with both wages and gender.\footnote{For example, two workers whose occupational category is “manager” may have different wages but also differ as to whether their career history includes being an engineer or an administrative assistant.} The literature studies both wages and occupation-related outcomes, e.g., being unemployed (fairlie_emergence_1999) or being employed in a particular industry (schmidt_prediction_1975). The literature studies decompositions of differences between groups of workers, which may be differentiated by gender (brown1980incorporating), race (fairlie_emergence_1999), industry (abowd2012persistent_industry), education (hotchkiss2011decomposing_education), union status (oaxaca1975estimation, freeman1984unions,lewis1986union), or other occupation characteristics. As discussed above, econometric theory tells us that the accurate prediction of, e.g., being in a unionized job is important for efficient estimation of decompositions. A future study with access to an accurate and granular occupation prediction model might decompose the differences in outcomes for workers transitioning out of AI-affected occupations.
Third, accurate prediction models can be an input for other types of applied econometric models. Some studies rely on counterfactual simulations of occupational transitions to support structural estimates of the impact of factors such as race (brown1980incorporating) or concentration (,schubert2024employer) on worker outcomes. Many dynamic structural models of labor markets also rely on predictions of worker job transitions among several categories of occupations and employment (keane1997career,sullivan2010dynamic). In the context of recommendation systems or automated job advice (de_ruijt_job_2021), accurate estimates of conditional transition probabilities may be a key building block.
\paragraph*{Career Trajectory Modeling and Next Occupation Prediction} In the economics literature, when studies of worker transitions analyze the relationship between worker characteristics and career histories to career transitions, they have traditionally relied on fairly simple predictive models and considered only a few occupation categories. For example, boskin_conditional_1974 uses a conditional logistic regression model to analyze the factors affecting workers' transitions among 11 occupational groups, where the factors included estimated earnings, training expenses, and costs due to unemployment. schmidt_prediction_1975 uses a multinomial logistic regression to analyze the impact of race, sex, educational attainment, and labor market experience on the probability that individuals transition into one of five different occupational categories, revealing significant effects of these variables on occupational outcomes. hall_turnover_1972 examines the dynamics of labor force turnover in the U.S., analyzing the influences of demographic factors, labor demand fluctuations, and job stability on unemployment. To study turnover, the authors consider factors such as race, counts and ages of children, estimated wage, income, age, marital status, location, and employment category (including private wage or salary, government roles, self-employment, and unpaid family work). blau_labor_1999 model labor force transitions among older married couples, showing that one spouse's employment status significantly impacts the employment status of the other, with financial incentives and preferences for shared leisure influencing these transitions. In addition to demographic characteristics, the authors incorporate human capital and education variables, including tenure on the current job and retirement benefits.
\paragraph*{Machine Learning Methods for Next Occupation Prediction} Prior to the use of foundation models, other authors (e.g., li_nemo_2017,meng_hierarchical_2019,zhang_attentive_2021) made use of various neural network architectures for the next-occupation prediction problem, sometimes training on large datasets. For example, li_nemo_2017 uses a Long Short-Term Memory (LSTM) neural network to predict occupation transitions, where the embedding dimension is 200, and the training set incorporates more than a million individuals. he2021your build a model to predict the next occupation position out of 32 frequent position names, as well as job salary and firm size for that position, using a dataset of 70{,}000 resumes. Another approach taken by zhang2019job2vec seeks to predict aggregate transition probabilities between pairs of job titles within the same firm. Their approach, which generates embeddings for each job title, does not attempt to condition on individual worker history. These papers do not make use of foundation models; a recent line of work builds domain-specific foundation models for careers, most notably CAREER developed by vafa_career_2024, which we discuss in more detail below.
\paragraph*{Foundation Models for Careers} Building on these machine-learning approaches, a growing literature applies foundation models to domains beyond text (savcisens2024using,wu2021prototransformer,radford2021learning). Within labor economics, the most closely related paper to ours is vafa_career_2024, who develop CAREER, a custom transformer-based foundation model whose vocabulary is restricted to occupation codes and that incorporates a dedicated state for remaining in the same occupation. The model is pre-trained on 24 million resumes of U.S. workers acquired from Zippia, Inc., a large but non-representative dataset, and is then fine-tuned using data from U.S. government surveys, including the Panel Study of Income Dynamics (PSID) (PSID2024) and two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) (BLS_NLSY79_2023,BLS_NLSY97_2024). vafa_career_2024 shows that CAREER substantially outperforms existing benchmarks for next-occupation prediction and that the learned embedding of career history transfers to related tasks: when the model is fine-tuned to predict wages, which are not available in the pre-training resume dataset, it improves wage prediction relative to popular wage regression models in labor economics. CAREER has fewer parameters (5.6 million) than FT-LABOR-LLM (7 or 13 billion), and its pre-training dataset, while highly relevant, is much smaller than the corpus used for Llama-2; moreover, CAREER does not make use of textual job titles or descriptions.
\paragraph*{Adapting LLMs to Build Domain-Specific Models}
Adapting pre-trained models to specific domains via fine-tuning has become a prevalent approach for improving the performance of LLMs for specific tasks. The (full parameter) fine-tuning approach involves further updating all weights of a pre-trained model using domain-specific data and optimization techniques such as gradient descent (wei_finetuned_2022). The pre-training and fine-tuning paradigm has produced state-of-the-art models for dialogue systems (yi_survey_2024), code generation (chen_evaluating_2021), music generation (agostinelli_musiclm_2023), scientific knowledge (taylor_galactica_2022), protein structure prediction (rives_biological_2021), chemistry (zhang_chemllm_2024), medicine (singhal_large_2022), and other settings. The literature on the adaptation of LLMs for recommendation systems is also closely related. geng_recommendation_2022 introduces a general paradigm to adapt the recommendation task to language processing.
Our paper compares our fine-tuning approach to one where LLM embeddings are extracted and treated as covariates in a multinomial logistic regression. This type of approach has been popular in language analysis for a long time; for example, it is used by sentiment classifiers (reimers-gurevych-2019-sentence).
Finally, prompt engineering and in-context learning are alternative approaches to fine-tuning LLMs that require minimal computation and avoid the need for direct model access (brown2020languagemodelsfewshotlearners). Prompt engineering involves designing specific queries, instructions, or examples within the prompt to direct the model's response. By tuning the language and structure of prompts, researchers can shape the model's output for different applications (maharjan2024openmedlm). Researchers can also use in-context learning by providing relevant example data within the prompt itself, priming the model to continue the pattern and apply similar logic to new inputs (yin2024deeper,bao2023tallrec). In this paper, we consider an approach in which we prompt off-the-shelf LLMs for a prediction of the next occupation using a textual representation of the worker's career history as the prompt. We show that including example resumes in the prompt helps improve the performance of off-the-shelf pre-trained LLMs, although performance is still worse than FT-LABOR-LLM.
\paragraph*{Other Applications of LLMs to Sequential Prediction Problems in Economics} LLMs have also been used to model time series data (jin_time-llm_2024) and in forecasting. For instance, faria2024artificial investigate the ability of LLMs to produce in-sample conditional inflation forecasts during the 2019–2023 period.
\paragraph*{The Biases and Representativeness of LLMs}A recent literature has emerged that aims to assess whether the outputs of foundation models are representative of larger populations, for example, whether the answers to opinion survey questions are representative of the population (santurkar_whose_2023, argyle_out_2023). One proposed approach is to query LLMs with survey responses from long-standing opinion surveys and see how aligned their responses are with the survey average. Our question differs in that we want to know whether the (fine-tuned) LLM can make predictions about occupation transitions that are representative of real-world transitions, conditional on history, which is a more complicated question to answer, as the population conditional probabilities are unknown due to the high-dimensional space of potential histories.
Our primary sources of data are three surveys of workers in the U.S. population. These surveys follow samples of individual workers, where workers are interviewed at regular intervals. The survey samples are constructed to be representative of the U.S. population at particular points in time.
The first dataset we consider is the Panel Study of Income Dynamics (PSID), which began in 1968. The sample of this dataset is intended to be representative of the United States as a whole, and new participants are added to the sample over time. Occupation information is consistently available starting in 1981, so we restrict our attention to survey years starting then. We further analyze data from two waves of the National Longitudinal Survey of Youth (NLSY). The NLSY 1979 follows a cohort of people aged 14-22 in 1979 throughout their careers. The NLSY 1997 began in 1997 and followed a cohort of individuals aged 12-16 at that time throughout their careers. We use extracts from these surveys to build longitudinal datasets for individual workers. Details of our dataset construction are reported in Appendix (ref).
We encode occupations using the occ1990dd system (occ1990dd) to map different versions of Census OCC occupational codes to a harmonized set of codes. In addition to the 332 occupations from the occ1990dd taxonomy, we include three special categories: “education,” “out of labor force,” and “unemployed.” We extract demographic characteristics, specifically gender, ethnicity, region of the country, and birth year. To simplify our analysis, we assign each worker a single, unchanging value for each demographic characteristic, typically the first valid value, and we do not allow it to change even if the original survey specifies different values in different survey years. We do not impute occupations for years without survey responses and focus on a single main occupation reported by the subject.
We refer to the cleaned versions of the three survey datasets as PSID81, NLSY79, and NLSY97. Table (ref) summarizes the total number of workers and transitions (individual-year survey observations, denoted by $\sum_i T_i$) in each survey dataset. The PSID81 dataset has 10.1 transitions per individual on average (median is 8 and maximum is 29), and the NLSY79 and NLSY97 track relatively fewer workers but have more transitions per individual, with 20.82 (median is 25 and maximum is 29) and 16.56 (median is 19 and maximum is 20), respectively, observations per worker on average.
As previously discussed, we convert individuals' complete career trajectories into natural language paragraphs using a text template. The total number of tokens ranges from 3.1 million to 7.9 million. The average length of career history $\textsc{Tmpl}(x_{i, \leq t}, y_{i, < t})$ ranges from 210 to 280 tokens depending on the dataset, while the average length of complete career history $\textsc{Tmpl}(x_{i, \leq T_i}, y_{i, \leq T_i})$ ranges from 250 tokens to 430 tokens; all of these templates fit well within Llama-2 model's context window of 4,096 tokens.
Transitions between two observations can be categorized into three types: first observation, moving (i.e., the current occupation is different from the occupation reported in the previous observation), and staying (i.e., the current occupation is the same as the occupation reported in the previous observation). Table (ref) shows the number of transitions by transition type, highlighting that between 50% and 60% of transitions involve staying in the same occupation.
We divide each of the three survey datasets into training (70%), validation (10%), and test (20%) samples, where all of an individual's transitions are allocated to the same set. Results about the performance of models are presented for the test set. Online Appendix (ref) and Table (ref) provide details and summary statistics.
Birth cohort affects workers' career trajectories (wachter2020persistent,doi:10.1177/0003122420966324). Figure (ref) shows that while birth years of PSID81 individuals span the range covered by NLSY79 and NLSY97, birth years of NLSY individuals are clustered within a small range due to the design of the NLSY surveys.
Table (ref) presents the top ten occupations in each dataset, highlighting commonalities and variations across datasets. Notable trends include “Not in labor force” ranking highest in all datasets, while occupations like “In education” show substantial variation, ranking 9$^{th}$ in PSID but 2$^{nd}$ and 1$^{st}$ in NLSY79 and NLSY97, respectively.
Figure (ref) illustrates the distribution of individual ages across calendar years for each dataset. In the NLSY datasets, the age distribution increases steadily over time, reflecting the longitudinal design that follows the same cohort of individuals. In contrast, the PSID dataset allows for dynamic changes in its subject pool, with individuals entering (e.g., upon becoming the head of a household) and exiting the study. Consequently, the PSID81 dataset exhibits a broader but more temporally stable age distribution. This figure highlights the degree of overlap in age distributions across the three datasets, suggesting that the occupation transition probabilities may be systematically different across the datasets.
In this paper, we re-implement the full pre-training and fine-tuning pipeline of the CAREER model so that we can carry out the fine-tuning step on identical survey datasets. Pre-training CAREER involves using a proprietary resume dataset of 24 million resumes acquired from Zippia Inc.\footnote{Zippia is a data-driven career intelligence platform that leverages analytics to provide personalized job recommendations, salary insights, and career development resources. The company aggregates labor market data to offer tailored guidance for job seekers, aiming to optimize their career decisions and employability. Other vendors providing similar data include Kaggle \url{https://www.kaggle.com/datasets/snehaanbhawal/resume-dataset} and Revelio \url{https://www.data-dictionary.reveliolabs.com/index.html}.} As described in Online Appendix (ref), we follow the approach of vafa_career_2024 to prepare and clean the data from Zippia Inc. This pre-training resume data represents resumes from the Zippia data as annual sequences of occ1990dd occupations, with tie-breaking rules for multiple jobs per year. Covariates include the year of each job, the latest educational degree, and the location, standardized following the approach we use for cleaning the survey datasets. Missing covariates are replaced by a special token, and missing occupational years are dropped. The final dataset comprises 245 million transitions (that is, individual-year observations).
We refer to a model that predicts an individual's next occupation as a function of career history and other individual characteristics as an occupation model. Let $t\in \{1,..,T_i\}$ correspond to a year in which an individual $i$ was surveyed and otherwise met our filtering requirements, where $T_i$ denotes the total number of individual-year observations of this individual. Note that our cleaned survey datasets do not, in general, have observations in every calendar year. We refer to an observation of an individual's occupation as a “transition,” with some abuse of terminology since we use the term for the first observation and also even when the individual stays in the same occupation. Let $\text{year}_{i, t}$ denote the calendar year corresponding to transition $t$ for individual $i$. We represent occupations as discrete variables, in particular following the occ1990dd, a variant of the OCC occupational classification system of occ1990dd, as described in Appendix (ref). Let $\mathcal Y$ denote the set of all occupations, and let $y_{i,t} \in \mathcal{Y}$ represent the occupation that an individual $i$ has at transition index $t$. We let $y_{i, < t} =(y_{i,1}, \dots, y_{i, t-1})$ denote an individual's occupation sequence prior to their $t$'th observation (for $t = 1$, define $y_{i, < t} = \emptyset$). Let $\mathcal{X}_\text{inv}$ be the support of time-invariant covariates (in our application, gender, race/ethnicity, region, and birth year, denoted by $x_i$), while $\mathcal{X}_\text{var}$ is the support of time-varying covariates (in our application, education and calendar year, denoted by $x_{i, t}$). Let $x_{i, \leq t} = (x_i, x_{i,1}, \dots, x_{i, t}) \in \mathcal{X}_\text{inv} \times \mathcal{X}_\text{var}^t$ denote the time-invariant covariates and time-varying covariates up to and including $t$. We refer to $(x_{i, \leq t}, y_{i, < t},)$ as the worker's career history at transition $t$.
The probability that the worker's next occupation is $y_{i,t}$, conditional on the worker's history, is written $P(y_{i,t} \mid x_{i, \leq t}, y_{i, < t})$.
We evaluate an occupation model's performance by comparing its predictions of an individual's next occupation to their actual next occupation. Specifically, we evaluate models by computing their perplexity, a commonly used metric in Natural Language Processing (NLP).\footnote{We primarily evaluate models using perplexity rather than more traditional metrics such as accuracy or F1 because occupational transitions are inherently imbalanced: around 50% of workers stay in the same occupation (see Table (ref)). Well-calibrated models will therefore almost always predict staying (i.e., $\text{argmax}_{y \in \mathcal{Y}} \hat{P}(y \mid x_{i, \leq t}, y_{i, < t}) = y_{i,t-1}$), making accuracy simply reflect the empirical staying rate rather than meaningful model differences. Perplexity instead uses the actual predicted probability $\hat{P}(y_{i,t} \mid x_{i, \leq t}, y_{i, < t})$, rewarding models that assign higher probabilities to correct occupations regardless of whether they involve staying or moving.} The perplexity is a negative monotonic transformation of the sample log-likelihood, with lower perplexity indicating that a model's predictions are more accurate. Formally, the perplexity of an occupation model $\hat{P}$ on a set of transitions (individual-year observations) for units $i=1,..,I$ is given by
where $w_{it}$ denotes the sampling weight for the individual relative to a population of interest. In this paper, for simplicity, we set $w_{it} = 1$. Note that a completely uninformative model that assigns uniform mass to each possible occupation would achieve a perplexity of $|\mathcal{Y}|$. We consider additional evaluation metrics (such as calibration) in Section (ref). We also consider predictive performance for several alternative outcomes of interest to economists, such as future unemployment status and coarse occupational categories; see Online Appendix (ref).
Model performance uncertainty comes from two sources: (i) randomness in model training (both the particular training sample and the stochastic elements of SGD), and (ii) randomness in the sample draw. We quantify (i) with fine-tuning multiple models on bootstrapped training sets and (ii) by evaluating the same model on bootstrapped test sets. Because full fine-tuning is computationally costly, we report test-set bootstrap standard errors for all models and provide training-set bootstrap standard errors only for the three focal models discussed in Section (ref). See Appendix (ref) for more details on both the training- and test-set-bootstrap standard errors.
The LLMs we use in this paper are trained to perform next-word prediction. Let $\mathcal{W}$ be the allowable set of words and punctuation, while $\cup_{j=1}^{\infty} \mathcal{W}^j$ is the set of sequences of words. In practice, LLMs work in the space of “tokens,” where words can be transformed into a sequence of tokens using a process called “tokenization.” We let $\mathcal{V}_\text{LLM}$ be the set of all possible tokens (also known as the vocabulary set) for a particular LLM. Popular commercial-scale LLMs typically use vocabulary sets with 10,000 to 200,000 tokens; for example, $|\mathcal{V}_\text{Llama-2}| = 32,000$. Let Tok$:\cup_{j=1}^\infty \mathcal{W}^j \rightarrow \cup_{j=1}^\infty \mathcal{V}^j$ denote the function mapping a sequence of words to a sequence of tokens.
The LLMs we consider can be viewed as estimating the probability that the next token equals $v_{j+1}$, conditional on a sequence of $j$ tokens (i.e., the prompt). LLMs also impose restrictions on the “context length,” or the maximum length of a sequence that can be conditioned on, which we denote $C$, with particular values $C_\text{LLM}$ imposed by different LLMs. Then, we let $\hat{P}^\mathcal{V}_\text{LLM}: \mathcal{V}_\text{LLM} \times \cup_{j \leq C_\text{LLM}} \mathcal{V}_\text{LLM}^j \to [0, 1]$ denote the LLM's estimate of the probability of the next token conditional on the input sequence, which for particular values of $v_1, \dots, v_{j+1}$ is written $\hat{P}^\mathcal{V}_\text{LLM}(v_{j+1} \mid v_1, \dots, v_j)$. The conditional probability $\hat{P}^\mathcal{V}_\text{LLM}(v_{j+1}, \dots, v_{j+j'} \mid v_1,\dots, v_j)$ for $j + j' \leq C_\text{LLM}+1$ can be derived from individual next-token predictions using the chain rule.
In this section, we describe the text template function we use to convert a worker's history of occupations and covariates into a sequence of words and punctuation. This text template can be combined with a tokenizer and an LLM to make next occupation predictions, as described in the next section.
We let $\textsc{Title}: \mathcal{Y} \to \cup_{j=1}^\infty \mathcal{W}^j$ denote the mapping from an occupation to its English-language title. Note that this mapping should be bijective (one-to-one). For example, the title of the occupation with occ1990dd code 95 is “nurse practitioners.”\footnote{Even though the occ1990dd system does not include job titles directly, one can crosswalk it to, for example, the Standard Occupational Classification (SOC) system, and use the list of job titles attached to each SOC code provided by the Bureau of Labor Statistics (\href{https://www.bls.gov/OES/CURRENT/oes_stru.htm}{https://www.bls.gov/OES/CURRENT/oes_stru.htm}).} The number of tokens needed to represent a job title depends on the tokenizer; using the Llama-2 tokenizer, the number ranges from 2 to 28 in the survey datasets we analyze, with an average length of 8.3 tokens (and a standard deviation of 4.8).\footnote{“Grinding, Lapping, Polishing, and Buffing Machine Tool Setters, Operators, and Tenders, Metal and Plastic” (28 tokens) and “Cutting, punching, and press machine setters, operators, and tenders, metal and plastic” (24 tokens) are the two longest job titles. The shortest job titles include “Cooks”, “Bakers”, “Tellers”, and “Designers”.}
To represent covariates as text, we express an individual's educational status using values such as graduate degree. Online Appendix (ref) provides the full mapping between occ1990dd codes and their job titles.
Building on this strategy, we define a text template function, $\textsc{Tmpl}(x_{i, \leq t}, y_{i, < t})$, that transforms an individual's career history into a textual summary. The text template incorporates additional punctuation, line breaks, and metadata, as detailed in Appendix (ref), and illustrated in the following example.
{ \scriptsize
}
The example above is defined as the text representation of the complete career history of the individual, denoted $\textsc{Tmpl}(x_{i, \leq t}, y_{i, \leq T_i})$, where $T_i$ represents the number of transitions for individual $i$. These complete career histories are used for model fine-tuning, as discussed in Section (ref).
Note that the individual can stay in the same occupation for multiple records (e.g., 1984 and 1985 in the example); the text representation explicitly reflects this information. Additionally, the dataset could miss an individual for certain years in her career trajectory; in this case, the text template will skip those years (e.g., 1987 and 1995 in the example).
We also create the text representation of the career history of the same individual prior to the $t^\text{th}$ occupation, denoted $\textsc{Tmpl}(x_{i, \leq t}, y_{i, < t})$, by truncating the complete career history. For example, to obtain an LLM's predictions of an individual's occupation in 1989 given the covariates and occupation history, we would use as input the text above, removing the text “Cleaners of vehicles and equipment" and everything afterward (i.e., the underlined part in the example). That is, we apply the text template function to all previous occupation and covariate information, and conclude with a partial row for the occupation to be predicted.
On average in the survey datasets we consider, the text representation of workers' complete career histories contains around 250 to 500 tokens using the Llama-2 tokenizer, which fits within the context window of Llama-2 models for fine-tuning. For inference, the prompt encoding of an individual's career history, i.e., $\textsc{Tok}(\textsc{Tmpl}(x_{i, \leq t}, y_{i, < t}))$, consists of 200 to 300 tokens on average. Detailed summary statistics on token counts can be found in Online Appendix (ref).
In this paper, we use LLMs in three ways. First, we use an LLM to directly produce a “predicted occupation” in response to a “prompt.” More precisely, if we first map job codes to text (the English language job title) using the text template function described in the previous subsection, and then use a tokenizer to translate the resulting sequences of past jobs into a sequence of tokens, an LLM will produce a textual “response” that is a sequence of tokens. That sequence may or may not correspond to a valid occupation, but we can, in principle, further transform the output in various ways to interpret it as an occupation. Of course, a textual response or a single predicted occupation is not an estimate of the probability of a sequence of tokens. Some commercial LLMs allow the user to set a “temperature” parameter when submitting a prompt, where a particular setting is designed to approximate sampling from the distribution of responses. Probabilities can then be estimated by repeatedly prompting the LLM. We do not follow this approach in this paper; instead, we restrict attention to LLMs where probabilities (or, where relevant, embeddings) can be directly obtained by the analyst.
Second, for those LLMs for which it is possible, we directly obtain the probability assigned to a given token. This functionality may be enabled in the setup of an open model such as Llama-2, or it may be exposed through an API in the case of a closed model such as ChatGPT-4.\footnote{For example, \url{https://cookbook.openai.com/examples/using_logprobs} explains how to use the logprobs parameter in OpenAI API requests to evaluate token probabilities, allowing analysis of model confidence and alternative predictions for improved understanding of text generation.} For example, for the LLM Llama-2 7 billion parameter model, denoted Llama-2-7B, we let $\hat{P}^\mathcal{V}_\text{Llama-2-7B}(\text{``Engineer''} \mid \text{``Software''})$ denote the estimated probability that “Engineer” follows the single-token sequence “Software”. To obtain the probability of the next occupation given a sequence of prior occupations, we first use the text template function and the tokenizer to translate the occupation history into a sequence of tokens; similarly, we translate the title of a particular next occupation $y_{i,t+1}$ into a sequence of tokens. The estimated next-token probability model associated with the LLM, denoted by $\hat{P}^\mathcal{V}_\text{LLM}(\cdot \mid v_1, \dots, v_j): \mathcal{V}_\text{LLM} \to [0, 1]$, can be applied several times to determine (using the chain rule of probability) an estimate of the probability that the sequence of tokens induced by $y_{i,t+1}$ follows the sequence of tokens induced by $(y_{i,1},\dots,y_{i,t})$. A language-based next-token prediction model thus induces an associated occupation model, as follows. See Appendix (ref) for details.
Third, some LLMs make it possible to extract a lower-dimensional “embedding” of text, where any sequence of tokens is associated with a real-valued vector. For example, for the Llama-2-7B LLM that we use in this paper, input text is represented as a vector of 4,096 floating-point numbers. Formally, we let $\mathcal{E}_\text{LLM}: \cup_{j \leq C_\text{LLM}} \mathcal{V}_\text{LLM}^j \to \mathbb{R}^{d_\text{LLM}}$ be the “embedding function”, where $d_\text{LLM}$ denotes embedding dimension. The composite function $\mathcal{E}_\text{LLM} \circ \textsc{Tok}$ generates the embedding of any input string of words (i.e., the “prompt”).
Both CAREER and our preferred model, FT-LABOR-LLM, make use of the transformer neural network functional form for the occupation transition model. In addition, both make use of the foundation model and fine-tuning approach popularized in the artificial intelligence literature. FT-LABOR-LLM builds on the Llama-2 series of LLMs. Here, we provide an overview of the general approach used to estimate and fine-tune Llama.
A transformer model can be viewed as a multinomial logistic regression model where the outcome $v_j$ is selected from the set of tokens $\mathcal{V}$ and the covariates are the previous tokens $v_{1:j}$ in the sequence. To estimate the probabilities associated with each next token given history, the model first maps the history $v_{1:j}$ into an embedding using a parameterized embedding function (defined in Section (ref)) $z_j = \mathcal{E}_\text{tsf}(v_{1:j};\gamma)\in \mathbb{R}^{d_\text{tsf}}$. The embedding function is specified as a transformer neural network where the parameters determine how strongly tokens in earlier and later positions should be weighted in forming the prediction of the next token, and these weights vary with the specific tokens in the history. Given an embedding $z_j$, the probability that the next token equals $v \in \mathcal{V}$ can be written as $$ \hat{P}^{\mathcal{V}}_{\text{Transformer}}(v_{j+1}=v \mid v_{1:j}) = \frac{\exp\!\big(z_j^\top \hat{\beta}_v\big)} {\sum_{v' \in \mathcal{V}} \exp\!\big(z_j^\top \hat{\beta}_{v'}\big)}, $$ where $\hat{\beta}_v \in \mathbb{R}^{d_\text{tsf}}$ is the parameter vector associated with the outcome that the next token is $v$. The parameters in the embedding function $\hat\gamma$ and the coefficients $\{\hat{\beta}_v\}_{v \in \mathcal{V}}$ are estimated jointly using maximum likelihood. Each token is an outcome and thus corresponds to an observation. It is standard practice to estimate the parameters using variations of stochastic gradient descent (touvron_llama_2023), where observations are batched into relatively small batches and for each batch, parameters are updated in the direction that increases the likelihood, where the direction is determined by calculating the gradient of the objective function evaluated at the current batch of observations.
Estimating the parameters of a foundation model, sometimes referred to as “pre-training,” refers to estimation using a large, potentially unrepresentative dataset. Fine-tuning the model refers to estimating the same parameters using a second, distinct dataset that better represents the eventual prediction tasks of interest, but where the starting values for parameter estimation are the ending values from the pre-training phase. The fine-tuning phase typically proceeds through multiple “epochs,” where an epoch is a complete pass through the fine-tuning data. Using more epochs is analogous to placing a higher weight on the fine-tuning data. Performance of the model on a held-out validation set (distinct from the test set) is used to select the number of epochs.
We use the term FT-LABOR-LLM to refer to the combination of a base model (either Llama-2-7B or Llama-2-13B) and fine-tuning data, as well as to refer to the union of the fine-tuned models we evaluate in this paper. Consider the implementation of fine-tuning for FT-LABOR-LLM (see Appendix (ref) for details). We fine-tune the LLMs to predict each token of a textual summary of worker careers, including punctuation and metadata, so that the FT-LABOR-LLM learns the structure of the text template as well as the conditional probabilities of tokens corresponding to occupations. For each individual $i$, we use the text template discussed in Section (ref) to build a text representation of their entire career, denoted as $\textsc{Tmpl}(x_{i, \leq T_i}, y_{i, \leq T_i})$. In estimation, for each $i$, each token $(i,t)$ is a distinct outcome and corresponds to a distinct observation in training.
We fine-tune the two Llama-2 models (7B and 13B) separately on each of the three training set text templates, resulting in three sets of fine-tuned models. We refer to the occupation models derived from these fine-tuned models through (ref) as FT-7B and FT-13B, dropping the “Llama-2” nomenclature because we only fine-tune Llama-2 models. The fine-tuning procedure is illustrated in Figure (ref). The resulting FT-LABOR-LLMs are themselves LLMs, and we create estimates of $\hat{P}_\text{LLM}$ based on each of them.
In practice, for estimation, we simply upload the textual resume files to the LLM hosting service Together AI and make use of their fine-tuning service. When fine-tuning FT-LABOR-LLM we use either three or five epochs, whichever has better performance in the validation set. For making predictions, we apply the estimated transformer neural network to resumes in the test set.
In this section, we provide details on the CAREER model, which builds on the same transformer architecture described in Section (ref). The CAREER model specializes this architecture for occupational sequences in several ways. First, CAREER restricts the vocabulary to the set of occupations, $\mathcal{V}_\text{CAREER} = \mathcal{Y}$, and directly models sequences of occupations rather than sequences of words. Second, at each step, CAREER embeds not only the previous occupation token but also covariates relevant for predicting the next occupation. These include static attributes (e.g., gender, region, birth year) and time-varying attributes (e.g., calendar year, most recent education level). Third, CAREER replaces the simple multinomial logistic over $\mathcal{Y}$ with a two-stage nested logistic structure: it first predicts whether the individual stays in the current occupation or moves; and conditional on a move, it outputs a distribution over new occupations in $\mathcal{V}_\text{CAREER} \setminus \{v_J\}$. This captures the empirical pattern that workers exhibit a high probability of remaining in the same occupation across periods. Fourth, CAREER is trained in two phases. It is first pre-trained on 24 million occupation sequences from resumes to learn general career patterns (i.e., the foundation model), and then fine-tuned on smaller, representative survey datasets to align the model with the target population. Online Appendix (ref) provides more details.
In this section, we compare the performance of alternative occupation models.
This section exploits LLMs for occupational modeling by training a multinomial logistic regression model with $\mathcal{Y}$ as the set of alternatives and embeddings derived from off-the-shelf (that is, not fine-tuned) LLMs as covariates. Formally, let $z_{i, t}$ be an embedding vector (defined in Section (ref)) of the career history $x_{i, \leq t}, y_{i, < t}$ derived from an LLM. For each occupation $y \in \mathcal{Y}$, the model estimates a parameter $\hat\beta_y$ such that $ \hat{P}_\text{MNL}(y_{i, t} \mid x_{i, \leq t}, y_{i, < t}) = \frac{\exp(z_{i, t}^\top \hat\beta_{y_{i, t}})}{\sum_{y' \in \mathcal{Y}} \exp(z_{i, t}^\top \hat\beta_{y'})}. \label{eq:MNL} $ The parameters $\{\hat\beta_y\}_{y \in \mathcal{Y}}$ are estimated via maximum likelihood.
We first convert the career history $(x_{i, \leq t}, y_{i, < t})$ to natural language using the text template described in Section (ref). We then pass the text to an LLM and extract the model's embedding, $\mathcal{E}_\text{LLM}(\textsc{Tmpl}(x_{i, \leq t}, y_{i, < t})) \in \mathbb{R}^{d_\text{LLM}}$. This approach requires that the researcher has access to the embeddings from the LLM either through an API or by using an open-weight model. We consider a wide range of off-the-shelf models to embed career histories into embedding vectors, including Llama-2-7B/13B, Llama-3.1-8B, Llama-3.2-1B/3B, as well as the text-embedding-3-large text embedding model provided by OpenAI.\footnote{This model was released January 2024 and was the latest available as of December, 2025.} We then train a multinomial logistic regression on top of these embeddings for the next occupation prediction task. See Appendix (ref) for details.
Table (ref) compares performance across baseline models. CAREER outperforms all of the embedding-based multinomial logistic regression models.\footnote{For CAREER, predictions were made directly; we do not use CAREER as an embedding engine and build multinomial logistic regression on top of the embeddings.}
To interpret these results and the value added from incorporating text embeddings, we also compare performance against another simple baseline, a simple empirical transition probability model. Let $\#^\text{(train)}\{y\}$ and $\#^\text{(train)}\{y \to y'\}$ denote the count of occupation $y$ and the transitions $y \to y'$ in the training data, respectively. This model estimates the probability of transitioning from occupation $y$ to $y'$ (where all individuals are in the “null” occupation when $t=0$) as $ \hat{P}_\text{Empirical}(y_{i, t} \mid x_{i, \leq t}, y_{i, < t}) = \frac{\#^\text{(train)}\{y_{i, t-1} \to y_{i, t}\}+1}{\#^\text{(train)}\{y_{i, t-1}\}+1}, $ adding 1 to the numerator and denominator to handle zero counts. This model predicts based solely on the immediately preceding occupation without using any covariates. Across all datasets, Table (ref) shows that the embedding-based models have better performance than the empirical transition model, but worse than CAREER.
In this section, we analyze the performance of occupational models based on LLMs fine-tuned on text templates created from our survey datasets.
Table (ref) reports the test set perplexity of the FT-LABOR-LLM occupation models along with several baselines. We see that FT-7B and FT-13B achieve substantially lower perplexity than CAREER, which was pre-trained on 24 million resumes and fine-tuned for occupation modeling on survey data. The differences between CAREER and FT-7B are about ten times larger than the test-set-bootstrap standard errors (defined in Section (ref)) for PSID81 and NLSY79, while they are similar in size to the standard error for NLSY97, a substantially smaller dataset. FT-13B exhibits even larger performance improvements. Online Appendix (ref) shows that FT-7B and FT-13B generally achieve lower perplexity than CAREER within education-defined subgroups, with CAREER slightly ahead only in the relatively small NLSY97 college subgroup.
As previewed in Section (ref), one question that naturally arises is whether sampling variation in the training set and randomness in the fine-tuning estimation algorithm lead to substantial variation in estimates of performance difference. In Appendix (ref), we carry out a small experiment with training-set-bootstrapping. The training-set-bootstrap standard errors for perplexity of FT-7B are 0.051, 0.058, and 0.020 for PSID81, NLSY79, and NLSY97, respectively (where to facilitate other comparisons, we included birth year in the estimation and these models are fine-tuned using pooled training set). These standard errors are smaller than those reported in Table (ref). We calculate the training-set-bootstrap standard error for the difference between FT-7B and FT-13B only for PSID81, and found a standard error of 0.029, larger than that corresponding test set standard error. This exercise suggests that variation due to training is not negligible, and small performance differences that appear to be statistically distinguishable from zero using test-set-bootstrap standard errors could in fact arise due to training uncertainty. Due to the large cost of training-set-bootstrapping, we report test-set-bootstrap standard errors in the rest of the paper, but we are cautious in interpreting marginally significant results.
Next, we turn to analyzing the sources of differences between the models. We compare performance for the task of predicting the binary outcome of whether workers move to a different job $\text{move}_{i,t} = \mathbf{1}\{y_{i,t} \neq y_{i,t-1}\}$; and separately, we analyze performance conditional on a transition involving a move. Changing occupations may be an outcome of particular interest in empirical studies of, e.g., the effects of layoffs, training programs, or technological change.
A standard way to evaluate the performance of alternative prediction models for binary outcomes is to compare the area under the Receiver Operating Characteristic curve (AUC-ROC) in the test set, which ranges from 0 (the worst possible model) to 1 (the best possible model). Table (ref) shows that the FT-7B model has AUC-ROC of 0.784, slightly greater than CAREER at 0.776. The empirical transition frequency benchmark model has a much lower AUC-ROC of 0.639.
To assess how well-calibrated each model is, we split observations into ten groups based on deciles of predicted probability of changing occupations $\hat{P}(\text{move}_{i,t})$ (i.e., the next occupation $y_{i, t}$ is different from the previous one $y_{i, t-1}$), denoted as $G_1, G_2, \dots, G_{10}$. Then, for each group, we compute the empirical percentage of movers. If a model is well-calibrated, the average predicted $\hat{P}(\text{move}_{i,t})$ should match the actual proportion of movers within the corresponding group in the test set. We further calculate the average (over deciles) of the calibration error \resizebox{0.5\linewidth}{!}{$\sqrt{\frac{1}{10} \sum_{i=j}^{10} \left[\left(\sum_{(i, t) \in G_j} \mathbf{1}\{\text{move}_{i, t}\} \right) - \left(\sum_{(i, t) \in G_j} \hat{P}_\text{model}(\text{move}_{i,t})\right) \right]^2}$}.
Figure (ref) illustrates calibration plots of the empirical transition frequency baseline, CAREER model, FT-7B model, as well as their corresponding calibration errors. The diagonal line in the plot represents a perfectly calibrated model.
We observe that our FT-7B model is better calibrated in predicting staying versus moving than the CAREER model, which underestimates moving in some groups and overestimates it in others. The CAREER model has a two-stage prediction design (i.e., predict staying versus moving, then next occupation sequentially), and the training process of CAREER pays special attention to enforcing the model calibration. In contrast, our LLM fine-tuning does not give special treatment to matching the empirical probability of staying, so it is somewhat surprising that it is better calibrated in this dimension than CAREER; with its extremely large parameter space, the FT-7B model appears to learn these probabilities without special treatment in the model.
Table (ref) reports perplexity conditional on moving for alternative models, while the bottom panel reports differences between FT-LABOR-LLM models and CAREER. We see that the FT-7B and FT-13B models outperform all other models on PSID81 and NLSY79, and FT-13B achieved comparable performance to CAREER on NLSY97. Note that occupation transitions conditional on moving are inherently harder to predict, explaining the higher level of perplexity in this table.
Beyond studying whether workers change occupation, it may be of interest to study other outcomes derived from occupational transitions, such as transitions into unemployment, or transitions out of working (either to unemployment, education, or out of the labor force). Online Appendix (ref) shows that FT-LABOR-LLM, despite being fine-tuned only for the more granular next-occupation prediction task, achieves similarly strong relative performance for these alternative outcomes of interest.
Because the surveys we study were typically conducted every other year, FT-LABOR-LLM typically needs to make predictions about transitions separated in time by two years. However, it is possible to use FT-LABOR-LLM to make predictions about transitions for years that we did not directly observe, including the years between surveys. A potential limitation of FT-LABOR-LLM is that it may not be internally consistent when predicting transitions when survey observations are separated in time; the prediction that comes out of the LLM is not constrained to be equal to the result if we were to make sequential predictions for each year and combine them via Bayes' rule. In particular, the probability FT-LABOR-LLM models assign to $y_{i,t}$ when $\text{year}_{i,t}=\text{year}_{i,t-1}+2$ is not necessarily equal to the estimated probability by applying the model year-by-year, composing its predictions about $\text{year}_{i,t}=\text{year}_{i,t-1}+1$ and predictions about $\text{year}_{i,t}$ conditional on potential occupations in $\text{year}_{i,t-1}+1$. In Appendix (ref), we compare the model's direct predictions about a transition across two years with those constructed based on a sequence of two one-year-ahead predictions and show that the correlation between the two predictions is 0.93, meaning that the model appears to correctly account for the gap in calendar time when making predictions. We leave further exploration of this issue for future work.
In this section, we evaluate the importance of fine-tuning. We report results about the performance of occupation models based on off-the-shelf (rather than fine-tuned) LLMs, applying Equation ((ref)) to estimate $\hat{P}_\text{LLM}$ for several alternative LLMs.\footnote{To improve computational efficiency for prediction, we quantize all LLMs in this paper to 8-bit precision (i.e., compress model weights to 8-bit integers to speed inference with minimal precision loss) while running model inference. We perform full-precision inference on a few subsamples, and the difference in performance was small. See Online Appendix (ref) for more details on full-precision versus quantized model analysis.} Because evaluating perplexity requires accessing a model's assigned probabilities, we restrict attention to open-source LLMs where it is possible to obtain predicted probabilities directly, with the exception of Section (ref) of Online Appendix (ref), where we evaluate the ability of OpenAI gpt-4o-mini to produce valid job titles in response to a prompt.
In particular, we study open-source LLMs from the Llama family of models: Llama-2, Llama-3.1, and Llama-3.2. For example, Llama-2 models were trained by Meta on approximately 2 trillion tokens of text, much of it from the Internet, and are among the most capable open-source LLMs currently available (touvron_llama_2023). We do not study bigger models such as Llama-2-70B and Llama-3.1-405B because evaluating this model across many variations requires substantial cost.
Table (ref) contains the perplexity of off-the-shelf LLMs, which are substantially higher than other models discussed. \footnote{For reference, a completely uninformative model that assigns uniform mass to each possible occupation would achieve a perplexity of $|\mathcal{Y}|$, which is 335.} The unsatisfactory performance of off-the-shelf LLMs can be attributed to two factors: off-the-shelf LLMs are not adapted to the career trajectory distributions in our survey dataset, and these LLMs do not know the set of valid job titles to predict. To better understand the poor performance of the model based on off-the-shelf LLMs, we assess the responses that the LLMs provide when prompted with examples of tokenized text templates. Online Appendix (ref) provides some examples. While the responses appear plausible, the LLMs also assign mass to strings that are not valid job titles.
Online Appendix (ref) explores alternative prompting strategies that direct LLMs to restrict prediction to valid occupations when computing occupation probabilities. While we observe notable reductions in perplexity with these prompting strategies, the resulting model perplexities remain orders of magnitude above those achieved by CAREER. As mentioned, LLMs can sometimes hallucinate and produce invalid job titles; however, these alternative prompting strategies help mitigate this issue. Off-the-shelf Llama variants and GPT-4o-mini generate valid job titles with probabilities climbing from roughly 0.6 to above 0.9 as prompts add explicit job-title lists and up to ten in-context resume examples. Online Appendix (ref) presents more details.
In this section, we analyze the roles of model complexity (number of parameters) and of quantity of data in determining performance. As discussed in the introduction, analysts using fine-tuned LLMs will need to consider costs of computation in the fine-tuning process, as well as when making predictions from the model, costs which increase with model complexity. These costs may be traded off against improved accuracy from more complex models. Another tradeoff arises when acquiring more data: extra data may come from a different context and thus correspond to a different data-generating process. Incorporating non-representative data in fine-tuning may or may not improve performance.
We empirically evaluate these tradeoffs by varying the datasets used for fine-tuning, for example, by combining datasets, while holding the three test sets fixed. To facilitate our discussion, we use $\mathcal{D}_{\text{data}}^\text{(split)}$ to denote a particular split of the dataset $\omega$, for example, $\mathcal{D}_\text{PSID81}^\text{(train)}$ represents the training split of the PSID81 dataset. We explore the consequences of fine-tuning based on a different survey dataset, or combinations of survey datasets, than the survey from which the test set is drawn. Recall that all three of the survey datasets we analyze are approximately representative of the U.S. population, but as shown in Section (ref), they incorporate different distributions of calendar year, as well as different conditional distributions of birth year for each calendar year, where we include both of these variables in the text templates for the analyses in this section.
In our first exercise, reported in Table (ref), we evaluate models fine-tuned using the training split of one survey data, $\mathcal{D}_\omega^\text{(train)}$, and the test split from another dataset, $\mathcal{D}_{\omega'}^\text{(test)}$, with $\omega \neq \omega'$. This exercise shows how training from data with very different distributions of birth year and calendar year affects performance; since FT-7B and FT-13B are trained using information about both of these variables and the transformer neural network allows for rich interactions, in principle, the model could be flexible enough to predict well across distributions. In particular, the PSID81 dataset has overlap in terms of calendar year and birth year with both NLSY datasets, and it is substantially larger overall. However, the results illustrate significantly degraded predictive performance when the training data and test data are from different survey datasets.
Next, we evaluate the value of data by first pooling all training data from survey datasets together, so that $\mathcal{D}_\text{all}^\text{(train)} = \bigcup_{\omega \in \text{\{PSID81, NLSY79, NLSY97\}}}\mathcal{D}_\omega^\text{(train)}$. Then, we sample $P\%$ of individuals from $\mathcal{D}_\text{all}^{\text{(train)}}$ and use the sample to fine-tune a Llama-2-7B model. Finally, we evaluate the FT-LABOR-LLM on the test split of each survey dataset separately. Table (ref) summarizes the performance of these models. The model's performance improves as we increase the amount of training data (i.e., raise the value of $P$), and the returns to data are diminishing. On the test split of dataset $\omega$, models fine-tuned on the aggregated dataset eventually outperform the model fine-tuned on the corresponding training set $\mathcal{D}_{\omega}^\text{(train)}$, when $P \geq 80$. In addition, the models fine-tuned on the pooled data with FT-7B eventually outperform FT-13B trained on the individual baseline training sets, showing that adding data, even data from different distributions, can substitute for model complexity. Note, however, that the improvement on PSID81 is small enough (0.06) relative to the test-set-bootstrap standard error that the uncertainty derived from training may be large enough to overturn the statistical significance of the result. Indeed, in Appendix (ref) we find a training-set-bootstrap standard error of 0.055 for this improvement, which together with test-set uncertainty would render the improvement not statistically significant.
In Appendix (ref), we consider another variation of the analysis, incrementally adding pooled data to the full baseline training set for a given survey. We find that adding the data from other surveys to the full baseline training set immediately improves performance, and increasing the training set size by 30% allows FT-7B to match or surpass the performance of FT-13B.
Our analyses demonstrate that our best-performing model, directly predicting occupations through text tokens using FT-LABOR-LLM, achieves superior perplexity scores compared to the previous state-of-the-art CAREER model. This section delves deeper into the sources of performance differences.
One key difference between LLMs and baseline models, besides the number of parameters, is that the LLMs have an understanding of textual data. This section examines whether LLMs' performance is driven by their rich, deep neural network architecture or their advanced understanding of the meaning of occupations based on textual data. To do so, we create an alternative prediction space with “numeric job titles” only. We assign each occupation $y \in \mathcal{Y}$ a randomly chosen numeric job title (in contrast to their original literal job title) from job_000, job_001, ....(e.g., Cashiers is mapped to job_045); all numeric job titles have three digits. Then, we replace all original literal job titles in the text representation with their corresponding numeric job titles, denoted as $\textsc{Tmpl}^\text{(numeric)}(x_{i, \leq t},y_{i, < t})$. Online Appendix (ref) provides an example of career history text representations with numeric job titles.
For each survey dataset, we fine-tune the Llama-2-7B model using the training corpus with numeric job titles only, and denote that fine-tuned model as FT-7B-NUMERIC; then, we compare FT-7B-NUMERIC to FT-7B fine-tuned on the corresponding training split of a single survey data. While evaluating performance, we use the conditional probability of numeric job titles assigned by the LLM. For example, the predicted probability of the next occupation being cashier is $P_\text{LLM}(\texttt{job\_045} \mid \textsc{Tmpl}^\text{(numeric)}(x_{i, \leq t},y_{i, < t}))$ instead of $P_\text{LLM}(\text{Cashier} \mid \textsc{Tmpl}(x_{i, \leq t},y_{i, < t}))$, where historical occupations are also replaced with numbers.
Table (ref) shows the performance of FT-7B-NUMERIC, which performed much worse than the FT-7B model using literal job titles. Our results indicate that an important contributor to the LLM's performance comes from the LLM's ability to use the textual content of the occupation title to make predictions; using numeric job titles disassociates this knowledge from the prediction task and hurts performance significantly.
This section describes several additional analyses that shed light on the sources of performance improvements.
We evaluate the importance of demographic variables (gender, ethnicity, and region) for the fine-tuned model's predictive performance by randomly replacing these covariates in the test set with values from the validation set and measuring the impact on perplexity. The analysis reveals that randomizing gender alone significantly degrades performance (e.g., increasing perplexity by $\sim 12\%$ in PSID81), while ethnicity alone has about one-quarter of gender's effect, and region has minimal impact. Simultaneously randomizing all three variables yields the greatest performance decline, demonstrating that complex interactions among demographic covariates are crucial for prediction. Appendix (ref) provides more details.
In Appendix (ref), we test how much historical information the model uses to predict the next occupation by predicting with prompts that only include the last $k \in \{5, 10, 15, 20, 25\}$ observations and evaluate the perplexity lift within age-banded test subsets. Results indicate that adding history from five to 10-15 prior records lowers perplexity sharply, while additional context delivers diminishing returns: for later career transitions ($t > 20$), most predictive information is captured within just 10-15 years of history.
Appendix Table (ref) shows the extent to which the embeddings created by FT-7B fine-tuned using PSID81, the largest dataset, incorporate more information about the meaning of job titles. One way to approach this analysis is to assess the predictive power of these embeddings on a task that relates to the interpretation of the titles. We consider a particular task that requires such an understanding: predicting which part of the occupation code hierarchy a particular occupation falls into (this information was not used in LABOR-LLM, although it may have been one part of the enormous pre-training corpus for the original Llama models). We compare the predictions derived from a multinomial logistic regression using as features embeddings extracted from FT-7B and CAREER. We show that the embeddings from FT-7B have a test-set accuracy of 82% for predicting the correct SOC group for an occupation, which is better than CAREER (78%).
In a second exercise (Appendix (ref)), we characterize the economic conditions under which the language-based LABOR-LLM (FT-13B) outperforms the CAREER baseline for “mover” transitions in the test split of the PSID81 dataset. While CAREER is a sophisticated transformer that effectively captures the “structural syntax” of occupational sequences via embeddings, it only relies on discrete categorical occupational codes. In contrast, LABOR-LLM leverages the semantic richness of natural language encoded in the literal titles. Using a Generalized Random Forest (GRF) to analyze heterogeneity in model performance, we find the language-based model provides the highest value-added where transitions are less predictable: during “complex” labor market adjustments that cross occupational boundaries (different SOC groups) or involve theoretically distant skill sets (low O*NET similarity). This suggests that the LLM uncovers semantic linkages between occupations that administrative codes obscure. Furthermore, LABOR-LLM's advantage grows with the length of the career history (i.e., for later transitions in the worker's career history), suggesting that textual representations better encode human capital accumulation than the categorical occupation codes of CAREER.
This paper proposes a novel approach, LABOR-LLM, to the problem of predicting a worker's next occupation conditional on history. The best-performing version of this approach, FT-LABOR-LLM, translates the tabular data about a worker's history from publicly available U.S. surveys (PSID and NLSY) into text files that resemble resumes, and then fine-tunes the Llama-2 open-weight foundation models on that corpus. Then, to estimate the probability that the next occupation is a particular occupation, say “engineer,” given worker history, the approach prompts the fine-tuned LLM with the textual version of the worker's history, and extracts the probability that the LLM assigns to the text “engineer” as the next word. We show that, off-the-shelf, without fine-tuning, this approach performs poorly even when OpenAI's API is used. However, the fine-tuning leads this approach to outperform all existing benchmarks. The fine-tuning eliminates the problem of occupation title “hallucinations,” but more importantly, it leads the model to make accurate predictions about conditional probabilities in held-out test data. Accurate, fine-grained predictions enable economists to ask and answer more nuanced questions, and to improve the quality of causal inference analyses that rely on accurate predictions.
The paper explores some of the sources of the strong performance of FT-LABOR-LLM, showing that representative data is important, but that adding more data (even non-representative data) can lead a smaller model (in terms of number of parameters) to outperform a larger one. The paper also shows that FT-LABOR-LLM makes use of many years of history, even a worker's early career history, to improve prediction quality. Our results illustrate that the approach can be effective in datasets of moderate size (tens of thousands of transitions), leveraging the general information about occupations embedded in the open-weight LLM's embeddings of the text of job titles and resumes.
An advantage of the FT-LABOR-LLM approach is that all data and software necessary to apply this approach is available publicly, including the weights of the LLM, so that the main cost in practice is the cost of the computing for fine-tuning and making predictions. Low-cost cloud-based services are available (we used the service provided by Together AI) that enable fine-tuning by simply uploading documents; with these services, no coding is required for the training step, and minimal original coding is required to obtain predictions from the fine-tuned LLM. Thus, researchers can focus on analyzing the results and performing downstream empirical exercises. However, a limitation to this approach is that fine-tuning can become expensive as the dataset size grows, and repeatedly fine-tuning (for example, to bootstrap standard errors) can be prohibitively expensive.
It is important to recognize that LLM-based approaches share a common limitation with all predictive models used in economic analysis: their performance can deteriorate when the underlying data-generating process changes. These models rely on historical patterns and, therefore, may generalize poorly during periods of structural change, such as rapid AI adoption or other macroeconomic disruptions, when past transitions no longer reflect future dynamics. This limitation underscores the need for caution in such settings and points to future work, such as combining LLM-based prediction with structural economic models.
Using a textual representation of resumes has the advantage of incorporating other types of data into the analysis. For example, some datasets include job descriptions or descriptions of accomplishments as part of a resume; the LABOR-LLM framework makes this type of information straightforward to incorporate.
An approach based on publicly available foundation models may also be useful in other settings, for example, any economic prediction problems that involve discrete outcomes with many alternatives and where the alternatives may be associated with meaningful textual descriptions. A sequence of purchases made by a consumer may have a similar structure. Our paper also illustrates the importance of fine-tuning: off-the-shelf LLMs may make plausible sounding predictions, but without fine-tuning they are unlikely to give accurate conditional probabilities for any particular dataset of interest.
When comparing the performance of alternative occupation models, we wish to quantify the uncertainty about estimates of performance. The randomness in measured perplexity for a given model arises from several sources: sampling variation in the training data, randomness in the fine-tuning pipeline (e.g., data shuffling for a stochastic gradient descent optimizer), and sampling variation of the test data.
To estimate the uncertainty arising from the first two sources, we bootstrap the training set used for fine-tuning (sampling at the individual level) and estimate the variation in measures of the performance of models across bootstrap samples. We refer to the resulting standard errors as “training-set-bootstrapped.” To capture sampling variation of the training set, we sample with replacement.
According to the support team of \href{https://www.together.ai/}{Together AI}, the platform we use to fine-tune LLMs, the randomness in their fine-tuning process arises mainly from randomizing the order of observations in the process of optimizing via stochastic gradient descent; each instance of re-tuning a bootstrap sample will include randomization of this type.\footnote{Unfortunately, at the time of this writing, there is no way to specify the random seed for reproducibility. The support team also mentioned “adapter weight initialization” as another source of randomness in the fine-tuning pipeline, which is only relevant if one is fine-tuning using the Low-Rank Adaptation (LoRA) technique. We are doing full-parameter fine-tuning instead.} Because fine-tuning is very expensive to carry out, we conduct an experiment for three of the models, as described below in Section (ref) and Appendix (ref).
To estimate the uncertainty arising from sampling variation in the test set, we bootstrap the test set and refer to the resulting standard errors as “test-set-bootstrap.” We sample at the individual level with replacement and, in the analysis we report below, use 100 bootstrap replications.
We employ a similar bootstrapping approach to calculate the test-set-bootstrap standard errors for the differences in perplexities between the two models. We select bootstrap samples at the individual level, compute the perplexities for both models on the bootstrap sample, and calculate the standard deviation of the difference in perplexities.
The purpose of our bootstrapping approach for the test set is to estimate the sensitivities of our metrics (e.g., perplexities and differences in perplexities) to changes in the test set distribution. In the main paper, we first report the metric (e.g., perplexity) computed using all observations in our test set. Then, we create $B$ bootstrap samples of the test set, sampled on the individual level, to estimate the standard error of the metric. For each bootstrap iteration $b$, we sample individuals in the test set with replacement, then we collect all individual-year observations associated with these sampled individuals and the log-likelihood values assigned to these observations by the model. We use these log-likelihood values to compute the $b^\text{th}$ bootstrap value of the metric (e.g., perplexity). After repeating the process above $B$ times, we estimate the standard error using the standard deviation of the $B$ bootstrap values.
We call this procedure the test-set-bootstrap, and report the standard error estimation from the test-set-bootstrap along with our metrics in this paper.
Similarly, the purpose of our bootstrapping approach for the training set is to quantify the uncertainty in model performance due to training set variation and randomness in the training pipeline, primarily due to data shuffling (according to the support team at Together AI). We create 12 bootstrapped training sets by sampling the pooled training set (i.e., the union of the training splits of PSID81, NLSY79, and NLSY97) with replacement. Let $\mathcal{D}_\text{mixture}^\text{(train, s)}$ denote the bootstrapped training data of the mixture dataset (sampled with replacement), generated using the random seed $s \in \{0, 1, ..., 11\}$. We fine-tune 12 versions of the Llama-2-7B models using these mixture dataset bootstrapped training sets and evaluate their performance using the complete test split of each dataset. We call this procedure the training-set-bootstrap.
To better understand how uncertainty from the training set impacts the comparisons we make in this paper, we also fine-tune 12 versions each of our Llama-2-7B and Llama-2-13B models using the PSID81, our largest survey dataset, subset of each $\mathcal{D}_\text{mixture}^\text{(train, s)}$, denoted by $\mathcal{D}_\text{PSID81}^\text{(train, s)}$.\footnote{We did not perform the same exercise for NLSY79 and NLSY97 due to the computational cost.} We fine-tune these additional models to make two comparisons. First, we compare the FT-7B fine-tuned on the mixed data to the same model fine-tuned on only one dataset to understand how the training-set-bootstrap impacts our value of information analysis. Second, we compare the Llama-2-7B model fine-tuned on PSID81 to the Llama-2-13B model fine-tuned on the same data to learn how our analysis of smaller versus larger models is impacted by the training-set-bootstrap. The results for all three models and the two comparisons are shown in Table (ref).
We observe that uncertainty from the training-set-bootstrap is lower than the uncertainty from the test-set-bootstrap for perplexity; however, the training-set-bootstrap uncertainty is relatively higher than the test-set-bootstrap uncertainty when it comes to the perplexity differences. Because the computational cost is prohibitive since it involves multiple rounds of large language model fine-tuning, we did not perform the training-set-bootstrap in the main paper.
This appendix describes the text template in more detail. The text template starts with a preamble that describes the individual's static covariates, then lists the individual's education level and occupation for each calendar year. Specifically:
The following example shows a complete text template of an individual worker. For more examples, see Online Appendix (ref).
{
}
The survey dataset may have missing data for certain individuals in some years, as described in Appendix (ref). This missingness can occur if a worker did not respond to a particular wave of the survey but participated in later waves. Additionally, some surveys, such as the NLSY and PSID, have transitioned from annual to biennial surveys in recent years, resulting in gaps for certain years.
The text template only has rows corresponding to the years when the individual was observed.
In this appendix, we explain the details of how to directly leverage LLMs' next token prediction capabilities to predict future occupations using job titles described in Section (ref). To obtain the predicted probability of the next occupation, we first tokenize each job title, $\text{title}_y$, into a sequence of tokens. Suppose the string $\text{title}_y$ is tokenized into $n$ tokens $\{ \text{token}_y^{(1)}, \text{token}_y^{(2)}, \dots, \text{token}_y^{(n)} \}$. Then, the unnormalized probability of predicting $y$ is the likelihood the language model assigns to the token sequence $\{ \text{token}_y^{(1)}, \text{token}_y^{(2)}, \dots, \text{token}_y^{(n)} \}$ as the continuation of the text representation $\textsc{Tmpl}(x_{i, \leq t}, y_{i, < t})$. The predicted probability can further be expanded using the chain rule of probability, as shown in Equation ((ref)).
The $\hat{P}^\mathcal{V}_\text{LLM}(\text{token}_y^{(j)} \mid \textsc{Tmpl}(x_{i, \leq t}, y_{i, < t}), \text{token}_y^{(1)}, \text{token}_y^{(2)}, \dots, \text{token}_y^{(j-1)})$ is operationalized by (1) appending all tokens $\text{token}_y^{(1)}, \text{token}_y^{(2)}, \dots, \text{token}_y^{(j-1)}$ to the text representation $\textsc{Tmpl}(x_{i, \leq t}, y_{i, < t})$ and (2) querying the likelihood the language model assigned to $\text{token}_y^{(j)}$ as the next token conditioned on all the previous tokens.
For example, the title “software engineer” may be tokenized into two tokens, one for “software” $\in \mathcal{V}_\text{LLM}$ and one for “engineer” $\in \mathcal{V}_\text{LLM}$.\footnote{This is for illustration purposes only, how the LLM's tokenizer splits the phrase “software engineer” depends on the exact LLM used.} Equation ((ref)) illustrates how to obtain the conditional probability assigned to “software engineer”.
It is worth noting that we cannot guarantee that the model only assigns positive probabilities to valid job titles. In fact, given the presence of the softmax function in our language model, $\hat{P}^\mathcal{V}_\text{LLM}(\cdot \mid \textsc{Tmpl}(x_{i, \leq t}, y_{i, < t}))$ is strictly positive for any sequence of tokens of any length. Therefore, the sum of all possible job titles' probabilities is not necessarily one. We would need the following normalization to calculate the probability of predicting $y_t$ so that predicted probabilities on all job titles sum to one.
The normalization operation in Equation ((ref)) is computationally expensive, since we need to perform LLM inference $|\mathcal{Y}|$ times. In this paper, we do not perform this normalization and we use the predicted probability from Equation ((ref)) directly. It is worth noting that since the denominator in Equation ((ref)) is less than one (since the total probability mass on the subset of job title tokens is less than the total probability mass on all tokens), $\hat{P}^\mathcal{V}_\text{LLM} \leq \hat{P}_\text{LLM}^\text{normalized}$. As a result, test perplexity for LLMs reported in the paper under-estimates the performance of these LLMs. To assess the extent of this potential issue, we analyze the total probability mass assigned to the set of all possible job titles on a randomly selected 10% subset of the data; we did not scale to the full dataset due to the computational cost of computing the predictive probability of all occupations. We compute the sum of predicted probabilities for all valid titles $y \in \mathcal{Y}$ for each observation $(i, t)$ in this subset, then compute the normalization constant $\sum_{y' \in \mathcal{Y}} \hat{P}^\mathcal{V}_\text{LLM}(\textsc{Tok}(\textsc{Title}(y') \mid \textsc{Tmpl}(x_{i, \leq t}, y_{i, < t}))$. The average values of normalization constants in all three datasets are close to 1, with detailed values in Table (ref). This indicates that the fine-tuned LLM concentrates nearly all of its probability mass (> 99%) on valid job title tokens. Furthermore, we compare the perplexity computed using un-normalized probabilities (i.e., $\hat{P}^\mathcal{V}_\text{LLM}(\textsc{Tok}(\textsc{Title}(y)) \mid \textsc{Tmpl}(x_{i, \leq t}, y_{i, < t}))$) against the perplexity using normalized probabilties (i.e., $\hat{P}_\text{LLM}^\text{normalized} (y_{i,t} \mid x_{i, \leq t}, y_{i, < t})$) on this 10% subset. Table (ref) summarizes the perplexity computed from unnormalized and normalized predictive probabilities. The results suggest that the lack of explicit normalization leads to a negligible underestimation of model performance. Note that these perplexity values are calculated on a subset of the test data and is different from the main results reported using the full test set.
This appendix provides the details of the embedding-based approach reported on in Section (ref). To extract embeddings from the Llama models (fine-tuned and off-the-shelf), we use the final-layer model embedding at the last token of each sequence of tokens. For OpenAI embeddings, we used the latest text-embedding-3-large model at the time the analysis was conducted (November $12^\text{th}$, 2024), which was released in January, 2024, and was still the latest as of December, 2025; details are available at \url{https://platform.openai.com/docs/guides/embeddings}.
We estimate the multinomial logistic regression using Bayesian Optimization to find the optimal learning rate in the log-uniform space $[10^{-6},10^{-2}]$. The embeddings are high-dimensional with thousands of dimensions. We also explore using embeddings of 16, 64, or 256 dimensions, using PCA to reduce our embeddings, in addition to the full-dimensional embeddings, and pick the best-performing model from our validation set.\footnote{We explore random forest with 50 Bayesian Optimization calls and uniform parameters $[20,400]$ estimators, $[5,50]$ maximum depth, $[0.01,0.9]$ minimum samples split, $[0.01,0.9]$ minimum samples leaf. Performance is significantly worse than that of multinomial logistic regression.}
Note that the embedding-based model cannot predict occupations that are not in the training set; therefore, we drop transitions of occupations that are present in the test set, but not the training set in Table (ref). The train/test split that we use to report results in this paper has 13 transitions in the test set for PSID81 and two for NLSY97 that are dropped due to having occupation codes that are not in the training set. These few observations have a negligible impact on our perplexity metric, as it is inherently robust to individual data points. The language-based model addresses this issue by producing predictive probabilities that are inherently valid for all job titles, including those not represented in the training set. In later tables, there will be 13 more transitions in the PSID81 and two more transitions in NLSY97.
This section discusses the details of fine-tuning LLMs in this paper and additional results showing how the number of epochs, i.e., complete passes through the entire training dataset during the training process, impacts model performance.
For each individual $i$ in the training split, we construct a text representation of her complete career history $\textsc{Tmpl}(x_{i, \leq T_i}, y_{i, \leq T_i})$ as described in Section (ref). We use these text representations as the corpus to fine-tune the language models. In the fine-tuning process, there is one observation for each token in each $\textsc{Tmpl}(x_{i, \leq T_i}, y_{i, \leq T_i})$ in the training corpus. The loss function not only considers the model's prediction on tokens corresponding to job titles, but also on tokens corresponding to everything else in the text representation to improve models' understanding of our text template data structure. We use $\textsc{Tmpl}(x_{i, \leq T_i}, y_{i, \leq T_i})$ from individuals in the validation split to evaluate the performance of the fine-tuned models after each fine-tuning epoch.
For each model reported in the paper, we deploy two different training strategies: full-parameter automated mixed precision fine-tuning for three epochs (where in the context of fine-tuning, an epoch is a single complete pass through a dataset) and the same for five epochs. During the fine-tuning, we evaluate the model's validation loss after each training epoch, and keep the model checkpoint (saved snapshot of a model's parameters) that attains the lowest validation loss for evaluation.
All models in this paper were fine-tuned using the two strategies mentioned, and we always report the model from the better-performing strategy.
Consider now some additional details about fine-tuning, which mirrors the pre-training process. First, note that our description of CAREER in Section (ref) gives a high-level overview of the functional form of a transformer model, where the “vocabulary” of CAREER is occupations instead of tokens from English words. Now consider estimation details. In current practice, LLMs are usually trained so that the parameters of the transformer neural network maximize log-likelihood, which in the case of language models, where outcomes are encoded as indicator variables for tokens, is equivalent to minimizing cross-entropy loss (touvron_llama_2023). In stochastic gradient descent, observations are grouped into small batches. Given parameter estimates from prior batches, within each new batch, the gradient of the loss with respect to the parameters is evaluated for each observation in the batch (where the gradient is evaluated at the previous parameter estimates). The parameters are then updated using an adaptive version of stochastic gradient descent where updates are made using moving averages; see touvron_llama_2023 for details.
In our fine-tuning, we use a batch size of 32, the initial learning rate of $10^{-5}$ (which determines the step size for each update of model parameters), and a linear learning rate decay (which determines how the learning rate changes across epochs, see e.g., jin2023rethinking) from the initial learning rate to zero learning rate. Such learning rate scheduling of linear decaying is enforced by Together AI, and we do not have control over it at the time of fine-tuning. It is worth noting that given the linear learning rate decay, the checkpoints corresponding to the first three epochs in the three epoch settings are different from the first three epochs in the five epoch settings.
We also experiment with fine-tuning the model for more epochs while taking the checkpoint corresponding to the lowest validation loss. We observed escalating validation loss (i.e., over-fitting) after four to five epochs. Due to the prohibitive computational cost, we only fine-tuned Llama-2-7B models using the pooled training data for five (reported in the main paper), six, eight, and ten epochs. Table (ref) summarizes the perplexities of the best model checkpoint, according to the validation loss, in these settings. We do not observe significant improvement in model performance, if any, while fine-tuning the model for more epochs.
This section provides additional analyses of FT-LABOR-LLM's prediction behavior when there is a gap between the calendar years of the current occupation and the previous occupation in a transition. Let $\text{year}_{i, t}$ denote the calendar year of the $t^{\text{th}}$ transition of individual $i$, where $\text{year}_{i, t-1}$ is the calendar year of the previous transition (only defined for $t > 1$). Specifically, we focus on transitions with $t > 1$ such that $\text{year}_{i, t} = \text{year}_{i, t-1} + 2$, i.e., the gap size is exactly one calendar year, to reduce computational resource requirements. To create the dataset, we randomly sample 500 transitions from the test split of each survey dataset, resulting in a total of 1,500 transitions.
Using the FT-LABOR-LLM model fine-tuned on the mixture training data, we compute the predicted probability of landing at occupation $y_{i, t}$ in calendar year $\text{year}_{i, t}$ as: $$ \hat{P}(y_{i, t} \text{ in } \text{year}_{i, t} \mid y_{i, t-1} \text{ in } \text{year}_{i, t}-2), $$ where covariates $x_{i, \leq t}$ and past occupations $y_{i, < t-1}$ are omitted in the conditional part for simplicity. This is referred to as the direct prediction. We also compute the compound prediction as: \[ \sum_{y' \in \mathcal{Y}} \hat{P}(y_{i, t} \text{ in } \text{year}_{i, t} \mid y' \text{ in } \text{year}_{i, t}-1 \land y_{i, t-1} \text{ in } \text{year}_{i, t}-2) \times \hat{P}(y' \text{ in } \text{year}_{i, t}-1 \mid y_{i, t-1} \text{ in } \text{year}_{i, t}-2). \] Computing the compound prediction for a single transition requires approximately $2 \times |\mathcal{Y}| \approx 700$ model inferences, making this experiment computationally expensive.
Finally, we compare the agreement between the direct prediction and the compound prediction using the 1,500 transitions; the log probabilities are found to be highly correlated, with a correlation coefficient of $0.93$.
In this section, we report on a complementary exercise to that conducted in Table (ref) of the main paper. Instead of either fine-tuning on a single dataset or the union of all datasets, we start from each baseline survey training dataset and create new training datasets that mix in additional data from the other two surveys. Specifically, we take the training split of dataset $\omega$, $\mathcal{D}_\omega^{\text{(train)}}$ and mix it with $P\% \times |\mathcal{D}_\omega^{\text{(train)}}|$ additional training samples from training splits of the other two datasets $\mathcal{D}_{\omega'}^{\text{(train)}} \cup \mathcal{D}_{\omega''}^{\text{(train)}}$. We fine-tune Llama-2-7B models using the merged training data, and then evaluate the model's performance on the test split $\mathcal{D}_{\omega}^{\text{(test)}}$.
Table (ref) summarizes the performance of these models fine-tuned with additional training data; adding sufficient non-representative data leads to improvements over the models fine-tuned with only data representative of the test set.
In this section, we evaluate the importance of demographic variables for predictive performance. This exercise is not straightforward for complex, nonlinear models. If we find that including a covariate in the estimation of a model improves predictive quality on a test set, that implies that the covariate both matters in the true (unknown) data generating process, and that the predictive model makes use of the covariate in prediction. However, if excluding a covariate does not affect predictive quality, we cannot be sure whether something in the estimation process failed to capture a relationship that is present in the true data generating process (e.g., mis-specification or noise), or whether the covariate is simply not important once other covariates are incorporated. Although it is straightforward to assess whether an individual covariate has predictive power in isolation using very simple models, understanding whether it has predictive power conditional on other covariates relies on modeling. Thus, negative results about the importance of a covariate require additional analysis to confirm whether, in fact, that covariate has predictive power. Here, we do not explore the latter question.
We evaluate the importance of demographic variables for FT-7B fine-tuned on $\mathcal{D}_\text{all}^\text{(train)}$, our best-performing predictive model. We apply an approach common in the machine learning literature, which entails holding fixed the estimated model, and replacing covariates with randomly assigned values in the test set, then assessing the impact on predictive performance of the model when the model is applied to the modified test set. \footnote{Note that this exercise is imperfect; an alternative would be to re-estimate the model omitting a covariate, since a model might increase the loadings on correlated covariates when a particular covariate is omitted. However, re-estimating the model comes with computational cost. Thus, we focus here on exercises that can be carried out without re-estimation.}
We explore the importance of three static variables in our text representations: gender, ethnicity, and indicators for four regions of the country. To implement the randomization of the test set demographics, we create an alternative version of the test set in which, for each unit, we replace the vector of demographics with a randomly drawn vector of demographics from units in the validation set and assign the unit those demographics. We repeat this exercise with alternative combinations of variables.
Table (ref) presents the results. Randomly modifying gender hurts the performance of FT-LABOR-LLM significantly. For PSID81, randomizing gender labels increases perplexity by 1 (about 12% above baseline), while ethnicity has about a quarter of the effect. For NLSY79 and NLSY97 test sets, gender has a similar impact, but ethnicity has a much lower effect. For PSID81, there is substantial additional degradation in performance from the interaction of gender and ethnicity, while NLSY79 sees ethnicity and region having larger effects when randomized jointly rather than individually. For all three survey datasets, the three-way interaction of gender, ethnicity, and region results in the largest impact, with the incremental effect of including all three covariates over two of them is substantial for PSID81 and NLSY79. These findings should be interpreted in light of the historical trends in the labor market participation relevant to the time periods covered by the different survey. Overall, these results suggest that complex interactions are important to consider when building predictive models of occupation, suggesting that simple, additive regressions of the type commonly used in labor market applications may omit important predictors.
In this section, we assess the predictive value of observing a worker's full history as recorded in the survey, relative to truncating the history to include only more recent observations. This question helps shed light on the sources of model performance with respect to the ability of the transformer model to capture relevant information from long histories; it also informs survey design, since following individuals over long time periods is expensive.
We proceed by evaluating how the predictive quality of FT-7B fine-tuned on $\mathcal{D}_\text{all}^\text{(train)}$, our best-performing predictive model, changes when we make predictions about $y_{i, t}$ using time-invariant covariates, $x_i$, time-varying covariates and occupations reported in the $k$ most recent observations of $\{x_{i, \tau}\}_{\tau=t-k}^t$, $\{y_{i, \tau}\}_{\tau=t-k}^{t-1}$, reporting $$\hat{P}_\text{LLM}\left(y_{i, t} \mid x_i, \{x_{i, \tau}\}_{\tau=t-k}^t, \{y_{i, \tau}\}_{\tau=t-k}^{t-1}\right).$$ With $k = t-1$, the model has access to all available history. We first create different subsets of individual-year observations from the test set of each dataset, defining the following non-overlapping subsets of individual-year observations $S^{\text{(test)}}_{t_\text{min} < t \leq t_\text{min} + 5} = \{(i, t) \in \mathcal{D}^{\text{(test)}} \mid t_\text{min} < t \leq \text{min} + 5\}$ for $t_\text{min} \in \{5, 10, 15, 20 ,25\}$. The NLSY97 dataset covers a shorter time span, therefore, $S^{\text{(test)}}_{20 < t \leq 25}$ and $S^{\text{(test)}}_{25 < t \leq 30}$ are defined as empty sets for NLSY97. Given a $S^{\text{(test)}}_{t_\text{min} < t \leq \text{min} + 5}$, for each observation $(i, t) \in S^{\text{(test)}}_{t_\text{min} < t \leq \text{min} + 5}$, we create text templates consisting of only the $k$ most recent observations of individual $i$ prior to her $t^\text{th}$ observation: $\textsc{Tmpl}(x_i, \{x_{i, \tau}\}_{\tau=t-k}^t, \{y_{i, \tau}\}_{\tau=t-k}^{t-1})$. For values of $k$, we consider multiples of five such that $k \leq t_\text{min}$ (e.g., $k \in \{5, 10, 15, 20\}$ if $t_\text{min} = 20$). A greater value of $k$ exposes the model to more information about the individual's career history and should lead to an improved prediction accuracy.
We then assess perplexity in the test set for different subsets of constructed test data defined by values of $(k,t_\text{min})$:
where each element of $\tilde{S}^{\text{(test)}}_{t_\text{min} < t \leq t_\text{min}+5, k}$ is a pair of (1) a prompt containing $k$ past observations prior to the $t^\text{th}$ record of individual $i$ and (2) the ground truth occupation individual $i$ has in her $t^\text{th}$ record.
We evaluate the FT-LABOR-LLM models using the prompt-label pair in each $\tilde{S}^{\text{(test)}}_{t_\text{min} < t \leq t_\text{min}+5, k}$ separately. Within each $\tilde{S}$ group, we query the likelihood that the language model assigns to the ground truth job title as the continuation of the text prompt, $\hat{P}_\text{LLM}(\textsc{Title} (y_{i, t}) \mid \textsc{Tmpl}(x_i, \{x_{i, \tau}\}_{\tau=t-k}^t, \{y_{i, \tau}\}_{\tau=t-k}^{t-1}))$, and compute the perplexity using all predictions within that $\tilde{S}$.
For example, consider the following prompt that would be fed into the LLM to predict the fifth occupation using the first four observations. {
}
If we set $k=2$ most recent previous observations, we would drop the first two observations in the years 2007 and 2009 and feed the following prompt into the LLM to predict the fifth occupation using only the two most recent observations instead of the full prompt above.
{
}
Finally, we build a matrix of perplexity metrics assessing model's performance under different levels of exposure to past information. Table (ref) summarizes model performance when it only has access to a limited number of past observations while predicting the next occupation. To better illustrate the result, we compute the perplexity difference between predictions made using prompts with $k \in \{10, 15, 20, 25\}$ and the baseline predictions made using prompts with $k=5$. For example, for PSID81, the data in row $t\in(15,20]$ and column $k=10$ indicates that predictions made on those observations using $k=10$ past observations in prompts for transitions indexed between 15 and 20 achieve a perplexity that is 0.193 (with a test-set-bootstrap standard error of 0.026) lower than the perplexity of predictions using $k=5$ past observations. Truncating the career history thus leads to a significant decrease in predictive performance, although for transitions at the end of a worker's career, most of the predictive benefit is achieved with 10 or 15 years of history.
In this appendix, we provide additional details on the two exercises we perform in Section (ref). First, we report the details of our analysis to learn the extent to which the embeddings created by FT-7B incorporate information about the meaning of job titles by assessing the predictive power of these embeddings on a task related to the interpretation of job titles.
Specifically, we use different transformer models to generate embedding vectors for all occupations $y \in \mathcal{Y}$, and set up a prediction task to explore how much SOC occupational hierarchy these embeddings encode. Since the Llama-2-7B embedding dimensions are much higher (e.g., 4,096) than CAREER's native embedding dimension (i.e., 192), we apply PCA dimension reduction to reduce all embeddings to 192 dimensions for fair comparison. Then, we build a multinomial logistic regression (with an elastic-net regularization) to predict which of the following six SOC groups an occupation belongs to: “Alternate aggregations”, “Management, Business, Science, and Arts Occupations”, “Service Occupations”, “Sales and Office Occupations”, “Natural Resources, Construction, and Maintenance Occupations Production, Transportation, and Material Moving Occupations”, and “Military Specific Occupations”. We regularize the multinomial regression using a convex combination of L1 and L2 regularization (i.e., the elastic-net, $\frac{\alpha||\mathbf{\beta}||_1 + (1-\alpha) ||\mathbf{\beta}||_2}{C}$); and we use five-fold cross-validation to choose the best regularization strength $C$ and weight $\alpha$.
Table (ref) shows that LLM embeddings can capture meaningful patterns in occupational hierarchies, highlighting the importance of prior knowledge in the predictions.
Second, we report the details of our analysis to learn for which types of transitions FT-13B outperforms CAREER in predicting whether an individual “moves” occupations.
Specifically, we ask the question: for what kind of mover observations $(i, t)$ with characteristics $(y_{i,t}, x_{i, \leq t}, y_{i, <t})$ do language models outperform the previous specialized transformer? We focus on “mover” transitions in the test split of the PSID81 dataset since it is our largest dataset.
To begin, we define our prediction target as the difference in the log-likelihood of the ground truth between predictions from FT-13B and CAREER, as follows:
where $\Delta \hat{P}_\text{job}$ quantifies the improvement of FT-13B over the CAREER model for a particular transition $(i, t)$ (i.e., individual-year observation).
We build a predictive generalized random forest (which embeds sample splitting to avoid overfitting as described in athey_generalized_2018) to predict this difference using as covariates the variables in Table (ref). We assign each realization of covariates to a quintile based on the resulting estimates of the difference between the models (i.e., $\Delta \hat{P}_\text{job}$). The presence of heterogeneity in the quintile-level test set mean differences in log-likelihood indicates that the intensity of differences in performance between FT-13B and CAREER vary as a function of the features of the individual-year observation, denoted $\Phi_{i,t}(y_{i,t}, x_{i, \leq t}, y_{i, <t})$. Note that logged variables are computed as $\log(x+1)$ to avoid $\log(0)$.
Then, we show the values of several features in each quintile, allowing us to understand the factors that vary systematically between higher and lower quintiles. The corresponding heat map is shown in Figure (ref); for example, Figure (ref) shows that fine-tuned Llama-2-13B performs better for movers as the transition index increases and the similarity between $y_{i, t-1}$ and $y_{i, t}$ decreases. This improvement can again be attributed to the attention mechanism and pre-training.