Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
47,043 characters · 10 sections · 63 citation commands
CAREER: A Foundation Model for Labor Sequence Data
In labor economics, many analyses rely on models for predicting an individual's future occupations. These models are crucial for estimating important economic quantities, such as gender or racial differences in unemployment hall_turnover_1972,fairlie1999emergence; they underpin causal analyses and decompositions that rely on simulating counterfactual occupations for individuals brown_incorporating_1980,schubert_employer_2021; and they also inform policy, by forecasting occupations with rising or declining market shares. More accurate predictive models opens the door to more reliable estimates of these important quantities.
In this paper we develop a representation-learning method---a transformer adapted for modeling jobs---for building such predictive models of occupation. Our model is first fit to large-scale, passively-collected resume data and then fine-tuned to more curated economics datasets, which are carefully collected for unbiased generalization to the larger population. The representation it learns is effective both for predicting job trajectories and for conditioning in downstream economic analyses.
In the past, labor economics analyses only involved fitting predictive models to small datasets, specifically longitudinal surveys that follow a cohort of individuals during their working career psid,nlsy. Such surveys have been carefully collected to represent national demographics, ensuring that the economic analyses can generalize, but they are also very small, usually containing only thousands of workers. As a consequence, prior models of occupation trajectories have been based on very simple sequential assumptions, such as where a worker's next occupation depends only on their most recent occupation hall_turnover_1972 or a few summary statistics about their past blau_labor_1999. \looseness=-1
In recent years, however, much larger datasets of online resumes have also become available. These datasets contain the occupation histories of millions of individuals, potentially revealing complex information and patterns about career trajectories. But, while one might hope these datasets can improve our economic analyses, there are fundamental difficulties to using them. First, they are passively collected and likely represent a biased sample of the population. Second, they are noisy, since the occupation sequences in the data are derived from text analysis of raw resumes. Finally, they generally omit important economic variables such as demographics and wage, which are essential for the kinds of quantities that economists would like to estimate.
To overcome these challenges, we develop CAREER, a machine learning model of occupation trajectories. CAREER is a foundation model bommasani2021opportunities: it learns an initial representation of job history from large-scale resume data that is then adjusted on downstream survey datasets. CAREER is based on the transformer architecture vaswani2017attention, which underpins foundation models in natural language processing Devlin2019Bert,radford2019language and other fields, such as music generation copet2023simple and coding li2023starcoder. We will show that CAREER's representations provide effective predictions of occupations on survey datasets used for economic analysis, and can be used as inputs to economic models for other downstream applications.
To study this model empirically, we pretrain CAREER on a dataset of 24 million passively-collected resumes provided by \href{https://www.zippia.com}{Zippia}, a career planning company. We then fine-tune CAREER's representations of job sequences to make predictions on three widely-used economic datasets: the National Longitudinal Survey of Youth 1979 (NLSY79), another cohort from the same survey (NLSY97), and the Panel Study of Income Dynamics (PSID). In contrast to resume data, these well-curated datasets are representative of the larger population. It is with these survey datasets that economists make inferences, ensuring their analyses generalize. In this study, we find that CAREER outperforms standard econometric models for predicting and forecasting occupations on these survey datasets.
We further find that CAREER can be used to form good predictions of other downstream variables; incorporating CAREER into a wage model provides better predictions than the econometric models currently in use. We release code so that practitioners can train CAREER for their own problems.\footnote{\url{https://github.com/keyonvafa/career-code}}
In summary, we demonstrate that CAREER can leverage large-scale resume data to make accurate predictions on important datasets from economics. Thus CAREER ties together economic models for understanding career trajectories with transformer-based foundation models. (See (ref) for details of related work.) \looseness=-1
Given an individual's career history, what is the probability distribution of their occupation in the next timestep? We go over a class of models for predicting occupations before introducing CAREER, one such model based on transformers and transfer learning.
Consider an individual worker. This person's career can be defined as a series of timesteps. Here, we use a timestep of one year. At each timestep, this individual works in a job: it could be the same job as the previous timestep, or a different job. (Note we use the terms “occupation” and “job” synonymously.) We consider “unemployed” and “out-of-labor-force” to be special types of jobs.
Define an occupation model to be a probability distribution over sequences of jobs. An occupation model predicts a worker's job at each timestep as a function of all previous jobs and other observed characteristics of the worker.
More formally, define an individual's career to be a sequence $(y_1, \dots, y_T)$, where each $y_t \in \{1, \dots, J\}$ indexes one of $J$ occupations at time $t$. Occupations are categorical; one example of a sequence could be (“cashier”, “salesperson”,\ ...\ , “sales manager”). At each timestep, an individual is also associated with $C$ observed covariates $x_t = \{x_{tc}\}_{c=1}^C$. Covariates are also categorical, with $x_{tc} \in \{1, \dots, N_c\}$. For example, if $c$ corresponds to the most recent educational degree, $x_{tc}$ could be “high school diploma” or “bachelors”, and $N_c$ is the number of types of educational degrees.\footnote{Some covariates may not evolve over time. We encode them as time-varying without loss of generality.} Define $\mb{y}_t = (y_1, \dots, y_t)$ to index all jobs that have occurred up to time $t$, with the analogous definition for $\mb{x}_t$.
At each timestep, an occupation model predicts an individual's job in the next timestep, $p(y_t|\mb{y}_{t-1}, \mb{x}_t)$. This distribution conditions on covariates from the same timestep because these are “pre-transition.” For example, an individual's most recent educational degree is available to the model as it predicts their next job. \looseness=-1
Note that an occupation model is a predictive rather than structural model. The model does not incorporate unobserved characteristics, like skill, when making predictions. Instead, it implicitly marginalizes over these unobserved variables, incorporating them into its predictive distribution.
An occupation model's predictions are governed by an individual's career history; both whether an individual changes jobs and the specific job they may transition to depend on current and previous jobs and covariates.
We consider a class of occupation models that make predictions by conditioning on a low-dimensional representation of work history, $h_t(\mb{y}_{t-1}, \mb{x}_t) \in \mathbb{R}^D$. This representation is assumed to be a sufficient statistic of the past; $h_t(\mb{y}_{t-1}, \mb{x}_t)$ should contain the relevant observed information for predicting the next job. \looseness=-1
Since individuals frequently stay in the same job between timesteps, we propose a class of models that make predictions in two stages. These models first predict whether an individual changes jobs, after which they predict the specific job to which an individual transitions. The representation is used in both stages. \looseness=-1
In the first stage, the career representation $h_t(\mb{y}_{t-1}, \mb{x}_{t})$ is used to predict whether an individual changes jobs. Define the binary variable $s_t$ to be 1 if a worker's job at time $t$ is different from that at time $t - 1$, and 0 otherwise. The first stage is modeled by
where $\sigma(\cdot)$ is the logistic function and $\eta \in \mathbb{R}^D$ is a vector of coefficients.
If the model predicts that an individual will transition jobs, it only considers jobs that are different from the individual's most recent job. To formulate this prediction, it combines the career representation with a vector of occupation-specific coefficients $\beta_j \in \mathbb{R}^D$:
Otherwise, the next job is deterministic:
Two-stage prediction improves the accuracy of occupation models. Moreover, many analyses of occupational mobility focus on whether workers transition jobs rather than the specific job they transition to kambourov_rising_2008. By separating the mechanism by which a worker either keeps or changes jobs ($\eta$) and the specific job they may transition to ($\beta_j$), two-stage models are more interpretable for studying occupational change. \looseness=-1
(ref) define a two-stage representation-based occupation model. In the next section, we introduce CAREER, one such model based on transformers.
We develop a two-stage representation-based occupation model called CAREER.\footnote{CAREER is short for “Contextual Attention-based Representations of Employment Encoded from Resumes.”} This model uses a transformer to parameterize a representation of an individual's history. CAREER is a foundation model: it is pretrained on a large resumes dataset and fine-tuned to make predictions on small survey datasets. \looseness=-1
\parhead{Transformers.} A transformer is a sequence model that uses neural networks to learn representations of discrete tokens vaswani2017attention. Transformers were originally developed for natural language processing (NLP), to predict words in a sentence. Transformers are able to model complex dependencies between words, and they are a critical component of large language models radford2018improving,Devlin2019Bert,radford2019language,brown2020language,hoffmann2022training,touvron2023llama.
CAREER is an occupation model that uses a transformer to parameterize a low-dimensional representation of careers. While transformers were developed to model sequences of words, CAREER uses a transformer to model sequences of jobs. The transformer enables the model to represent complex career trajectories.
CAREER is similar to the transformers used in NLP, but with two modifications. First, as described in (ref), the model makes predictions in two stages, making it better-suited to model workers who stay in the same job through consecutive timesteps. (In contrast, words seldom repeat.) Second, while language models only condition on previous words, each career is also associated with covariates $\mb{x}$ that may affect transition distributions (see (ref)). We adapt the transformer to these two changes. \looseness=-1
\parhead{Parameterization.} CAREER's computation graph is depicted in (ref). Note that in this section we provide a simplified description of the ideas underlying the transformer. (ref) contains a full description of the model.
CAREER iteratively builds a representation of career history, $h_t(\mb{y}_{t-1}, \mb{x}_{t}) \in \mathbb{R}^D$, using a stack of $L$ layers. Each layer applies a series of computations to the previous layer's output to produce its own layer-specific representation. The first layer's representation, $h_t^{(1)}(\mb{y}_{t-1}, \mb{x}_t)$, considers only the most recent job and covariates. At each subsequent layer $\ell$, the transformer forms a representation $h_t^{(\ell)}(\mb{y}_{t-1}, \mb{x}_t)$ by combining the representation of the most recent job with those of preceding jobs. Representations become increasingly complex at each layer, and the final layer's representation, $h_t^{(L)}(\mb{y}_{t-1}, \mb{x}_t)$, is used to make predictions following (ref). We drop the explicit dependence on $\mb{y}_{t-1}$ and $\mb{x}_t$ going forward, and instead denote each layer's representation as $h_t^{(\ell)}$.
The first layer's representation combines the previous job, the most recent covariates, and the position of the job in the career. It first embeds each of these variables in $D$-dimensional space. Define an embedding function for occupations, $e_y: [J] \to \mathbb{R}^D$. Additionally, define a separate embedding function for each covariate, $\{e_c\}_{c=1}^C$, with each $e_c: [N_c] \to \mathbb{R}^D$. Finally, define $e_t: [T] \to \mathbb{R}^D$ to embed the position of the sequence, where $T$ denotes the number of possible sequence lengths. The first-layer representation $h_t^{(1)}$ sums these embeddings:
For each subsequent layer $\ell$, the transformer combines representations of the most recent job with those of the preceding jobs and passes them through a neural network:
where $W^{(\ell)} \in \mathbb{R}^{D \times D}$ is a model parameter and $\text{FFN}^{(\ell)}$ is a two-layer feedforward neural network specific to layer $\ell$, with $\text{FFN}^{(\ell)}: \mathbb{R}^D \to \mathbb{R}^D$.
The weights $\{\pi_{t,t'}^{(\ell)}\}$ are referred to as attention weights, and they are determined by the career representations and $W^{(\ell)}$. The attention weights are non-negative and normalized to sum to 1. The matrix $W^{(\ell)}$ can be interpreted as a similarity matrix; if $W^{(\ell)}$ is the identity matrix, occupations $t$ and $t'$ that have similar representations will have large attention weights, and thus $t'$ would contribute more to the weighted average in (ref). Conversely, if $W^{(\ell)}$ is the negative identity matrix, occupations that have differing representations will have large attention weights.\footnote{In practice, transformers use multiple attention weights to perform multi-headed attention ((ref)).} The final computation of each layer involves passing the intermediate representation $\tilde h_t^{(\ell)}$ through a neural network, which ensures that representations capture complex nonlinear interactions.
The computations in (ref) are repeated for each of the $L$ layers. The last layer's representation is used to predict the next job:
where “$\textrm{two-stage-softmax}$” refers to the operation in (ref), parameterized by $\eta$ and $\beta$.
All of CAREER's parameters --- including the embedding functions, similarity matrices, feed-forward neural networks, and regression coefficients $\eta$ and $\beta$ --- are estimated by maximizing the likelihood in (ref) with stochastic gradient descent (SGD), marginalizing out the latent variable $s_t$.
\parhead{Transfer learning.} Economists build occupation models using survey datasets that have been carefully collected to represent national demographics. In the United States, these datasets contain a small number of individuals. While transformers have been successfully applied to large NLP datasets, they are prone to overfitting on small datasets kaplan2020scaling,dosovitskiy2020image,varivs2021sequence. As such, CAREER may not learn useful representations solely from small survey datasets.
In recent years, however, much larger datasets of online resumes have also become available. Although these passively-collected datasets provide job sequences of many more individuals, they are not used for economic estimation for a few reasons. The occupation sequences from resumes are imputed from short textual descriptions, a process that inevitably introduces more noise and errors than collecting data from detailed questionnaires. Additionally, individuals may not accurately list their work experiences on resumes wexler2006successful, and important economic variables relating to demographics and wage are not available. Finally, these datasets are not constructed to ensure that they are representative of the general population.
Between these two types of data is a tension. On the one hand, resume data is large-scale and contains valuable information about employment patterns. On the other hand, survey datasets are carefully collected, designed to help make economic inferences that are robust and generalizable.
Thus CAREER incorporates the patterns embedded in large-scale resume data into the analysis of survey datasets. It does this through transfer learning: CAREER is first pretrained on a large dataset of resumes to learn an initial representation of careers. When CAREER is then fit to a small survey dataset, parameters are not initialized randomly; instead, they are initialized with the representations learned from resumes. After initialization, all parameters are fine-tuned on the small dataset by optimizing the likelihood. Because the objective function is non-convex, learned representations depend on their initial values. Initializing with the pretrained representations ensures that the model does not need to re-learn representations on the small dataset. Instead, it only adjusts representations to account for dataset differences.
CAREER is designed to be a foundation model bommasani2021opportunities: after pretraining on large-scale resume data, CAREER can be transferred to downstream survey datasets. CAREER takes inspiration from foundation models developed in other domains, such as NLP Devlin2019Bert,radford2018improving,radford2019language,brown2020language, image generation ramesh2021zero, music generation copet2023simple, and coding li2023starcoder. For example, in NLP, transformers are pretrained on large corpora, such as unpublished books or web crawl data, and then fine-tuned to make predictions on small datasets such as movie reviews Devlin2019Bert. Our approach is analogous. Although the resumes dataset may not be representative or carefully curated, it contains many more job sequences than most survey datasets. This volume enables CAREER to learn representations that transfer to survey datasets.
Many economic analyses use log-linear models to predict jobs in survey datasets boskin1974conditional,schmidt_prediction_1975. These models typically use small state spaces consisting of only a few occupation categories. For example, some studies categorize occupations into broad skill groups keane_career_1997,cortes2016have; unemployment analyses only consider employment status (employed, unemployed, and out-of-labor-force) hall_turnover_1972,laureova_what_2007; and researchers studying occupational mobility only consider occupational change, a binary variable indicating whether an individual changes jobs kambourov_rising_2008,guvenen_multidimensional_2020. Although transitions between occupations may depend richly on history, many of these models condition on only the most recent job and a few manually constructed summary statistics about history to make predictions hall_turnover_1972,blau_labor_1999. In contrast to these methods, CAREER is nonlinear and conditions on every job in an individual's history. The model learns complex representations of careers without relying on manually constructed features. Moreover, CAREER can effectively predict from among hundreds of occupations. \looseness=-1
Recently, the proliferation of business networking platforms has resulted in the availability of large resume datasets. schubert_employer_2021 use a large resume dataset to construct a first-order Markov model of job transitions; CAREER, which conditions on all jobs in a history, makes more accurate predictions than a Markov model. Models developed in the data mining community rely on resume-specific features such as stock prices xu2018dynamic, worker skill ghosh2020skill, network information meng2019hierarchical,zhang2021attentive, and textual descriptions he2021your, and are not applicable to survey datasets, as is our goal in this paper (other models reduce to a first-order Markov model without these features dave2018combined,zhang_job2vec_2020). The most suitable model for survey datasets from this line of work is NEMO, an LSTM-based model that is trained on large resume datasets li2017nemo. In contrast to these approaches, our primary concern is not modeling resume datasets directly, but rather modeling survey datasets. Our experiments demonstrate that a transfer learning approach, from resume data to survey data, improves modeling performance. \looseness=-1
Recent work in econometrics has applied machine learning methods to sequences of jobs and other discrete data. In the context of consumer grocery shopping behavior, ruiz2020shopper and donnelly2021counterfactual introduce a method called SHOPPER, based on matrix factorization, for creating embeddings of grocery products. SHOPPER can be thought of as a foundation model, and donnelly2021counterfactual show that the approach of building embeddings using data from unrelated product categories improves predictions for a given category relative to traditional economic models focused only on one category, while ruiz2020shopper expand the setup to predict the next product in a sequence added to a shopping basket. One of the baselines we consider in this paper is a “bag-of-jobs” model similar to SHOPPER. Like the transformer-based model, the bag-of-jobs model conditions on every job in an individual's history, but it uses relatively simple representations of careers. Our empirical studies demonstrate that CAREER learns complex representations that are better at modeling job sequences. karthik_thesis build on SHOPPER and propose a Bayesian factorization method for predicting job transitions. Similar to CAREER, they predict jobs in two stages. However, their method only conditions on the most recent job in an individual's history. In our empirical studies, we show that models like CAREER that condition on every job in an individual's history form more accurate predictions than Markov models.
CAREER is a foundation model based on a transformer, an architecture initially developed to represent sequences of words in natural language processing (NLP). In econometrics, transformers have been applied to the text of job descriptions to predict their salaries bana2021using or authenticity naude2022machine; rather than modeling text, we use transformers to model sequences of occupations. Transformers underpin foundation models developed for text Devlin2019Bert,radford2018improving,radford2019language,brown2020language and other domains such as image generation ramesh2021zero, music generation copet2023simple, and coding li2023starcoder. Occupational trajectories have simpler structure than the data modeled in these other domains; for example, per-year career trajectories are shorter than the text sequences used to train large language models, and there are less possible occupations than words in a vocabulary. Thus, foundation models for occupations may not require as much data as those in other domains. Indeed, CAREER is smaller than foundation models in other domains and uses less data, but still forms effective predictions on downstream tasks. \looseness=-1
Our paper also contributes to the growing field of transfer learning in economics. Most existing transfer learning approaches in the economics literature have focused on fine-tuning existing models on image or text data, where the effectiveness of transfer learning is better understood. For example, one existing line of work has fine-tune pretrained convolutional neural networks on satellite images in order to predict economic indicators price2022global,yeh2020using,persello2020towards, while another has fine-tuned BERT on financial and economic text araci2019finbert,bana2021using. Our work differs from these papers in that we focus on a new domain (job sequences on survey data instead of images or text) and also identify a novel pretraining corpus that helps with this task (large-scale, passively-collected resume data).
We assess CAREER's ability to predict jobs and provide useful representations of careers. We pretrain CAREER on a large dataset of resumes, and transfer these representations to small, commonly used survey datasets. We study straightforward prediction (predicting jobs in a career with random train and test set splits) and forecasting (training on jobs before a certain year and predicting on future years). With the transferred representations, the model is better than econometric baselines at both prediction and forecasting. Additionally, we demonstrate that CAREER's representations can be incorporated into standard wage prediction models to make better predictions.
\parhead{Resume pretraining.} We pretrain CAREER on a large dataset of resumes provided by \href{https://www.zippia.com}{Zippia Inc.}, a career planning company. This dataset contains resumes from 23.7 million working Americans. Each job is encoded into one of 330 occupational codes, using the coding scheme of david2013growth. We transform resumes into sequences of jobs by including an occupation's code for each year in the resume. For years with multiple jobs, we take the job the individual spent the most time in. We include three covariates: the year each job in an individual's career took place, along with the individual's state of residence and most recent educational degree. We denote missing covariates with a special token. ((ref) contains an exploratory data analysis of the resume data.) CAREER uses a 12-layer transformer with 5.6 million parameters. Pretraining CAREER on the resumes data takes 18 hours on a single GPU. (See (ref) for more details on the model and hyperparameters.) \looseness=-1
\parhead{Survey datasets.} We transfer CAREER to three widely-used survey datasets: two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97)\nocite{nlsy,nlsy97} and the Panel Study of Income Dynamics (PSID)\nocite{psid}. These datasets have been carefully constructed to be representative of the general population, and they are widely used by economists for estimating economic quantities. NLSY79 is a longitudinal panel survey following a cohort of Americans who were between 14 and 22 when the survey began in 1979, while NLSY97 follows a different cohort of individuals who were between 12 and 17 when the survey began in 1997. PSID is a longitudinal survey following a sample of American families, with individuals added over the years. These surveys are publicly available and allow for linking individuals over time. See (ref) for more information about how the data is formed.
Compared to the resumes dataset, these survey datasets are small: we use slices of NLSY79, NLSY97, and PSID that contain 12 thousand, 9 thousand, and 12 thousand individuals, respectively. We convert all datasets so that occupations follow the occ1990dd encoding david2013growth, which contains 330 discrete occupations. Both the resume dataset and survey datasets contain education, location, and year covariates, while the survey datasets additionally include demographic variables (gender and race/ethnicity). The distribution of job sequences in resumes differs in meaningful ways from those in the survey datasets; for example, manual laborers are under-represented and college graduates are over-represented in resume data. (ref) includes more details for how these datasets differ.
We pretrain CAREER on the large resumes dataset and fine-tune on the smaller survey datasets. We fine-tune by initializing each model's weights with those of the pretrained model. The fine-tuning process is efficient; although CAREER has 5.6 million parameters, fine-tuning on one GPU takes 13 minutes on NLSY79, 7 minutes on NLSY97, and 23 minutes on PSID.
We compare CAREER to several baseline models: a second-order linear regression with covariates and hand-constructed summary statistics about past employment; a bag-of-jobs model inspired by SHOPPER ruiz2020shopper that conditions on all jobs and covariates in a history but combines representations linearly; and several baselines developed in the data-mining community for modeling worker profiles: NEMO li2017nemo, job representation learning dave2018combined, and Job2Vec zhang_job2vec_2020. As described in (ref), the baselines developed in the data-mining community for modeling worker profiles cannot be applied directly to economic survey datasets and thus require modifications, described in detail in (ref). These baselines are all trained on survey data. We also compare to two additional versions of CAREER --- one without pretraining or two-stage prediction, the other only without two-stage prediction --- to assess the sources of CAREER's improvements. All models use the covariates we included for resume pretraining, in addition to demographic covariates (which are recorded for the survey datasets but are unavailable for resumes).
We divide all survey datasets into 70/10/20 train/validation/test splits, and train all models by optimizing the log-likelihood with Adam kingma2014adam. We evaluate the predictive performance of each model by computing held-out perplexity, a common metric in NLP for evaluating probabilistic sequence models. The perplexity of a sequence model $p$ on a sequence $y_1, \dots, y_T$ is $\exp\{-\frac{1}{T}\sum_{t=1}^T \log p(y_t | \mb{y}_{t-1}, \mb{x}_t)\}$. It is a monotonic transformation of log-likelihood; better predictive models have lower perplexities. We train all models to convergence and use the checkpoint with the best validation perplexity. Detailed experimental settings and training details are described in (ref).
(ref) compares the test-set perplexity of each model. With the transferred representations, CAREER makes the best predictions on all survey datasets. The baselines developed in the data mining literature, which were designed to model large resume datasets while relying on resume-specific features, struggle to make good predictions on these small survey datasets, performing on par with standard econometric baselines. Pretraining is the biggest source of CAREER's improvements. Although the resume data is noisy and differs in many ways from the survey datasets used for economic prediction, CAREER learns representations of work experiences that aid its predictive performance. In (ref) we show that modifying the baselines to incorporate two-stage prediction ((ref)) improves their performance, although CAREER still makes the best predictions across datasets. We include further qualitative analysis of CAREER's predictions in (ref).
To assess how the volume of resumes used for pretraining affects CAREER's predictions on survey datasets, we downsample the resume dataset and transfer to survey datasets. The scaling law for NLSY79 is depicted in (ref). When there are less than 20,000 examples in the resume dataset, pretraining CAREER does not offer any improvement. The relationship between pretraining volume and fine-tuned perplexity follows a power law, similar to scaling laws in NLP kaplan2020scaling.
We also assess CAREER's ability to forecast future career trajectories. In contrast to predicting held-out sequences, forecasting involves training models on all sequences before a specific year. To predict future jobs for an individual, the fitted model is used to estimate job probabilities six years into the future by sampling multi-year trajectories. This setting is useful for assessing a model's ability to make long-term predictions, especially as occupational trends change over time.
We evaluate CAREER's forecasting abilities on NLSY97 and PSID. (These datasets are more valuable for forecasting than NLSY79, which follows a cohort that is near or past retirement age.) We train models on all sequences (holding out 10% as a validation set), without including any observations after 2014. When pretraining CAREER on resumes, we also make sure to only include examples up to 2014. (ref) and (ref) compare the forecasting performance of all models (results are averaged over three random seeds). CAREER makes the best overall forecasts. CAREER has a significant advantage over baselines at making long-term forecasts, yielding a 17% advantage over the best baseline for 6-year forecasts on NLSY97. Again, the baselines developed for resume data mining, which had been developed to model much larger corpora, struggle to make good predictions on these smaller survey datasets.
\parhead{Using CAREER's representations to predict wage.} In addition to forming job predictions, CAREER learns low-dimensional representations of job histories. These representations can be used as inputs to economic models for downstream applications.
As an example of how CAREER's representations can be incorporated into other economic models, we use CAREER to predict wages. Economists build wage prediction models in order to estimate important economic quantities, such as the adjusted gender wage gap blau_gender_2017, wage effects in labor market discrimination neal1996role, and union wage premiums linneman1990evaluating. For example, to estimate the adjusted gender wage gap, blau_gender_2017 regress an individual's log-wage on observable characteristics such as education, demographics, and current occupation on PSID. Rather than including the full, high-dimensional job-history, the model summarizes an individual's career with summary statistics such as full-time and part-time years of experience (and their squares).
We incorporate CAREER's representation into the wage regression by adding the fitted representation for an individual's job history, $\hat h_i$. For log-wage $w_i$ and observed covariates $x_i$, we regress
where $\alpha$, $\theta$, and $\gamma$ are regression coefficients. CAREER's representations are trained to model occupation sequences on 80% of the samples on PSID, and are then frozen on the held-out samples; this ensures that the representations do not encode an individual's future occupations. To estimate regression coefficients, we again split the sample into five splits and evaluate $R^2$ on each held-out split, averaging over splits. We estimate standard errors over three random training seeds.
(ref) shows that adding CAREER's representations improves wage predictions for each year. Although these representations are trained to model occupational sequences and are not adjusted to account for wage, they contain information that is predictive of wage. Moreover, pretraining is important to CAREER's predictive advantage. That models with these representations outperform simpler models in predictive performance, both for occupation modeling (the objective the representations are trained to optimize) and wage prediction (which the representations are not trained to optimize), underscores the complexity of the learned representations.
We introduced CAREER, a foundation model for occupational sequences. CAREER is first trained on large-scale resume data and then fine-tuned on smaller datasets of interest. When we fine-tuned CAREER to common survey datasets in labor economics, it outperformed econometric baselines in predicting and forecasting career outcomes. We demonstrated that CAREER's representations could be incorporated into wage prediction models, outperforming standard econometric models.
One direction of future research is to incorporate CAREER's representations of job history into methods for estimating adjusted quantities, like wage gaps. Underlying these methods are models that predict economic outcomes as a function of observed covariates. However, if relevant variables are omitted, the adjusted estimates may be affected; e.g., excluding work experience from wage prediction may change the magnitude of the estimated gap. In practice, economists include hand-designed summary statistics to overcome this problem, such as in blau_gender_2017. CAREER provides a data-driven way to incorporate such variables---its representations of job history could be incorporated into downstream prediction models and lead to more accurate adjustments of economic quantities.
As discussed, passively-collected resume datasets are not curated to represent national demographics. Pretraining CAREER on these datasets may result in representations that are affected by sampling bias. Although these representations are fine-tuned on survey datasets that are carefully constructed to represent national demographics, the biases from pretraining may propagate through fine-tuning ravfogel-etal-2020-null,jin2021transferability. Moreover, even in representative datasets, models may form more accurate predictions for majority groups due to data volume pmlr-v81-dwork18a. Thus, we encourage practitioners to audit noisy resume data, re-weight samples as necessary kalton1983compensating, and review accuracy within demographics before using the model for downstream economic analysis.
Although resume datasets may contain personally identifiable information, all personally identifiable information had been removed before we were given access to the resume dataset we used for pretraining. Additionally, none of the longitudinal survey datasets contain personally identifiable information.
This work is supported by NSF grant IIS 2127869, ONR grants N00014-17-1-2131 and N00014-15-1-2209, the Simons Foundation, Open Philanthropy, and the Golub Capital Social Impact Lab. Keyon Vafa is supported by the Harvard Data Science Initiative Postdoctoral Fellowship. We thank Elliott Ash, Sune Lehmann, Suresh Naidu, Matthew Salganik, and the reviewers for their thoughtful comments and suggestions. We thank \href{http://zippia.com}{Zippia} for generously sharing the resume dataset. We also thank the Stanford Institute for Human-Centered Artificial Intelligence. Finally, we thank Lilia Chang, Karthik Rajkumar, and Lisa Simon upon whose research we build in this project, and especially Lisa Simon who helped obtain the data and encourage this line of research.