Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
66,105 characters · 15 sections · 80 citation commands
Estimating Wage Disparities Using Foundation Models
{14pt} {6pt}
Foundation models have revolutionized the machine learning approach to prediction devlin2018bert,radford2019language,bommasani2021opportunities. In contrast to traditional predictive models, which are trained to make predictions on specific, individual tasks, foundation models are typically trained in two steps: they first are trained on massive, passively-collected datasets and then are adapted to specific tasks. The success of these models stems from their ability to transfer information learned during the initial training period to new prediction problems through approaches like supervised fine-tuning --- adjusting a model's parameters to minimize prediction error on labeled examples from a target task devlin2018bert. For example, large language models devlin2018bert,radford2019language are foundation models that were originally trained to predict the next word of Internet articles, but can be fine-tuned to make other predictions involving text, like the next word of a conversation or the sentiment of a movie review.
While foundation models have been successful at making predictions about social science data vafa2023career,savcisens2024using, many core problems in social science require more than just accurate predictions. For example, social scientists often aim to estimate causal effects under the assumption of unconfoundedness imbens2004nonparametric or decompose observed differences between groups into explained and unexplained components based on observable factors kitagawa1955components,oaxaca1973male,blinder_wage_1973 --- isomorphic problems that use prediction as an intermediate step but ultimately require different criteria for success. While fine-tuning foundation models may be useful for these analyses, optimizing for predictive accuracy alone does not guarantee valid decompositions or causal estimates.
In this paper, we develop methods for adapting foundation models to perform decomposition and causal effect estimation by modifying how they are fine-tuned. Rather than fine-tuning foundation models to minimize prediction error, we develop objectives specifically designed for these estimation problems. Our first contribution is characterizing a statistical bias that arises when a foundation model discards information that may not be important for prediction but is relevant for the estimation problem. We then provide a novel set of conditions for fine-tuning under which estimates derived from a foundation model are not only unbiased but also consistent at a fast asymptotic rate. These conditions motivate new debiased fine-tuning methods. Our key insight is that fine-tuning foundation models for these applications requires addressing an omitted variable bias that standard supervised fine-tuning does not address.
To demonstrate these ideas, we focus on an application that addresses a classic decomposition problem from labor economics: estimating the difference between how individuals with the same labor market experience get paid when they belong to different demographic groups (see blau_gender_2017 and altonji_chapter_1999 for reviews). Accurately estimating this unexplained wage gap is important to help guide policy for reducing disparities. But the unexplained gap is challenging to estimate with traditional econometric models. It involves predicting an individual's wage from their labor market history, a high-dimensional and complicated variable. Our paper demonstrates that foundation models of labor market history can improve the predictions that underlie wage gap estimates.
We use CAREER, a foundation model of labor market history vafa2023career, to estimate unexplained wage gaps. CAREER is initially fit to a massive resume dataset to predict the next job an individual will have, rather than their wage. Naively, we can fine-tune CAREER to make accurate predictions of wage on the datasets used for wage gap estimation. However, using this approach to estimate the unexplained wage gap can amplify a classical problem: omitted variable bias. Instead, we develop new debiased fine-tuning methods to fine-tune foundation models so they can properly estimate unexplained wage gaps. The key is to fine-tune foundation models not to minimize predictive error but rather to reduce omitted variable bias. In semi-synthetic experiments, we show that debiased fine-tuning methods form better estimates of the unexplained wage gap than the standard fine-tuning approach.
We use our methods to estimate the explained gender wage gap on survey data from the Panel Study of Income Dynamics (PSID) psid. We first demonstrate that foundation models form accurate predictions of wage and gender; they outperform standard econometric models for predicting wage by 10-15%. We then use debiased fine-tuning methods to estimate the gender wage gap. We find that history consistently explains more of the gap than the variables typically included in standard econometric models. We conclude by studying which aspects of work history, captured by foundation models but omitted from prior approaches, are important for explaining the wage gap.
While this paper studies unexplained wage gaps in detail, the results and methods we develop are applicable to a broader set of problems, such as causal estimation. In particular, as observed by fortin2011decomposition and others in the literature, the problem of estimating a decomposition of a wage gap into explained and unexplained components is isomorphic to the problem of estimating the average effect of a treatment under the assumption of unconfoundedness. Although the interpretation of the estimate is distinct for decompositions, the statistical theory that applies to estimation is the same (see imbens2015causal for a review). Thus, our results also provide new theory and methods for the problem of incorporating foundation models into the estimation of treatment effects.
Relative to both the causal inference and decomposition literatures, our theory is adapted to a scenario where a foundation model may bring in information from a distinct, larger dataset, and where we fine-tune the model to avoid omitted variable bias. If we solve the latter problem well enough, then the traditional semi-parametric theory (e.g. chernozhukov2018double) can be applied as if the representations of high-dimensional covariates derived from the fine-tuned foundation model are sufficient statistics for the full high-dimensional covariate vector. The methods we introduce thus provide a widely applicable new framework for leveraging the capabilities of foundation models while mitigating biases due to omitted variables that they may introduce.
The unexplained wage gap is the wage gap between two groups of individuals with the same observed characteristics. We estimate an unexplained wage gap that arises when individuals in different groups have the same labor market history.
Consider the gender wage gap. In the United States, females earn roughly 80% the male hourly wage blau_gender_2017. Motivated by the fact that the male and female labor forces differ in observable ways, a large literature seeks to explain this wage gap through differences in these observable factors kitagawa1955components,blinder1973wage,oaxaca1973male,blau_gender_2017. One of the most important factors for explaining the gender wage gap is differences in the number of years that males and females have spent in the labor force blau_gender_2017. Understanding the difference in wages between males and females with the same career histories can help guide policy: if the unexplained gap is large, attempts to close the gap may involve interventions to address problems in bargaining or fairness in wage setting. On the other hand, if the gap can be accounted for by gender differences in career histories, these interventions might target career pathways.
More generally, consider $N$ individuals, indexed by $i = 1, \dots, N$. Each individual belongs to a binary group $A_i \in \{0, 1\}$ (e.g. $A_i=0$ denotes males and $A_i=1$ denotes females). Each individual also has a career history $X_i$, a sequence of $T$ discrete occupations and years, $X_i=((J_{i1}, D_{i1}), \dots, (J_{iT},D_{iT})) \in \mathcal X$, and where each occupation label $J_{it} \in \{1, \dots, N_J\}$ encodes the occupation an individual worked in during year $D_t$ (or their labor status if they're not working, e.g. “unemployed” or “student”). Finally, denote an individual's log-wage by $Y_i \in \mathbb{R}$. Each individual is sampled i.i.d. from a joint distribution $P(X, A, Y)$. Define the conditional expectation function, $\mu_a(x) = \mathbb{E}_P[Y|A=a, X=x]$, and the propensity function, $e(x) = P(A=1|X=x)$.
The raw wage gap is
This is the difference in the average wage between the two groups. Our goal is to estimate the wage gap that is unexplained by history:
This is the average difference in the expected wage between individuals in the two groups who have the same career histories. The unexplained and raw wage gaps are linked by a classic decomposition kitagawa1955components,blinder1973wage,oaxaca1973male:
The unexplained wage gap is the portion of the raw wage gap that cannot be attributed to gender differences in career histories.
The explained wage gap could be conditioned on factors in addition to labor market history, e.g. an individual's educational background. For simplicity our notation only includes history, but we incorporate additional observed characteristics of individuals in our empirical analyses. We also note that the unexplained wage gap is only non-parametrically identifiable under an overlap condition (see, e.g. imbens2004nonparametric): $P(A=1 | X) < 1$. We assume overlap throughout this paper and we limit the sample of workers we analyze empirically to those with histories where the condition is satisfied.
A common approach for estimating the unexplained wage gap involves constructing an estimator $\hat \mu_g(x)$ for $\mu_g(x)$, which in turn can be used to form an estimate of the unexplained wage gap:
where $N_1 = \sum_i A_i$.
However, estimating the relationship between history and wage is challenging with realistic data sizes because career histories are high-dimensional: the number of possible career histories grows exponentially in the number of years someone has worked. This challenge is compounded by the fact that unexplained wage gaps are commonly estimated using small survey-based datasets, particularly in the U.S. where administrative data about worker histories are not generally available to researchers. For this reason, traditional econometric approaches have used a small number of hand-constructed summaries of labor market experience, which keeps the number of covariates in the predictive model small relative to the data set size. But these summary statistics do not capture the full complexity of labor market history, and in particular they may omit factors of history that are important for explaining the wage gap. \looseness=-1
For example, a large body of literature has focused on decomposing the gender wage gap in the United States by applying (ref) to small survey datasets (see blau_gender_2017 and altonji_chapter_1999 for surveys). Rather than including an individual's career history, most analyses include summary statistics about an individual's career history, such as the years of experience or tenure in the current job mincer1974family,manning2008gender,blau2013feasibility,blau_gender_2017. Even though many occupational taxonomies contain hundreds of fine-grained categories, it is most common to include coarse-grained occupational categories containing 20-30 categories blau1999analyzing,blau_gender_2017,boheim2021decomposition,hegewisch2014occupational. Because these models rely on a relatively small number of covariates, $\hat \mu_0$ and $\hat \mu_1$ are typically constructed using relatively simple models models, such as linear regressions oaxaca1973male,blinder_wage_1973,blau1999analyzing,blau_gender_2017 or LASSO models boheim2021decomposition,bonaccolto2022gender.
However, these incomplete measures of experience discard factors that help explain the wage gap. For example, regan2009work and blau2013feasibility find that potential experience (an inexact measure of experience that does not measure workforce interruptions) explains less of the wage gap than years of actual work experience. Moreover, light1995early estimates a wage model with detailed measures of year-by-year experience, finding that the timing of work experience explains a substantial portion of the wage gap. While incorporating full histories into gender wage gap analyses could ensure these factors aren't discarded, the predictive models used for gender wage gap analyses are too simple to include them.
Foundation models bommasani2021opportunities offer an alternative approach. Foundation models are machine learning models that learn low-dimensional representations of high-dimensional variables from data. These representations are initially learned on massive, passively-collected data after which they can be adapted on specific datasets of interest. For example, in natural language processing, foundation models that are trained to predict words using terabytes of Internet text can be adapted to generate responses to human questions achiam2023gpt,touvron2023llama. While initially developed for text, foundation models have successfully addressed seemingly intractable prediction problems in domains such as computer vision dosovitskiy2020image, music huang2018music, and protein generation madani2023large
A foundation model of labor market history can help estimate the unexplained wage gap by providing a low-dimensional representation of history that is predictive of wage. Because representations are learned from data, these estimates are not limited to the features a researcher knows to include.
Formally define a representation to be a function $\lambda(X): \mathcal{X} \to \mathbb{R}^D$. Given a representation $\lambda$, the wage gap unexplained by the representation of history is
where $\mu_{a}(\lambda(x))=\mathbb{E}[Y|A=a,\lambda(X)=\lambda(x)]$ is the expected wage as a function of the representation $\lambda(x)$.
For the rest of the paper, we consider estimating the unexplained wage gap using CAREER, a foundation model of labor market history vafa2023career. CAREER is trained to learn representations that can predict the next occupation a worker will have from a dataset of 24 million resumes posted online. When these representations are adapted to small survey datasets, CAREER makes more accurate predictions of an individual's next occupation than standard econometric approaches. We consider using these representations to predict an individual's wage. \looseness=-1
Foundation models are effective because they compress high-dimensional information into low-dimensional representations. However, we demonstrate that replacing an individual's history with a representation can introduce an omitted variable bias chernozhukov2022long.
We say the wage gap unexplained by a representation $\lambda$ is biased if it differs from the wage gap unexplained by the full history. Define this bias as $\text{OVB}(\lambda) = \text{UWG}(\lambda) - \text{UWG}$. It has a closed-form expression:
where
for representation-based propensity function $e(\lambda(x))=P(A=1|\lambda(X)=\lambda(x))$. (ref) contains a detailed derivation. (ref) can also be seen as a special case of the general omitted-variable-bias formula in chernozhukov2022long, where $\alpha(\lambda(X))$ and $\alpha(X)$ correspond to the short and long Riesz representers, respectively.
(ref) provides intuition for how a representation can induce bias. The omitted variable bias is a covariance of two differences: the first term is the difference in expected wage as a function of history and the representation of history, while the second term is the difference in the group propensity odds ratio as a function of history and the representation of history. A low-dimensional representation by definition discards information; for there to be no omitted variable bias, the discarded information that's related to wage should be unrelated to group propensity, and vice-versa. (ref) is closely related to econometric results about the extent of omitted variable bias in semiparametric models (see e.g. chernozhukov2022long); while this literature focuses on whether individual variables are included or not in models, we focus on representations, which can still omit variables despite being functions of all variables. Focusing on representations, veitch2020adapting provide a sufficient condition under which (ref) is 0, but do not characterize the exact level of bias; meanwhile, (ref) exactly characterizes the extent of omitted variable bias.
Although CAREER is a foundation model that is trained to learn representations from data, it is not trained to minimize omitted variable bias ((ref)). One reason is that its representations are trained to optimize a single objective that doesn't naturally appear in (ref): the predictability of an individual's next job. Moreover, the representations are trained on a different population of individuals than those for whom we'd like to estimate the unexplained wage gap.
Even biased, a foundation model can still be useful for estimating the unexplained wage gap because it can be fine-tuned. Empirically, when a foundation model's representations are adjusted to optimize a related but distinct objective from the one they were initially trained to optimize, they often outperform models trained on only the new objective devlin2018bert,lewis2019bart. We do not have to learn unbiased representations of career history from scratch; we can adjust the representations of a pretrained foundation model to debias it.
The standard approach for modifying foundation models is supervised fine-tuning devlin2018bert. In our setting, supervised fine-tuning would entail modifying a foundation model's representation $\lambda$ to be predictive of wage on the survey data used for wage gap estimation. But while a foundation model would likely form better wage predictions after supervised fine-tuning, it can still be biased for estimating the unexplained wage gap; unless the foundation model recovers the exact relationship between labor market history and wage, supervised fine-tuning can still introduce arbitrarily large omitted variable bias.
We now describe a set of conditions for fine-tuning under which an estimator of a wage gap that conditions on representations derived from a foundation model is not only unbiased and consistent but also converges at a rate proportional to $n^{-1/2}$:
The first condition is about omitted variable bias: it requires that the omitted variable bias of the representations converges to 0 at a rate proportional to $n^{-1/2}$. This will be trivial for some representations: for example, $\lambda(X) = X$ has no omitted variable bias by definition. However, the second assumption imposes restrictions about modeling wage and group membership: these models must approximate the true relationship between the representation of history and these outcomes such that error goes to zero at a combined root-n rate. Note that these modeling assumptions are with respect to the representation: the true relationships between history and the outcomes do not need to be reconstructed, only those between the representation and the outcome. Therefore, satisfying the first two assumptions involves striking a balance: representations must be detailed enough to not have omitted variable bias, but also low-dimensional enough so that outcomes can be efficiently estimated as a function of the representation. (ref) contains more details and a proof.
Although (ref) is stated in terms of the size $n$ of the single dataset used for fine-tuning, note that much larger datasets are typically used to pretrain the foundation model. In practice, training a high-dimensional representation $\lambda(X)$ from scratch on moderate-sized survey dataset would be intractable; however, the fact that a much larger dataset contributes to the initial estimation of $\lambda(X)$ suggests that it may be feasible to adequately control omitted variable bias with a larger dimensional representation $\lambda(X)$ than would be possible without the foundation model. Indeed, under a repeated-sampling framework in which both the pretraining and fine-tuning samples are repeatedly resampled, we expect most of the sampling variation to arise from the smaller fine-tuning sample rather than from the larger pretraining sample used to train the foundation model. Consequently, our formal results condition on the pretrained representation and focus on the sampling variation arising from the fine-tuning sample.
Empirical evidence in domains such as computer vision, natural language processing, and protein structure suggests that such large-scale pretraining often yields robust and transferable representations for predictions bommasani2021opportunities. At the same time, we do not require the foundation model to deliver a perfect or “true” representation; we instead require that fine-tuning from the pretrained representation allows for omitted variable to go to 0 at a $\sqrt{n}$ rate. A general-purpose foundation model may still omit some information relevant for a particular downstream task, and it is precisely the role of subsequent fine-tuning to adapt the representation to the problem at hand.
(ref) relates to results from the causal inference literature robins1997toward,tsiatis2006semiparametric,athey2016efficient,chernozhukov2018double,kennedy2022semiparametric. Although the unexplained wage gap is not a causal quantity, it is mathematically identical to an average treatment effect on the treated (ATT). Specifically, Assumption 2 is similar to assumptions for $\sqrt{n}$-consistency in the doubly-robust/double machine learning literature, which often assume $o(n^{-1/2})$ combined error of outcome and propensity models chernozhukov2018double,kennedy2022semiparametric:
This is similar to Assumption 2, with one key difference: (ref) requires $o(n^{-1/2})$ combined error as a function of the full history, while Assumption 2 only requires $o(n^{-1/2})$ combined error as a function of the representation $\lambda_n(X)$. When $\lambda_n(X)$ is lower-dimensional than the full history $X$, Assumption 2 will be more realistic than (ref). In fact, when $\lambda_n(X) = X$, Assumptions 2 and 3 are trivially satisfied and our result reduces to the standard double-robustness result chernozhukov2018double. Assumption 2 lifts the requirement for full combined error to go to 0 while Assumption 1 imposes restrictions on what constitutes a valid representation.
(ref) relates to results from the variable selection literature in causal inference belloni2014high,shortreed2017outcome,tang2023ultra,cho2023variable. This literature is motivated by a classic result: to make valid causal inferences, it is sufficient to condition only on variables that affect both treatment assignment and outcome rosenbaum1983central. Like Theorem 1, the results in this literature do not necessarily assume that the full outcome or propensity model can be consistently estimated as a function of the full set of covariates. While this literature proposes techniques when individual variables are shared in outcome and treatment models, these techniques do not apply when there is more complicated shared structure; for example, the number of years spent in a blue collar job may affect both treatment and outcome, but this is a transformation rather than a single variable. In contrast, our method is based on representations, or potentially complicated functions of variables, rather than individual variables. A set of selected variables is an example of a representation; but representations can be more complex than a set of variables constructed by a researcher.
Other methods from the causal inference and econometrics literature have also proposed using representations or latent variables from machine learning models. For example, battaglia2024inference demonstrate that latent variables from a machine learning model should be jointly optimized with the econometric outcome of interest rather than first estimated separately and then plugged into an econometric model. Related to our method, veitch2020adapting provide a sufficient condition under which a representation is unbiased for estimating a causal effect, which motivates empirical methods used by shi2019adapting and chernozhukov2022riesznet (and is the basis of the multi-task debiased fine-tuning objective we consider). In contrast, we provide an if-and-only-if condition under which there is no bias, and we characterize the exact level of bias with a connection to omitted variable bias chernozhukov2022long. Additionally, we provide conditions about the level of omitted variable bias under which estimation is $\sqrt{n}$-consistent and asymptotically normal.
A strand of literature in supervised machine learning has also focused on integrating ideas from the causal inference literature into predictive methods in order to improve the properties of predictive models, such as stability (see cui2022stable for a review of this literature). Similar to the approach in our paper, these methods adjust the training of a predictive model to avoid regularization-induced omitted variable bias, but the literature on stable prediction considers reducing such bias for many covariates simultaneously in a cross-sectional prediction problem. For example, some such methods reweight data to reduce the correlation among features.
The exact level of omitted variable bias ((ref)) cannot be computed from data; it involves calculating the same high-dimensional function the representation is meant to approximate, $\mu_A(X)$. However, even if the bias cannot be computed exactly, we can still learn representations that are targeted to minimize it. Below, we develop three fine-tuning methods for minimizing this bias. Each method addresses omitted variable bias from a distinct angle. In principle, the optimal approach may vary across applications. For instance, multi-task fine-tuning is straightforward to implement but can require tuning an extra hyperparameter; projection fine-tuning removes that hyperparameter but may converge more slowly; difference-based fine-tuning can capture group disparities more directly but does not deliver a direct wage or propensity predictor. To choose among them, we recommend using validation metrics such as the R-Learner metric nie2021quasi described (and subsequently used for model selection) in (ref).
The expression for omitted variable bias recalls a classic result from causal inference: to make valid causal inferences, it is sufficient to condition only on variables that affect both treatment assignment and outcome rosenbaum1983central. Thus, if a representation captures all the features of history that are predictive of both wage and group membership, it will result in zero omitted variable bias. While a representation that perfectly captures all of the variables related to wage (or equivalently to group membership) will result in zero omitted variable bias, it might be more realistic to capture the potentially smaller set of variables that affects both wage and group membership.
We therefore fine-tune the representations to be predictive of both wage and group membership. We consider two approaches. In Multi-Task Fine-Tuning, we use the method proposed by veitch2020adapting and shi2019adapting to learn a representation that jointly minimizes wage and group membership predictive errors:
where $\beta\in\mathbb{R}^+$ is a hyperparameter, $\ell_Y$ is the mean squared-error loss, and $\ell_A$ is the binary cross-entropy loss. We also consider a similar approach, Projection Fine-Tuning, that alternates between losses: $\lambda$ is optimized to minimize mean-squared error loss until convergence, then $\lambda$ is optimized to minimize binary cross-entropy loss until convergence, and this process repeats until the procedure converges. This procedure is based on projected gradient descent calamai1987projected and does not require choosing a hyperparameter $\beta$. See (ref) for more details.
An alternative debiased fine-tuning method is motivated by the heterogeneous treatment effect literature athey2016recursive,wager2018estimation,nie2021quasi, where the goal is to model the difference in outcome functions for each group rather than each group's function individually. If the difference in these functions is simpler than each individual function, a method that's targeted to capture this difference will be more effective than modeling each function separately.
Thus, we propose a method for fine-tuning foundation models meant to capture group differences. This method is based on the R-learner approach for estimating heterogeneous treatment effects in causal inference robinson1988root,nie2021quasi. It is based on the observation that group differences can be written as the solution to an objective:
where $m(x) = \mathbb{E}[Y|X=x]$ is the conditional wage function averaged over the two groups.
The Difference-Based Fine Tuning procedure begins by estimating the conditional wage function $\hat m(\lambda_m(x))$ and the propensity function $\hat e(\lambda_e(x))$ using supervised fine-tuning. Starting with the wage, we estimate a function $\hat m$ and fine-tuned representation $\lambda_m$ in order to minimize the squared loss in predicting the wage $Y$. We estimate $\hat e$ and $\lambda_e$ analogously. Treating these functions and their respective representations as fixed, we then fine-tune a new representation to minimize the squared error:
where $\rho: \mathbb{R}^D \to \mathbb{R}$ is a flexible function; in the empirical studies we use a two-layer feed-forward neural network. The unexplained wage gap is then estimated as
where $N_1 = \sum_i A_i$. We refer to this method as difference-based fine-tuning. This approach is based on the R-learner approach, but is based on representations: separate representations are used for the wage, propensity, and wage-difference models, so that the representation used for the last model is optimized to only capture the differences in groups. See (ref) for more details.
This optimization procedure encourages a representation that captures differences in group wages. The true relationship between wage and labor market history may be complicated for both groups. However, if the difference in relationships is not as complicated, it will be easier to learn a representation that captures the difference.
In practice, we implement each fine-tuning method by optimizing the respective objective using Adam kingma2014adam. Each model is initialized at the foundation model's parameters and all parameters are re-optimized with respect to the new objective. The same neural architecture from pretraining (e.g. the transformer layers of CAREER) is retained, but we typically add extra parameters on top of the representation for our downstream tasks (e.g. predicting wages or propensities) and jointly update all parameters during fine-tuning. These steps enable the representation to adapt to the new objectives while preserving its broad, pretrained knowledge. See (ref) for more details.
Our empirical analysis uses data from one of the leading U.S. administrative surveys, the Panel Study of Income Dynamics psid, or PSID. PSID is a longitudinal survey that has followed a cohort of American families since 1968. It is constructed to be nationally representative and is frequently used to estimate unexplained wage gaps blau_us_2006,blau_gender_2017. Because the same individuals are interviewed over the course of the survey, labor market histories can be constructed by tracking the trajectory of reported occupations each year an individual is in the survey.
We encode occupations into one of 330 “occ1990dd” occupational categories david2013growth. Since the PSID includes information about individuals who are not working, we add seven categories for when an individual's occupation is not listed but their employment status is available (e.g. employed, laid off).
Following blau_gender_2017, we restrict our sample to the surveys conducted between 1990-2019, consisting of 91,391 observations over 19 surveys, and further restrict our sample to non-farm and non-military wage and salary workers between 25 and 64 years old who worked for at least 26 weeks in non-farm jobs. We incorporate longitudinal sample weights into our analysis, which are designed to adjust for differences in the probability of selection into the sample. (ref) contains more details about how we construct the dataset.
We first validate our proposed methods for debiased fine-tuning. Typically, machine learning models are evaluated using measures of predictive accuracy on held-out data. However, our ultimate goal isn't forming more accurate wage predictions, but rather more accurate estimates of the unexplained wage gap. To this end, we turn to semi-synthetic experiments, a common method to assess causal estimation strategies in a controlled setting (e.g. athey2021using). Semi-synthetic experiments allow us to evaluate the performance of different methods because the data generating process is known.
In order for the semi-synthetic experiments to reflect real-world data, we use real labor market histories from the PSID sample (Section (ref)). Our semi-synthetic experiments are based on forming a “ground-truth” representation of the history and then generating group labels and wages as a function of this history. To mimic the fact that nature is often more complicated than the model we might select, we use a more complicated representation to generate data than the one used by the foundation model to estimate wage gaps. Specifically, we use a transformer architecture that is 20 times larger than the one used for estimation to simulate a “ground-truth” representation $\lambda^*: \mathcal X \to \mathbb{R}^D$. For each setting, we simulate group labels and wages as a function of the representation $\lambda^*$. We control how much of the representation is shared between these functions by introducing binary variables $\mathbf{u}, \mathbf{v} \in \{0, 1\}^D$ that mask the dimensions of the representation used for the group and wage models, accordingly. We then simulate from the following model:
where $\beta \in \mathbb{R}^D$ is a random vector of regression coefficients, $\tau \in \mathbb{R}$ is the true unexplained gap, $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ is the outcome noise, and $\sigma(\cdot)$ is the inverse-logit function. In this setup, $\mathbf{u}^\top \mathbf{v}/D$ is the proportion of the representation that is shared. For each experiment, we control the shared proportion ($\mathbf{u}^\top \mathbf{v}/D$), the true gap ($\tau$), and the level of outcome noise $(\sigma^2)$. We consider 27 different settings, and perform multiple samples in each setting by resampling $A_i$, $\epsilon_i$, $\mathbf{u}$, $\mathbf{v}$, and $\beta$. See (ref) for more details.
We compare four methods for estimating the unexplained wage gap from semi-synthetic data. Our baseline is Supervised Fine-Tuning devlin2018bert: fine-tuning a foundation model to predict wage without an explicit debiasing objective. We compare this baseline to the three debiasing methods described above: Multi-Task Fine-Tuning, Projection Fine-Tuning, and Difference-Based Fine-Tuning. We perform 400 simulations, re-generating data and re-training each method for each simulation. Here we estimate the unexplained gap using the outcome-only estimator (analogous to (ref)), while (ref) shows results for the AIPW estimator ((ref)), which we find to perform slightly worse in practice. Although AIPW is theoretically advantageous in large samples robins1997toward,tsiatis2006semiparametric, it can suffer from higher variance when propensity models are difficult to estimate, thereby degrading its finite-sample performance kang2007demystifying.
(ref) compares the MSE of the estimate of the gender wage gap (relative to the oracle gender wage gap) derived from alternative estimation approaches. All three methods for debiasing foundation models consistently outperform the standard supervised fine-tuning approach. The advantage of debiasing is largest when more of the representation is shared across the wage model and group labels. This reinforces the motivation behind representation learning; as there is more shared structure in how group labels and wages relate to history, sharing representations can improve estimates. Projection fine-tuning and difference-based fine-tuning are both more successful than multi-task fine-tuning, especially when more of the representation is shared. The full set of results is in (ref).
How should we validate models on real-world data? While wage and gender predictive metrics are important, they do not directly assess estimation quality. While matching-based methods stuart2010matching can also be used to validate estimation in principle, these require low-dimensional covariates. We instead consider another validation metric, inspired by the R-Learner objective in (ref). Because (ref) is minimized when $\tau(X)$ is the true expected difference female and male wages, we evaluate (ref) given a model's estimate of $\hat \tau(X) = \hat \mu_1(X) - \hat \mu_0(X)$,
where $i=1, \dots, n$ index $n$ held-out samples and $\hat m(X_i)$ and $\hat e(X_i)$ are models trained to predict wage and gender, respectively, using supervised fine-tuning of CAREER. We refer to (ref) as the R-Learner Metric. We note that because this metric relies on estimates of wage and propensity models, it is sensitive to the specification of $\hat m$ and $\hat e$. However, we validate this metric on semi-synthetic data, finding it to be a useful proxy for model performance (see (ref)).
We now apply our methods to the (actual) PSID data. We begin by evaluating the quality of wage predictions derived from alternative models, including the standard econometric models from the wage gap literature blau_gender_2017, since predictive accuracy can be easily evaluated using held-out test data. We show that foundation-based representations substantially improve predictive performance relative to standard regression-based econometric models, suggesting that our methods can capture variables that have the potential to cause omitted variable bias when estimating unexplained wage gaps. We then directly demonstrate that representations derived from our methods capture elements of history that are predictive of both wage and gender, so that they indeed meet the criteria for omitted variable bias, and that these are quantitatively important for explaining wage gaps.
As baselines, we consider econometric models that use hand-constructed summaries of an individual's career but not their full history to predict wage. Following the econometric literature, we consider two linear models: regression (fit with OLS) and LASSO. Given covariates $Z_i \in \mathbb{R}^P$, these models estimate the wage function as
for $A \in \{0, 1\}$, an intercept $\theta_A \in \mathbb{R}$, and regression coefficients $\beta_A \in \mathbb{R}^P$. Following blau_gender_2017, the covariates included in $Z$ are: years of full-time and part-time experience (and their squares), years of schooling, indicators for bachelors and advanced degrees, race and ethnicity indicators, gender indicators, census and region indicators, an indicator for collective bargaining coverage, 15 industry category indicators, and 21 occupation category indicators. We also consider two different methods of encoding occupations: “coarse-grained”, which uses the 21 coarse-grained occupational categories above, and “fine-grained”, with an additional 330 fine-grained occupational categories.
We compare these models from the economics literature to predictions based on foundation models that use CAREER vafa2023career to represent labor market history. CAREER is pretrained to learn representations of career trajectories on a dataset of 23.7 million resumes. We consider both supervised fine-tuning devlin2018bert and debiased fine-tuning approaches to modify CAREER's representations. When we fine-tune CAREER, we use both its representations of history and the covariates $Z_i$ described above to predict an individual's wage; see (ref) for more details. In order to understand how different methods of including history affect predictions, we train two additional versions of CAREER: one that uses the neural network to encode an individual's current job but not their history (“CAREER (current job only)”), and one that includes an individual's current job but only their workforce participation status for previous jobs (e.g. “unemployed”, “out-of-labor force”), which we refer to as “CAREER (participation only)”.
Table (ref) shows the held-out $R^2$ for each model's wage and gender predictions. (Since gender isn't real-valued, we use pseudo $R^2$ based on negative log-likelihood.) CAREER outperforms all the econometric baselines. With the standard supervised fine-tuning, CAREER has a held-out wage $R^2$ of 0.515 and held-out gender $R^2$ of $0.510$. Its predictive performance is not stemming from including a better functional form for an individual's current job or capturing employment spells more fully. For the two debiased fine-tuning methods, we find that wage $R^2$ slightly worsens, while gender predictions might slightly improve, although this improvement is within the standard errors. (We do not have wage or gender predictions for difference-based fine-tuning because it predicts the difference in group wages rather than individual wages or genders.) These results extend a finding from vafa2023career, which also shows that transformer-based methods can improve wage predictions relative to econometric baselines. What (ref) additionally shows is that gender predictions are improved by a larger margin and that these gains are still present for debiased fine-tuning methods. This result demonstrates another benefit of representation learning; if group membership and wage are correlated with similar transformations of input data, then learning representations that are predictive of both can improve predictions.
We now compare gender wage gaps estimated with standard econometric techniques to those estimated with a foundation model's representations of labor market history, following the approaches described above. For the econometric models, we use a linear regression using the same covariates described in (ref), encoding occupation into one of 21 coarse-grained labels. For the foundation models, we fine-tune CAREER using each of the three debiased fine-tuning methods described in (ref). In addition to using the machine learned representations of history, these methods also incorporate the same hand-constructed covariates as the linear model. Because the unexplained wage gap is only identified when there is overlap (e.g. when there are workers with similar histories in both groups), we trim the study population. Specifically, we fine-tune CAREER with supervised fine-tuning to estimate a propensity model $\hat e(\lambda(X_i))$, and only include individuals $i$ such that $0.01 < \hat e(\lambda(X_i)) < 0.99$. We consider other trimming strategies in (ref), finding similar results. We compute standard errors with bootstrapping, where the standard errors reflect sampling uncertainty conditional on the fine-tuned model. We do not re-train models for each bootstrap sample. Instead, we keep models fixed, evaluating each model on the bootstrapped sample; we refer to this as test-set bootstrapping.
The results are summarized in (ref). Across all fine-tuning methods, full history explains more of the gender wage gap than hand-constructed summaries of history, which are typically used to explain gender wage gaps blau_trends_2013,goldin_grand_2014,blau_gender_2017. While the wage ratio unexplained by summaries of history is 88.6%, the ratio is above 90% for all methods that use representations of full history, ranging from 90.4% for projection fine-tuning to 93.4% to difference-based fine-tuning. Note that these numbers vary a little depending on the trimming threshold and whether the AIPW estimator ((ref)) is used instead of the outcome-only estimator; additional results are presented in (ref). (ref) also shows the R-Learner metric in (ref) for each model, which is minimized by difference-based fine-tuning. The last column of (ref) shows that the difference-based fine-tuning method's R-Learner metric improvement over the the regression and multi-task fine-tuning improvement is significant at the 95% level. Our analysis for the remainder of the paper considers results from the difference-based fine-tuning approach.
To further investigate where history is explaining the gender wage gap, we consider different cuts of the survey data. (ref) shows the gender wage ratios over time. Compared to the methods that adjust the wage gap for summary statistics of history, the learned representations of history are explaining the least of the gap in 1990-1995 and the most of the gap in 2014-2019. Overall, we find that compared to methods that adjust the gap for summary statistics of history, full history explains more of the gap for later years. While this may reflect changes in the underlying wage dynamics between these two periods, it may also reflect the fact that more history is available in the later versions of the survey. To further investigate, (ref) fixes the time period to 2014-2019 and considers different wage ratios by age. For the youngest workers, history explains the least of the gap; the history-explained ratio is almost the same as the unadjusted ratio for 25-34 year-old workers. However, history consistently explains more of the gap as workers become older. Further, (ref) breaks down the adjusted wage gap by the occupational categories considered in blau_gender_2017. Compared to the wage gap explained by summary statistics of history, the wage gap explained by representations of full history is smoother across occupational categories. The occupational category where history explains the most of the wage gap is in non-physician healthcare-related occupations; the category where it explains the least of the wage gap is for computer-related occupations.
The promise of incorporating more complete representations of history into wage gap analyses is that they can include variables that are typically omitted (recall from (ref) that omitted variables include variables that are correlated with both wage and gender). This is not unique to machine learning methods; for example, prior studies have found that potential experience (an inexact measure of experience that does not measure workforce interruptions) explains less of the wage gap than years of actual work experience regan2009work,blau2013feasibility, as do representations of history that do not include timing of labor force participation light1995early. What machine learning offers is the ability to automatically learn these variables without prespecifying them.
Here, we investigate the aspects of history that are captured by the foundation model but omitted by traditional methods that summarize history with summary statistics. While our model learns representations of history for each individual, they are difficult to interpret directly because they are continuous. To better understand these representations of histories, we form clusters of histories. We constrain each history in a cluster to have the same current occupational category but allow the kinds of history in a cluster to vary. We then study which clusters are important for predicting wages by building a regression tree to predict wage from the clusters. The regression tree adds clusters one-at-a-time in the order that is most useful for minimizing wage error. This ordering allows us to identify and interpret which occupation patterns --- beyond an individual's current job --- are most predictive of wage.
(ref) shows the six most predictive groups (see (ref) for more details and (ref) for the top 15 groups). These groups reveal important aspects of history that are omitted by hand-constructed summaries. For example, one of the most predictive types of histories consists of managers who were previously computer scientists and engineering technicians. This is a predominantly male group (83% male), and managers with these jobs in their histories get paid more than managers without them. There are multiple interpretations for why managers with these histories are paid more than managers without them; perhaps these managers have different skills than other managers, or perhaps they are performing different jobs that are not captured by the occupational encoding scheme (e.g. there is no occupational category for engineering manager). In addition to this group, the other top groups that are important for wage predictions are also correlated with gender. Omitting these variables --- like standard econometric methods do --- induces omitted variable bias. (ref) includes additional analyses of omitted variables found at the fine-grained occupational level.
We used foundation models to study a classic problem from labor economics: estimating the difference between how individuals with the same labor market experience get paid when they belong to different groups. With foundation models, wage predictions improve over econometric baselines by up to 15%. We also showed that an omitted variable bias arises when a foundation model discards relevant information about group differences. To mitigate this problem, we proposed procedures for debiasing foundation models, which we validated on semi-synthetic data. On survey data from the Panel Study of Income Dynamics, we found that labor market history explains more of the gender wage gap than the summary statistics of history used by standard econometric methods.
These findings are suggestive of how to use foundation models in social science research. One direct application is using the debiasing methods we propose to estimate causal effects with foundation models. While we study a foundation model of labor market history trained on resume data, these methods can extend to analyses involving other foundation models. For example, foundation models trained on rich, nationwide administrative data can help answer a variety of descriptive and causal questions savcisens2024using.
Additionally, our methods can help address questions about the representativeness of large language models (LLMs), the most common type of foundation model. A recent literature has found that large language models, when queried to answer questions from surveys, do not respond in ways that are representative of the national population santurkar2023whose. The problem of representative predictions and debiased foundation models are closely relate. Our methods could be adapted to improve the representativeness of LLMs.
\paragraph{Acknowledgements.} Keyon Vafa is supported by the Harvard Data Science Initiative. Susan Athey is supported by the Golub Capital Social Impact Lab, the Stanford Institute for Human-Centered Artificial Intelligence, and the Business, Governance, and Society Initiative at Stanford’s Graduate School of Business. David M. Blei is supported by NSF IIS-2127869, NSF DMS-2311108, ONR N000142412243, and the Simons Foundation.