Keyon Vafa, Susan Athey, David M. Blei
arXiv 15 Sep 2024 · Machine Learning · publishedProceedings of the National Academy of Sciences (2025) · 3 citations (OpenAlex)
arXiv:2409.09894 · PDF · DOI · OpenAlex · Extracted main text
The rise of foundation models marks a paradigm shift in machine learning: instead of training specialized models from scratch, foundation models are first trained on massive datasets before being adapted or fine-tuned to make predictions on smaller datasets. Initially developed for text, foundation models have also excelled at making predictions about social science data. However, while many estimation problems in the social sciences use prediction as an intermediate step, they ultimately require different criteria for success. In this paper, we develop methods for fine-tuning foundation models to perform these estimation problems. We first characterize an omitted variable bias that can arise when a foundation model is only fine-tuned to maximize predictive accuracy. We then provide a novel set of conditions for fine-tuning under which estimates derived from a foundation model are root-n-consistent. Based on this theory, we develop new fine-tuning algorithms that empirically mitigate this omitted variable bias. To demonstrate our ideas, we study gender wage decomposition. This is a statistical estimation problem from econometrics where the goal is to decompose the gender wage gap into components that can and cannot be explained by career histories of workers. Classical methods for decomposing the wage gap employ simple predictive models of wages which condition on coarse summaries of career history that may omit factors that are important for explaining the gap. Instead, we use a custom-built foundation model to decompose the gender wage gap, which captures a richer representation of career history. Using data from the Panel Study of Income Dynamics, we find that career history explains more of the gender wage gap than standard econometric models can measure, and we identify elements of career history that are omitted by standard models but are important for explaining the wage gap.
appendix boundary found by appendix_command · 55% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K (2018) BERT: Pre-training of deep bidirectional transformers for language understanding | 1.000 | 8 | 4 | 100% |
| 2 | Vafa, K., Palikot, E., Du, T., Kanodia, A., Athey, S., and Blei, D. M (2023) CAREER: A foundation model for labor sequence data self | 0.894 | 7 | 4 | 71% |
| 3 | Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C… (2018) Double/debiased machine learning for treatment and structural parameters | 0.874 | 6 | 4 | 67% |
| 4 | Chernozhukov, V., Cinelli, C., Newey, W., Sharma, A., and Syrgkanis, V (2022) Long story short: Omitted variable bias in causal machine learning | 0.811 | 5 | 2 | 80% |
| 5 | Oaxaca, R (1973) Male-female wage differentials in urban labor markets | 0.811 | 4 | 2 | 100% |
| 6 | Blau, F. D. and Kahn, L. M (2017) The gender wage gap: Extent, trends, and explanations | 0.784 | 29 | 7 | 48% |
| 7 | Panel Study of Income Dynamics (2023) Public use dataset, produced and distributed by the Survey Research Center, Institute for Social Research, University of Michiga… | 0.737 | 3 | 3 | 67% |
| 8 | Blau, F. D. and Kahn, L. M (2013) The feasibility and importance of adding measures of actual experience to cross-sectional data collection | 0.737 | 3 | 2 | 100% |
| 9 | Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von… (2021) On the opportunities and risks of foundation models | 0.737 | 3 | 2 | 100% |
| 10 | Kitagawa, E. M (1955) Components of a difference between two rates | 0.737 | 3 | 2 | 100% |
Showing the top 10 of 68 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.