Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan
arXiv 9 Dec 2024 · Econometrics · publishedAnnual Review of Economics (2026) · 19 citations (OpenAlex)
arXiv:2412.07031 · PDF · DOI · OpenAlex · Extracted main text
How can we use the novel capacities of large language models (LLMs) in empirical research? And how can we do so while accounting for their limitations, which are themselves only poorly understood? We develop an econometric framework to answer this question that distinguishes between two types of empirical tasks. Using LLMs for prediction problems (including hypothesis generation) is valid under one condition: no “leakage” between the LLM's training dataset and the researcher's sample. No leakage can be ensured by using open-source LLMs with documented training data and published weights. Using LLM outputs for estimation problems to automate the measurement of some economic concept (expressed either by some text or from human subjects) requires the researcher to collect at least some validation data: without such data, the errors of the LLM's automation cannot be assessed and accounted for. As long as these steps are taken, LLM outputs can be used in empirical research with the familiar econometric guarantees we desire. Using two illustrative applications to finance and political economy, we find that these requirements are stringent; when they are violated, the limitations of LLMs now result in unreliable empirical estimates. Our results suggest the excitement around the empirical uses of LLMs is warranted -- they allow researchers to effectively use even small amounts of language data for both prediction and estimation -- but only with these safeguards in place.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Ludwig, Jens and Mullainathan, Sendhil (2024) Machine learning as a tool for hypothesis generation self | 0.843 | 3 | 3 | 100% |
| 2 | Paul Glasserman and Caden Lin (2023) Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis | 0.811 | 4 | 2 | 100% |
| 3 | Anastasios N. Angelopoulos and Stephen Bates and Clara Fannjiang and… (2023) Prediction-powered inference | 0.737 | 3 | 2 | 100% |
| 4 | Sarkar, Suproteem K and Vafa, Keyon (2024) Lookahead bias in pretrained language models | 0.737 | 3 | 2 | 100% |
| 5 | Egami, Naoki and Hinck, Musashi and Stewart, Brandon M. and Wei, Han… (2024) Using imperfect surrogates for downstream inference: design-based supervised learning for social science applications of large l… | 0.644 | 2 | 2 | 100% |
| 6 | Lung-fei Lee and Jungsywan H. Sepanski (1995) Estimation of Linear and Nonlinear Errors-in-Variables Models Using Validation Data | 0.644 | 2 | 2 | 100% |
| 7 | Liu, Pengfei and Yuan, Weizhe and Fu, Jinlan and Jiang, Zhengbao and… (2023) Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing | 0.644 | 2 | 2 | 100% |
| 8 | Suproteem Sarkar (2024) StoriesLM: A Family of Language Models With Time-Indexed Training Data | 0.644 | 2 | 2 | 100% |
| 9 | Siruo Wang and Tyler H. McCormick and Jeffrey T. Leek (2020) Methods for correcting inference based on outcomes predicted by machine learning | 0.644 | 2 | 2 | 100% |
| 10 | Jacob Carlson and Melissa Dell (2025) A Unifying Framework for Robust and Efficient Inference with Unstructured Data | 0.644 | 2 | 2 | 100% |
Showing the top 10 of 110 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.