George Gui, Olivier Toubia
arXiv 24 Dec 2023 · Artificial Intelligence · 37 citations (OpenAlex)
arXiv:2312.15524 · PDF · DOI · OpenAlex · Extracted main text
Large Language Models (LLMs) have shown impressive potential to simulate human behavior. We identify a fundamental challenge in using them to simulate experiments: when LLM-simulated subjects are blind to the experimental design (as is standard practice with human subjects), variations in treatment systematically affect unspecified variables that should remain constant, violating the unconfoundedness assumption. Using demand estimation as a context and an actual experiment as a benchmark, we show this can lead to implausible results. While confounding may in principle be addressed by controlling for covariates, this can compromise ecological validity in the context of LLM simulations: controlled covariates become artificially salient in the simulated decision process, which introduces focalism. This trade-off between unconfoundedness and ecological validity is usually absent in traditional experimental design and represents a unique challenge in LLM simulations. We formalize this challenge theoretically, showing it stems from ambiguous prompting strategies, and hence cannot be fully addressed by improving training data or by fine-tuning. Alternative approaches that unblind the experimental design to the LLM show promise. Our findings suggest that effectively leveraging LLMs for experimental simulations requires fundamentally rethinking established experimental design practices rather than simply adapting protocols developed for human subjects.
appendix boundary found by appendix_command · 64% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Toubia, Olivier and Gui, George Z and Peng, Tianyi and Merlau, Danie… (2025) Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions self | 1.000 | 7 | 3 | 100% |
| 2 | Argyle, Lisa P. and Busby, Ethan C. and Fulda, Nancy and Gubler, Jos… (2023) Out of One, Many: Using Language Models to Simulate Human Samples | 0.928 | 4 | 3 | 100% |
| 3 | Hewitt, Luke and Ashokkumar, Ashwini and Ghezae, Isaias and Willer,… (2024) Predicting results of social science experiments using large language models | 0.811 | 4 | 2 | 100% |
| 4 | Horton, John J (2023) Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? | 0.737 | 3 | 2 | 100% |
| 5 | Brand, James and Israeli, Ayelet and Ngwe, Donald (2023) Using GPT for Market Research | 0.644 | 2 | 2 | 100% |
| 6 | Berke, Alex and Calacci, Dan and Mahari, Robert and Yabe, Takahiro a… (2024) Open e-commerce 1.0, five years of crowdsourced US Amazon purchase histories with user demographics | 0.511 | 2 | 2 | 50% |
| 7 | DellaVigna, Stefano and Gentzkow, Matthew (2019) Uniform Pricing in U.S. Retail Chains* | 0.511 | 2 | 2 | 50% |
| 8 | Ludwig, Jens and Mullainathan, Sendhil and Rambachan, Ashesh (2024) Large language models: An applied econometric framework | 0.511 | 2 | 1 | 100% |
| 9 | Angrist, Joshua D. and Pischke, Jörn-Steffen (2009) Mostly Harmless Econometrics: An Empiricist's Companion | 0.405 | 1 | 1 | 100% |
| 10 | Imbens, Guido W. and Rubin, Donald B (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction | 0.405 | 1 | 1 | 100% |
Showing the top 10 of 35 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.