Joel Persson, Mårten Schultzberg, Sebastian Ankargren
arXiv 15 Jun 2026 · Statistics — Methodology
arXiv:2606.17165 · PDF · DOI · OpenAlex · Extracted main text
Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost. We study when a treatment effect estimated on LLM outcomes can recover the effect for the human population of interest. Distributional equivalence between LLM and human outcomes would make any standard estimator valid but is unrealistic. We therefore develop a statistical framework that adapts surrogate endpoint theory to LLMs, showing that calibrating LLM outcomes to human outcomes identifies the average treatment effect under surrogacy and comparability conditions that are jointly weaker than distributional equivalence. We present a falsification test for surrogacy and a bound on the worst-case bias from limited overlap between the LLM and human samples. We further show that the stochasticity inherent to LLMs can weaken surrogacy for identification while also introducing bias and variance during estimation, but that using an average over multiple LLM draws per unit as the surrogate mitigates these issues. Simulations validate the results, and an empirical application to the Upworthy Research Archive dataset shows that raw LLM outputs recover only 39% of the human treatment effect while nonparametric calibration closes the gap. A central takeaway is that A/B testing on LLM responses is correct only by assumption, whereas A/B testing on humans is correct by design, and that the required assumptions are hardest to justify precisely where LLMs promise the greatest benefit. We discuss the choice of LLM, prompting, and temperature as design variables, the compounded challenge posed by long-term outcomes, and how to size human pilot studies for validation.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang (2019) The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely | 0.928 | 4 | 3 | 100% |
| 2 | Wayne A. Fuller (1987) Measurement Error Models | 0.843 | 3 | 3 | 100% |
| 3 | Yuan Gao, Dokyun Lee, Gordon Burtch, and Sina Fazelpour (2025) Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina | 0.737 | 3 | 2 | 100% |
| 4 | Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi (2020) The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations (ICLR) | 0.644 | 2 | 2 | 100% |
| 5 | Nathan Kallus and Xiaojie Mao (2020) On the Role of Surrogates in the Efficient Estimation of Treatment Effects with Limited Outcome Data | 0.644 | 2 | 2 | 100% |
| 6 | Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang (2025) The Surrogate Index: Combining Short-term Proxies to Estimate Long-term Treatment Effects More Rapidly and Precisely | 0.511 | 2 | 1 | 100% |
| 7 | Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee,… (2023) Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Representations (IC… | 0.511 | 2 | 1 | 100% |
| 8 | Naoki Egami, Musashi Hinck, Brandon M. Stewart, and Hanying Wei (2023) Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large L… | 0.511 | 2 | 1 | 100% |
| 9 | George Gui and Olivier Toubia (2023) The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective | 0.511 | 2 | 1 | 100% |
| 10 | Anne Lundgaard Hansen, John J. Horton, Sophia Kazinnik, Daniela Puzz… (2024) Simulating the Survey of Professional Forecasters | 0.511 | 2 | 1 | 100% |
Showing the top 10 of 32 scored citations.