Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
60,608 characters · 15 sections · 70 citation commands
AI-Assisted Variance Reduction in Randomized Experiments
The rise of large language models (LLMs) has renewed interest in “silicon sampling,” “digital twins,” and related frameworks that use AI to simulate or predict human responses in surveys and experiments argyle2023out, horton2023large, brand2023using. While these approaches are fast and scalable, they are often poor substitutes for human respondents cummins2025threat,peng2025digital. This has motivated a growing methodological literature on hybrid estimators that combine AI predictions with ground-truth human data, for example, via prediction-powered inference angelopoulos2023prediction or semiparametrically efficient estimation ruan2025calm.
This paper focuses on randomized experiments, where AI predictions can be used purely for variance reduction. Our central message is that no new methods are needed: researchers can include AI predictions as covariates in standard regression adjustment, analogous to adjusting for the estimated prognostic score hansen2008prognostic,schuler2022increasing. In fact, we can view regression adjustment as the (linearly) calibrated form of several recently proposed bias-corrected estimators vdL2026calibeating.
Relative to the unadjusted estimator, regression adjustment “does no harm,” in the sense that it retains unbiasedness and (weakly) reduces variance lin2013agnostic. When predictions are uninformative, the fitted coefficient shrinks toward zero and the estimator reverts to the difference-in-means; when they are informative, precision improves. Uncalibrated approaches, such as the model-assisted estimator sarndal2003model and variants of PPI angelopoulos2023prediction, lack this guarantee and can inflate variance when prediction quality is modest. Regression adjustment also accommodates baseline covariates alongside AI predictions.
How much AI predictions improve precision depends directly on their quality. Pre-trained foundation models tend to be more predictive than traditional tabular methods when the signal comes from complex or unstructured inputs that are otherwise difficult to featurize. With already-structured tabular data, ridge regression and XGBoost can match or outperform LLMs alone. Even here, adjusting for both AI predictions and tabular features gives the best of both worlds. More broadly, LLMs can also featurize unstructured inputs, yielding additional covariates that further improve precision.
We also discuss practical implementation issues that arise with binary and count outcomes. In these cases, we argue that researchers should adjust for continuous predicted probabilities rather than discretized predictions. We give practical strategies for obtaining such scores, including extracting log-probabilities from the output token distribution and averaging over multiple stochastic draws.
We demonstrate these ideas in synthetic experiments and three empirical applications: a mega-study of survey experiments with digital twins, an email marketing A/B test with tabular covariates, and a large-scale technology platform experiment with sequential user data. Across these settings, incorporating AI predictions via regression adjustment yields real if modest gains beyond traditional covariates. By contrast, less careful uses of AI predictions can lead to bias or inflated standard errors.
\paragraph{Synthetic surveys and digital twins.} A growing literature explores using LLMs to generate “silicon samples” as substitutes for human participants argyle2023out, horton2023large, brand2023using. While promising for rapid prototyping, synthetic samples often exhibit substantial bias cummins2025threat,peng2025digital, hindering their use in real-world deployments. Recognizing these limits, several recent methods instead focus on combining AI predictions with ground-truth human responses. These methods divide into two broad camps.
\paragraph{Hybrid estimation: finite-sample approaches.} Motivated by survey calibration and control variates, several papers combine randomization and human responses to guarantee exact (or near-exact) unbiasedness without relying on model correctness egami2023using. Prediction-powered inference angelopoulos2023prediction has emerged as a particularly popular framework, with many extensions song2026demystifying. As we discuss below, regression adjustment in randomized experiments is a special case lin2013agnostic,deng2023augmentation. vdL2026calibeating explore calibrated forms of PPI and connect that framework to the semiparametric causal inference literature, including documenting the equivalence between linear calibrated PPI++ and linear regression. Separately, bashari2025statistical propose using conformal methods for finite-sample guarantees when combining human and silicon samples.
\paragraph{Hybrid estimation: asymptotic and semiparametric approaches.} Motivated by semiparametric efficiency theory, several papers instead propose AIPW-style estimators that optimally combine AI predictions with real data ruan2025calm, bartolomeis2025efficient, engh2025llm. See also byun2025valid, who propose a flexible strategy based on generalized method of moments. These approaches are more general than the finite-sample approaches above, allowing for observational data and weaker identification assumptions. However, they typically require stronger regularity conditions and more complex estimation procedures. Importantly, in randomized experiments these estimators are asymptotically equivalent to the methods we discuss below.
\paragraph{Propagating uncertainty from AI-generated variables.} A separate line of work in econometrics addresses inference when AI- or ML-generated variables are used as regressors or outcomes in observational analyses, where prediction error can bias estimates and invalidate standard errors battaglia2024inference,ludwig2024large. These concerns do not arise in our setting: under randomization, AI predictions enter only as auxiliary covariates for variance reduction, and the resulting ATE estimators are unbiased (model-assisted) or consistent (regression-adjusted) regardless of prediction quality. The black-box predictor's error is absorbed into the residual variance rather than the point estimate, so explicitly modeling prediction uncertainty is unnecessary for valid inference on the ATE.
\paragraph{Complementary uses of AI in experiments.} There is a growing literature on using AI to improve other aspects of experimental design and analysis. For example, gui2025leveraging introduce generative stratification, using LLMs to synthesize covariate information for stratified randomization. saenger2024autopersuade use LLMs to optimize persuasive interventions. broska2025mixed consider hybrid designs that combine human and synthetic samples.
\paragraph{Empirical studies of variance reduction in randomized experiments.} We add to a substantial literature in the biomedical and social sciences on empirically assessing the plausible variance reduction from covariate adjustment in randomized trials. schochet2010regression reviews eight large-scale social science experiments, finding that covariate adjustment reduces variance by a median of roughly 25%, ranging from under 10% to over 50% depending on the outcome. shao2025benchmarking benchmark 18 covariate-adjustment strategies across 50 biomedical RCTs, finding a median variance reduction of 13% for continuous outcomes and 5% for binary outcomes; notably, ML-based methods do not improve over simple linear regression in their settings.
We consider a standard finite population framework for randomized experiments freedman2008regression, lin2013agnostic with $n$ units indexed by $i = 1, \ldots, n$. We refer to these $n$ units interchangeably as the “finite population” and “sample”. Each unit has potential outcomes $Y_i(1)$ and $Y_i(0)$, representing the outcomes under treatment and control, respectively. In a completely randomized experiment, we randomly assign $n_1$ units to treatment and $n_0 = n - n_1$ units to control, where the treatment assignment is denoted by $Z_i \in \{0, 1\}$; our setup immediately extends to other randomized designs. The observed outcome for unit $i$ is $Y_i^{\text{obs}} = Y_i(0) + Z_i (Y_i(1) - Y_i(0))$. Our estimand of interest is the sample average treatment effect (SATE), \[ \tau \;=\; \bar{Y}(1) - \bar{Y}(0) \;=\; \frac{1}{n} \sum_{i=1}^n Y_i(1) - \frac{1}{n} \sum_{i=1}^n Y_i(0), \] where $\bar{Y}(z) = \frac{1}{n} \sum_{i=1}^n Y_i(z)$ denotes the mean outcome under treatment $z$.
We have access to baseline covariates $X_i \in \mathcal{X}$. Importantly, we also have access to predictions from a black-box model, which we denote as $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z)$ for unit $i$ under treatment $z \in \{0, 1\}$. Under the finite-population framework, both potential outcomes $Y_i(z)$ and predictions $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z)$ are taken as fixed; the only randomness is in the assignment vector $Z$. Such predictions typically condition on the baseline covariates; we suppress this in the notation to emphasize the generative AI use case that typically abstracts away from baseline features. These black-box predictions could come from a variety of sources, including traditional supervised learning models trained on historical data, large language models generating “digital twins” of experimental participants, or AI agents simulating human behavior. We explicitly allow the predictions to depend on possible treatment assignment $z$, capturing potential treatment effect heterogeneity in the black-box model. Importantly, we do not assume that these predictions are unbiased or correctly specified. We simply view them as auxiliary information that we can use to improve efficiency.
\paragraph{Extension to complex and adaptive randomization.} While we focus on completely randomized experiments for clarity, the framework extends to stratified, clustered, and other complex randomization designs, as well as adaptive designs with always-valid confidence sequences. There is a substantial literature on regression adjustment and model-assisted estimation for such designs ding2024first,su2021model,molitor2025anytime that can readily accommodate AI predictions.
\paragraph{Extension to super-population estimands.} Our design-based framework targets the sample average treatment effect (SATE), with randomness coming solely from treatment assignment in the finite population. Researchers are sometimes interested in super-population estimands such as the population average treatment effect (PATE) imbens2015causal, where units are drawn i.i.d.\ from a population and the target is $\ensuremath{\mathbb{E}}[Y(1) - Y(0)]$. The estimators we discuss are consistent for the PATE under standard sampling assumptions, with the usual inflation of standard errors to account for sampling variability of $X$. Recent semiparametric work on hybrid estimation, including ruan2025calm and bartolomeis2025efficient, develops efficient AIPW-style estimators for the PATE and related functionals that are asymptotically equivalent to calibrated regression adjustment in randomized experiments; these approaches extend more naturally to observational data and non-i.i.d.\ sampling at the cost of stronger regularity conditions.
\paragraph{Extension to textual and complex treatments.} This framework also naturally accommodates experiments with textual or high-dimensional treatments, increasingly common in survey research and online platforms saenger2024autopersuade, lin_optimizing_2024. When treatments are text-based---such as different message framings or policy descriptions---LLMs can interpret these directly when generating predictions, extending variance reduction to settings where traditional adjustment would require manual coding. More broadly, LLMs can serve as flexible “codebooks” for high-dimensional data, automatically extracting features or categorizing treatments into meaningful groups egami2023using.
\paragraph{Notation.} We use the following notation throughout: $\bar{Y}(z) = \frac{1}{n} \sum_{i=1}^n Y_i(z)$ denotes the finite population mean outcome under treatment $z$; $\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(z) = \frac{1}{n} \sum_{i=1}^n \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z)$ is the finite population mean prediction under treatment $z$; $\bar{Y}_z^{\text{obs}} = \frac{1}{n_z} \sum_{i: Z_i = z} Y_i^{\text{obs}} $ is the observed mean outcome in treatment group $z$; and $\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_z^{\text{obs}} = \frac{1}{n_z} \sum_{i: Z_i = z} \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z)$ is the sample mean prediction in treatment group $z$.
As a warmup, we first consider estimation of one component of the ATE, the average treated potential outcome $\bar{Y}(1)$. We start with the population mean AI prediction $\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(1)$ as an initial, likely biased estimate of $\bar{Y}(1)$. Following a long literature on survey calibration and control variates sarndal2003model,angelopoulos2023prediction,egami2023using,song2026demystifying, we can augment this biased estimate with a correction term. Under random assignment, the difference between sample predictions and observed outcomes among treated units, $$ \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_1^{\text{obs}} - \bar{Y}_1^{\text{obs}}, $$ is an unbiased estimate of the prediction bias $\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(1) - \bar{Y}(1)$, regardless of the prediction quality of the AI. Subtracting this estimated bias from the AI prediction yields the model-assisted estimator sarndal2003model:
This estimator, equivalent to the unweighted form of prediction-powered inference angelopoulos2023prediction and to design-based supervised learning egami2023using, is unbiased for $\bar{Y}(1)$ regardless of prediction quality. If the model is perfect, the correction vanishes; if the model is poor, the correction pulls the estimate back toward the observed data.
We now apply the same bias-correction strategy to each treatment arm to estimate the full average treatment effect $\tau = \bar{Y}(1) - \bar{Y}(0)$. As before, we begin with a pure “synthetic experiment” estimate using only the AI predictions: $$ \hat{\tau}_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}} = \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(1) - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(0). $$ In general, this estimate will be biased for $\tau$. However, we can again use the difference between sample predictions and observed outcomes within each treatment group to estimate and correct for this bias. Bias-correcting the AI prediction separately for treated and control groups yields the model-assisted estimator ding2024first,poulet2025ppi_for_rct,deng2023augmentation:
The second line highlights that the model-assisted estimator can also be framed in terms of control variates deng2023augmentation,ding2024first: we augment the unbiased difference-in-means estimator with a mean-zero correction term based on the AI predictions. The correction term has mean zero by randomization, preserving unbiasedness; whether augmentation reduces variance depends on prediction quality, with the precise condition stated below.
Finally, this estimator is equivalent to the augmented inverse propensity weighted (AIPW) estimator with known (constant) propensity scores and the AI predictions $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z)$ playing the role of the outcome model ruan2025calm,engh2025llm,bartolomeis2025efficient. Specifically, the AIPW estimator simplifies to:
where $n_1/n$ is the known propensity score under complete randomization. Rearranging terms recovers $\hat{\tau}_{\text{ma}}$.
We now discuss how standard regression adjustment is a calibrated version of the model-assisted estimator. An important limitation of the model-assisted estimator is that it can increase variance when predictions are poor freedman2008regression, ding2024first. In particular, the model-assisted estimator reduces variance in both treatment arms if and only if \[ \frac{\text{Cov}(Y(z), \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z))}{\text{Var}(\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z))} \geq \frac{1}{2}. \] If predictions are standardized to have the same variance as the outcomes, this is equivalent to requiring that the correlation between outcomes and predictions within each treatment arm exceeds $1/2$. When this condition fails, which is common in empirical settings peng2025digital, the bias correction actually inflates variance.
Calibration resolves this variance inflation problem. Rather than fully correcting the bias, we estimate the optimal scaling via calibration coefficients \[ \hat{\beta}_z = \frac{\widehat{\text{Cov}}_z(Y, \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z))}{\widehat{\text{Var}}_z(\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z))}, \] the within-arm OLS slope of $Y$ on $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z)$ among units assigned to arm $z$, computed separately within each treatment group. The calibrated estimator is then: \[ \hat{\tau}_{\text{cal}} = \bar{Y}_1^{\text{obs}} - \bar{Y}_0^{\text{obs}} + \hat{\beta}_1(\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(1) - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_1^{\text{obs}}) - \hat{\beta}_0(\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(0) - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_0^{\text{obs}}), \] which we can view as a single control variate adjustment with a scaling factor based on the estimated coefficients $\hat\beta_0$ and $\hat\beta_1$ li2020rerandomization.
A remarkable fact due to lin2013agnostic is that we can directly estimate $\hat{\tau}_{\text{cal}}$ via the coefficient on the treatment indicator $Z$ in the following interacted regression \[ Y \sim 1 + Z + \dot{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}(Z) + Z \times \dot{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}(Z), \] where $\dot{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}(z) = \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z) - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_z^{\text{obs}}$ are group-mean-centered predictions.\footnote{As freedman2008regression emphasized, the OLS-based estimator carries a small finite-sample bias of order $O(1/n)$. lin2013agnostic showed that the interacted specification renders this bias asymptotically negligible while guaranteeing (weak) efficiency gains over the unadjusted estimator under arbitrary model misspecification. In what follows we treat $\hat{\tau}_{\text{cal}}$ as approximately unbiased. Readers desiring exact unbiasedness can use the closed-form bias-correction of chang2024exact, which removes the $O(1/n)$ term while preserving the limiting distribution.} In practice, including the interaction term often leads to negligible change, and researchers can simply fit the regression: $Y \sim 1 + Z + \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(Z)$. This form is known as the ANCOVA adjustment for the estimated prognostic score in the causal inference literature; see, for example, hansen2008prognostic, schuler2022increasing, guo2022machinelearningvariancereduction. The linear regression estimator is also equivalent to a version of CUPED deng2023augmentation, and to the calibrated version of CALM ruan2025calm. See vdL2026calibeating for further equivalences among calibrated estimators.
Given the regression estimator, we can construct valid confidence intervals using standard heteroskedasticity-robust standard errors lin2013agnostic,ding2024first. We can quantify the variance improvement via the design effect in treatment arm $z$, the ratio of the calibrated estimator's variance to the unadjusted variance: $$ \text{Deff}_z = \frac{\text{Var}\left(\bar{Y}_z^{\text{obs}} + \hat{\beta}_z(\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(z) - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_z^{\text{obs}})\right)}{\text{Var}\left(\bar{Y}_z^{\text{obs}}\right)} = 1 - \rho(z,z)^2 \leq 1, $$ where $\rho(z,z) \equiv \text{corr}(Y_i(z), \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z))$. Since $\rho(z,z)^2 \geq 0$, calibration never increases variance relative to the unadjusted estimator.
\paragraph{Alternative regression specifications.} There are many potential regression specifications for adjusting for AI predictions. Following cohen2024noharm, we can adjust for the black-box predictions of both potential outcomes, $(\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0), \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1))$: $Y \sim 1 + Z + \dot\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}} + Z \cdot \dot\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}} $, where $\dot{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}} = \left( (\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0) - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_0^{\text{obs}}), (\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1) - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_1^{\text{obs}}) \right)$. This has a design effect of $\text{Deff}_z = 1 - \rho_{\text{mr}}(z)^2$, where $\rho_{\text{mr}}(z)$ denotes the multiple correlation between $Y(z)$ and both predictions $(\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0), \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1))$.\footnote{Alternatively, we can adjust for only a single black-box prediction, either $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0)$ or $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1)$. Using $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z')$ for treatment arm $z$ has a design effect of $\text{Deff}_z = 1 - \rho(z,z')^2$.} Many other variants are common in the literature on heterogeneous treatment effect estimation; see wager2024causal for a review.
\paragraph{Digital twin estimator.} We briefly discuss an alternative variance reduction approach based on directly predicting the missing potential outcome with “digital twins” argyle2023out, horton2023large, schuler2022increasing,alam2024digitaltwin. This is known as the imputation or matching estimator ding2024first,lin2023nn_ecma. First, construct unit-level treatment effect estimates by imputing the missing potential outcome using the AI prediction: \[ \hat{\tau}_i =
\] Averaging over all units yields the digital twin estimator: \[ \hat{\tau}_{\text{dt}} = \frac{1}{n}\sum_{i=1}^n \hat{\tau}_i = \frac{n_1}{n}\left(\bar{Y}_1^{\text{obs}} - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_1(0)\right) - \frac{n_0}{n}\left(\bar{Y}_0^{\text{obs}} - \mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_0(1)\right), \] where $\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu_z(z') = \frac{1}{n_z} \sum_{i: Z_i = z} \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z')$ is the sample mean prediction for units in group $z$ under treatment $z'$.
This estimator is biased unless the predictions are correct on average: $\mkern 4mu\overline{\mkern-2mu\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}\mkern-2mu}\mkern 4mu(z) = \bar{Y}(z)$ for $z \in \{0,1\}$. This is a strong assumption that LLMs and other AI models rarely satisfy; indeed, if it were true, there would be no need for non-synthetic data. This makes the digital twin estimator risky in practice despite its intuitive appeal.
\paragraph{Nonlinear calibration.} For binary or count outcomes, continuous AI predictions $(\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0), \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1))$ can be calibrated nonlinearly, for example, by including them in a logistic or Poisson regression, or fitting a generalized additive model. The generalized Oaxaca-Blinder framework basse2024oaxaca, with an extension from cohen2024noharm, shows that such nonlinear calibration inherits the same robustness properties: the resulting estimator is unbiased regardless of calibration model quality and reverts to the unadjusted estimator when predictions are uninformative.
Table (ref) summarizes the bias and design effect for each estimator discussed above.
\paragraph{Calibration guarantees robustness.} The calibrated estimators always achieve $\text{Deff}_z \leq 1$; that is, incorporating AI predictions does not increase variance. When predictions are uninformative ($\rho = 0$), the design effect equals 1 and we simply recover the unadjusted estimator's precision. When predictions are highly correlated with outcomes, variance drops substantially.
\paragraph{Uncalibrated approaches are fragile.} Uncalibrated estimators only achieve efficiency gains with very strong predictions. For instance, the model-assisted estimator improves over the baseline within arm $z$ only when $\rho(z,z) > 0.5$, and otherwise inflates that arm's variance. The digital twin estimator can reduce variance with weaker correlations, but introduces bias that may be arbitrarily large.
\paragraph{Both predictions maximize gains.} When the model generates predictions for both potential outcomes $(\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0), \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1))$, adjusting for both will typically achieve greater variance reduction than adjusting for the observed prediction alone, $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(Z_i)$. The additional gain beyond the factual prediction depends on whether the counterfactual prediction $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1-z)$ provides information about $Y(z)$ not already captured by $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(z)$.
We briefly discuss how to generate the AI predictions, $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z)$.
\paragraph{Black-box models.} We consider three prediction approaches:
\paragraph{Generating predictions.} To maximize variance reduction, we query the model separately under each treatment condition to obtain both $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(0)$ and $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(1)$. Prompt design can substantially affect prediction quality; pilot data can guide prompt engineering by optimizing for the within-arm correlation $\rho(z,z)$, though analysts should be cautious about overfitting the prompt to a small pilot sample.
\paragraph{Cross-fitting.} Zero-shot LLM predictions are constructed without access to the realized $Y^{\text{obs}}$ or $Z^{\text{obs}}$. We can therefore view $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i$ as if it were a fixed pre-treatment covariate, no different from age or baseline survey responses. Thus standard regression adjustment applies with no cross-fitting or sample-splitting required. By contrast, models trained on experimental outcome data (TabPFN, XGBoost, Ridge, fine-tuned LLMs) require cross-fitting to preserve design-based validity guo2022machinelearningvariancereduction; in our experiments, we use 5-fold cross-fitting for all ML baselines. Nonlinear calibration models fit on top of predictions may also require cross-fitting when the calibration function is estimated from the data.
\paragraph{Combining prediction sources.} LLMs can also featurize unstructured data, extracting embeddings or categorizing free-text responses, ${\bm{X}}_\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}$ engh2025llm, egami2023using. These can then be fed into downstream models such as TabPFN or used alongside the AI predictions as additional baseline covariates: \[ Y \sim 1 + Z + \dot{\bm{X}}_\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}} + Z \cdot \dot{\bm{X}}_\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}} + \dot{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}(Z) + Z \cdot \dot{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}(Z), \] where $\dot{\bm{X}}_\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}$ denotes group-mean-centered AI features. The resulting estimator is approximately unbiased and achieves a design effect of $1 - \rho_{\text{mr}}(z)^2$, where $\rho_{\text{mr}}(z)$ is now the multiple correlation between $Y(z)$ and the full covariate and prediction vector. Including both can only improve (or maintain) precision relative to using either alone, except when features are very high-dimensional lei2021regression.
\paragraph{Binary and discrete outcomes.} For binary outcomes, continuous AI predictions can yield greater variance reduction than binarized predictions (Section (ref)). Directly eliciting a probability $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}} \in [0,1]$ from the LLM is one option, though such probabilities can be poorly calibrated. We therefore favor two alternatives that yield a well-scaled continuous score: extracting the log-probability of the positive-class token (e.g., “Yes”), which needs no extra queries, or averaging multiple completions sampled at positive temperature ($10$--$20$ draws, batchable in a single call). See Appendix (ref) for comparisons.
We run a series of controlled synthetic experiments to assess the plausible range of precision gains from AI assistance and how this varies with both the quality of the AI and the difficulty of the prediction task.
In each set of experiments, we generate a finite population of $n = 1,000$ units with covariates $X_i \sim \text{Uniform}(0, 1)$ for $i = 1, \ldots, n$. We consider a completely randomized design with $n_1 = n_0 = 500$ units assigned to treatment and control, respectively. In the first set of experiments, we generate potential outcomes under control as $Y_i(0) = m_0(X_i) + \varepsilon_i$ where $\varepsilon_i \overset{iid}{\sim} \mathcal{N}(0, \sigma^2_y)$ and $m_0(x) = \sin(3 \cdot \pi \cdot x) / (3 \cdot \pi \cdot x)$ is a non-linear but smooth function. We then generate potential outcomes under treatment as $Y_i(1) = Y_i(0) + \tau + X_i$, where $\tau$ is chosen so that the average treatment effect is 0. Note that the potential outcomes are perfectly correlated conditional on the covariate $X$, due to the shared noise term $\varepsilon_i$. The variance of this noise term, $\sigma^2_y$, controls the inherent difficulty of the prediction task: lower values correspond to easier prediction problems. We use $R^2_m$ to denote the proportion of variance in $Y(0)$ explained by the true underlying model $m_0(X)$, i.e., $R^2_m = \text{Var}(m_0(X)) / (\text{Var}(m_0(X)) + \sigma^2_y)$. Higher values of $R^2_m$ correspond to tasks where the outcome is more predictable from the covariate.
To simulate AI predictions, we generate predicted potential outcomes as $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(0) = m_0(X_i) + b + \eta_i$ and $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(1) = \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(0) + \tau + X_i$, where $\eta_i \overset{iid}{\sim} \mathcal{N}(0, \sigma^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}})$ is independent noise and $b$ is a bias term that reflects systematic over- or under-prediction. We use $R^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}$ to denote the signal-to-total-variance ratio of the AI prediction, $R^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}} = \text{Var}(m_0(X)) / (\text{Var}(m_0(X)) + \sigma^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}})$; higher values correspond to more accurate AI predictions. The squared correlation between $Y(0)$ and $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0)$, which directly governs variance reduction, is $\text{corr}(Y(0), \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0))^2 = R^2_m \cdot R^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}$. Because the AI predictions share the same functional form as the true outcome model, idiosyncratic errors in the AI predictions come from the noise term $\eta_i$, and so $\sigma^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}$ controls the quality of the AI predictions.
For the second and third sets of experiments, we consider the case where outcomes are binarized so that $\tilde{Y}_i(z) = \mathbbm{1}\{Y_i(z) \geq 0\}$. In the second set of experiments, the AI predictions are also binarized, so that $\tilde{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}_i(z) = \mathbbm{1}\{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z) \geq 0\}$. In the third set of experiments, the AI predictions remain continuous.
Figure (ref) shows the design effect, $\frac{\text{Var}(\hat{\tau}_{\text{adj}})}{\text{Var}(\hat{\tau}_{\text{unadj}})}$, from AI-assisted variance reduction across a range of adjustment techniques, prediction quality ($R^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}$) and outcome predictability ($R^2_m$). (Here and below we report the overall design effect rather than separate per-arm design effects.) For continuous outcomes (a), we see that when the underlying outcomes are difficult to predict (low $R^2_m$), even AI predictions that are as predictive as possible (i.e., $R^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}$ close to $R^2_m$) yield only modest efficiency gains, while larger gains are possible when both the outcome is very predictable (high $R^2_m$) and the AI predictions are of high quality ($R^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}$ close to $R^2_m$). Consistent with the theoretical results in Table (ref), the uncalibrated model-assisted estimator inflates variance at low $R^2_{\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}}$, while linear regression adjustment maintains $\text{Deff} \leq 1$ throughout.
For case (b), with both binary outcomes and AI predictions, we see that the potential efficiency gains are even more muted. This is because even if the underlying continuous outcomes are highly predictable, the binary outcome may still have high entropy when the continuous outcome is near the binarization threshold. Binary AI predictions exacerbate this issue. However, when the AI predictions remain continuous (c), we see that efficiency gains are more similar to the continuous outcome case, although attenuated somewhat due to higher variability of the outcomes.
Appendix (ref) reports complementary simulations varying the AI-prediction bias $b$. Varying $b$ symmetrically across arms, the uncalibrated model-assisted estimator's variance inflates rapidly with $|b|$, while calibrated regression adjustment is unaffected (Figure (ref)). Appendix (ref) further compares against an AIPW estimator with estimated propensity score under both symmetric and asymmetric bias (where only the estimated control outcome is shifted): calibrated regression is the only estimator that remains simultaneously unbiased, well-calibrated, and powerful across regimes. By contrast, AIPW is point-unbiased but substantially overcovers; the digital twin fails under asymmetric bias.
Our first empirical comparison uses Twin-2K-500 Mega-Study peng2025digital, which constructed digital twins of 2,000 participants by having GPT-4 simulate responses based on detailed persona profiles derived from 500 prior survey questions.
We analyze two randomized experiments that test opt-in vs.\ opt-out framing, Organ Donation and Green Energy ($n{=}600$ per experiment, approximately balanced across arms). For each participant we observe the actual outcome under the realized treatment and generate LLM predictions under both treatment conditions.
We generate predictions using GPT-5.4 following the Twin-2K-500 protocol, providing each participant's full persona profile (${\sim}$127K characters) without truncation. We extract continuous predictions from token logprobs in a single API call per persona-condition pair, obtaining $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(z) \in [0,1]$ as the model's estimated probability of the target behavior (Appendix (ref)).
For ML baselines, we engineer 26 composite psychological scale scores from the raw survey items, following peng2025digital, and use GPT-5.4 to select 10 theoretically relevant features per outcome. Importantly, we do so without using any outcome data, avoiding data leakage. All ML models (Ridge, XGBoost, TabPFN hollmann2022tabpfn) use T-Learner with leave-one-out cross-validation. Appendix (ref) gives full details on feature engineering, model specifications, and cross-fitting.
Table (ref) presents the main results, which align with our discussion above. The unbiased difference-in-means estimates are $\hat{\tau} \approx 0.04$ and $\hat{\tau} \approx 0.25$ for Organ Donation and Green Energy, respectively. The digital twin estimators show substantial bias, with $\hat{\tau} = -0.28$ for Organ Donation and $\hat{\tau} = -0.20$ for Green Energy. The uncalibrated model-assisted estimator increases variance by 36% for Organ Donation ($\rho \approx 0.13$) and 8% for Green Energy ($\rho \approx 0.37$), consistent with our result that this estimator requires $\rho > 0.5$ to reduce variance. By contrast, the standard regression estimator achieves 13.7% variance reduction for Green Energy and 2.1% for Organ Donation. Conditioning on both $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0)$ and $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(1)$ slightly improves variance for both, achieving 14.3% vs.\ 13.7% for Green Energy and 2.7% vs.\ 2.1% for Organ Donation. Additionally adjusting for baseline covariates (27 composite scales from the pre-treatment survey wave) further improves variance, with a 16.4% reduction for Green Energy and 4.5% for Organ Donation.
Table (ref) compares LLM predictions against a ridge regression with 10 LLM-selected features. For the Organ Donation study, LLM predictions are between $\rho = 0.11$ and $\rho = 0.13$, depending on whether predictions are based on the extracted log-probabilities or on averaging over many binary predictions. Ridge regression with the LLM-extracted features outperforms these, with $\rho = 0.20$. For the Green Energy study, where the GREEN attitude scales directly measure the construct of interest, all methods perform comparably, with $\rho$ between 0.36 and 0.38. Combining the LLM logprob predictions with the LLM-selected features in a single ridge regression yields a small further gain on Green Energy ($\rho = 0.40$, +16% VR) and no detectable gain on Organ Donation ($\rho = 0.20$, +4% VR), confirming the substantial overlap between the two signals. Thus, using LLMs as a pre-processing step to extract relevant features can be a competitive alternative to using the AI predictions directly.
We next evaluate variance reduction in a traditional marketing setting with tabular covariates, using the Hillstrom MineThatData E-mail Analytics dataset. The dataset contains 64,000 customers randomly assigned to one of three treatments (1/3 each), referred to as: Men's email, Women's email, or Control (no email). We analyze Men's Email vs.\ Control ($n \approx 42{,}613$) for the Visit outcome. Each customer has 8 pre-treatment covariates capturing purchase history, demographics, and channel preferences (Appendix (ref)). For LLM predictions, we convert tabular features to structured text and use GPT-5.4 with direct probability elicitation, prompting for a numeric 0--100 probability estimate rather than binary classification. We found this produces better-calibrated predictions than binary or logprobs modes, which were overconfident and yielded near-zero variance reduction (see Appendix (ref) for prompt design).
Table (ref) presents the results. Ridge and XGBoost on the original features both yield reasonable predictions: with $\rho = 0.170$ (2.9% VR) for ridge and $\rho = 0.148$ (2.2% VR) for XGBoost. By contrast, the direct probability predictions from an LLM (GPT-5.4) achieve $\rho = 0.105$ (1.1% VR). Unlike with the previous Green Energy and Organ Donation studies, here eight tabular features directly measure purchase behavior, the primary driver of future visits, and ML methods outperform the LLM by more than a factor of two.
Finally, we evaluate a large-scale randomized experiment conducted on a consumer-facing online platform. The experiment tested a new search ranking algorithm against an incumbent ranker, with users randomly assigned to see search results generated by one of the two. In this setting, key business outcomes, such as successful downstream conversions, are sparse and highly variable at the user level, and treatment effects are typically small. Achieving adequate statistical power often requires large sample sizes and long experiment runtimes, making this a natural testbed for variance reduction techniques.
\paragraph{Data and experimental design.} Users who searched on the platform during the experiment were randomly assigned at the user level, with equal probability, to treatment or control. Treatment users were shown search results based on the new ranking algorithm, while control users continued to see search results from the incumbent ranker. Assignment was persistent for the duration of the experiment, so each user consistently experienced the same ranker across all search sessions.
The experiment ran for one month and included 13.2 million users, who entered on a rolling basis. From this population, we construct a focused analysis cohort by randomly sampling $n_{1} = 2500$ treated users and $n_{0} = 2500$ control users who entered the experiment on its first day. We analyze this fixed subsample to facilitate a direct comparison of estimator performance under limited sample sizes. Restricting to users with a common entry time also simplifies the estimand by ensuring that all users are exposed to the assigned treatment for the full experiment duration, rather than a mixture of exposure lengths.
For each user, we observe a pre-treatment outcome $Y_{i, \text{pre}}$, a rich set of pre-treatment user activity $\bm{X}_i$, and an LLM-based prediction of conversion likelihood $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(0) \in [0,1]$ constructed from pre-treatment user activity.
The outcome of interest is the number of successful downstream conversions per user. This metric is highly zero-inflated, with the large majority ($93\%$) of users having no more than one conversion during the experiment; most users ($70\%$) have zero conversions. We include implementation and evaluation details in Appendix (ref), including the construction of LLM predictions of user conversion from raw activity data.
\paragraph{Results.} Table (ref) shows the results for the unadjusted estimator and various regression estimators. (Due to data limitations, we are unable to report other estimators for this experiment.) The Unadjusted estimator finds a positive but non-significant effect of the new search ranking algorithm on downstream conversions ($\hat{\tau} = 0.0380$, SE $= 0.0250$, pval $= 0.13$). Regression with $Y_{\text{pre}}$ is equivalent to the standard CUPED estimator deng2013improving, deng2023augmentation used in large-scale industry A/B testing platforms, where the pre-treatment value of the outcome metric (e.g., a user's prior conversion count) is used for covariate adjustment. This industry-standard estimator provides modest efficiency gains, with a $15.5\%$ reduction in variance. By contrast, regression with the pre-treatment LLM prediction alone, $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}(0)$, yields only a $1.3\%$ variance reduction, and adding it on top of $Y_{\text{pre}}$ contributes little beyond CUPED ($16.4\%$). In this setting, the pre-treatment LLM prediction provides limited additional signal over the pre-treatment outcome itself, and the estimated effect remains inconclusive after adjustment.
We argue that linear regression adjustment is a simple and effective way to incorporate AI predictions into randomized experiments, building on a long causal inference literature on covariate and prognostic score adjustment. We can also motivate regression adjustment as a calibrated version of recently popular bias-corrected estimators, such as prediction-powered inference. Regression has an appealing “do no harm” property: it remains unbiased regardless of prediction quality and (weakly) reduces variance relative to the unadjusted estimator. By contrast, uncalibrated approaches such as model-assisted and digital twin estimators can inflate variance when predictions are poor, and digital twin estimators can also be arbitrarily biased. In applications with substantial text and other unstructured data, LLM predictions can outperform standard tabular models; in other settings, the gains are often modest and context-dependent. In our technology platform application, for example, LLM predictions based on pre-treatment user activity added little beyond the pre-treatment outcome itself. Regardless of prediction quality, however, researchers can always incorporate this information into existing covariate adjustment frameworks alongside other baseline covariates.
While regression calibration protects against variance inflation and model misspecification, it does not by itself address other important guardrails around AI deployment. The most prominent is to avoid pipeline failures from data leakage, which invalidate the statistical guarantees of the design-based framework and can lead to biased estimates. In particular, researchers should ensure that AI predictions are generated from pre-treatment information only and have no access to the realized treatment assignments or outcomes. A related danger is the use of AI tools to enable $p$-hacking by repeatedly tuning prompts or other LLM choices; here too, researchers should pre-specify any adjustment procedure to maintain validity.
We suggest several directions for future work:
\paragraph{Subgroup analysis with predicted treatment effects.} AI predictions under both treatment conditions yield unit-level predicted treatment effects $\text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(1) - \text{\textcolor{gray}{\scalebox{0.9}{\faRobot}}}_i(0)$, which can define subgroups for heterogeneous effect analysis. LLMs can also generate otherwise latent covariates, such as imputed party affiliation, user intent, or personality traits, that serve as subgroup moderators, analogous to using LLMs for survey imputation egami2023using. While promising, a key danger is that such LLM-generated moderators might be biased, which would complicate estimates of treatment effect heterogeneity.
\paragraph{Surrogate outcomes.} When treatment effects on the primary outcome are slow to materialize, AI predictions may serve as surrogate outcomes that enable earlier decisions. Conversion likelihood scores, such as those in our technology platform experiment (Section (ref)), are a candidate when they can be computed early and sequentially, and reliably track downstream behavior. A promising direction is to formalize this connection to the surrogacy literature and to characterize when such scores add signal beyond pre-treatment outcomes.
\paragraph{Randomization-based testing with statistics based on AI predictions.} Our estimators target point estimation and Wald-type intervals, but the same design-based logic extends to Fisher-style randomization tests. For instance, guo2025ml construct ML-assisted randomization tests that compare cross-validation errors of models with and without the treatment indicator, retaining finite-sample validity under the sharp null while gaining power from flexible learners. A natural complement to the regression adjustment approach we consider is to use AI predictions to construct similarly informed test statistics or to define more powerful covariate-adjusted randomization tests.
\paragraph{Mixed subject designs.} A recent line of work, including the mixed subjects design of broska2025mixed, instead treats AI predictions as “additional subjects” alongside human participants. The implied estimand then averages over a mixed population of human and synthetic units rather than the human-only SATE. A natural way forward is to treat this as a problem of generalizability: estimate the human-only effect as discussed here, then formally transport it to the mixed target population, drawing on the literature on transporting treatment effects across populations degtiar2023review. This framing keeps the target estimand and its assumptions explicit.