Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
89,420 characters · 20 sections · 65 citation commands
Causal Inference on Outcomes Learned from Text
We run an increasing number of expensive randomized controlled trials (RCTs) to study the causal effect of interventions, yet their analysis remains limited by which outcomes are measured. While the interventions we study may affect study subjects in many ways, our inference may miss important effects that go beyond the narrow aperture of the outcome variables we have collected or constructed. For example, a new drug may affect patients' well-being not only in terms of the few health measurements we focus our analysis on and a micro-credit program may affect the financial lives of the poor beyond the investment and consumption data we collect. Instead, such broader outcomes can be captured by unstructured responses of study subjects, especially in the form of natural language. These responses may encode subtle, context-specific nuances, including emotional tone and emphasis of themes or concepts not anticipated ex ante. Yet analyzing effects based on such unstructured data requires new tools that combine the complexity of language with the discipline of causal inference.
In this article, we provide a framework for leveraging AI tools to provide valid inference on differences across two groups of text documents, such as the treatment and control groups in a randomized trial. We first ask whether the two groups are systematically different, which we tackle by combining machine-learning predictors and sample splitting. We then conceptualize the task of describing what the systematic differences across texts are as a problem of finding and validating simple descriptions of texts in small datasets, which we implement using large language models (LLMs) followed by human validation. Finally, we quantify how complete our resulting descriptions are by comparing them against a model-free benchmark.
Conventional econometric tools are limited in making inference on text. A first approach would be to extract a small number pre-specified features. However, such an approach would have to rely on strong priors about which part of the text is affected by a treatment or demonstrates differences across groups, and may miss signal outside of a narrow set of such measures. A second approach could instead use high-dimensional features such as word counts. In this latter case, we may be worried about spurious results when we search through many potential ways that only a few documents may differ, necessitating heavy multiple-testing penalties that may again result in limited power. Furthermore, despite their high dimensionality, features such as word counts may lose crucial contextual information.
Instead of relying on pre-specified text features or simple word counts, we deploy state-of-the-art techniques from artificial intelligence and machine learning in order to extract signals from text, including large language models (LLMs). These approaches can contextualize words in their context and effectively extract signals even when the number of documents we analyze is small, thanks to pre-training on large corpora of texts. At the same time, these tools come with a separate challenge: How can we interpret and ultimately provide statistical guarantees for output extracted from extremely complex black-box machine-learning systems that were furthermore trained on text we do not have access to and refined using human input that we do not observe?
To turn intransparent AI systems into a valid econometric tool, we first translate inference tasks into machine-learning questions and then validate their answers using sample splitting. We start with asking whetehr there is a systematic difference across different groups of text documents. We reframe this task into a question that is more amenable to the use of machine learning: Given a document, can we predict which group it is from better than some trivial baseline? For example, for a text document in a randomized trial, can we classify whether it is from the treatment or control group? If we can do so on a hold-out dataset that was not used to train our machine-learning algorithm, then we can provide evidence for a causal effect through a valid permutation test, without relying on assumptions about the underlying algorithm. This allows us to obtain valid inference even when a large language model provides predictions.
Having established that two groups of text documents vary significantly, we next ask what the main differences are. We turn the question of describing differences into the task of generating hypotheses about which themes are different across groups. These themes optimally represent a low-dimensional summary of outcomes captured by text, with the goal of providing a meaningful and quantifiable low-dimensional description of differences. These themes can include what texts talk about (topics) as well as how texts represent topics (sentiment). To obtain such features, we provide instructions to a large language model (LLM) to identify themes that describe differences across groups. For each of the themes, we allow the algorithm to propose a flexible scale for evaluating individual documents, such as the degree to which a topic appears or the tone as a scale that ranges from negative to positive sentiment. These themes are learned based on training data that includes group assignments, such as which documents come from the treatment group of a randomized trial, to ensure that themes are informative about differences across groups.
To validate our description of differences between text documents, we combine sample splitting and human validation. If we run a complex large language model on a small experimental dataset, we may be worried that it returns theme classifications that do not represent systematic differences and instead are the result of overfitting to the existing data. Beyond this statistical concern, AI systems like the LLM we deploy do not come with guarantees that the machine-provided summaries are accurate and substantively refer to the described concepts. We, therefore, provide a sample-splitting scheme that separates (machine) hypothesis generation on the training from (human) hypothesis validation on held-out data. On the training data, our algorithm learns systematic differences across treatment and control groups and suggests ways of scoring themes. On the test data, an expert scores held-out texts according to the proposed themes before group labels became available. As a result, we obtain valid inference on the proposed themes without having to make the assumption that the proposed themes are correct or that an AI would be able to score them correctly, neither of which we think can be guaranteed.
Having obtained and validated descriptions of differences across groups of text documents, we finally ask how complete these descriptions are. To do so, we return to our reverse-prediction approach and compare how well the simple theme scores alone predict group labels relative to an unconstrained benchmark. If our measure of completeness is close to zero, then the themes do not contain a lot of information about systematic differences between text documents, and further analysis may be needed to improve descriptions. If, on the other hand, completeness is close to one, then the simple themes express most of the differences that we can find from the texts in the training sample.
As a proof of concept, we apply our method to abstracts of academic manuscripts on arXiv. Starting with a classification into two groups, we analyze their systematic differences using Google's Gemini large language model. We first demonstrate that these groups are significantly different by showing that the LLM can determine which document comes from which group better than a trivial benchmark. We then prompt Gemini to describe differences across both in terms of a few core themes. We then have a member of the research team label the held-out documents according to these themes and validate differences on the hold-out set. Finally, we benchmark how well these themes distinguish between groups based on logistic regression and interpret our results as saying that the intuitive descriptions capture almost all of the differences in our example. While this proof of concept does not showcase an application to a randomized trial directly, it nevertheless provides suggestive evidence that the method can uncover the kind of systematic differences that allow for causal inferences based on text outcomes.
The approach outlined in this work comes with questions about the replicability of AI-based statistical inference as well as the cost of human validation. We propose a pre-specification strategy for empirical analysis with LLMs based on successive data access, where it is not the algorithm itself but instead its intermediate output that is publicly archived before additional data is accessed and the final validation is performed. Furthermore, we show how recent proposals for combining human and machine labels can be implemented within our workflow, and quantify trade-offs between validation cost and statistical precision in our empirical illustration.
Our work builds upon approaches across the social sciences, computer science, and econometrics on using machine learning to make inferences on text specifically and on high-dimensional outcomes more broadly. Our approach to detecting effects on text via reverse prediction is building on Ludwig2017-xk, while the sample-splitting scheme is similar to egami2022make. The text-scoring approach goes back to pmlr-v162-zhong22a and is most closely related to the D5 approach from zhong2023goaldrivendiscoverydistributional, while our approach to integrating and validating LLM outputs builds upon llmapplied and has parallels to the work on images in ludwig2024machine. The completeness measure is informed by fudenberg2022measuring, and our approach to combining machine and human labels follows robins1994estimation and angelopoulos2023prediction. We review some related literature in more detail in (ref) before laying out our setup in (ref). We then discuss testing for effects in (ref), describing differences in (ref), and estimating the completeness of descriptions in (ref). (ref) provides our empirical illustration using academic abstracts, (ref) discusses replicability and pre-specification, and (ref) concludes.
Our project builds upon advances in natural language processing and large language models in computer science, textual analysis in the social sciences, and work in economics, econometrics, and statistics on integrating causal inference and machine learning into causal inference.
\paragraph{Natural language processing and large language models in computer science.}
Natural language processing (NLP) methods have rapidly evolved, enabling automated analysis of textual data. As a foundational model, Latent Dirichlet Allocation blei2003latent is a generative probabilistic model of text as a mixture of a small number of topics, making it possible to uncover hidden thematic structures within large text corpora. The Structural Topic Model robertssocial extends topic models by incorporating document-level metadata. Such topic-modeling methods complement approaches that summarize and differentiate text by classifying documents according to pre-defined categories fuchunbayes. The emergence of large language models (LLMs) has opened new opportunities for identifying patterns within large texts based on limited task-specific training data. Inspired by brownfewshot, which demonstrates that LLMs can learn from examples included in their input (a concept known as in-context learning), lin2023unlockingspellbasellms shows that just a few carefully chosen examples in the input prompt can guide pre-trained base models to follow instructions and interact with users comparably to fine-tuned models.
Closely related to the techniques we apply to randomized trials, pmlr-v162-zhong22a proposes an LLM-based hypothesis generation procedure in which an LLM is queried to learn descriptions that are true more often for one group of text, relative to another. These hypotheses are then validated via another question--answer AI agent. Further work by zhong2023goaldrivendiscoverydistributional introduces the D5 system, designed to automatically discover differences between extensive text collections based on user-specified goals. This system combines language models to generate hypotheses about differences and validate these hypotheses, leveraging a diagnostic benchmark and a meta-dataset for evaluation. Our approach is closely related, particularly since we also use LLMs to uncover systematic differences between groups of documents. It builds on these approaches by extending the learned descriptions to “causal themes", implementing verification via ground-truth labeling, and explicitly focussing on causal inference from experimental data. Related to this line of work, zhu2022gsclipframeworkexplaining introduces a framework for explaining changes between images using natural language, followed by an encoding step to quantify differences across groups.
\paragraph{Text analysis in the social sciences.}
The bulk of textual analysis in the social sciences relies upon more traditional NLP methods, largely due to their superior interpretability relative to more modern transformer-based deep learning architectures. gentzkow_congressional_2018 compiles a corpus of US congressional floor speeches and goes on to parse and analyze the text of these speeches to measure trends in partisanship over time. voelkel_changing_2022 measures the impact of different textual interventions on attitudes towards immigration.
There has also been a burgeoning literature on textual data methods for causal inference. keith_text_2020 outlines a method for using textual data to remove confounding in causal inference settings. mozer_matching_2020 considers textual data as a covariate in matching estimation of treatment effects. fong_discovery_2016 puts forth a generative model of text used to discover latent topics that are particularly relevant for treatment effect estimation. egami2022make provides a detailed overview of current methods for the use of text in causal inference, including using text as treatment, and outlines both the difficulties inherent in using latent representations of text, such as topic model proportions, when conducting causal inference. That work builds on fong_causal_2023 in constructing relevant assumptions to obtain causal estimates from latent features derived from textual data. feder2022causal surveys work on bridging causal inference and natural language processing from the perspective of the NLP literature.
Finally, recent literature has also sought to leverage LLMs for descriptive analysis in textual settings in the social sciences. kim_ai-augmented_2024 utilizes LLMs in survey data to impute user responses to unasked questions. durvasula2024counting fine-tunes an LLM to provide high-quality annotations of unstructured text. Related to our approach, wu_elicitationgpt_2024 uses LLMs to produce summary points of text corpora, which are then used to produce measures of text similarity. egami2024using provides a framework for working with LLM-annotated texts. imai2024_llmtreat extends the use of LLMs to text-as-treatment settings in causal inference, leveraging generative AI to produce sample texts that modify selected features of interest in order to assess causal effects.
\paragraph{Integration of machine learning and AI into causal inference and economics.}
Recent work in economics, econometrics, and statistics has proposed innovative ways of integrating AI and machine learning into applied data anlysis mullainathan2017machine,athey2018impact,athey2019machine,dell2024deep. This includes work that uses supervised machine learning to identify causal effects on high-dimensional outcomes Ludwig2017-xk, test for balance between covariates gagnon2019classification, and impute missing observations rambachan2024programevaluationremotelysensed,battaglia2024inference, angelopoulos2023prediction,egami2024using. We build upon this work by applying a similar approach to text outcomes, where we use LLMs to predict missing labels. Furthermore, we adapt the approach of fudenberg2022measuring that evaluates the completeness of economic theories to quantify how well our themes describe differences between treatment and control outcomes.
We specifically build upon recent approaches that learn from image and text data in economics. ludwig2024machine builds a deep-learning tool for creating new hypotheses from image data and communicating them to researchers. horton2023large,manning2024automated leverage LLMs to generate novel hypotheses and accelerate the research process. Similarly, we propose an LLM-based tool that forms hypotheses based on text (and statistically validates them on held-out documents). vafa2022career and du2024labor demonstrate the value of purpose-built LLM tools for the analysis of labor-market data. Similarly, we aim to build a tool that is specifically aligned with solving a causal-inference task. Finally, we apply the econometric framework from llmapplied to capture the use and limitations of LLMs across the causal-inference tasks in our application.
We consider causal estimation based on text in a potential-outcomes framework with binary treatment $W \in \{1,0\}$. For each observation, we assume that the outcomes of interest are represented within a text document $Y \in \mathcal{Y}$, where documents may vary in length and we do not make any ex-ante restrictions on their structure. Following standard causal-inference notation neyman1923applications,rubin1974estimating, we assume that each of these documents is equal to the potential document $Y(1)$ or $Y(0)$, depending on whether the corresponding observation is treated ($W=1$) or in control ($W=0$), so that $Y = Y(W)$. We aim to describe overall differences between the random variables $Y(1)$ and $Y(0)$.
We assume that we have access to data from a randomized trial where treatment $W$ is assigned randomly, that is, $(Y(1),Y(0)) \perp W$. As a consequence, describing differences between the distributions of $Y(1)$ and $Y(0)$ is equivalent to describing differences between the distributions of $Y|W{=}1$ and $Y|W{=}0$. Our approach, therefore, can be seen as making statistical inferences on the difference between two distributions over text more broadly, where $W$ denotes group membership. If these two groups indeed come from a randomized experiment, then we provide statistical inference on causal effects; if the two groups denote any other two sources of texts (as is the case in our illustration in (ref) below), then we still provide statistical inference on differences across the two distributions, even if the differences can no longer be interpreted causally.
To describe differences between the distributions $Y|W{=}1$ and $Y|W{=}0$ of text, we assume that we have a random sample available in which text from both distributions are sampled independently.
Throughout, we perform our analysis conditional on the sizes $n_1$ and $n_0$ of the treated and control samples. This covers randomized trials with a fixed fraction of treated units as well as trials with a fixed probability of assignment. Our results are easily extended to popular alternative randomization schemes, such as stratified and cluster randomization.
Relative to a standard causal-inference framework in which realizations of the outcome $Y$ take the form of a single number or a low-dimensional vector, we face a fundamental challenge: if outcomes are represented in text form (or, more generally, are very complex), then we cannot simply summarize the effect of treatment $W$ on outcomes $Y$ in terms of simple estimands like differences $\operatorname{E}[Y|W{=}1] - E[Y|W{=}0]$ (corresponding to the average treatment effect $\operatorname{E}[Y(1) - Y(0)]$). Instead, we need to find ways of describing core aspects of the differences across $Y|W{=}1$ and $Y|W{=}0$ in a manageable and statistically valid way, based on a potentially small sample $\{(Y_i,W_i)\}_{i=1}^n$. In the following three chapters, we tackle this challenge by providing an approach to addressing whether there is a difference between $Y|W{=}1$ and $Y|W{=}0$ ((ref)), what the difference is ((ref)), and how complete our description of said difference is ((ref)).
Our first goal is to detect whether the two distributions $Y|W{=}1$ and $Y|W{=}0$ over text documents are the same. That is, before asking how the two distributions differ, we first want to check whether there is any statistically significant difference at all. Even for this seemingly more effortless task, standard approaches for testing for differences across two distributions are limited. First, we could choose a few features of the text documents by hand that we think could be affected by treatment and then run a joint test for average differences in those features across the two groups. For example, we could hypothesize that the texts are longer in one group than the other, that a specific word (or group of words) is more likely to appear in one group, and/or that the sentiment of the text (as measured by some pre-defined algorithm) differs in a systematic way. But the number of ways in which we could provide simple features of complex texts is so vast that this approach is likely to miss important signals unless we have strong and correct priors about which aspect of text is most affected. Second, we could instead search through many such features automatically by trying them one by one. However, this approach risks leading to spurious findings, requiring us to apply severe corrections to avoid excess false positives. In the context of a very high number of potential features relative to a small or moderate number of text documents, these corrections may make detecting significant effects very hard, if not impossible. Finally, we could apply general non-parametric tests that have been proposed for detecting the difference between two distributions, such as multi-dimensional generalizations of the Kolmogorov--Smirnov test. Yet even such tests would rely on specific representations of the text, as they typically focus on relatively low-dimensional outcome vectors and may quickly lose power otherwise.
Instead of relying on tests that compare text features across the two groups, we adapt a trick from Ludwig2017-xk and ask: Given a document $Y_i$ in our sample, can we predict its group assignment $W_i$ better than some trivial benchmark? That is, rather than asking whether $W$ has a causal effect on $Y$, we ask whether $Y$ is predictive of $W$. This change in framing allows us to leverage non-parametric tools from machine learning directly, since these tools excel at predicting some simple outcome (here: $W_i$) from complex covariates (here: $Y_i$) by systematically searching over good predictors. Of course, we now have to adapt this idea for the specific case of a small sample of complex text documents. First, not all machine learning methods work well with text. Second, text methods specifically may be very complex and do not come with any statistical guarantees that allow us to assess predictive signal without additional tests. We, therefore, adapt this idea for text data and provide a specific testing scheme that ensures validity without making restrictive assumptions on the machine-learning method we use, remains computationally manageable, and still works well with small sample sizes.
We construct a two-step procedure to test whether text documents can predict group assignments (and thus, whether the two distributions are different). In the first step, we let an algorithm learn the relationship of text documents to group assignments. In the second step, we validate whether the learned relationship outperforms a trivial benchmark. To ensure that our inferences are statistically valid, we separate these two steps by sample splitting, following egami2022make. Specifically, we split the full sample $\{1,\ldots,n\}$ randomly into a training sample $\mathcal{T}$ and a hold-out sample $\mathcal{H}$. Throughout, all of our results are conditional on the split into training and hold-out.
We first apply a machine-learning algorithm to the training data $(Y_i,W_i)_{i \in \mathcal{T}}$ and have it produce predictions $\widehat{W}_i$ for the (unseen) group assignments $W_i$ of each text $Y_i$ in the hold-out, $i \in \mathcal{H}$. These predictions can be classifications ($\widehat{W}_i \in \{1,0\}$) or probabilities ($\widehat{W}_i \in [0,1]$). On the hold-out data, we then estimate how well these predictions $\widehat{W}_i$ line up with the actual group assignments $W_i$ by calculating a loss $L((W_i,\widehat{W}_i)_{i \in \mathcal{H}})$. For classifications $\widehat{W}_i \in \{1,0\}$, this loss could be the misclassification rate or the F1 score. For probabilities $\widehat{W}_i \in [0,1]$, it could be the mean-squared error, log-likelihood, or the area under the curve (AUC). In any case, we can gauge the quality of these predictions by comparing them to the loss $L((W_i,\widehat{w})_{i \in \mathcal{H}})$ for a trivial benchmark $\widehat{w} \in \{0,1\}$ or $\widehat{w} \in [0,1]$ obtained from the training data (for example, by minimizing $L((W_i,\widehat{w})_{i \in \mathcal{T}})$) that does not depend on the text data. Intuitively, if we obtain lower loss from using the text documents, $L((W_i,\widehat{W}_i)_{i \in \mathcal{H}}) < L((W_i,\widehat{w})_{i \in \mathcal{H}})$, then we have found evidence that there is signal for predicting group assignments -- and thus that the distributions over the two groups must differ.
As our preferred implementation, we use an LLM to form classifications $\widehat{W}_i \in \{0,1\}$, and then evaluate prediction quality using the accuracy or, for very imbalanced groups, the F1 score. \footnote{ The accuracy is the fraction $\text{Acc}((W_i,\widehat{W}_i)_{i \in \mathcal{H}}) = \sfrac{\sum_{i \in \mathcal{H}} \Ind{\widehat{W}_i = W_i}}{h}$ of correctly labeled instances. The F1 score is the harmonic mean $\text{F1}(\cdot) = 2 \frac{\text{Pre}(\cdot) \: \text{Rec}(\cdot)}{\text{Pre}(\cdot) + \text{Rec}(\cdot)}$ of precision $\text{Pre}((W_i,\widehat{W}_i)_{i \in \mathcal{H}}) = \sfrac{\sum_{\widehat{W}_i = 1} W_i}{\sum_{\widehat{W}_i = 1} 1}$ and recall $\text{Rec}((W_i,\widehat{W}_i)_{i \in \mathcal{H}}) = \sfrac{\sum_{W_i = 1} \widehat{W}_i}{\sum_{W_i = 1} 1}$. In these two cases, we choose $L((W_i,\widehat{W}_i)_{i \in \mathcal{H}}) = - \text{Acc}((W_i,\widehat{W}_i)_{i \in \mathcal{H}})$ and $L((W_i,\widehat{W}_i)_{i \in \mathcal{H}}) = - \text{F1}((W_i,\widehat{W}_i)_{i \in \mathcal{H}})$, respectively. The F1 score has the downside that it is less intuitive and not invariant to switching group labels, but it may be the better choice when group labels are very imbalanced, in which case the standard practice is to designate the smaller group the positive class ($W_i = 1$). } In principle, the above procedure works with any machine-learning tool that can extrapolate the relationship of text documents $Y_i$ to assignments $W_i$ from the training to the hold-out samples. In practice, the procedure is particularly easy to implement by prompting a state-of-the-art LLM with a large context window.\footnote{In our application in (ref), we use Google Gemini 1.5 Pro, which allows for a context window of around one million tokens or around 250,000 words, with implementation details and prompts provided in (ref).} Specifically, we provide all training documents with their group assignment along with all held-out documents without their assignment in a single prompt and query the LLM to provide the most likely group assignments for each of the held-out text documents. Beyond the simplicity of its implementation, one advantage of this LLM-based approach is that it is effectively pre-trained on a large corpus and on related text-analysis tasks, making this approach effective even with a small number of documents. At the same time, prompting the LLM in this way only produces binary classifications (rather than group probabilities) and does not directly optimize for any specific loss function. An alternative approach that still leverages LLMs would be to obtain low-dimensional embeddings from an LLM, and then obtain more granular probability estimates via non-parametric prediction.
Next, we construct a valid test of a difference in distributions from the chosen measure of out-of-sample prediction quality. Rather than relying on assumptions on the machine-learning tool or large-sample arguments, we use a permutation test to turn prediction quality into a $p$-value that is valid irrespective of method and sample size, and instead relies on random sampling and sample splitting alone. Specifically, we consider random permutations $\pi$ of the hold-out units $\mathcal{H}$, and then compare the improvement in performance \[ \Delta = L((W_i,\widehat{w})_{i \in \mathcal{H}}) - L((W_i,\widehat{W}_i)_{i \in \mathcal{H}}) \] over trivial predictions to the distribution of improvements \( \Delta_\pi = L((W_{\pi(i)},\widehat{w})_{i \in \mathcal{H}}) - L((W_{\pi(i)},\widehat{W}_i)_{i \in \mathcal{H}}) \) where actual treatment assignments are randomly permuted. Intuitively, if there is no true difference between the two groups, then the loss improvement $\Delta$ on the actual data should not look systematically different from the improvement $\Delta_\pi$ when group assignments are shuffled. If, on the other hand, the predictions $\widehat{W}_i$ contain true signal about the assignments $W_i$, then we hope that the improvement $\Delta$ in loss is larger than many of the other draws $\Delta_\pi$. This suggests the $p$-value
with $B$ permutations $\pi_b$ of $\mathcal{H}$ drawn independently and uniformly at random, and the associated test with significance level $\alpha$ that rejects the null hypothesis $Y|W{=}1 \stackrel{d}{=} Y|W{=}0$ for $\widehat{p} \leq \alpha$. \footnote{ The $p$-value measures the fraction of draws that lead to an improvement that is at least the same as $\Delta$, including the original data. Therefore, we add one to the numerator and denominator. As a result of this adjustment, the test is conservative for small $B$, and has asymptotically exact size as $B \rightarrow \infty$. } As long as the algorithm that produces the treatment assignments $(\widehat{W}_i)_{i \in \mathcal{H}}$ never has access to the held-out actual assignments $(W_i)_{i \in \mathcal{H}}$, the resulting test provides valid size control.
This test has three advantages in our setting. First, it is valid without further assumptions on the algorithm that produces the predictions. This includes LLMs, provided we can guarantee that the held-out assignments $(W_i)_{i \in \mathcal{H}}$ never entered the training process (which includes the LLM training data as well as any fine-tuning exercise). Second, the test does not require the repeated computation of predictions, so it is computationally efficient even when we use very complex algorithms to obtain predictions $(\widehat{W}_i)_{i \in \mathcal{H}}$. Third, the test is valid even when the number of documents is small. At the same time, the test remains inherently limited since it does not provide a clear indication of what the differences across groups are -- it only provides evidence that some differences exist.
Above, we have proposed an approach to testing whether two distributions over texts are systematically different. We now tackle the challenge of describing what the systematic differences across the distributions $Y|W{=}1$ and $Y|W{=}0$ are.
We note that the task of describing systematic differences is challenging for at least two reasons. First, the space of potential differences is amorphous and huge, and even (or especially) a human analyst would struggle searching through it systematically. Second, unlike the approach in the previous section that relied on a statistical test of differences across texts, making substantive claims about which outcomes are affected relies on a definition of economic outcomes that is hard to formalize. While the former challenge points to the massive potential of using machine-learning tools that search through complex data and hypotheses, the latter challenge also foreshadows their inherent limitations.
We first define what we mean by describing differences across text distributions, leading to the notion of causal themes ((ref)). We then propose an LLM-based implementation to hypothesize such themes ((ref)), before discussing their validation (Sections (ref) and (ref)). We close this section by emphasizing the role of human analysts beyond scoring ((ref)).
To tackle the challenge of describing systematic differences in complex outcomes across different groups, we consider a simple description vector $Y^f = f(Y) \in \mathcal{D}$ that includes core aspects of documents across the two groups. Specifically, we score texts according to $k$ core themes, where $k$ is a small number. Each of these themes may correspond to a specific topic covered in a document, to its style, or to the sentiment it expresses. For each topic $j$, we obtain a score $Y_j^f \in \mathcal{D}_j$ within a simple set $\mathcal{D}_j$. That simple set will typically be a continuous or discrete scale, such as when the prevalence of a topic is scored from 0 to 100 or sentiment as $\{-1, 0, +1\}$. Or it can be a discrete classification into a few groups, for example whether the poetic form of a document is a sonnet, a haiku, or neither ($\mathcal{D}_j = \{ \textnormal{sonnet}, \textnormal{haiku}, \textnormal{other} \}$). In any case, we assume that descriptions are easily understood and that we can use standard statistical tools to estimate causal effects on the low-dimensional vector $(Y^f_1,\cdots,Y^f_k) = Y^f = f(Y) \in \mathcal{D} = \mathcal{D}_1 \times \cdots \times \mathcal{D}_k$. By using these descriptions, we turn the task of considering the intractable effect of $W$ on full documents $Y$ into describing differences between scores $Y^f_j |W =1$ and $Y^f_j |W =0$.
An optimal scoring $Y^f = f(Y)$ that summarizes a complex document $Y$ in terms of core themes should provide mutually exclusive, differentially complete, and causally meaningful descriptions of the text. By mutually exclusive, we mean that components $Y_j^f$ describe themes that do not overlap in meaning and are entirely separate. By differentially complete, we specifically refer to the ability of the given description to capture the systematic differences across the two groups as well as possible. (In this sense, a good set of causal themes differs markedly from a complete description of the text, which instead may include features that are common across groups.) Finally, by causally meaningful, we mean that the aspects of text captured by the description are descriptions of plausible outcomes described in the text. This latter criterion is meant to rule out cases where descriptions do not have a clear relationship to the content of the text or represent aspects that are unhelpful in describing effects of interest. These criteria represent an ideal that may be infeasible in practice since simple scorings of complex documents may inherently lose relevant information and it may not be feasible to formalize what makes a description accurate or meaningful.
How can we construct simple summaries of documents that represent systematic differences between treatment and control groups? One approach would be to rely on pre-specified scoring functions applied to the documents. For example, we could classify documents based on the counts of specific words, use a generic topic model trained on some other text data, or apply algorithms for sentiment analysis. However, the use of pre-specified classifications is limited in at least three ways. First, such scores may not be a good fit for the specific documents in our experiment. Second, even if they are good at classifying documents overall, they may not represent the systematic differences across groups well. Finally, pre-specified ways of classifying our documents may fall short of our goal of expressing flexibly what the (possibly unexpected) differences across documents are, since they may reflect our preconceptions more than the data.
Instead of relying on pre-specified classifications of documents, we learn plausible causal themes from the data itself, similar to pmlr-v162-zhong22a,zhong2023goaldrivendiscoverydistributional. That is, rather than pre-specifying themes and the associated scoring function $f$, which maps text $Y$ to descriptions $Y^f$, we learn themes directly from the training data $(Y_i,W_i)_{i \in \mathcal{T}}$. We do not take the stance that there is a unique, true set of themes that we aim to learn. Instead, we hope to obtain some suitable description of documents, and will later provide valid statistical inference based on scoring additional documents $(Y_i,W_i)_{i \in \mathcal{H}}$ in the hold-out sample. In fact, having flexibility with respect to the technology that proposes themes is a feature of our approach, and our inference framework still delivers robust causal statements regardless of which representation is used.
For our implementation, we feed into a large-language model (LLM) the documents in the training data, along with their classification into treatment and control groups. From this LLM, we obtain themes that represent systematic differences between treatment and control documents in terms of core themes that differentiate them (similar to a D5 system, zhong2023goaldrivendiscoverydistributional, and earlier work in pmlr-v162-zhong22a). Practically, we can exploit very large context windows of modern LLMs (e.g.\ more than one million tokens in Google Gemini 1.5 Pro) to obtain such themes by prompting, where we request the LLM to provide descriptions of systematic differences across groups and to provide a list of themes and corresponding scales, along with examples. Details on our specific settings for the LLM and prompts are provided in (ref), and (ref) provides a worked-out empirical example.
While a prompting-based approach is straightforward to implement using publicly available LLMs, there are natural extensions that are likely to further align the model with the task of finding causal themes at the cost of additional computational and/or human effort. First, providing more detailed examples of successful scorings according to causal themes has the potential to improve the quality and reliability of output. For example, Zhou2023LIMALI shows that feeding a language model a curated set of high-quality prompts and responses, even in a limited quantity, can significantly enhance its performance by aligning it more closely with the specific task requirements. As an additional limitation, next-token prediction is at best a crude proxy for aligning the LLM with being able to find classifications that distinguish well between treatment and control groups, and adding explicit rewards to the optimization for finding themes that distinguish well between groups may further improve the LLM's ability to find differentially complete summaries. An important criterion for successful causal themes is not only their ability to distinguish between groups statistically but also whether these themes correspond to meaningful economic outcomes. Since that question is ultimately one of communication to human researchers, further aligning causal theme classifications based on reinforcement learning from human feedback (RLHF) could be helpful in ensuring that the LLM provides effective descriptions of meaningful outcomes.
Our approach can be related to supervised topic modeling, which provides an alternative implementation. Topic models like those based on Latent Dirichlet Allocation blei2003latent estimate common topics across documents, where each topic corresponds to a distribution over words and each document is assumed to come from a mixture of topics. Since we are interested in those topics that are related to differences across groups, we relate more directly to supervised topic models mcauliffe2007supervised that implement a version of LDA in which topics are also optimized for predicting an outcome of interest (here, the assignment to groups). A main downside of (supervised or unsupervised) topic models is that their performance may be unreliable at low sample sizes. We document such behavior in our empirical evaluation in (ref) below. Relative to such an implementation based on supervised topic models, leveraging readily available LLMs simplifies our technical implementation and still provides reliable results when the number of documents is low. The LLM approach may also capture themes beyond simple distributions over words, which allows us to integrate aspects of sentiment or more complex distributions over text. At the same time, this more vaguely defined concept of a “theme” may be less mathematically tractable than the narrower notion of a “topic” in LDA models.
Having obtained themes from the training data, our goal is now to obtain valid statistical inference on the differences in scores across the two groups in the held-out data. In doing so, we face a challenge: for the themes obtained from the training data, we need to obtain scores for the held-out documents. Once we have these scores, we can then obtain inference on their differences across treatment and control using standard statistical tools (like the permutation test from (ref)), where our sample-splitting procedure egami2022make ensures that our inference remains statistically valid.
A first approach to obtaining score labels would be to use a machine-learning tool to provide scores for the held-out text documents $Y_i$, while withholding their group assignments $W_i$. In our implementation, this could be achieved by prompting the LLM to provide scores $\widehat{Y}^{\hat{f}}_{ij}$ for each held-out document $i \in \mathcal{H}$ and each theme $j \in \{1,\ldots,k\}$. These score labels $\widehat{Y}^{\hat{f}}_{ij}$ then estimate the true scores $Y^{\hat{f}}_{ij} = \hat{f}_{j}(Y_i)$ corresponding to the theme $\hat{f}_j$ suggested by the LLM based on training data. For example, the LLM may suggest “text talks about cars” as a theme $\hat{f}_j$ with a classification into $\widehat{\mathcal{D}}_j = \{\text{yes},\text{no}\}$. The classification $\widehat{Y}^{\hat{f}}_{ij}$ then corresponds to the LLM's claim whether a specific text $Y_i$ talks about cars or not.
Is sample splitting enough to make valid inference on differences in theme scores across text distributions based on LLM-generated labels? In general, the answer is clearly no. While sample splitting allows us to provide statistical inferences for differences in machine scores $\widehat{Y}^{\hat{f}}_j$ across the two groups, it does not provide any guarantee that these inferences are valid for the true scores $Y^{\hat{f}}_j$. For example, an LLM may label any text that includes a vehicle as a “text that talks about cars”, even if that vehicle is a boat rather than a car -- a phenomenon we call theme score bias. In this case, we could obtain statistically valid inference that there must be a difference across the two groups, even if that difference is about boats rather than cars. But crucially, we do not obtain substantively valid inference for the claim that the difference is related to cars. This problem persists even if we know that the LLM can score text well on average based on some training distribution, since average scoring performance does not guarantee valid inference conditional on the chosen theme ${\hat{f}}_j$. In other words, the kind of guarantee we would need for being able to make substantively correct (causal) inferences based on LLM-generated scores is at odds with those typically available for such machine-learning tools, and would at least require some true scores $Y^{\hat{f}}_{ij}$ for verification (an idea we expand on in (ref) below). In the parlance of llmapplied, we are applying an LLM to an estimation problem for which machine-generated scores are not sufficient to obtain valid inference.
Instead of relying on machine-learning scores for the held-out documents, we therefore take the stance that providing scores for the final validation should ultimately be left to human experts. That is, we assume that the themes provided by the procedure in (ref) can be scored correctly by experts, yielding true scores $Y^{\hat{f}}_{ij}$. Of course, such scoring may be costly, but we argue that it cannot be avoided if our goal is to make valid inference on these themes. Furthermore, the cost appears small relative to two counterfactual procedures that do not leverage machine learning: first, a procedure in which we generate costly scores for a small number of pre-specified themes, risking that we waste costly scores on aspects of the text that may only have a small chance of being affected; or second, a procedure that generates costly scores for a very large number of themes first and then inspects effects on all of these themes. In the first case, we spend a similar amount of effort on generating scores, but risk achieving much smaller power; in the second case, we spend a considerably larger cost on scoring without a clear gain in yield.
If we have true (human-generated) scores for the proposed (machine-generated) themes available, and none of the hold-out data entered the construction of the themes, then inference on differences across the two groups is straightforward. For example, as long as the hold-out sample is sufficiently large, we can obtain conditionally valid inference based on a central limit theorem for the score differences across the two groups.
This result justifies constructing standard-error estimates, confidence intervals, and tests by estimating the variance as
on the hold-out, or by bootstrap inference based on resampling the hold-out sample only (holding $h_1,h_0$ fixed). In both cases, our conditional-inference approach means that we do not have to re-derive themes themselves, so no additional computation is necessary beyond straightforward calculations on the hold-out. The resulting inference is valid conditional on the scoring function $\hat{f}$ and does not require any guarantees about how topics were constructed. Instead, this approach only relies on sampling and sample splitting as a guarantee (Assumptions (ref) and (ref)).
Our method analyzes causal effects based on text data by summarizing text across treatment and control groups in terms of core themes. While these summaries may be suggestive of causal effects, we argue above that typical AI systems like the LLM we deploy in our empirical example in (ref) do not come with guarantees about their validity, necessitating the use of human-generated scores.
In practice, scoring all hold-out documents may be costly. An alternative approach, going back to at least bound1991extent,pepe1992inference,robins1994estimation and recently popularized in machine learning as “prediction-powered inference” (angelopoulos2023prediction; see {ji2025predictions} for a recent survey), is to combine costly (human-generated) true scores with cheap (machine-generated) approximations to correct for biases in the latter. In our context, a natural way of combining true scores $Y^{\hat{f}}$ with machine-generated ones $\widehat{Y}^{\hat{f}}$ is the estimator
where $\mathcal{L} \subseteq \mathcal{H}$ is a subset of the hold-out sample for which true scores $Y^{\hat{f}}_i$ are available. Here, $\mathcal{L}$ is chosen uniformly at random among subset of $\mathcal{H}$ of size $\ell = \ell_1 + \ell_0$ with $\ell_1>0$ units with $W_i{=}1$ $\ell_0>0$ units with $W_i{=}0$. The estimator in (ref) has an intuitive form, similar to the doubly-robust semi-parametric estimators from robins1994estimation and chernozhukov_doubledebiased_2018: The first part can be thought of as a potentially biased estimator based on the machine-generated scores. The second part estimates and corrects for the bias of the first part. As a result, the full estimator is exactly unbiased (conditional on themes) for the true expected difference $\tau$ across groups. This estimator is a special case of the bias-corrected estimator in angelopoulos2023prediction and egami2024using, and is also proposed by llmapplied for the combination of human and LLM labels in the context of estimation problems.
If machine-generated scores closely resemble those by human experts, then the bias-correction term in (ref) is small, and the estimator performs similarly to an estimator that uses true scores from the full sample. In this case, adding human-generated scores can improve precision. But the estimator is still unbiased and allows for valid inference even if the machine-generated scores are systematically off, without any substantive assumption on how the machine scores are generated.
In particular, this result justifies estimating the variance of the estimator $\widehat{\tau}^\dagger$ by
on the hold-out (where $L_i = \Ind{i \in \mathcal{L}}$), or by a hold-out bootstrap (that holds $h_1,h_0,\ell_1,\ell_0$ fixed). The variance expression has two components for each of the groups. The first part involves the variance of $Y^{\hat{f}} = \hat{f}(Y)$, which is the part of the variance that cannot be avoided since it stems from inherent within-group variation of the true scores. The second part is about the difference $\widehat{Y}^{\hat{f}} - Y^{\hat{f}} = \hat{f}^\dagger(Y) - \hat{f}(Y)$ between estimated and true scores. This part is small if the estimated scores $\widehat{Y}^{\hat{f}}=\hat{f}^\dagger(Y)$ are a good approximation of the true scores $Y^{\hat{f}}=\hat{f}(Y)$. Its contribution goes down as the fraction $\sfrac{\ell_1}{h_1}$ or $\sfrac{\ell_0}{h_0}$, respectively, of true scores increases.
Above, we have considered a scheme by which an AI system proposes themes that may vary systematically across groups. We have evaluated how these themes actually vary across groups in a validation step on the hold-out data. However, in practice, not all proposed themes may provide the desired scientific insight, and it may be infeasible to prompt a large language model to achieve all the properties specified in (ref). We, therefore, now consider an additional human validation step before scores are generated and effects estimated.
One reason why proposed themes may miss the mark is that they may not express relevant outcomes of interest. For example, if our procedure is applied to estimate causal effects in an experiment, then a theme proposed in the first stage may express differences related to the administration of treatment itself rather than plausible downstream differences in outcomes. As an example, consider an application where treatment consists of attending a job-training program. Texts written by program participants may differ between treatment and control groups by whether they mention having participated in a job-training program, which represents a statistically correct difference across texts, but may not constitute a causally meaningful one in the parlance of (ref).
Our procedure allows for human intervention to select and adjust themes before validation, provided that the “firewall principle” mullainathan2017machine is not violated. In general, different themes may be of different scientific value, which only researcher judgement may be able to assess. Our procedure remains valid if the researcher excludes, modifies, or adds themes after they are generated from the hold-out sample, but before validation. As long as the sample is split randomly between the training and hold-out samples and the researcher does not have access to any data from the hold-out sample when making these decisions, the downstream (conditional) inference on themes remains valid. Formally, these modifications become part of the themes $\hat{f}$ produced from the training data, only that they are now resulting from a joint human––AI procedure.
Given a set of themes that describe systematic differences across groups, we now ask how complete that description is. That is, we aim to estimate which fraction of the differences in text is captured by the differences in these specific themes. To do so, we go back to the prediction approach from Ludwig2017-xk and ask how well the theme scores predict group membership. Following the approach of fudenberg2022measuring, we then benchmark this prediction performance against two alternatives: first, a trivial predictor that does not use any text information; and second, the non-parametric benchmark from (ref).
By construction, this completeness measure is always between 0 and 1. This is because using themes to predict group membership has at least as much (expected) predictive power as a constant prediction but restricts how documents can be used for predicting and thus has (expected) predictive power that is not better than the non-parametric benchmark in the denominator. When applied to themes $\hat{f}$ learned on the training data, a completeness of close to 1 means that the information in those themes can separate groups as well as possible. On the other hand, if the theme-based predictor performs considerably worse than the unconstrained benchmark, then the themes are limited in capturing systematic differences.
In practice, we can estimate completeness on the hold-out data by computing
where $\hat{w}, \hat{\phi}, \hat{g}$ are estimates of the respective minimizers that are learned based on the training data only. As a result, the empirical measure can, in principle, be outside the unit interval. This can happen because having more detailed information available to predict group assignment does not necessarily translate to better prediction performance in practice. For example, the complex non-parametric prediction $\widehat{W}_i = \hat{g}(Y_i)$ may overfit to the training sample and have worse out-of-sample performance than the simpler prediction $\widehat{W}^f_i = \hat{\phi}(Y^f_i)$ based on theme scores, assuming that the theme representation provide an effective low-dimensional representation of the text.
In the implementation we demonstrate in (ref) below, we use the same LLM as in (ref) to form classifications $\widehat{W}_i = \hat{g}(Y_i) \in \{0,1\}$, obtain classifications $\widehat{W}^f_i = \hat{\phi}(Y^f_i)$ by logistic regression on the training sample, and evaluate prediction quality using accuracy. We can apply this approach to the themes $f = \hat{f}$ proposed based on the training data, but also to other themes (such as those proposed by a human analyst) to compare their completeness.
We now evaluate the above approach by describing and evaluating systematic differences between two groups of documents. Specifically, we consider two groups of academic abstracts. We use our approach to provide statistical evidence that the two groups are different; to describe systematic differences across the two; and to calculate how complete our description of these differences is. We also show the return to mixing human and machine scores, and compare the main results to an alternative approach based on supervised topic models. Our illustration demonstrates that our LLM-based implementation can provide clear evidence across the two groups of abstracts as well as easily understandable summaries, even when sample sizes are very low.
In designing a plausible dataset to apply our method, we face the challenge that our underlying LLM may already have a large amount of publicly available text in its training data. We, therefore, create our own grouping of abstracts of academic manuscripts from scratch to avoid contamination and choose a labeling that does not obviously align with common categorizations. While the LLM may have seen the actual text in these abstracts before, it has not seen this (human-generated) classification into groups. Our sample-splitting approach, therefore, still provides valid statistical inference.
We start with 200 arXiv abstracts that are randomly chosen from the econometrics category. A member of our research team then subjectively labels each of the abstracts by whether they align with their own research interests (“treatment”, Group A, $W=1$) or not (“control”, Group B, $W=0$). We employ this human-generated, subjective classification to see whether the causal LLM can capture structure that goes beyond easily measurable differences, such as differences across pre-defined categories or the usage of specific keywords. We (randomly) divide these 200 abstracts into a first half of “training” abstracts and a second half of “hold-out” abstracts.
First, we check whether the two groups are indeed different. We implement the reverse-prediction approach from (ref), where we obtain a non-parametric classification of documents via an LLM (Google Gemini 1.5 Pro via prompting as in (ref)). We measure performance via accuracy (the fraction of correctly labeled instances). We also report F1 scores (the harmonic mean of precision and recall, see (ref)) as an alternative measure of performance that takes into account the imbalance in class assignment in our dataset (the treatment group is smaller than the control). The LLM-based classification achieves an out-of-sample accuracy of 86% (F1 score of .74), relative to a trivial classification that always guesses Group B (“control”) and achieves an out-of-sample accuracy of 71% (F1 score of .00 since its recall is necessarily zero). A permutation test on the hold-out shows that this difference is significant, with none of 10,000 random permutations achieving a better improvement in accuracy (or in F1 score) over the trivial benchmark ($p$-value of 0.000).
Having established that the groups do not follow the same distribution, we next describe differences across groups. As a starting point, we first prompt the LLM to describe systematic differences across treatment and control groups based on the training sample only, with results reported in (ref). Following (ref) with the template in (ref), we then have the LLM propose up to six specific themes and scales according to which documents can be scored. These themes and scales are reported in (ref), which is produced by the LLM. (We again leverage Google Gemini 1.5 Pro via prompting, using a new instance relative to (ref).) We then have a different member of our research team score hold-out abstracts according to these scales. In addition, we also obtain machine-generated scores for both training and held-out abstracts. As a result, we can tabulate average machine-generated scores across treatment and control groups in both the training and hold-out samples, and on the latter, we can compare machine- to human-generated scores.
The differences in average theme scores are provided in (ref), with conditional standard error estimates based on a bootstrap on the hold-out dataset only Our main results are the differences in human-generated scores ((ref)), which show clear and statistically significant differences across groups for the first four of the five themes, while the last theme only shows small and noisy differences that could be consistent with equal means. In addition, we also inspect machine-generated scores for both the hold-out ((ref)) and the training abstracts ((ref)), with similar findings. A joint Wald test of the null hypothesis that the score averages are the same across treatment and control group clearly rejects with $p$-value of close to zero, for both the human- and machine-generated hold-out scores. As we would expect, the differences in the training sample (where treatment status was available during training) tend to be slightly larger than those in the held-out data (where treatment assignment was not available to the LLM). This indicates slight overfitting and reinforces the need to use sample splitting to obtain valid causal inference.
To quantify how well the themes describe differences across treatment and control, we use the completeness metric from (ref) with reverse-classification F1 scores as a metric. As an upper benchmark, we use the reverse LLM-based classification from (ref) above, which we define as a completeness of 100%. The lower benchmark is obtained by trivial classification into the most common group, which yields an accuracy of 71% and corresponds to a completeness of 0%. The results of the completeness exercise are reported in (ref).
A logistic classification of treatment status from the human-labeled theme scores achieves an accuracy score of 85%, where the logistic regression is learned on training data only. This performance corresponds to a completeness score of 93%, suggesting that the LLM-based themes capture differences exceptionally well. Interestingly, a logistic regression based on machine-generated scores performs somewhat worse (accuracy of 81%, completeness of 67%) despite both the regressions being trained on LLM scores on the training data, consistent with an error-in-variables interpretation where LLM scores represent noisy versions of human scores.
In addition, we also evaluate a reverse classification based on a supervised topic model with five topics, as discussed in (ref). The supervised topic model only achieves an out-of-sample accuracy of 73% (completeness of 13%), which shows the potential gain of scoring based on our causal LLM approach over summaries based on conventional topic models when sample sizes are very small.
We further test the hypothesis that each model performs equivalently to the trivial classifier, with corresponding $p$-values based on a permutation test for accuracy executed on the hold-out provided in (ref). All the approaches described here document significant differences across groups, with the $p$-value for the supervised topic model at 4.1% compared to 0.0% for the LLM-based implementation. These conclusions would be similar if we used F1 scores, where we note, however, that the best trivial classification would be to assign all instances to Group A (“treatment”) in this case for a benchmark F1 score of .29.
We can further conceptualize the cost--benefit tradeoff of human scoring in terms of a tradeoff between the cost of producing score labels and the benefit of improved precision in our average treatment effect estimate at the theme level. Using the approach outlined in (ref), we can simulate this tradeoff by estimating $\hat{\tau}^\dagger_j$ for various sizes of the set $\mathcal{L} \subseteq \mathcal{H}$ of human-scored instances in the hold-out set. In particular, for a given $\ell = |\mathcal{L}| \leq |\mathcal{H}| = 100$, we estimate the variance of $\hat{\tau}^\dagger_j$ via bootstrap, and plot how that value changes with $\ell$ for each of our five identified themes (holding the fraction of “treated” fixed, $\sfrac{\ell_1}{\ell} \approx \sfrac{h_1}{h} = 29\%$). The resulting plot ((ref)) shows that the precision of the estimator for each theme is monotonically decreasing in $\ell$. Furthermore, the curvature of each line illustrates decreasing marginal benefit to additional human scoring, implying that it may be optimal (from a decision-theoretic point of view) to human-label a certain number of held-out documents, with the stopping point characterized by the point at which the marginal benefit (in terms of estimator precision) outweighs the marginal cost (in terms of scoring cost).
We propose a procedure that uncovers and validates systematic differences across complex text documents. In theory, the procedure provides statistical validity even when we lack guarantees about the underlying AI tools. However, when the procedure is used in practice, we may be concerned about our ability to replicate the procedure and verify its results. In this section, we propose a flexible pre-specification approach based on sample splitting that balances the value of flexibility with the need for statistical validity and ex-post verification.
Concerns around replicability arise when the procedure is run multiple times and some estimation and reporting choices are adjusted based on the results, such as when prompts are changed, themes selected, or scores modified after looking at some of the hold-out evaluation. With the use of large language models, there is an additional concern that the LLM explicitly or implicitly had access to data from the hold-out dataset. A standard approach to mitigate such concerns is to rely on pre-specification, where all steps of the analysis are pre-registered in a pre-analysis plan before the data is accessed, and the procedure is then run under controlled conditions according to that pre-specification.
We believe that strict pre-analysis plans are limited in working for AI procedures like ours, both because they do not capture all concerns around replicability and because they can be too restrictive. First, even when the analysis is pre-specified and all plans are followed, complex analysis based on LLMs (or other AI systems) may fail to replicate because it may produce different results when applied to the same data. This can be because the underlying LLM was changed by the developer or even because two runs of the same model produce different outputs due to inherent randomness or instability (a challenge not usually overcome by setting randomness parameters such as the “temperature” to their minimum). Second, strict pre-analysis plans can also be too restrictive because some corrections and adjustments based on a human inspection of preliminary results may be essential to producing meaningful output, such as expert judgment about which themes represent meaningful causal effects ((ref)). We, therefore, see pre-specifying the entire algorithm before any data becomes available as a solution that is simultaneously insufficient and impractical.
Instead of pre-specifying the complete analysis, we propose a sequential-access analysis plan where the analyst first obtains access to the training data only, and a third party witholds access to the hold-out data (similar to split-sample analysis plans proposed in anderson2017split). The analyst then runs all the preliminary analysis on the training data, adjusts prompts as necessary, and chooses themes. Before the hold-out data becomes available, the analyst commits to the themes to be evaluated out of sample. Only then is the remaining data made available on which the specified analysis is performed and reported, with the third party certifying its result. \footnote{ One challenge with sample splitting arises from our approach to testing based on LLMs ((ref)). Specifically, our proposed test for differences across groups requires access to texts from the hold-out data to predict group labels. For the testing part, the pre-specification may therefore either include detailed instructions about how to implement scoring once the documents become available, possibly including prompts. Or the procedure may involve an intermediate step where held-out documents are made available and predictions of group labels are registered before the remaining held-out data (that is, held-out group labels) are made available as well. } In our eyes, this approach balances the need for replicability with the benefit of adjustments, and does not rely on the full replicability of hypothesis generation.
We provide a tool for estimating causal effects on qualitative outcomes captured in text. This method combines the power of modern AI tools with statistical guarantees. In data from a randomized trial, it summarizes individual responses in terms of core themes and then provides causal inference on the resulting summaries.
\paragraph{Role of AI.} Within this procedure, we view the role of generative AI as a hypothesis-generation machine. This view does not give AI a role in validating evidence by itself. Instead, it gives LLMs a role in generating plausible hypotheses that are then validated, similar to the approach of zhong2023goaldrivendiscoverydistributional. For this procedure to work, hypotheses have to be formulated in a way that can be validated based on separate ground-truth data. In our framework, AI-generated themes play the role of these hypotheses, and we then use sample splitting and human scoring for validation. We do not see this as a limitation of the current generation of AI solutions specifically but instead as a general structure for integrating machine learning and AI.
\paragraph{Researcher--AI complementarity.} While the AI can suggest plausible causal effects, our validation and inference necessarily rely on human judgment: we cannot automatically evaluate whether causal themes are correct, meaningful, and relevant representations of causal effects expressed in text. Beyond this need for validation, our approach also faces remaining challenges with respect to the interpretation of its output. On the one hand, an AI tool may be more systematic in picking up differences between treatment and control groups than a human analyst. At the same time, the analysis may still be driven by pre-conceptions embedded in the corpus the LLM is trained on and is thus not neutral. Ultimately, we see the AI as a communication tool ludwig2024machine that proposes summaries that can be evaluated statistically, but still have to be contextualized by an expert researcher to be evaluated substantively.