Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
45,160 characters · 9 sections · 25 citation commands
Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective
Large language models (LLMs) can be powerful tools for creating content that supports business goals. From enhancing user engagement to increasing purchase likelihood, companies often aim to generate content that delivers measurable economic value. Prior research has shown that pretrained LLMs can perform well in certain tasks, such as generating creative product ideas castelo2024ai and predicting the likelihood of purchases arora2025ai. However, these models often struggle to capture human preferences and directly optimize for business outcomes goli2024frontiers, ye2024lola, emphasizing the need for fine-tuning with labeled data to causally guide the models towards desired business outcomes. Yet obtaining the right kind of labeled data to support this alignment is difficult; human-labeled data and surveys can introduce bias due to artificial contexts yeh2024reliable, and randomized experiments are often infeasible due to logistical and opportunity costs quin2024b. This paper explores how to bridge this gap using an abundant but underutilized source of supervision available to firms: historical observational data.
Consider a news website that aims to improve the click-through rates (CTR) of news headlines. While they may not have the capacity to run controlled experiments, they may track how users respond to different headlines over time, which could be used for fine-tuning. However, directly fine-tuning on this data could be challenging because external factors such as time trends may influence both the content and the outcome. In this paper, we examine both the opportunities and risks associated with using observational data, and we propose a novel method that corrects for confounding effects in the fine-tuning process.
Prior work has primarily relied on experimentation data, such as A/B tests, as supervision signals for fine-tuning language models. For example, angelopoulos2024value fine-tune LLMs on A/B test outcomes to generate email subject lines that outperform human-written ones in click-through rate (CTR), while ye2024lola introduce a system that combines LLMs with adaptive experimentation to optimize headline performance. In a different vein, wang2024using develop a distillation framework where smaller models learn response behaviors from larger LLMs, assuming that the teacher model's feedback is a reliable proxy for optimal performance. While these approaches highlight the potential of LLMs for business optimization, they rely on supervision signals that are either derived from controlled experiments, which are often expensive and limited in scope feit2019test, miller2020empirical, or from synthetic feedback, which may inherit the limitations of the teacher model. Our work instead explores how to fine-tune LLMs using abundant observational data while addressing the confounding factors that can distort model learning.
Broader trends in machine learning progress further highlight the importance of leveraging observational data. As gains from architectural improvements begin to plateau, the frontier of advancement is increasingly shifting toward improving data collection, supervision, and the quality of training signals. In this new regime, the informativeness of the alignment signal becomes central to further progress ouyang2022training, rafailov2024direct, zhou2023lima. While many alignment approaches rely on curated human-labeled data liu2023importance, large-scale observational data remains an untapped resource due to challenges related to confounding and reward misspecification concerns.
Reward misspecification and related manifestations of Goodhart’s Law, wherein optimizing proxy metrics leads models to diverge from the intended objective, have been studied in previous research gao2023scaling, rafailov2024scaling. These works have primarily focused on settings where supervision is derived from preference labels, mostly collected from humans. This paper, however, investigates scenarios where the reward signal originates from observational outcomes, which are especially vulnerable to confounding. Our work thus focuses on Causal Goodhart, where optimizing for observationally valid but causally flawed signals can lead to undesirable or misleading model behavior. We further elaborate on how our framework connects to prior literature in Appendix (ref).
Our results in this paper show:
We conclude by discussing the broader implications of these findings and how our proposed method can inform safer and more effective use of historical data for model fine-tuning.
Fine-tuning LLMs to align their outputs with user preferences is a common approach for enhancing their performance. This process typically relies on labeled preference data, which may be collected through human annotations ziegler2019fine, automated feedback mechanisms (e.g., RLAIF lee2024rlaif), or structured reasoning tasks (e.g., guo2025deepseek). Two major paradigms are commonly used to incorporate preference data into LLM training: (i) Reward Modeling followed by Reinforcement Learning, and (ii) Direct Preference Optimization.
In the former, a reward model \( r_\phi(x, y) \) is first trained to predict human preferences between outputs given an input \( x \). The model is typically trained using pairwise comparisons, optimizing a Bradley-Terry likelihood: \[ \mathcal{L}_{\text{RM}}(\phi) = - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left(r_\phi(x, y_w) - r_\phi(x, y_l)\right) \right], \] where \( \sigma \) is the logistic sigmoid. Once trained, this reward model is used to fine-tune the language model \( \pi_\theta(y \mid x) \) using reinforcement learning algorithms such as Proximal Policy Optimization (PPO), which maximize expected reward while regularizing against a reference policy: \[ \max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)} \left[ r_\phi(x, y) - \beta \, \mathrm{KL}\left(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\right) \right]. \]
In contrast, Direct Preference Optimization (DPO) bypasses reward modeling entirely and directly updates the policy to prefer higher-rated responses using a contrastive objective over preference pairs rafailov2024direct.
In industrial applications such as optimizing click-through rates for headlines, boosting booking rates on rental platforms, or improving adherence in health messaging, the gold standard for evaluating outcomes is randomized controlled trials, which allow unbiased estimation of causal effects. However, such experiments are expensive and often infeasible in practice. Meanwhile, organizations often have abundant observational data: historical logs of content such as page titles, messages, or headlines and their associated outcomes. This data can be used directly for fine-tuning, either by training a reward model or by constructing preference pairs for methods like DPO. The challenge, however, is that observational data is subject to confounding: unobserved variables may influence both the textual content and the observed outcome, leading to spurious correlations. For example, consider an LLM deployed at Airbnb to generate listing titles aimed at increasing reservation rates. Historical data may show that listings with the word “affordable” in the title perform better. However, this could reflect the underlying confounding effect of the price, as lower-priced listings generally get higher reservations. A model fine-tuned naively on this data may learn to associate “affordable” with success in all contexts, leading to unsuitable generations. Say you are asking the model to generate a title for a luxury riverside property, and the model generates “Affordable log chalet – perfect for solo travelers”!
This is a clear instance of reward misspecification: the model learns to optimize a proxy signal that only partially reflects the true objective. While prior work gao2023scaling, rafailov2024scaling has investigated reward over-optimization in both classical RLHF and Direct Alignment settings, these studies have largely focused on empirical scaling behavior and optimization dynamics. In this work, we take a step further by analyzing the role of confounding and causal misalignment in fine-tuning large language models using observational data. In the following section, we investigate how relying on historical preference data can lead to biased and potentially flawed model fine-tuning, by examining a real-world case study.
In this section, we present an example illustrating how confounding can lead to reward misspecification when fine-tuning language models using historical data. Specifically, we construct a dataset similar to that used by askell2021general, based on data from the \href{http://academia.stackexchange.com/}{Academia Stack Exchange}. In their study, the authors fine-tune a question-answering model and, in one step of their fine-tuning, Preference Model Pre-Training (PMP), use historical data to guide learning. They treat answer scores as preference signals and train the model to prefer higher-scored answers in cases where multiple answers are available for a question. This PMP step is followed by fine-tuning with human feedback, to ensure the alignment of the model's preferences with human judgments. In our experiment, we investigate what happens when human feedback is unavailable and only observational data is used for fine-tuning.
While using the scores can signal which answer is more helpful, these scores are not the outcomes of randomized experiments, rather could be affected by user engagement patterns. For example, answers posted earlier may receive more views and thus more votes. One confounder we investigate is periodicity in platform engagement across different weekdays. To investigate this, we analyze the average answer scores by weekday. As shown in Figure (ref), answers posted on Mondays receive significantly higher scores than those posted on Fridays. While one might speculate that this might be causal and reflect differences in writing quality, we observe a similar pattern in the average scores of questions themselves (Figure (ref)), suggesting that broader engagement trends may be at play.
We further examine the number of views per question as a proxy for user exposure. Since the dataset does not include view counts for individual answers, we cannot directly assess the effect of exposure at the answer level. Figure (ref) displays both question views and scores over time. The strong correlation between the two suggests that the observed temporal trends are more likely driven by fluctuations in user activity than by differences in content quality.
To test whether this bias can influence model behavior, we simulate a fine-tuning setup similar to that of askell2021general. We construct answer pairs based on user scores and designate the higher-scoring answer as preferred. For answers posted on Mondays, we prepend a neutral “Happy Monday!” phrase to introduce a content marker correlated with engagement rather than quality. We then evaluate model generations on 3000 held-out questions and count how often the words “Happy” and “Monday” appear. Table (ref) summarizes the results. The base pre-trained model rarely generates these terms. The supervised fine-tuned (SFT) should ideally capture the distribution in the data (generation temperature is set to $1$). Our results show a frequency of 13.7% ± 0.2% for the first word “Happy” which is consistent with the distribution in the data ($\sim 1/7$). In comparison, the DPO model generates “Happy” in 21.2% ± 0.8% and “Monday” in 11.8% ± 0.8%, representing substantial increases of approximately 7.5 and 2.7 percentage points, respectively. To assess whether these increases are statistically significant, we perform independent two-sample t-tests over the 25 generation rates from each model. The difference in “Happy” usage is highly significant ($p = 6.5\times10^{-10}$), and the increase in “Monday” usage is also statistically significant ($p = 3.4\times10^{-3}$). These results show that the model has internalized and amplified a spurious temporal signal. Additional details of this experiment, as well as further details about data and training characteristics, are provided in Appendix (ref).
This case highlights how confounding variables in observational datasets can lead to reward misspecification and unintended behavior in fine-tuned models. Without accounting for causal structure, models may learn to exploit spurious signals that correlate with success, even when they do not contribute to genuine task quality.
In the previous section, we saw how using historical data can lead models to learn from spurious correlations. In this section, we turn to a different question: What is the potential value of using historical data? Suppose a firm lacks access to experimental data, perhaps due to budget constraints or engineering limitations. Can it still use historical logs of content and their observed performance to fine-tune a model in a way that improves future predictions?
This question cannot be answered by studying settings where only observational data is available (such as observational data on Airbnb reservation rates), because we cannot know whether the observed outcomes are causally attributable to the content. To assess the effectiveness of observational fine-tuning, we need a setting where ground-truth causal effects are available. For this purpose, we use the Upworthy dataset matias2021upworthy, which includes click-through rates (CTR) collected from controlled A/B tests comparing alternative headlines. Because the dataset is built on randomized experiments, it allows us to evaluate how well models trained under different data conditions can recover causal preferences.
To simulate a scenario where a firm only has access to historical logs, i.e., data from a single version of content shown to users, we modify the dataset by randomly retaining just one headline package and its CTR from each A/B test, discarding the alternative. This mirrors a typical observational setting where the firm knows how a particular piece of content performed, but lacks counterfactual information about what could have happened with a different option. By training models with and without access to experimental variation, and evaluating both on held-out A/B test data, we aim to quantify the value and the limits of using observational data alone for learning preferences. A summary of the dataset statistics for both experimental and observational setups as well as implementation details and preprocessing steps are described in Appendix (ref).
Before moving on to the modeling setup and results, we briefly highlight a potential confounder that can affect observational CTRs: temporal variation in user engagement and topic salience. The probability that a user clicks on a given package depends not only on the quality or attractiveness of the headline, but also on who the viewers are and how relevant or important the topic is at the time. For instance, if a major sporting event occurs, the site may receive a surge of sports fans, whose preferences disproportionately influence overall CTRs. Similarly, politically themed headlines may receive more engagement during election periods. Figure (ref) shows the average CTR by month for three data subsets: experimental training packages, observational training packages, and observational validation packages. We observe a clear temporal pattern, with certain months getting substantially higher CTRs. Moreover, the CTR trends are highly correlated across subsets (pairwise correlations of monthly averages are between 96% and 97%), suggesting that these fluctuations are systematic rather than random.
This raises a concern for models trained directly on raw CTRs. They may overfit to superficial, time-related artifacts rather than learning meaningful properties of headline quality. As previously discussed, variations in CTR may partly reflect changes in the user population and taste rather than differences in content effectiveness. Figure (ref) provides evidence of these changes, showing substantial variation in the number of impressions per package across months. Notably, there is a marked increase in impressions toward the end of 2023, coinciding with the U.S. election period. These fluctuations suggest that the volume and potentially the composition of website traffic change over time. As a result, shifts in user demographics or interests could introduce biases into the observed CTRs, potentially misleading models trained on such observational data.
To evaluate how well models can learn from observational data versus experimental data, we fine-tune large language models with a classification/regression head on top of the final hidden embeddings. For models trained on experimental data, we use a classification head and optimize using a logistic loss to predict which of two headlines in a pair had the higher click-through rate (CTR). For the observational setup, where only individual headline packages are available without comparisons, we use a regression head and optimize mean squared error (MSE) to predict the observed CTR directly. In both cases, we apply $L_2$ weight regularization and select the regularization hyperparameter $\lambda$ using the validation set.
\paragraph{Performance.} We evaluate all models on the same held-out set of headline pairs, where each pair comes from an A/B test with a known preferred headline. The evaluation objective is to assess whether the model correctly ranks the preferred headline higher. To do this, we compute the ROC AUC (Area Under the Receiver Operating Characteristic Curve), which reflects the model’s ability to distinguish between better and worse-performing headlines. We use the Pythia suite of open-weight language models for all experiments biderman2023pythia. Figure (ref) shows the ROC AUC results for the Pythia-12B model. Training on the experimental dataset yields an AUC of 0.82, whereas the observational dataset produces a lower, but still above-chance, AUC of 0.74. This result is encouraging: it suggests that historical data, even without experimental variation and with only about 26% of the training packages, can still provide meaningful signals for preference learning. However, the performance gap also highlights the value of randomized feedback; exposure to counterfactual comparisons enables better generalization and more reliable preference estimation.
Furthermore, Figure (ref) shows the effect of model size on ROC AUC in both experimental and observational settings. We observe a consistent improvement in the performance of hyperparameter-tuned models as their size increases across both training settings. However, the gap between observational and experimental training remains stable, reinforcing the importance of causal data when it is available.
\paragraph{Importance of regularization.} We study the role of regularization in observational learning and find that strong regularization is critical for generalization. As shown in Figures (ref) and (ref), optimal validation loss occurs at $\lambda = 18{,}000$, yet the best test ROC AUC is achieved at $\lambda = 50{,}000$. This discrepancy suggests that in the presence of confounding factors, tuning hyperparameters solely based on validation loss may not suffice. The model may overfit to patterns influenced by spurious correlations in the validation data, rather than learning features that generalize causally to unseen headline comparisons. Figure (ref) shows that this gap holds across model sizes: stronger regularization consistently yields better test performance than what validation loss would suggest. We further find that larger models generally require stronger regularization for optimal test performance. This observation implies that using a fixed regularization setting across models of different sizes is suboptimal. Figure (ref) demonstrates this by plotting test performance against model size under fixed regularization levels. The figure shows a non-monotonic effect, larger models begin to overfit more if regularization is kept constant. These results emphasize the need to scale regularization appropriately with model capacity in order to maintain generalization, which is often overlooked in practice.
While these results underscore the critical role of regularization, they also raise a practical challenge when access to held-out experimental data for tuning hyperparameters is often limited. In such cases, alternative strategies are needed to remove the effect of confounders. We address this issue in Section (ref), where we introduce a method for explicitly correcting for confounding effects in observational fine-tuning.
\paragraph{Temporal pattern overfitting.} As discussed in Section (ref), temporal variation in CTRs is a potential confounder in observational data. To assess how much models internalize these patterns, we compute the correlation between monthly average CTR estimates on the validation set and observed monthly CTRs in the training data. Figure (ref) shows this correlation across values of $\lambda$ for the Pythia-12B model. We observe that moderate regularization improves alignment with temporal patterns, but higher regularization reduces it. Interestingly, the $\lambda$ that yields the best test performance comes well after this drop, indicating that suppressing temporal patterns helps the model on the causal evaluation of headlines. This trend holds across model sizes. As shown in Figure (ref), models consistently show lower temporal correlation at their optimal test-time $\lambda$, further suggesting that failing to effectively account for confounding patterns can impair generalization performance.
We now introduce a formal framework to analyze the effect of confounding in observational fine-tuning. Suppose we have access to historical interaction data $\mathcal{D} = \{(\boldsymbol{X}_i, y_i)\}_{i \in \mathcal{I}}$, where $\boldsymbol{X}_i$ denotes the context vector for interaction $i$ and $y_i$ is the associated outcome (e.g., click-through rate or reservation rate). We assume that $\boldsymbol{X}_i$ can be decomposed into a textual decision variable $T_i$ (e.g., a generated title or headline) and auxiliary features $\tilde{\boldsymbol{X}}_i$, such that:
where $f$ is the unknown outcome function and $\epsilon_i$ is zero-mean noise. The goal is to train a generative model $G: \tilde{\boldsymbol{X}} \rightarrow T$ that produces high-reward textual actions for new inputs.
To capture confounding, we assume that the outcome function can be decomposed as:
where $\boldsymbol{F}_i$ is the set of observed features including $T_i$, and $\boldsymbol{C}_i$ represents observed confounders that influence both the action and the outcome. The function $g$ captures the causal effect of the textual action and other features, while $h$ captures the contribution of confounders. Crucially, if $g$ and $h$ are entangled, estimating them independently may result in biased models and reward misspecification.
\paragraph{Proposed method: DeconfoundLM} Our proposed approach, Deconfounded Language Model Fine-Tuning (DeconfoundLM), involves first identifying and modeling the effect of confounders, and then explicitly removing their contribution from the observed outcomes. This allows the model to learn the causal impact of the textual input and other features, without being influenced by confounding effects. In our experiments, we apply an instrumental variable strategy to estimate the confounding component. However, the framework is flexible: other methods such as Double Machine Learning chernozhukov2018double or Adversarial GMMs dikkala2020minimax can be used, provided the researcher is mindful of the assumptions each method entails. We further discuss the potential impacts and limitations of our theoretical framework and proposed method in Appendix (ref).
In our earlier real-world case studies, Upworthy and the Monday effect, we identified potential confounding variables (e.g., post date and news month). However, we cannot determine whether these factors have purely confounding or partially causal effects. For instance, monthly changes in Upworthy news CTRs could come partly from changes in editorial members in different periods, which causally affect CTRs. Hence, we turn to simulation experiments for a controlled evaluation of the proposed DeconfoundLM\ method.
We base our simulation experiments on the MIND dataset wu2020mind, which contains over 160{,}000 English-language news articles with both titles and full text. To ensure contextual consistency and controlled evaluation, we focus on the sports category, which includes the largest number of articles. Each article consists of a body of text and an associated headline. We treat the article body as the input context and aim to generate a headline $T_i$ that maximizes a synthetic performance score $y_i$, interpreted as a proxy for engagement or click-through rate.
To simulate realistic challenges in observational fine-tuning, we design two scenarios in which the observed outcome $y_i$ depends on both the textual quality and a confounding variable $p_i$ representing topic popularity (e.g., how much fan interest a team garners). We use the headline sentiment $s(T_i) = g(T_i, \tilde{\boldsymbol{F}}_i)$ as a measure of quality, as it is interpretable and easily measured. In both scenarios, we model the outcome as:
where $p_i$ is the confounder, and $\nu_i \sim \mathcal{N}(0, 0.1)$ represents observational noise.
We vary how $p_i$ is constructed across two settings:
These synthetic scenarios allow us to explicitly test whether models can recover the true effect of sentiment when the observed performance signal is partially corrupted by a structured confounder.
\paragraph{Comparative Methods.} We evaluate six approaches: (1) a base pre-trained model, (2) supervised fine-tuning (SFT), (3) RL with access to ground-truth sentiment (which serves as a baseline) (4) RL using observed performance without controlling for confounders, (5) RL models that incorporate popularity either as input text or as a scalar feature in the final layer, and (6) our proposed method DeconfoundLM-IV, which estimates and removes the confounder effect using an instrumental variable.
\paragraph{Results.} We train the reward models with five different seeds and evaluate all models' generations after the RL step on a held-out set of 3{,}000 news articles. Table (ref) summarizes the average sentiment of generated headlines and the frequency of team name mentions, which serve as a proxy for reliance on the popularity-based confounder. In the orthogonal setting, the model trained on observed performance (without accounting for confounding) is able to improve headline sentiment, indicating that it learns part of the true signal. However, it also shows a marked increase in the frequency of team name mentions, suggesting reliance on popularity cues. Incorporating popularity information, either via text prompts or as an input feature, reduces this effect. Among all methods, DeconfoundLM-IV\ achieves the best results: it more closely matches the sentiment gains of the true-reward model (with no statistically significant difference at $P < 0.05$ level compared to the noisy baseline) while minimizing unnecessary references to team names caused by the confounding variable.
The entangled case presents a more challenging scenario. Here, the naive model trained on observed performance fails to improve sentiment and again heavily generates team names. While models that include popularity in the input text or final layer performed well in the orthogonal setting, they struggle to recover the sentiment-performance relationship in this setting. In contrast, DeconfoundLM-IV\ demonstrates strong robustness. It successfully suppresses the influence of the confounder and generates headlines with high sentiment scores. This highlights the method’s ability to decouple the effect of confounders, which are correlated with the target signal. Full experimental details, including prompt formats and data splits, are provided in Appendix (ref).
Our findings suggest that using historical data to fine-tune language models can be a double-edged sword: while it provides valuable information without the need for experimentation, it could also introduce the risk of learning from confounded outcomes. Through both real-world and synthetic experiments, we show that models trained on observational data may internalize spurious correlations that are not causally linked to content quality. To mitigate this, we introduce DeconfoundLM, a method that explicitly adjusts for observed confounders in the fine-tuning process. By separating confounding influences from the outcome signal, our approach enables more causally grounded learning. Across multiple settings, we find that DeconfoundLM\ improves fine-tuning outcomes and better captures the true effects of textual inputs. That said, fully eliminating confounding, especially from the high-dimensional treatment of textual content, remains an open challenge. As a next step, we aim to explore techniques for detecting the causal impact of content in historical data. Finally, while our primary focus is performance and causal inference, we note that confounding can also introduce fairness concerns. If unaddressed, it may lead models to replicate or amplify structural biases in the data. We view causal deconfounding as a promising direction for aligning language models not only with user preferences but also with broader values of equity and accountability.