Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
68,220 characters · 19 sections · 38 citation commands
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
\lhead \hypersetup{pdftitle={Causal Evaluation of Language Models under Confounded Model Choice}}
Offline evaluation of language models increasingly relies on deployment logs, preference data, and judge annotations collected in the wild Zheng2023LMSYSChat1M,Zhao2024WildChat,Zheng2023LLMJudge,Chiang2024ChatbotArena. Once model choice is user-driven, those logs are confounded: the same unobserved factors that influence which model is used can also influence how its output is judged Kallus2018ExperimentalGrounding,Rosenman2023Shrinkage,Cheng2021Adaptive. Raw comparisons of logged outcomes therefore mix different self-selected subpopulations rather than a common target population. In summarization, a reader who prefers concise prose may select a model known for that style and rate it more favorably; in coding assistance, repository difficulty or developer familiarity may affect both which assistant is invoked and whether the resulting patch succeeds.
A randomized experiment breaks this link by overriding model choice. In practice, such experiments are costly and disruptive, so they remain small. Prior work studies how to combine a large confounded observational sample with a small randomized one Kallus2018ExperimentalGrounding,Rosenman2023Shrinkage,Cheng2021Adaptive,Lin2025PowerLikelihood,Yang2025CVCI,Colnet2024Review.
The problem has two parts: outcome generation and outcome scoring. Outcome generation asks which output a model would have produced on a given context; in many deployments an offline simulator (SIM) handles this by replaying models on cached contexts. Outcome scoring asks how the realized output should be evaluated without inheriting the bias in logged model choice. Separating the two yields a three-source design: a large observational log (OBS) with self-selected model choices, a small randomized experiment (EXP) with unconfounded outcome labels, and SIM outputs under alternative model choices. The goal is to estimate how each generative model would perform when logged model choice is confounded, randomized outcome labels are limited, and offline replay can regenerate candidate outputs.
\noindentContributions. The paper makes three contributions.
Section (ref) defines the causal graph, data sources, target estimands, and assumptions. Section (ref) states and proves the SIM-plus-EXP identification result. Section (ref) introduces the post-identification estimator families and the theory-to-evidence map. Section (ref) describes the controlled validation and cached benchmarks; Sections (ref) and (ref) report the cached summarization and cached coding probes. Sections (ref), (ref), and (ref) cover related work, scope and limitations, and conclusions.
OBS provides scale and auxiliary supervision from self-selected usage, EXP provides unbiased outcome labels, and SIM provides counterfactual outputs. Theorem (ref) shows that EXP and SIM identify the target, while OBS only improves statistical efficiency afterward. Figures (ref) and (ref) illustrate the causal structure and the division of labor.
We study a five-variable graph $(X,U,A,M,Y)$ shown in Figure (ref), where $X$ is the observed context, $U\in\mathbb{R}^{d_u}$ is an unobserved user state, $A\in\mathcal{A}$ indexes the chosen generative model, $M$ is the generated output, and $Y\in[0,1]$ is the scalar outcome. Confounding arises because the latent state $U$ affects both model choice and outcome. Beyond ordinary confounding, the graph imposes two structural restrictions: conditional on $(X,A)$, the latent user state $U$ does not directly affect the mediator $M$, and $A$ affects the outcome $Y$ only through $M$. These assumptions are natural in stateless generative systems, where the model's behavior is determined by its input and internal randomness rather than by who is asking.
Three data sources are available, each contributing something the others cannot provide alone.
\paragraph{Observational log (OBS).} OBS is a large observational dataset $D_{\mathrm{OBS}}=\{(X_i,A_i,M_i,Y_i,Z_i)\}_{i=1}^{n_{\mathrm{OBS}}},$ where $A_i$ may depend on the corresponding latent state $U_i$ and $Z_i\in[0,1]^K$ are $K$ auxiliary labels that may or may not be available. The auxiliary labels $Z_i$ provide extra supervision for learning a compact proxy representation of the context--output pair; user feedback ratings and pairwise preferences are natural sources and can often be collected at scale Christiano2017HumanPreferences,Ouyang2022InstructGPT,Chiang2024ChatbotArena.
\paragraph{Randomized experiment (EXP).} EXP is a much smaller dataset $D_{\mathrm{EXP}}=\{(X_j,A_j,M_j,Y_j)\}_{j=1}^{n_{\mathrm{EXP}}},$ in which model choice is randomized and therefore unconfounded. Since $A_j$ is drawn from the experimental action set $\mathcal{A}_{\mathrm{EXP}}$ according to a fixed distribution, $A\perp U$ and $A\perp U\mid X$, so EXP outcomes are causally interpretable. Randomizing model choice disrupts user experience, which is why these experiments remain small in practice.
\paragraph{Offline simulator (SIM).} The simulator reruns any generative model on any context and returns a mediator draw from $M \sim p_{\mathrm{sim}}(\cdot\mid X=x, A=a).$ Operationally, SIM reruns generative model $a$ on held-out context $x$ and records the generated output. In generative systems, SIM is especially natural because prompts, retrieved documents, tool traces, and other environment inputs can often be stored as deterministic inputs, while the remaining randomness comes from the generative model itself.
Two practical targets matter.
\paragraph{Marginal model value.} How well would each model perform on average if used by the full target population? The marginal model value is the main target for offline comparison of a fixed set of generative models, and it corresponds to the target estimand \[ \mu(a) := \mathbb{E}\!\left[Y\big(\mathrm{do}(A=a)\big)\right]. \]
\paragraph{Context-conditional model value.} Given a particular observed context $x$, what outcome would each model deliver on average? The context-conditional value is the key target for recommendation and personalization, and it corresponds to the context-level function \[ q(x,a) := \mathbb{E}\!\left[Y\big(\mathrm{do}(A=a)\big)\mid X=x\right]. \]
Section (ref) shows how both targets are recovered: SIM supplies the model-specific mediator distribution and EXP provides unconfounded scores for realized outputs.
In real cached benchmarks, we instead evaluate the finite cached target \[ q_{\mathrm{cache}}(x,a):=r^\star\!\bigl(x,m_{\mathrm{cache}}(x,a)\bigr), \] where $m_{\mathrm{cache}}(x,a)$ is the stored output for that context--model pair. The finite cached target approximates $q(x,a)$; the superpopulation estimand remains latent.
\paragraph{Assumption A1 (Randomization in EXP).} In EXP, $A$ is randomized over $\mathcal{A}_{\mathrm{EXP}}$ so that $A\perp U$ and $A\perp U\mid X$ hold.
Randomization is the sole source of unconfounded outcome information; without it, identification is impossible regardless of OBS size.
\paragraph{Assumption A2 (SIM validity and support).} The simulator matches the interventional mediator distribution, $p_{\mathrm{sim}}(m\mid x,a)=\mathbb{P}(M=m\mid X=x,\mathrm{do}(A=a))$, and is supported within the EXP mediator support: for every $x$ in the EXP context support and every $a\in\mathcal{A}_{\mathrm{EXP}}$, \[ \operatorname{supp} p_{\mathrm{sim}}(\cdot\mid x,a)\;\subseteq\;\operatorname{supp} \mathbb{P}(M\mid X=x,A=a,\mathrm{EXP}). \] Throughout the paper, “the shared EXP/SIM support” refers to this support condition. SIM validity is satisfied whenever any randomness in the generated output comes from the generative model itself and all deployment-time inputs that affect the generative model's behavior are included in $X$.
\paragraph{Assumption A3 (Outcome consistency under intervention).} OBS and EXP measure the same outcome $Y$, and the conditional outcome law given $(X,M,U)$ is invariant across OBS, EXP, and the post-intervention world: for every $(x,m,u)$ on the relevant support,
Only the model-choice mechanism differs across OBS, EXP, and intervention. Without this, EXP supervision could not transport either to the intervention or to the OBS-trained reward models.
\paragraph{Assumption A4 (No latent mediator confounding; single-round generation).} Conditional on $(X,A)$, the latent state $U$ is independent of the mediator $M$ in both OBS and EXP, and under $\mathrm{do}(A=a)$: \[ U \perp M \mid X, A. \] Equivalently, the SCM has no edge $U\to M$ once $(X,A)$ is fixed, and any deployment-time inputs that drive the generative model are absorbed into $X$. The same structural restriction appears in Figure (ref); promoting it to a formal assumption combines with A1's randomization in the mediator step of the proof of Theorem (ref). A4 restricts the framework to single-round, stateless generation. In repeated, adaptive, or multi-round interactions the residual dependence of $M$ on $U$ given $(X,A)$ is generally nonzero, so A4 fails.
\paragraph{Assumption A5 (Context distribution alignment).} The marginal model value $\mu(a)$ is defined with respect to a target context distribution $P_X^{\mathrm{tgt}}$ that coincides with the EXP context distribution: $P_X^{\mathrm{tgt}}=P_X^{\mathrm{EXP}}$. Equivalently, the EXP sample is drawn from the same context population over which $\mu(a)$ is averaged. When this fails, only $q(x,a)$ on the shared support is identified, and any reweighting to a different $P_X^{\mathrm{tgt}}$ requires standard transportability and trial-generalizability arguments ColeStuart2010,PearlBareinboim2011Transportability,BareinboimPearl2016DataFusion that are outside the scope of this paper.
The remaining identification problem is to score generated outputs without bias. EXP provides that scoring rule, while SIM provides the interventional mediator distribution. Throughout, we evaluate only actions in the experimental action set $\mathcal{A}_{\mathrm{EXP}}$ and mediator values on the shared support of the EXP law of $M\mid X=x,A=a$ and the corresponding SIM law $p_{\mathrm{sim}}(\cdot\mid x,a)$. Define \(r^\star(x,m) := \mathbb{E}[Y \mid X=x, M=m, \mathrm{EXP}]\).
\paragraph{Proof sketch.} The proof uses the assumptions one at a time in four steps.
The full proof is in Appendix (ref). Identification requires both SIM, to regenerate outputs, and EXP, to provide outcome labels for those outputs. Conditioning on $(X,M)$ in OBS does not block confounding through $U$ when Assumption A4 fails or OBS routes $A$ on $U$. Appendix (ref) gives the explicit derivations.
\paragraph{From identification to the experimental design.} The controlled semi-synthetic validation and the real cached benchmarks probe different parts of the theorem. The controlled semi-synthetic validation (Section (ref)) directly probes the theorem-level target: a known latent reward generator varies only $\beta$, $n_{\mathrm{OBS}}$, and $n_{\mathrm{EXP}}$, so estimator gaps reflect SIM-plus-EXP identification followed by post-identification estimation. The real cached benchmarks (Sections (ref) and (ref)) report recommendation regret against the finite cached target $q_{\mathrm{cache}}$, with one cached output per context-action pair and synthetic OBS/EXP resampling. They diagnose the post-identification estimator families within those benchmark constructions rather than testing Theorem (ref) directly. In all settings, the estimator design in Section (ref) uses OBS only after identification to reduce variance or improve approximation.
After SIM and EXP identify the target, the remaining question is how OBS can reduce estimation error. The six estimator families differ only in how they use OBS after identification: some ignore OBS outcomes entirely, some use OBS only to learn a lower-dimensional proxy representation, and some use OBS outcomes directly through a baseline or pooled fit. Exact objectives, proxy-learning details, and value-aggregation formulas are deferred to Appendix (ref) and Appendix (ref).
Throughout, $z=(x,m)$ denotes a context--mediator pair, and all reward models start from text features $\varphi(z)$. Hybrid methods use OBS auxiliary labels only to learn a lower-dimensional proxy representation $\psi(z)$, in the same spirit as text embeddings adapted for causal adjustment VeitchSridharBlei2020. The proxy only makes the EXP-side fit lower-dimensional and leaves the identified target unchanged.
The estimator comparison is guided by three evidence tiers. The identification theorem is part of the main paper. The estimator-comparison results in Appendix (ref) provide formal support for several oracle and approximation statements. Target-alignment patterns in the benchmarks diagnose whether a target reward aligns with a proxy, baseline, or correction class. They provide empirical rather than formal evidence. Table (ref) summarizes this map.
All six reward models minimize penalized squared loss with predictions clipped to $[0,1]$. They differ only in whether and how they use OBS. Table (ref) summarizes the six families; Appendix (ref) gives exact objectives and benchmark-specific instantiations.
\paragraph{Prediction-form summary.} Each family produces a clipped predicted reward $\widehat r(x,m)$ from one of the following forms; here $\varphi(z)$ is the raw text feature, $\psi(z)$ is the OBS-derived proxy representation, $f_{\mathrm{OBS}}(z)$ is the OBS-trained predictor (so $\widehat r_{\mathrm{OBS}}(z)=f_{\mathrm{OBS}}(z)$ for $\OBSOnly{}$), and $\alpha_{\mathrm{corr}}\in[0,1]$, $\lambda\in[0,1]$ are EXP-side tuning weights chosen by causal cross-validation:
In words: $\widehat r_{\mathrm{EXP}}$ regresses $Y$ on $\varphi$ using EXP; $\widehat r_{\psi}$ regresses $Y$ on the OBS-derived proxy $\psi$ using EXP; and $\widehat r_{\mathrm{G}}$ adds an EXP-fit correction to the OBS baseline $f_{\mathrm{OBS}}$ in proxy space. $\widehat r_{\mathrm{C}}$ pools the OBS and EXP regression problems on $\varphi$ with mixing weight $\lambda$ ($\lambda=1$ gives $\OBSOnly{}$, $\lambda=0$ gives $\EXPOnly{}$), and $\widehat r_{\mathrm{CR}}$ adds an EXP residual fit $\widehat g_{\mathrm{res}}$ in proxy space to $f_{\mathrm{OBS}}$ on the residual target $Y-f_{\mathrm{OBS}}(z)$. Appendix (ref) gives the exact objectives and the tuning rule for $\alpha_{\mathrm{corr}}$, $\lambda$, and the basis $B$.
\paragraph{From reward models to model values.} Given a fitted reward $\widehat r$, the direct method (DM) and doubly robust (DR) value estimators RobinsRotnitzkyZhao1994,BangRobins2005,Dudik2011DoublyRobust,Dudik2014DRStatSci,Jiang2016DROffPolicy plug $\widehat r$ into the identification formula (ref)--(ref). For each held-out context $x$ and action $a$, the cached benchmarks compute \[ \widehat q^{\mathrm{DM}}(x,a)=\widehat r(x,m_{\mathrm{cache}}(x,a)), \] i.e.\ the fitted reward at the cached mediator; the controlled semi-synthetic validation, which has access to a real SIM, instead averages $\widehat r$ over a SIM mediator sample. The marginal value is $\widehat\mu^{\mathrm{DM}}(a)=\mathbb{E}_{X\sim P_X^{\mathrm{EXP}}}[\widehat q^{\mathrm{DM}}(X,a)]$, with the expectation replaced by an empirical average over the held-out EXP context split.
The doubly robust estimator (DR) corrects $\widehat\mu^{\mathrm{DM}}(a)$ on EXP. With $K$-fold cross-fitting and uniform EXP randomization $p_{\mathrm{EXP}}(a)=1/|\mathcal A_{\mathrm{EXP}}|$, write $\widehat r^{(-k)}$ for the reward model trained on EXP folds $\{1,\ldots,K\}\setminus\{k\}$ and let $k(j)$ be the fold containing EXP point $j$. The DR estimator for the marginal model value is
where $\widehat\varepsilon_j(a):=Y_j-\widehat r^{(-k(j))}(X_j,M_j)$ is the cross-fitted EXP residual. Both terms in (ref) target the marginal value $\mu(a)$ under Assumption A5. The complete DM/DR formulas and cross-fitting protocol are given in Appendix (ref).
3.1 Grounded Correction (\Grounded). Inspired by experimental grounding Kallus2018ExperimentalGrounding, \Grounded starts from the OBS predictor $f_{\mathrm{OBS}}$ and uses EXP to learn a correction in proxy space. In the benchmark implementation, that correction is chosen from a small basis library rather than from a single linear head. Its final prediction is \[ r_{\mathrm{ground},B,\alpha_{\mathrm{corr}}}(z) \;=\; \mathrm{clip}_{[0,1]}\!\Big(f_{\mathrm{OBS}}(z)-\alpha_{\mathrm{corr}}\big(\widehat\theta_{B}^{\top}B(\tilde\psi(z))+\widehat c_{B}\big)\Big), \] where $\tilde\psi(z)$ is the low-dimensional proxy representation and $B$ is selected from a small fixed basis library on that proxy space. The method is most attractive when OBS already captures the coarse reward surface and EXP mainly needs to remove a lower-dimensional bias. Appendix (ref) gives the exact fitting and tuning details.
3.2 Cross-validated Causal Inference (\CVCI). CVCI Yang2025CVCI, building on the broader OBS--EXP combination literature Rosenman2023Shrinkage,Cheng2021Adaptive,Lin2025PowerLikelihood, pools the OBS and EXP regression problems directly and chooses the pooling weight $\lambda\in[0,1]$ by causal cross-validation on EXP. Concretely, the pooled coefficient solves
where $\mathcal L_{\mathcal D}(w,b;\phi,Y):=\sum_{i\in\mathcal D}(Y_i-(w^{\top}\phi(z_i)+b))^{2}$ is the squared-error loss on dataset $\mathcal D$ in feature map $\phi$, so $\lambda=1$ recovers the $\OBSOnly{}$ regression and $\lambda=0$ recovers the $\EXPOnly{}$ regression on $\varphi$. The pooling weight $\lambda$ is selected by minimizing held-out causal-CV loss on EXP at the model level (Appendix (ref)), anchoring the bias--variance interpolation to randomized outcomes.
3.3 CVCI-Residual. CVCI-Residual extends the CVCI pooling rule of Yang2025CVCI by first residualizing around the OBS baseline (replacing the regression target $Y$ with $Y-f_{\mathrm{OBS}}(z)$) and then running a $\lambda$-pooled fit in the proxy space $\psi(z)$: \hfuzz=5pt
yielding $\widehat g_{\mathrm{res}}(z)=(\widehat w^{\mathrm{res}}_\lambda)^{\top}\psi(z)+\widehat b^{\mathrm{res}}_\lambda$ and the final prediction $\mathrm{clip}_{[0,1]}\!\big(f_{\mathrm{OBS}}(z)+\widehat g_{\mathrm{res}}(z)\big)$. It targets settings where OBS already captures the hard part of the reward surface and EXP is mainly needed to calibrate a simpler discrepancy. When residual-CVCI hyperparameter CV scores tie within numerical tolerance, the tie is broken toward the more regularized candidate; Appendix (ref) and Appendix (ref) give the full tuning protocol and additional implementation details.
\paragraph{Family-level interpretation.} The six items above are estimator families with benchmark-specific instantiations. The proxy channel, the OBS baseline, the basis library, the pooling rule, and the EXP-side tuning protocol differ across benchmarks (see Table (ref) in Appendix (ref)). The empirical comparisons in Sections (ref)--(ref) diagnose those family-level instantiations.
Across all empirical benchmarks, the task is the same: estimate a reward surface, recommend the best action on each held-out context, and measure held-out recommendation regret. Two quantities recur throughout the benchmarks: $\beta$ controls the OBS choice law, and the real cached benchmarks replace the latent target $q$ with the finite cached target $q_{\mathrm{cache}}$. Each benchmark setting specifies a training-context distribution, a held-out test set, an action set, a simulator, an experimental reward map, and an observational logging family. Within one such cell, the observational law factors as \[ p_{\mathrm{OBS}}(x,u,a,m,y) = P_{\mathrm{tr}}(x)\,P(u\mid x)\,p_{\mathrm{OBS}}^{\beta}(a\mid x,u)\,p_{\mathrm{sim}}(m\mid x,a)\,P_Y(y\mid x,m,u), \] and the experimental law replaces the choice model by an unconfounded randomization rule \(\pi_{\mathrm{EXP}}(a\mid x)\): \[ p_{\mathrm{EXP}}(x,u,a,m,y) = P_{\mathrm{tr}}(x)\,P(u\mid x)\,\pi_{\mathrm{EXP}}(a\mid x)\,p_{\mathrm{sim}}(m\mid x,a)\,P_Y(y\mid x,m,u). \] Here \(P_Y\) is the benchmark-specific outcome law. In the main-text benchmark settings, \(\pi_{\mathrm{EXP}}(a\mid x)=1/|\mathcal A|\), and \(r^\star(x,m)=\mathbb{E}[Y\mid X=x,M=m,\mathrm{EXP}]\) integrates out \(U\mid X=x\) under \(p_{\mathrm{EXP}}\). The identified target remains \[ q(x,a)=\mathbb{E}_{M\sim p_{\mathrm{sim}}(\cdot\mid x,a)}[r^\star(x,M)]. \] Each estimator returns \(\widehat q\), inducing the recommendation rule \(\widehat\pi(x)=\arg\max_{a\in\mathcal A}\widehat q(x,a)\). We evaluate that rule by held-out recommendation regret, \[ \mathrm{Regret}_{\mathrm{test}} = \frac{1}{|\mathcal{X}_{\mathrm{test}}|} \sum_{x\in\mathcal{X}_{\mathrm{test}}} \Big[ \max_{a\in\mathcal{A}} q(x,a)-q(x,\widehat\pi(x)) \Big]. \]
The \(\beta\)-indexed observational choice law is benchmark-specific. In the appendix controlled validation, \(p_{\mathrm{OBS}}^{\beta}(a\mid x,u)=(1-\beta)\pi_X(a\mid x)+\beta\,\pi_U(a\mid u)\), where \(\pi_X\) is a context-driven policy and \(\pi_U\) is a latent-user policy. Thus \(\beta=0\) removes latent-user routing, whereas \(\beta=1\) uses only the latent-user component. Section (ref) instead uses a softmax router in which \(\beta\) scales the latent-user logit contribution.
The real cached benchmarks fix one realized mediator \(m_{\mathrm{cache}}(x,a)\) for each context-action pair and report the finite cached target \(q_{\mathrm{cache}}(x,a)=r^\star\!\bigl(x,m_{\mathrm{cache}}(x,a)\bigr)\). Their reported recommendation regret therefore replaces \(q\) by \(q_{\mathrm{cache}}\) on the held-out cache. Randomness then comes from OBS/EXP sampling and the benchmark's latent draws, not from regenerating mediators at evaluation time.
The controlled semi-synthetic validation uses the summarization task family but replaces judged rewards by a known latent reward generator. The validation keeps the SIM/EXP/OBS structure fixed while varying confounding strength \(\beta\), the observational budget \(n_{\mathrm{OBS}}\), and the randomized budget \(n_{\mathrm{EXP}}\). The validation connects the estimator theory to observable behavior. When EXP is scarce, OBS-assisted estimators can improve held-out recommendation performance. With a larger randomized sample, EXP-only and directly pooled estimators become competitive. No single estimator dominates every regime. The full generator, protocol, and \(24\)-cell winner map are in Appendix (ref) and Appendix (ref).
This controlled validation isolates how data budget and confounding strength change performance when the true interventional surface is known. The real summarization and coding benchmarks below then test the same estimator families on finite cached targets \(q_{\mathrm{cache}}\), where reward definition and auxiliary-feature alignment become the main explanatory variables.
The main paper reports the controlled validation and two real-task cached benchmarks with synthetic OBS/EXP resampling. The first cached benchmark is summarization on CNN/DailyMail\footnote{\url{https://huggingface.co/datasets/ccdv/cnn_dailymail}} Hermann2015CNN,Nallapati2016Sum,See2017Pointer. The second is coding on SWE-bench Verified\footnote{\url{https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified}}, a human-validated 500-instance subset of SWE-bench Jimenez2024SWEBench,OpenAI2024SWEBenchVerified, with candidate patches drawn from the BouncerBench public patch pool\footnote{\url{https://github.com/uw-swag/BouncerBench/releases/download/paper/all_patches.csv}} Mathews2025BouncerBench. Both benchmarks use real tasks, real candidate model outputs, and real LLM-judged or program-test rewards on those outputs. Their OBS and EXP mechanisms are benchmark constructions obtained by resampling and reweighting the same cached pool under specified choice and randomization rules. Table (ref) lists the source data, cached objects, and targets. All benchmarks use family-level instantiations of the estimators in Section (ref), and the reported results diagnose those families within the benchmark constructions.
The cached summarization benchmark uses CNN/DailyMail articles and LLM-judged rubric scores. Its OBS and EXP samples are constructed by resampling and reweighting the same cached pool under specified routing and randomization rules. The summarization benchmark fixes one judged cached summary for each article--model pair and evaluates two reward maps on that same cache (Table (ref)). Each cached summary carries a rubric vector \(s(x,a)=\bigl(s_{\mathrm{faith}}(x,a),\,s_{\mathrm{cov}}(x,a),\,s_{\mathrm{clar}}(x,a),\,s_{\mathrm{conc}}(x,a)\bigr)\in[0,1]^4\) for faithfulness, coverage, clarity, and conciseness, together with an unsupported-claims count \(u(x,a)\). Let \(\mathcal T\) denote the four user segments and let \(w_\tau\in\Delta^4\) be the segment-specific weight vector over these rubric dimensions. For a sharpening exponent \(\gamma\ge 1\) and unsupported-claims penalty \(\lambda\ge 0\), define \(\bar w_{\tau k}^{(\gamma)}=\frac{w_{\tau k}^{\gamma}}{\sum_{\ell=1}^4 w_{\tau \ell}^{\gamma}}\) and \(\ell_\tau^{(\gamma,\lambda)}(x,a)=\sum_{k=1}^4 \bar w_{\tau k}^{(\gamma)} s_k(x,a)-\lambda \frac{u(x,a)}{20}\), and write \([t]_{[0,1]}:=\min\{1,\max\{0,t\}\}\). The smooth map used in Tables (ref) and (ref) is \[ g_\tau^{\mathrm{sm}}(x,a)=\ell_\tau^{(1,0)}(x,a)=\langle w_\tau, s(x,a)\rangle, \qquad q_{\mathrm{cache}}^{\mathrm{sm}}(x,a)=\frac{1}{|\mathcal T|}\sum_{\tau\in\mathcal T} g_\tau^{\mathrm{sm}}(x,a). \] The sharpened map used in Table (ref), with \(\sigma(t)=(1+e^{-t})^{-1}\), is \[ g_\tau^{\mathrm{sh}}(x,a) = \sigma\!\left( \frac{[\ell_\tau^{(16,0.1)}(x,a)]_{[0,1]}-0.9}{0.02} \right), \qquad q_{\mathrm{cache}}^{\mathrm{sh}}(x,a)=\frac{1}{|\mathcal T|}\sum_{\tau\in\mathcal T} g_\tau^{\mathrm{sh}}(x,a). \]
OBS and EXP are sampled only from training contexts. User segments are drawn uniformly from \(\mathcal T\), EXP randomizes uniformly over \(\mathcal A\), and OBS uses the \(\beta\)-indexed router defined by setting \(\zeta_a(x,\tau)=\eta_a(x)+\beta\,\xi_a(\tau)\) and writing
where \(f_{\mathrm{ctx}}(x)\) is a fixed article feature vector, \(e(\tau)\in\{0,1\}^{|\mathcal T|}\) is the one-hot segment code, and \(c_a\) is the normalized cost of model \(a\). The term \(\eta_a(x)=\langle \theta_a^{X},f_{\mathrm{ctx}}(x)\rangle-\kappa c_a\) is an article-dependent logit, whereas \(\xi_a(\tau)=\langle \theta_a^{U},e(\tau)\rangle\) is a segment-affinity logit.
In this benchmark, \(\beta\) changes only the OBS routing policy: \(\beta=0\) removes the latent-user term, and larger \(\beta\) makes routing more segment-specific. Reported regret is computed on the finite cached target \(q_{\mathrm{cache}}^{\mathrm{sm}}\) or \(q_{\mathrm{cache}}^{\mathrm{sh}}\). All reported numbers are means over 30 seeds (held-out split, OBS/EXP resamples, and EXP randomization combined). Appendix (ref) gives the exact slate construction, judged-reward aggregation, split details, the budget grid, and per-cell standard errors.
In the cached summarization benchmark, $\CVCI{}$ has the best average rank, top-3 coverage, and excess regret across 30 seeds and 48 settings (Table (ref)). The gap to the next family (about 0.04 in excess regret to $\OBSOnly{}$) is small relative to the spread of cell-level standard errors reported in Appendix (ref), so this result reads as family-level evidence within this benchmark instantiation. The remaining tables explain that pattern. Table (ref) shows that increasing \(n_{\mathrm{EXP}}\) lowers regret for the EXP-based estimators much more than increasing \(n_{\mathrm{OBS}}\) under the smooth reward. That pattern matches the estimator view in Table (ref): once EXP and SIM identify the target, additional randomized labels can improve EXP-side reward fits. Table (ref) then changes the target reward while holding the cached outputs fixed. The regret increase and the drop in rubric-linear fit serve as an empirical diagnostic of target alignment rather than a formal oracle statement.
In Table (ref), the two held-out \(R^2\) values are out-of-sample linear fits of the cached target surface using the auxiliary proxy representation and the raw rubric vector \(s(x,a)\), respectively. Table (ref) reports model-level value RMSE for DM and DR. In this benchmark, DR mainly helps OBS-heavy methods; DM remains stronger for the other families. Appendix (ref) gives the budget-specific breakdown and implementation details.
\paragraph{Benchmark-instantiation reading.} In this cached summarization benchmark, the raw judged dimensions already give a high held-out \(R^2\) for the target reward (Table (ref)). Table (ref) shows two patterns: $\CVCI{}$ already attains the lowest regret at $n_{\mathrm{EXP}}=20$ and stays near that level as $n_{\mathrm{EXP}}$ grows, while $\EXPOnly{}$ and $\ProxyEXP{}$ improve substantially with more EXP labels and approach the $\CVCI{}$ floor at $n_{\mathrm{EXP}}=200$. Additional EXP labels mainly help the EXP-anchored families catch up to the OBS-pooled $\CVCI{}$ floor rather than lower that floor.
The cached coding benchmark uses SWE-bench Verified issues Jimenez2024SWEBench,OpenAI2024SWEBenchVerified and one cached candidate patch for each issue--agent pair (Table (ref)). Candidate patches come from BouncerBench Mathews2025BouncerBench, and the agent baselines follow the SWE-agent setup YangEtAl2024SWEAgent. The cached patches and program-test outcomes are real, and the OBS and EXP mechanisms are benchmark constructions over the cached pool. Evaluation averages over a fixed set of benchmark user types and reports regret on the finite cached target \(q_{\mathrm{cache}}\).
Each cached patch is scored by four components \(c_1,c_2,c_3,c_4\), where \(c_1\) records true fix success and \(c_2,c_3,c_4\) summarize patch size, locality, and issue focus. These patch-quality components can disagree with true fix success, which is why changing the reward definition matters here.
Method rankings change as the reward shifts weight from patch quality to true fix success. The shift is controlled by \(\alpha_{\mathrm{fix}}\), which weights true fix success, and by \(\omega_{\mathrm{weak}}\), which determines whether the non-success components are aggregated additively or by a weakest-link rule. The first, \(\alpha_{\mathrm{fix}}\), controls how much weight the reward places on true fix success. The second, \(\omega_{\mathrm{weak}}\), controls whether the non-success components are aggregated additively or through a weakest-link rule.
For user type \(u\) with weights \(w_u=(w_{u1},w_{u2},w_{u3},w_{u4})\), normalized non-success weights are \(\bar w_{uk}=w_{uk}/(w_{u2}+w_{u3}+w_{u4})\) for \(k\in\{2,3,4\}\), and \(g_u(c_2,c_3,c_4)=(1-\omega_{\mathrm{weak}})\sum_{k=2}^4 \bar w_{uk}c_k+\omega_{\mathrm{weak}}\min\{c_2,c_3,c_4\}\). The benchmark utility for user type \(u\) is \(Y_u(x,a)=w_{u1}\,\alpha_{\mathrm{fix}}\,c_1(x,a)+(1-w_{u1})\,g_u\!\big(c_2(x,a),c_3(x,a),c_4(x,a)\big)\), and the evaluation target is the cached user-average surface \(q_{\mathrm{cache}}(x,a)=\mathbb{E}\!\left[Y_U(x,a)\mid X=x\right]\). \RepStruct maps each issue--patch pair into an auxiliary representation learned from observational patch statistics and fits EXP rewards on that representation. Appendix (ref) gives the public-data construction and the complete \(5\times 5\) results table.
With the data budget fixed, the ranking changes because the reward puts more or less weight on true fix success. The crossover occurs near \(\alpha_{\mathrm{fix}}\approx 0.25\): below that point \RepStruct is among the lowest-regret methods, and above it the best OBS-based methods overtake it. On the additive slice, the regret gap between \RepStruct and the better of \OBSStruct and \CVCIStruct moves from \(-0.0039\) at \(\alpha_{\mathrm{fix}}=0\) to \(+0.0003\) at \(\alpha_{\mathrm{fix}}=0.25\) and \(+0.0415\) at \(\alpha_{\mathrm{fix}}=1\). At \(\omega_{\mathrm{weak}}=0.25\), the corresponding values are \(-0.0034\), \(+0.0024\), and \(+0.0375\). Figure (ref) (left) shows the same crossover on the full \(5\times 5\) grid. This sweep tests whether the auxiliary patch statistics retain the target signal as the target moves from patch quality toward true fix success.
Held-out \(R^2\) comes from linear fits of the cached target using either the auxiliary patch statistics or \(c_1\) alone; Appendix (ref) gives the exact protocol. On the additive slice, the auxiliary-feature fit drops from \(R^2=0.347\) at \(\alpha_{\mathrm{fix}}=0\) to \(0.081\) at \(\alpha_{\mathrm{fix}}=1\), while the \(c_1\)-only fit rises from essentially zero to \(R^2=0.967\). The $R^2$ shift matches the change in method ranking: \RepStruct helps when the target still follows patch-quality features, but it loses that advantage once the reward is driven mainly by fix success.
Changing \(\omega_{\mathrm{weak}}\) changes only how the non-success components are combined. Its largest effect is on \GroundedStruct: at zero fix-success weight, the grounded-minus-OBS regret gap shrinks from about \(+0.0495\) under additive scoring to about \(+0.0126\) under the pure weakest-link rule. Figure (ref) (right) shows the same effect. Even so, \GroundedStruct remains worse than the best OBS-based method throughout this fixed-budget setting.
\paragraph{Benchmark-instantiation reading.} In this fixed-budget coding probe over the BouncerBench cached patch pool, \RepStruct is the most competitive family in the regions where the auxiliary representation predicts the held-out cached target better than fix success alone (Figure (ref), low-\(\alpha_{\mathrm{fix}}\) cells). Once \(c_1\) alone predicts the target at least as well, the OBS-based families overtake \RepStruct on held-out regret in the same probe (Figure (ref)). The pattern is a target-alignment diagnostic for this fixed-budget coding probe. Choosing among families in deployment would still require a live randomized trial.
The paper combines three literatures: off-policy evaluation, hybrid estimation from observational and randomized data, and LLM evaluation. Each covers only part of the setting. The setting combines confounded logging on real tasks, replayable text mediators that can be regenerated for any candidate model, and expensive outcome labels that are unbiased only inside a small randomized experiment.
\paragraph{Off-policy evaluation.} \DMFull and \DRFull are standard estimators in contextual-bandit and policy-evaluation settings Dudik2011DoublyRobust,Dudik2014DRStatSci,Swaminathan2015CRM,Jiang2016DROffPolicy,Thomas2016DataEfficient, and they descend from the foundational doubly robust and missing-data lineage of RobinsRotnitzkyZhao1994 and BangRobins2005. The OPE setting these papers analyze starts from logged action--outcome pairs in which outcomes are assumed unconfounded given context: the central technical issue is action coverage under the logging policy, and the recorded outcomes faithfully represent the causal target. In our setting, even for observed actions, the recorded outcomes are confounded because user-side factors influence both who selects which model and how its output is judged. Because SIM can regenerate outputs for any model on any held-out context, DM and DR enter only after the reward surface itself has been estimated. We use SIM and EXP to identify the causal scoring rule and then use OBS to reduce variance.
\paragraph{Experimental grounding and hybrid estimation.} The closest methodological antecedents are experimental grounding Kallus2018ExperimentalGrounding, data-fusion and transportability formalisms PearlBareinboim2011Transportability,BareinboimPearl2016DataFusion, and trial-to-target generalizability ColeStuart2010. A related line of work studies how to combine large biased samples with smaller randomized ones Rosenman2023Shrinkage,Cheng2021Adaptive,Lin2025PowerLikelihood,Yang2025CVCI,Colnet2024Review. That literature asks when observational signal can be used to reduce variance without giving up the causal credibility of the randomized sample; in those papers, the unit-level outcome is observed on both samples and the modeling work is done on a fixed outcome label. The estimator families in Section (ref) are concrete instantiations of the same bias--variance question. Our setting differs in one specific way: the target outcome is not directly observed even on the randomized sample because the unit “output” must first be regenerated by SIM and then scored. Identification in our setting depends on SIM validity (Assumption A2) together with EXP randomization.
\paragraph{Summarization and LLM-based evaluation.} Our experiments use CNN/DailyMail contexts Hermann2015CNN,Nallapati2016Sum,See2017Pointer,Stiennon2020SummarizeHF and rely on LLM-judged rubric scores in the pool-based study. Recent LLM-evaluation work covers benchmark design Liang2023HELM, judge reliability and pairwise preference platforms Zheng2023LLMJudge,Chiang2024ChatbotArena, and real-world usage logs Zheng2023LMSYSChat1M,Zhao2024WildChat. It also documents biases in automatic evaluators, including positional unfairness WangEtAl2023FairEvaluators, length bias Dubois2024LengthControlledAlpacaEval, and self-preference PanicksseryBowmanFeng2024, and studies panel-based mitigations VergaEtAl2024Juries. Preference data is central in alignment pipelines, both with explicit learned reward models Christiano2017HumanPreferences,Stiennon2020SummarizeHF,Ouyang2022InstructGPT and with implicit reward formulations such as direct preference optimization Rafailov2023DPO. The LLM-evaluation work cited above primarily treats judged scores as outcome data and studies how reliable those scores are and how to average them. We use cached outputs and judged scores as inputs to a causal evaluation problem; they become the causal target only after EXP-identified scoring.
A complementary line of work in causal NLP studies how text can serve as treatment, outcome, mediator, or proxy for unobserved confounders FederEtAl2022CausalNLP; OBS-derived auxiliary labels in our setting play exactly the proxy role identified there.
Taken together, the setting sits across the three literatures rather than fitting cleanly inside any one of them.
Ranking diagnostics can diverge sharply from recommendation regret. In the cached summarization benchmark, \Grounded wins every top-1 and top-3 ranking setting but wins regret in only 6 settings. Regret is driven by a small number of costly recommendation mistakes that average ranking metrics do not isolate. The benchmarks therefore center $\mathrm{Regret}_{\mathrm{test}}$, with context-action RMSE and model-level RMSE as secondary diagnostics.
\paragraph{What our cached benchmarks are and are not.} The two real-task benchmarks use real candidate model outputs and real LLM-judged rubric or program-test scores on those outputs. Their OBS and EXP mechanisms are benchmark constructions over the cached pool. The empirical comparisons therefore diagnose how the estimator families behave under controlled synthetic OBS/EXP resampling on real tasks.
\paragraph{Scope of the empirical benchmarks.} The empirical scope is deliberately narrow: cached summarization isolates supervision scarcity, and cached coding isolates reward geometry. Family-level wins in these benchmarks are benchmark-specific diagnostics.
\paragraph{Methodological implications.} Estimator choice depends on what limits performance in the benchmark at hand. In practice, the key checks are whether more EXP labels materially improve EXP-based methods and whether a structured representation predicts the held-out target better than a simple success-only baseline. Deployment decisions should still be confirmed by live randomized evaluation.
\paragraph{Limitations and future work.} These conclusions are specific to the task families and benchmark constructions studied here. Future work includes broader tasks, repeated mediator draws, naturally observed deployment logs paired with in-deployment randomized trials, and multi-round human--agent interaction, which would require relaxing or replacing Assumption A4.
Offline evaluation separates into identification and post-identification estimation. In the three-source design, the simulator and the randomized experiment together suffice to identify causal model values; the observational log helps only with estimation afterward, when its signal genuinely matches the causal target.
Across the controlled semi-synthetic validation and the two real-task cached benchmark constructions, different estimator families win in different regimes on held-out recommendation regret. The two benchmark families point to the same practical lesson: which estimator works best depends on both the amount of experimental supervision and on what the reward is actually trying to capture.
The results apply to one-shot cached-evaluation settings, not yet to richer interactive settings with repeated feedback and broader population-level uncertainty. Future work should extend the framework beyond one-shot cached evaluations to richer interactive settings and live deployment with randomized evaluation.