Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
39,440 characters · 11 sections · 11 citation commands
HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
\let\arabic{footnote}\relax \footnotetext{
} \let\arabic{footnote}\arabic
\thispagestyle{empty}
\thispagestyle{plain}\setcounter{page}{1}
\doublespacing
Human evaluation is at the heart of developing and deploying reliable generative AI models chiang2024chatbot. Many organizations rely heavily on human annotations to evaluate model output quality, assess safety, calibrate moderation systems, and compare candidate model versions before release snow2008cheap,northcutt2021confident,frenay2013classification. Small differences in estimated model quality, even on the order of a few percentage points in win rate or defect rate, determine which model is shipped to hundreds of millions of users weidinger2025toward,stein2023exposing.
As widely documented in natural language processing and related fields, a key challenge within an annotation task is that expert-annotated “gold" labels (a.k.a. ground truth labels) are expensive and scarce snow2008cheap, so organizations often instead rely on judgments from a pool of less specialized “silver" annotators whose labeling reliability varies. The so-called gold labels may be produced by a single expert annotator or by consensus among multiple experts within the organization, and they serve as the best available reference standard. In contrast, large-scale evaluations typically rely on silver labels collected from broader, less specialized sources, including crowd or vendor annotators and automated judges such as LLMs gu2024survey or VLMs chen2024mllm. Throughout this paper, we view silver labels as the primary operational labels used in evaluation, and gold labels as a more accurate but limited supervision source used to assess and calibrate them. Because silver annotators differ in reliability, simply aggregating their labels without adjustment can reduce the reliability of evaluation and introduce bias in the estimated model performance.
Many methods have been developed to account for annotator variability DawidSkene1979,snow2008cheap,whitehill2009whose,raykar2010learning, but most operate within a single evaluation round and do not fully leverage the unique structure of modern model evaluation settings. We note that two novel features distinguish modern AI model evaluations from classical annotation tasks. First, organizations that evaluate generative models accumulate large-scale historical annotation data across prior evaluation rounds. These datasets often include repeated observations of annotator behavior and subsets of instances with gold labels, which together reveal patterns in labeler reliability. Second, the historical and current model evaluation tasks may share content that transfers across rounds, such as prompt features. Generation prompts often persist across rounds because organizations evaluate successive model versions on a stable prompt distribution drawn from real user traffic or curated benchmarks chiang2024chatbot, so the same prompt features that appeared in earlier rounds also characterize the current evaluation set. With these two features, in principle, labeler reliability learned from historical gold labels can calibrate current annotations, and the shared generation prompts available in a large historical sample can stabilize current model evaluation estimates. We defer a more complete comparison with existing literature to Appendix F.
To fully exploit historical data for reliable model evaluation, we propose HERO (History Enhanced RObust model evaluation), a framework that adjusts noisy annotation-based estimators with historical evaluation data to reduce both bias and variance in model performance estimates. The key contributions of our work include:
We organize common model evaluation scenarios into a unified observational structure. Consider $n$ instances indexed by $i = 1, \ldots, n$. For each instance, $Y_i^*$ denotes the operational ground-truth gold label, the best available adjudicated assessment of the generated output, and $Z_i$ denotes the noisy silver labels collected from less specialized annotators or automated AI systems as proxies for $Y_i^*$. The target parameter is a function of $Y_i^*$, and the task-specific outcome $Y_i$ is constructed from $Y_i^*$ according to the scenario. Table (ref) summarizes four evaluation scenarios, listing the outcome variable $Y_i$, the corresponding target parameter, and the available covariates for each. We summarize each case in Table 1 below and defer its detailed explanation to the appendix.
Estimating the target parameter $\mathbb{E}[Y_i]$ from gold labels alone is rarely feasible in practice. This is because many evaluation tasks provide no gold labels at all. Even when a subset of instances is annotated by a gold labeler, the labeled fraction is typically small. The sample mean over this subset is unbiased, but its variance remains too large to resolve the differences in model performance that guide practical decisions. For these reasons, practitioners often construct a proxy for $Y_i$ by aggregating the available silver labels $Z_i$. Let $\widehat{\mu}$ denote an estimator of $\mathbb{E}[Y_i]$ constructed from a silver-labeler-based proxy. To assess estimator quality, we decompose the mean-squared error (MSE) into bias and variance:
Both components introduce distinct challenges. We start by examining the bias component through two sources that arise in generative model evaluation. The first source of bias is annotator heterogeneity. The inter-rater reliability literature consistently finds variation in annotator accuracy across domains from medical imaging Warfield2004, Raykar2010 to natural language processing Snow2008, even under shared guidelines Cohen1960, Fleiss1971, DawidSkene1979, Whitehill2009. Differences in expertise, interpretation standards, and decision thresholds cause some labelers to favor precision over recall, while others show the opposite tendency. Aggregation without calibration, such as the majority vote-based estimators, allows these systematic tendencies, particularly from lower-quality labelers, to shift the estimate of $\mathbb{E}[Y_i]$ away from the true target.
As shown in Figure 1 Panels A-C, the seven silver labelers in our benchmarking data span agreement rates from below 0.4 to above 0.8, and the sensitivity $\alpha_j$ and specificity $\beta_j$ panels reveal asymmetric error profiles behind that spread, with some labelers favoring positives, others favoring negatives, and a third group noisy on both axes. A majority-vote estimator weights these labelers equally and therefore inherits the mixture rather than the gold-labeler target.
A second source of bias arises when the composition of the labeler pool changes between evaluation rounds. For example, in comparative evaluation tasks (Use Case 2 in Table (ref)), organizations may evaluate a newer model against an earlier version at different times. The labeler pools that annotate each model may only partially overlap. A reliable annotator who contributes a large fraction of Model A labels may complete only a small fraction of Model B labels, and vice versa. When the proportion of reliable annotators differs between the two rounds, the observed label difference reflects both true differences in model quality and shifts in the composition of the labeler pool. An estimator that treats the two label sets as exchangeable confounds these two sources of variation. Figure 1 Panel D illustrates this shift in our benchmarking data. Two labelers active in the Model A round do not appear in the Model B round, and the labelers active in both rounds contribute different proportions of labels across the two models.
The variance component of the MSE in Equation (ref) remains a challenge even after addressing the bias issues mentioned above. Even with a well-calibrated silver label based proxy, a small labeled set or high labeling outcome heterogeneity across instances can produce confidence intervals too wide to resolve the performance gaps that guide practical decisions. A variance reduction technique that halves the estimator variance is equivalent to doubling the effective sample size, so that an evaluation previously requiring 1{,}000 labeled instances can reach the same precision with 500, cutting annotation cost and turnaround time proportionally.
In this paper, we propose a unified framework to reduce both bias and variance for an accurate (improved reliability) and efficient (improved sensitivity) estimation of the target parameter $\mu$. Achieving both goals requires an external source of information beyond the only current silver labels. As organizations that evaluate generative models often maintain historical evaluation datasets from prior rounds, our framework leverages these historical data to construct an estimator of $\mu$ with reduced MSE. We term the proposed framework History Enhanced RObust (HERO) model evaluation and organize it as a two-step procedure that first reduces the bias and then controls the variance.
At a very high level, HERO works as follows. When we analyze current evaluation data with only silver labels, HERO first constructs calibrated gold proxies that weight each silver labeler by the reliability learned from historical data, yielding a bias-reduced estimator $\widehat{\mu}_{\text{HERO-1}}$. HERO then incorporates historical covariates (e.g., prompt features) as control variates to further reduce the variance of $\widehat{\mu}_{\text{HERO-1}}$, yielding the final estimator $\widehat{\mu}_{\text{HERO-2}}$. This estimator satisfies:
To distinguish historical data from the current evaluation task, we introduce a consistent notation. We use a tilde to denote quantities associated with historical data. The historical dataset consists of $N$ instances collected from earlier evaluation rounds and is defined as follows:
Meanwhile, the current evaluation dataset contains $n$ instances, where $n \ll N$:
Throughout this section, all outcomes and labels are binary, so that $Y_i \in \{0,1\}$ and $\widetilde{Y}_i \in \{0,1\}$. The target parameter remains $\mu = \mathbb{E}[Y_i]$ as defined in Section (ref), but gold label $Y_i$ is unobserved.
Scope of historical data. Our framework allows two kinds of mismatch between current and historical data. First, the current annotator pool can differ, but it is a subset of the labelers who contributed to the historical data. Second, the historical task itself may differ from the current task, since HERO calibrates labeler reliability rather than item-level labels.
To account for model evaluation bias introduced by silver labeler heterogeneity, historical data provide a natural source for revealing persistent patterns in labeler behavior. HERO Step 1 extracts these reliability patterns from the historical data and transfers them to the current evaluation task, where no gold labels are available. More concretely, we characterize each silver labeler $j$ by two parameters that describe intrinsic properties of that labeler's decision process,
Here $\alpha_j$ is the sensitivity and $\beta_j$ is the specificity. Because these parameters describe the labeler rather than the task, they are expected to transfer from the historical dataset to the current dataset even when the underlying evaluation content/task differs.
HERO estimates $(\alpha_j, \beta_j)$ for each labeler $j$ from the historical dataset $\{(\widetilde{X}_i, \widetilde{R}_i\widetilde{Y}_i, \widetilde{Z}_i, \widetilde{R}_i, \widetilde{A}_{ij})\}_{i=1}^{N}$ using an EM algorithm that treats unobserved gold labels as latent variables, and use observed gold labelers as anchors. Given a candidate value $y \in \{0,1\}$ for the gold label, the likelihood of the observed silver labels for historical instance $i$ is
where $P(\widetilde{Z}_{ij} \mid \widetilde{Y}_i = 1) = \alpha_j^{\,\widetilde{Z}_{ij}}\,(1 - \alpha_j)^{1 - \widetilde{Z}_{ij}}$ and $ P(\widetilde{Z}_{ij} \mid \widetilde{Y}_i = 0)= (1 - \beta_j)^{\widetilde{Z}_{ij}}\,\beta_j^{\,1 - \widetilde{Z}_{ij}}$. An instance without a gold label ($\widetilde{R}_i = 0$) contributes the marginal $\widetilde{\mu}\,\widetilde{\ell}_i(1) + (1-\widetilde{\mu})\,\widetilde{\ell}_i(0)$, where $\widetilde{\mu} = \mathbb{E}[\widetilde{Y}]\neq \mathbb{E}[Y]$. The full historical log-likelihood is
To stabilize the EM updates, HERO initializes the parameters using only the gold-labeled subset $\mathcal{G}=\{i:\widetilde{R}_i=1\}$. Specifically, we set \[ \widetilde{\mu}^{(0)}=|\mathcal{G}|^{-1}\sum_{i\in\mathcal{G}}\widetilde{Y}_i, \qquad \alpha_j^{(0)} = \frac{\sum_{i\in\mathcal{G}}\widetilde{A}_{ij}\widetilde{Y}_i\widetilde{Z}_{ij}} {\sum_{i\in\mathcal{G}}\widetilde{A}_{ij}\widetilde{Y}_i}, \qquad \beta_j^{(0)} = \frac{\sum_{i\in\mathcal{G}}\widetilde{A}_{ij}(1-\widetilde{Y}_i)(1-\widetilde{Z}_{ij})} {\sum_{i\in\mathcal{G}}\widetilde{A}_{ij}(1-\widetilde{Y}_i)}. \] HERO then maximizes $\widetilde{\mathcal{L}}$ via the EM algorithm (see Appendix Algorithm 1, Stage A), producing estimates $(\widehat{\alpha}_j,\widehat{\beta}_j)$ that are treated as fixed when estimating $\mu$ on the current data.
The current-data likelihood $l_i(y)$ follows the same structure as defined in (ref), with $(Z_{ij}, A_{ij})$ replacing $(\widetilde{Z}_{ij}, \widetilde{A}_{ij})$ and the estimates $(\widehat{\alpha}_j, \widehat{\beta}_j)$ replacing the unknown parameters. Because no gold labels are observed, every instance contributes through the marginal form. The HERO Step 1 estimator maximizes the log-likelihood over $\mu \in (0,1)$,
The first-order condition is
which can be solved by any standard one-dimensional optimizer because $\mu$ is scalar. Here, $\psi_i(\mu)$ denotes the individual score contribution from instance $i$.
On top of bias reduction, HERO can further reduce estimator variance by leveraging auxiliary covariates that are shared between the historical and current datasets. Prompt features are a natural shared covariate, since organizations typically evaluate successive model versions on a stable prompt distribution drawn from real user traffic or curated benchmarks, and the same prompt features that appeared in earlier rounds also characterize the current evaluation set.
The key observation is that the historical dataset ($N \gg n$), being much larger than the current evaluation sample, provides a substantially more precise estimate of the covariate mean. This historical mean can therefore serve as a stable reference in a control-variate adjustment without changing the target parameter.
Yet, the classical control variate adjustment does not transfer directly in our task, because the Step 1 estimator is a calibrated functional of EM posteriors rather than a sample mean. We thus propose to apply the control variate adjustment to the estimating equation that defines the Step 1 HERO estimator, with a shared covariate function whose population mean is supplied by the historical sample. Let $X_i$ denote a covariate observed in the current evaluation task, and let $ \widehat{m}_X = \frac{1}{N} \sum_{r=1}^N \widetilde{X}_r$ denote the sample mean of the same covariate in the historical data. We construct a control-variate estimator based on the score equation in (ref). After obtaining the pilot HERO estimator $\widehat{\mu}_{\textsc{hero-1}}$ from Step 1, define the estimated score contribution for instance $i$ as
We then estimate the control-variate coefficient by regressing the pilot score $\widehat \psi_i$ on $X_i$:
where $\widehat{\mathrm{Var}}(X) = \tfrac{1}{n}\sum_{i=1}^n (X_i-\bar X)(X_i-\bar X)^\top$ and $\widehat{\mathrm{Cov}}(X,\widehat{\psi}) = \tfrac{1}{n}\sum_{i=1}^n (X_i-\bar X)\widehat{\psi}_i$.
Whenever the covariate mean is shared between the historical and current datasets, the difference $\bar X-\widehat{m}_X$ is asymptotically mean zero. Therefore, the adjustment term $\widehat{\gamma}^\top(\bar X-\widehat{m}_X)$ can be used to reduce variance without changing the asymptotic expectation of the estimator. We define the control-variate adjusted estimator $\widehat{\mu}_{\textsc{hero-2}}$ as the solution to
As in Step 1, this equation can also be solved using any standard one-dimensional optimizer.
This adjustment reduces variance whenever the covariate $X_i$ is correlated with the score contribution $\psi_i$, which quantifies how each observation influences the estimator. If $X_i$ carries little information about the outcome, then $\widehat{\gamma}$ is close to zero, and the estimator effectively reduces to $\widehat{\mu}_{\textsc{hero-1}}$. Consequently, HERO Step 2 improves efficiency when informative auxiliary covariates are available, while leaving the estimator essentially unchanged when they are not.
On top of being practical and easy to implement, HERO enjoys theoretical guarantees along three dimensions: reliability (Theorem 1), sensitivity (Theorem 2), and valid statistical inference (Theorem 3). Reliability means that HERO accurately targets the gold-label estimand, rather than a proxy estimand distorted by heterogeneous silver labelers. Sensitivity means that HERO reduces variance, which may improve the detection of small but practically meaningful model-performance differences. Valid statistical inference means that the estimator admits an asymptotic distribution, providing the basis for credible confidence intervals. Due to space limit, we defer the formal theorem statements and proofs to Appendix C.
In this section, we evaluate the performance of HERO method against several benchmark methods. We summarize our takeaway as follows: First, HERO-1 remains accurate even when the historical gold-label rate is small, whereas the sample-proportion calibration baseline is unstable when only a few gold-routed examples are available per labeler. This shows HERO-1 can extract useful reliability information from the full historical label matrix, not only from the gold-labeled subset. Second, HERO-2 further improves sensitivity by reducing variance and RMSE relative to HERO-1 when the shared covariate is informative. Third, under increasing labeler heterogeneity, HERO-1 and HERO-2 remain substantially more robust than baseline methods, confirming that historical reliability calibration and score-level control variates reduce estimation errors. Due to the page limit, we defer the data generating process details to Appendix. In what follows, we introduce the methods in comparison, evaluation metrics, and simualtion study results.
Methods in comparison. We compare five estimators: (1) Majority vote. (2) Dawid--Skene method with current data only: The current-only Dawid--Skene baseline fits the standard latent-class model using the current silver labels alone, jointly estimating $\mu$ and the labeler-specific parameters $\{(\alpha_j,\beta_j)\}_{j=1}^{J}$. (3) Historical sample-proportion calibration. We estimate each labeler's sensitivity and specificity using only the gold-labeled subset of the historical data. With Laplace smoothing. We then maximize the current-data likelihood over $\mu$ with these calibration estimates fixed. (4) HERO-1 only use historical data for bias reduction. (5) HERO-2 applies the score-anchored control-variate adjustment to HERO-1. In this simulation, the population mean $\mathbb{E}[X]=0$ is known, but we use $\bar X_{\mathrm{hist}}$ to mirror the practical setting in which HERO relies on a high-precision historical estimate rather than oracle knowledge. Appendix Table (ref) summarizes the simulation settings. The target $\mu_0$ is computed from the data-generating model, $\mu_0=\mathbb{E}_{X}\{\sigma(\beta_0+\beta_1X)\}$, and is fixed within each experimental setting.
Experiment 1: Historical gold-label rate. The first experiment varies the amount of historical gold supervision. In Setting (ii), we use a larger labeler pool and a stronger covariate signal: $n=300$, $N=800$, $J=10$, and $\beta_1=3$. We vary $\rho\in\{0.05,0.06,\ldots,0.10\}$. Experiment 2: Labeler heterogeneity. The second experiment studies robustness to heterogeneity in silver-labeler accuracy. We use the same baseline configuration as Experiment 1 Setting (ii), with $n=300$, $N=800$, $J=10$, $\beta_0=-0.2$, and $\beta_1=3$. We hold the mean reliability fixed at $\mu_\alpha=0.85$ and $\mu_\beta=0.55$, and vary only the Beta concentration parameter $k$. Under the parametrization $ \alpha_j\sim \mathrm{Beta}\{k\mu_\alpha,k(1-\mu_\alpha)\}, \ \beta_j\sim \mathrm{Beta}\{k\mu_\beta,k(1-\mu_\beta)\}, $ the means satisfy $\mathbb{E}[\alpha_j]=\mu_\alpha$ and $\mathbb{E}[\beta_j]=\mu_\beta$ for every $k$, while $ \mathrm{Var}(\alpha_j)=\frac{\mu_\alpha(1-\mu_\alpha)}{k+1}, \ \mathrm{Var}(\beta_j)=\frac{\mu_\beta(1-\mu_\beta)}{k+1}. $ Thus, decreasing $k$ increases labeler heterogeneity without changing the average sensitivity or specificity. We consider the five heterogeneity levels in Appendix Table (ref). For each level, we evaluate historical gold-label rates $\rho\in\{0.05,0.10\}$.
Evaluation metrics. For each configuration, we run $B=200$ Monte Carlo replications. We report absolute bias, Monte Carlo standard deviation, and root mean squared error (RMSE). All EM algorithms use convergence tolerance $10^{-6}$ and a maximum of 200 iterations.
Simulation study results. Figure (ref) shows that Majority vote has nearly constant bias as $\rho$ varies, since it does not use historical gold labels. The current-only Dawid--Skene baseline also does not improve with $\rho$ and remains limited by the identifiability and finite-sample challenges of estimating all labeler parameters from the small current round alone. The historical sample-proportion baseline improves as $\rho$ increases, but it is unstable when the number of gold-routed examples per labeler is small. In contrast, HERO-1 is substantially less sensitive to low gold-label rates because its anchored EM step uses both the gold-routed and ungolded historical instances. HERO-2 further reduces SD and RMSE by exploiting the correlation between $X_i$ and the score contribution, with larger variance reduction in the stronger-signal setting. We provide additional results in Appendix Figure (ref).
Figure (ref) shows that majority vote becomes increasingly unreliable as labeler heterogeneity grows, because it assigns equal weight to labelers with different error profiles. The current-only Dawid--Skene baseline partially adapts to heterogeneous reliability but remains unstable because the current sample is small and unanchored by gold labels. Historical sample-proportion calibration improves when enough gold labels are available, but it can still have high variance when each labeler receives few gold-routed examples. HERO-1 maintains low bias across heterogeneity levels by pooling information from the full historical label matrix, and HERO-2 achieves the lowest RMSE by combining this calibration with score-level variance reduction.
In this section, we evaluate the proposed HERO method on two real-world datasets provided by a technology company that routinely conducts large-scale generative model evaluations. The datasets were collected to assess the performance of 3D asset generation models and contain both historical and current evaluation data. The case studies cover two representative tasks: generation defect evaluation and safety evaluation. In each task, every prompt--asset pair is evaluated by a small number of randomly selected silver labelers according to the task-specific evaluation objective, and a random subset of pairs is additionally evaluated by a gold labeler, who is a well-trained expert. To assess statistical performance and robustness, we use a resampling-based approach to report absolute bias, SD, and RMSE for the five estimators described in Section (ref). Additional details on the real-world datasets are provided in Appendix E.
Table (ref) and Figure (ref) show that HERO method reduces absolute bias and RMSE relative to the baseline methods. This improvement is driven by the presence of labeler heterogeneity and labeler-pool shift between the historical and current datasets, as illustrated in Figure (ref). HERO is designed to exploit precisely these sources of information by estimating historical labeler reliabilities and transferring them to the current evaluation task. Furthermore, HERO-2 achieves additional variance reduction through control variates. In Case Study I, shown in Table (ref), HERO-2 uses GPT-5-predicted labels as the control variate, as all generated contents were evaluated by the same model. In Case Study II, shown in Figure (ref), HERO-2 uses TF-IDF features as the control variate. In both cases, HERO-2 reduces the SD of HERO-1 by approximately 5%--10%, demonstrating that the proposed variance-reduction framework is effective across different choices of control variates.
In contrast, the classical Dawid--Skene baseline does not effectively learn labeler heterogeneity in either case. As a result, it performs worse than HERO and is often closer to Majority Vote in terms of bias and RMSE.
Limitation. Despite these promising real-world results, HERO Step 1 requires labeler reliability to transfer from historical data to the current evaluation task and Step 2 requires the shared covariates to have comparable distributions across the two periods. We view these requirements as mild, but they may not hold uniformly across model evaluation settings.