Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
91,205 characters · 18 sections · 136 citation commands
Calibeating Prediction-Powered Inference
We study semisupervised mean estimation with one labeled dataset \(\{(X_i, Y_i)\}_{i=1}^n\) and one unlabeled dataset \(\{\widetilde X_i\}_{i=1}^N\), where outcomes are observed only for a subset sampled completely at random. This is a classical missing-outcome problem: rich covariate information is available for many individuals, but the outcome of interest is observed only for a smaller subset. Such settings arise when gold-standard outcomes are expensive to collect but prediction scores are readily available, for example in randomized trials with prognostic models trained on historical data schuler2022increasing, biobank analyses using phenotype predictors built from EHR or omics measurements hong2019semisupervised,zhang2019phecap, and computational biology pipelines that supplement slow or costly experimental measurements with model-based predictions ppi2023,crossppi2024.
The goal is to use the larger unlabeled sample to improve estimation of a population mean while accounting for prediction error. This problem has long been studied in missing-data and semiparametric statistics, where methods such as AIPW, TMLE, and related debiased estimators combine flexible machine-learning prediction with valid inference robins1994estimation,vanderlaanunified,van2006targeted,vanderLaanRose2011,DoubleML. More recently, closely related ideas have reappeared in the machine-learning literature under the name prediction-powered inference ppi2023,angelopoulos2023ppi,crossppi2024. Similar questions also arise in covariate adjustment for randomized trials, where machine-learning predictors trained on auxiliary data can improve precision in small samples hansen2008prognostic,rubin2008empirical,moore2009covariate,lin2013agnostic,benkeser2021improving,schuler2022increasing,demirel2024prediction,jin2026prognostic.
A standard approach is to use the labeled sample to debias a plug-in estimator based on a prediction model. If \(m(X)\) is a black-box prediction score, the plug-in estimator averages \(m(X)\) over the pooled covariate sample, while the labeled sample is used to estimate the residual correction term \(\mathbb{E}\{Y-m(X)\}\). For example, the augmented inverse-probability weighted (AIPW) estimator robins1994estimation averages \(m(X)\) over both labeled and unlabeled samples and then adds the average residual over the labeled sample, whereas the standard PPI estimator ppi2023 averages \(m(X)\) only over the unlabeled covariates before applying the same residual correction. The latter is generally less efficient because it discards information in the labeled covariates.
These estimators protect against bias from an imperfect prediction model, but they typically treat \(m(X)\) as fixed. This can be problematic because efficiency depends on how well \(m(X)\) serves as a regression adjustment, that is, on how well it approximates the conditional mean of \(Y\), not merely on whether it ranks observations correctly. A score may be informative for ranking individuals while still being poorly calibrated, in the sense that its numerical values systematically overstate or understate the outcome. In that case, the residuals \(Y-m(X)\) remain unnecessarily variable, and the resulting semisupervised estimator can be far from efficient. Such miscalibration may arise because the model was trained on a different population, or because of temporal drift, distribution shift, or regularization bias.
This motivates post-hoc calibration. Rather than correcting only the marginal bias \(\mathbb{E}\{Y-m(X)\}\), one can use the labeled sample to transform \(m(X)\) so that its numerical values better align with the outcomes they are meant to predict zadrozny2001obtaining,zadrozny2002transforming,niculescu2005predicting. An ideal target is a perfectly calibrated score \(m_n^\star(X)\) satisfying \[ \mathbb{E}\!\left[Y \mid m_n^\star(X)\right] = m_n^\star(X). \] Post-hoc calibration can improve the regression adjustment, and thus efficiency, without retraining the underlying model. Classical post-processing methods include Platt scaling platt1999probabilistic, linear calibration mincer1969evaluation, histogram binning zadrozny2001obtaining, and isotonic regression zadrozny2002transforming. Such methods are especially relevant for modern machine-learning predictors, such as neural networks, which are often informative yet systematically miscalibrated bella2010calibration,guo2017calibration,wang2023calibration. Figure (ref) illustrates this mechanism in a simple example: post-hoc calibrating a mis-scaled prediction score improves both predictive accuracy and the efficiency of prediction-powered mean estimation.
\paragraph{Contributions.} We make three main contributions.
First, in Section (ref), we introduce calibrated prediction-powered inference for semisupervised mean estimation with a small labeled sample and a larger unlabeled one. The basic idea is to use the labeled data to post-hoc calibrate an existing prediction score before using it for semisupervised estimation. This framework accommodates familiar calibration methods, including linear calibration, Platt scaling, histogram binning, and isotonic regression; see, in particular, Sections (ref) and (ref). We focus especially on linearly calibrated PPI and isotonic-calibrated PPI, which are simple to implement, require no tuning, and can be added to an existing prediction pipeline without retraining the original model.
Second, we show that calibration addresses an important limitation of standard AIPW and PPI estimators: a prediction score may rank observations well while still be poorly aligned with the outcome scale, thereby leaving efficiency gains unrealized. In our setting, post-hoc calibration can improve the score as a regression adjustment, yielding sharper point estimates and confidence intervals, while still leading to a simple plug-in estimator. It can also make the fitted score first-order optimal against further post-processing within a specified class, both for prediction and for semisupervised estimation. In the spirit of foster2023calibeating, we refer to these two properties as prediction calibeating and functional-estimation calibeating, respectively.
Third, we develop asymptotic theory for linearly calibrated and isotonic-calibrated PPI estimators. For linear calibration, we show that PPI++ is asymptotically equivalent to linearly calibrated PPI, a form of prognostic-score regression adjustment. For isotonic calibration, Section (ref) establishes asymptotic normality, valid inference, and efficiency guarantees. In particular, isotonic-calibrated PPI achieves the smallest asymptotic variance among estimators based on monotone transformations of the original score, and is therefore at least as efficient as AIPW and PPI based on the uncalibrated score. We also show that the fitted isotonic score cannot be improved by further post-processing for prediction, and that the resulting mean estimator is first-order equivalent to the oracle estimator based on the conditional mean of the outcome given the calibrated score.
Our calibrated PPI framework can be viewed as a semisupervised specialization of the calibrated debiased machine learning framework developed by van2024automatic. Our estimators can also be interpreted as targeted minimum loss estimators van2006targeted,vanderLaanRose2011, with calibration playing the role of the targeting step.
Section (ref) reviews the two-sample semisupervised model, the AIPW, PPI, and PPI++ estimators, and the main efficiency concepts used throughout the paper. Section (ref) presents synthetic and real-data experiments.
Code. The accompanying Python package, reproduction code, and documentation are publicly available at \href{https://github.com/Larsvanderlaan/ppi-aipw}{github.com/Larsvanderlaan/ppi-aipw}. An interactive package website with examples and API documentation is available at \href{https://larsvanderlaan.github.io/ppi-aipw/}{larsvanderlaan.github.io/ppi-aipw/}. Self-contained code is provided in Appendix (ref).
Although PPI has been presented as a recent development, the underlying statistical problem and several closely related estimators lie within a much broader literature on semiparametric debiasing and regression adjustment, including work on missing data, causal inference, and debiased machine learning. In particular, the use of machine-learning predictions to improve efficiency under missing at random outcomes is closely connected to the literature on covariate adjustment in randomized trials and causal inference moore2009covariate,benkeser2021improving,hojbjerre2025powering, while power-tuning ideas can be viewed as a form of empirical efficiency maximization rubin2008empirical.
Semiparametric statistics and missing data. This paper builds on foundational ideas from the missing-data and semiparametric literatures. rubin1976 formalized missing at random as a key identification condition for recovering full-data targets from partially observed outcomes. Semiparametric efficiency theory then characterized efficient influence functions, regular asymptotically linear estimators, and one-step debiased procedures levit1975efficiency,pfanzagl1985contributions,hasminskii1979nonparametric,klaassen1987consistent,bickel1993efficient. In missing-data and causal-inference problems, these ideas led to influence-function-based estimating equations robins1994estimation,robins1995analysis,robins1995semiparametric,vanderlaanunified,tsiatis2006semiparametric, often with doubly robust structure bang2005doubly. As a result, the efficiency theory for missing-data models under missingness or coarsening at random is now well understood, and the class of regular asymptotically linear estimators in these models can be characterized through their influence functions robins1994estimation,robins1995analysis,robins1995semiparametric,vanderlaanunified,tsiatis2006semiparametric.
Debiased machine learning and targeted learning. A large body of work combines the semiparametric framework above with machine-learning-based nuisance estimation, often under the label of debiased machine learning. Flexible learning methods within estimating equations and AIPW-style procedures appear in early work such as vanderlaanunified. Targeted minimum loss estimation (TMLE), or targeted learning, updates an initial nuisance estimator in a parameter-targeted way to produce a plug-in estimator with reduced first-order bias; in the present setting, the primary nuisance estimated by machine learning is the outcome regression van2006targeted,vanderLaanRose2011,gruber2009targeted,hoffman2020tmle,ross2025constructing. These approaches are often combined with sample splitting and cross-fitting to accommodate generic machine-learning estimators schick1986asymptotically,klaassen1987consistent,zheng2010asymptotic,vanderLaanRose2011,DoubleML. Double machine learning emphasizes similar ingredients---orthogonality, cross-fitting, and influence-function expansions---for inference with modern nuisance estimators DoubleML,chernozhukov2022automatic, and is a closely related semiparametric framework vanderlaan2019cvtmle,diaz2020machine,chen2026equivalence. For sufficiently stable estimators, cross-fitting can itself be relaxed chen2022stable. Closely related work studies balancing-weight estimators hainmueller2012entropy,imai2014covariate,zubizarreta2015stable,lendle2015balancing,chattopadhyay2020balancing,hejazi2023revisiting, which often admit equivalent regression-adjustment or augmented representations chattopadhyay2023implied,bruns2025augmented,rotnitzky2025note. In missing-at-random settings with known missingness probability, including semisupervised learning and randomized trials, doubly robust estimators typically permit valid inference under flexible nuisance estimation even when the learned regression adjustment is misspecified moore2009covariate,moore2009increasing,rosenblum2009using,hojbjerre2025powering,hojbjerre2026within.
Semisupervised inference and improving power in randomized trials. Related work studies semisupervised or surrogate-assisted estimation in settings where the primary outcome is observed only on a limited subset, while auxiliary outcomes or surrogate measurements are available more broadly pepe1992inference,pepe1994auxiliary,chen2000miscellanea,chen2008improving,cheng2021robust,ji2025predictions,kallus2025role. Two-sample semisupervised inference is a special case of two-stage sampling designs scott1982effect,rose2011targeted,hejazi2021efficient,qiu2026efficient and, more generally, of data fusion, for which a substantial semiparametric efficiency literature has been developed li2023efficient,li2025data,graham2024towards,xu2025unified. More recent papers have used the term prediction-powered inference (PPI) for the semisupervised mean-estimation setting in which a black-box predictor is combined with a small labeled sample and a large unlabeled sample ppi2023,angelopoulos2023ppi,crossppi2024,xu2025unified,song2026demystifying,poulet2025prediction,wang2025efficient,lee2026mec. These papers also brought such methods to a broader machine-learning audience and provided accessible software. Closely related questions have long been studied in randomized trials, where prognostic scores or other regression adjustments learned from larger historical, external-control, or auxiliary data sources are used to improve precision in smaller trials hansen2008prognostic,rosenblum2009using,moore2009covariate,moore2009increasing,moore2011robust,vanderweele2019selecting,benkeser2021improving,burger2021use,schuler2022increasing,li2023estimating,balzer2024adaptive,van2024covariate,hojbjerre2025powering,yang2026improving. Methodologically, however, PPI estimators can often be viewed as instances or refinements of existing semiparametric procedures. For example, PPI++ is an AIPW estimator robins1994estimation,vanderlaanunified obtained by empirical efficiency maximization rubin2008empirical over a scaling parameter, while cross-PPI crossppi2024 is a cross-fitted AIPW estimator zheng2010asymptotic,DoubleML. More fundamentally, in settings with known missingness probability, the ability to combine black-box prediction models with valid semiparametric inference is closely tied to the robustness properties of AIPW and is therefore not unique to semisupervised mean estimation bang2005doubly,seaman2018introduction,rotnitzky2021characterization.
Calibrated DML and Recalibrated PPI. In our mean-estimation setting, the closest precursor is the calibrated debiased machine-learning framework of van2024automatic, which studies calibration of the outcome regression in AIPW estimators for doubly robust inference. Our setting can be viewed as a semisupervised specialization of that perspective, adapted to prediction-powered inference and aimed at improving efficiency in randomized designs. This connection also motivates our term calibrated PPI. A related paper is ji2025predictions, who study recalibrated prediction-powered inference for general estimating equations. Rather than correcting the predictive model itself, their approach modifies the imputed loss in the PPI estimating equation, thereby “recalibrating” the estimator rather than the predictions. In the mean-estimation setting under squared loss, the target correction reduces to the conditional mean of the outcome given the initial prediction score, which is also the efficient regression adjustment in the reduced model based only on that score. Thus, this conditional mean may be viewed as an oracle calibrator. The main differences lie in the estimators, emphasis, and theory. ji2025predictions use flexible machine learning to estimate this oracle target and thereby pursue semiparametric efficiency robins1994estimation. They do not study specific calibration procedures, and their method remains in AIPW form. By contrast, we study simple, widely used post-hoc calibration maps applied directly to a fixed score, and ask what can be guaranteed even without consistent estimation of the oracle target. Our estimators retain a simple plug-in form, yielding lightweight methods that can be incorporated into an existing prediction pipeline without retraining. We also develop calibration-specific results: an exact AIPW interpretation of the plug-in estimator, first-order equivalence between linear calibration and PPI++, and a first-order calibeating guarantee for isotonic calibration.
From missing data to causal inference. From the perspective of estimation and efficiency theory, semisupervised mean estimation with missing-at-random outcomes is equivalent to randomized experiments, controlled trials, and survey-sampling designs once the missingness indicator is reinterpreted as a treatment or sampling indicator, so that the relevant potential outcomes are viewed as missing rubin1978bayesian,vanderlaanunified,licausal,mozer2026ppi. This equivalence allows tools from covariate adjustment, causal inference, and model-assisted estimation to be transferred directly to the semisupervised setting; see, for example, mozer2026ppi for results relating prediction-powered inference to model-assisted survey-sampling estimators. The two-sample formulation we study is also equivalent to a one-sample i.i.d.\ model li2023efficient. In particular, our methods apply to inference on the counterfactual mean \(\mathbb{E}[Y(1)]\) in a randomized trial with binary treatment, where treated units form the labeled sample and control units contribute covariate information only; see Appendix (ref).
Calibration. A final relevant thread concerns calibration. Classical post-processing methods such as Platt scaling platt1999probabilistic,cox1958two,gupta2023online, linear calibration mincer1969evaluation,chernozhukov2018generic,leng2021calibration, isotonic regression zadrozny2002transforming,niculescu2005predicting, and histogram binning zadrozny2001obtaining,gupta2021distribution improve predictive reliability without retraining the underlying model. The idea has deeper roots in the forecasting literature mincer1969evaluation,lichtenstein1977calibration,vovk1992universal,vovk2005algorithmic,gneiting2007probabilistic,lambert2011elicitation; see also the historical discussion in lee2023t. More recent work studies calibration under broader losses and prediction tasks jung2021moment,noarov2023statistical,van2023causal,whitehouse2024orthogonal,van2025venn, as well as applications to conformal prediction van2024self,van2025venn, importance-weight stabilization gutman2022propensity,deshpande2023calibrated,ballinari2024improving,van2024stabilized, treatment effect estimation chernozhukov2018generic,van2023causal,whitehouse2024orthogonal, and value prediction in offline reinforcement learning van2025bellman,van2025semiparametric. Distribution-free guarantees for histogram binning are developed by gupta2021distribution, while van2023causal,van2025venn establish asymptotic distribution-free guarantees for isotonic calibration.
We observe two independent datasets, \[ \mathcal D_L=\{(X_i,Y_i)\}_{i=1}^n \qquad\text{and}\qquad \mathcal D_U=\{\widetilde X_j\}_{j=1}^N. \] The labeled sample consists of i.i.d.\ draws from a distribution \(P_0\) on \(\mathcal X\times\mathbb R\), while the unlabeled sample consists of i.i.d.\ draws from the distribution \(P_{0,X}\) of \(X\) under \(P_0\). Our target is the population mean \[ \psi_0 := \mathbb{E}_{P_0}[Y], \qquad \mu_0(x):=\mathbb{E}_{P_0}[Y\mid X=x]. \] This can also be written as \[ \psi_0=\int y\,dP_0(x,y)=\int \mu_0(x)\,dP_{0,X}(x). \]
For functions \(a:\mathcal X\times\mathbb R\to\mathbb R\) and \(b:\mathcal X\to\mathbb R\), define the empirical means over the labeled and unlabeled samples by \[ \mathbb{P}^{L}_n a := \frac{1}{n}\sum_{i=1}^n a(X_i,Y_i), \qquad \mathbb{P}^{U}_N b := \frac{1}{N}\sum_{j=1}^N b(\widetilde X_j). \] For readability, we will sometimes write \(\mathbb{P}^{L}_n\{a(X,Y)\}\) instead of \(\mathbb{P}^{L}_n a\). When a function depends only on \(X\), we also write \[ \mathbb{P}^{L}_n\{b(X)\}:=\frac{1}{n}\sum_{i=1}^n b(X_i). \] We write expectations and variances as \(\mathbb{E}_{P_0}[\cdot]\), \(\mathbb{E}_{P_{0,X}}[\cdot]\), \(\mathrm{Var}_{P_0}[\cdot]\), and \(\mathrm{Var}_{P_{0,X}}[\cdot]\).
Throughout, let \(M:=n+N\) and \(\rho_n:=n/M\) denote the labeled fraction. Then \[ \frac{1}{n+N}\left\{\sum_{i=1}^n f(X_i)+\sum_{j=1}^N f(\widetilde X_j)\right\} = \rho_n \mathbb{P}^{L}_n\{f(X)\} + (1-\rho_n)\mathbb{P}^{U}_N\{f(\widetilde X)\}. \] In other words, the empirical mean of \(f(X)\) over the pooled covariate sample is a weighted average of the empirical means from the labeled and unlabeled samples.
The AIPW class. We begin with the augmented inverse-probability weighted (AIPW) estimator in the unrestricted two-sample model robins1994estimation. A useful perspective is that AIPW is not a single estimator, but a family indexed by a score function \(f:\mathcal X\to\mathbb{R}\) with finite second moment. For any such \(f\), define
This estimator combines a plug-in term based on \(f(X)\) with a residual correction from the labeled sample. For any fixed \(f\), \(\widehat\psi(f)\) is unbiased in finite samples for \(\psi_0=\mathbb{E}_{P_0}[Y]\). In particular, taking \(f(X)=0\) yields the empirical mean estimator \(\mathbb{P}^{L}_n\{Y\}\). Thus, within the AIPW class, the choice of \(f\) affects efficiency but not validity: even a poor score does not induce bias, but it can leave substantial precision gains unrealized. In this sense, AIPW is a “safe” estimator xu2025unified. This property can be viewed as a particularly simple instance of double robustness\footnote{In more general missing-at-random settings with unknown, covariate-dependent missingness probabilities, the analogous AIPW estimator remains doubly robust asymptotically, but it no longer enjoys the same finite-sample unbiasedness as in the present known-\(\rho_n\) setting seaman2018introduction.} bang2005doubly.
This viewpoint is important for our purposes. Since validity is guaranteed for any \(f\), the central question is how to choose or improve the score to make \(\widehat\psi(f)\) as efficient as possible. In particular, efficiency depends on how well \(f(X)\) approximates the conditional mean of \(Y\), not merely on whether it ranks observations correctly. The same principle underlies the use of flexible machine learning for covariate adjustment in randomized trials while retaining valid inference moore2009covariate,moore2009increasing,schuler2022increasing,hojbjerre2025powering.
Asymptotic linearity, efficiency, and the role of residual variance. To characterize the leading large-sample error in the two-sample model, we decompose the estimation error into one contribution from the labeled sample and one from the unlabeled sample. We say that \(\widehat\psi\) is asymptotically linear with influence pair \((D^L,D^U)\) if \[ \widehat\psi-\psi_0 = \frac{1}{M}\sum_{i=1}^n D^L(X_i,Y_i) + \frac{1}{M}\sum_{j=1}^N D^U(\widetilde X_j) + o_p(M^{-1/2}), \] where \(\mathbb{E}_{P_0}[D^L(X,Y)]=0\) and \(\mathbb{E}_{P_{0,X}}[D^U(\widetilde X)]=0\). By the central limit theorem and the independence of the two samples, \[ \sqrt{M}\,(\widehat\psi-\psi_0) \rightsquigarrow N\!\left(0,\, \rho_0 \mathrm{Var}_{P_0}[D^L(X,Y)] + (1-\rho_0)\mathrm{Var}_{P_{0,X}}[D^U(\widetilde X)]\right). \] The influence pair therefore determines the estimator's asymptotic variance and provides a convenient way to compare efficiency across different choices of \(f\). The next result uses this representation to characterize the AIPW class, identify the efficient influence pair in the unrestricted two-sample model, and thereby obtain the semiparametric efficiency bound bickel1993efficient.
A key implication of Proposition (ref) is that the asymptotic variance of the AIPW estimator based on \(f\) exceeds the efficiency bound by \[ \frac{1-\rho_0}{\rho_0}\mathrm{Var}_{P_0}[f(X)-\mu_0(X)]. \] Thus, within the AIPW class, efficiency is governed entirely by how well \(f(X)\) approximates the conditional mean \(\mu_0(X)\), up to an additive constant; a related observation was made in wang2025efficient. In particular, ranking alone is not enough: a score may order observations correctly and still be inefficient if its numerical values are poorly aligned with the outcome scale. Full efficiency therefore generally requires access to the full covariate information \(X\), or at least to a summary rich enough to recover \(\mu_0(X)\) robins1994estimation,robins1995semiparametric. This motivates using a flexible black-box prediction score \(m(X)\) for regression adjustment in practice, and motivates post-hoc calibration when that score is informative but misaligned with the conditional mean.
Two particularly popular choices in (ref) are AIPW and PPI. The most common choice is \(f(X)=m(X)\), where \(m(X)\) is a prediction score, yielding the usual augmented inverse-probability weighted (AIPW) estimator robins1994estimation: \[ \widehat{\psi}_{\mathrm{AIPW}} = \frac{1}{M}\left\{ \sum_{i=1}^n m(X_i) + \sum_{j=1}^N m(\widetilde X_j) + \frac{1}{\rho_n}\sum_{i=1}^n \{Y_i-m(X_i)\} \right\}, \] where \(\rho_n^{-1}\) is the inverse probability of observing the outcome.
Another estimator in this class was considered by ppi2023 under the name PPI,\footnote{The PPI and PPI++ estimators are algebraically equivalent to the model-assisted survey-design estimators of cassel1976some and sarndal2003model, respectively; see mozer2026ppi for a review.} defined by \[ \widehat{\psi}_{\mathrm{PPI}} = \mathbb{P}^{U}_N\{m(\widetilde X)\}+\mathbb{P}^{L}_n\{Y-m(X)\}, \] which implicitly corresponds to the choice \(f(X)=m(X)/(1-\rho_n)\). Relative to \(\widehat{\psi}_{\mathrm{AIPW}}\), \(\widehat{\psi}_{\mathrm{PPI}}\) places the plug-in term entirely on the unlabeled sample and rescales the score by the factor \((1-\rho_n)^{-1}>1\). By Proposition (ref), this improves efficiency only when the rescaled score is closer to \(\mu_0(X)\) in mean squared error. Consequently, PPI is inefficient when \(m=\mu_0\) and is typically less efficient when \(m\) is already reasonably accurate han2012note,rotnitzky2012improved. For this reason, we view AIPW as the more natural default in practice.
Improving over a class of AIPW estimators through empirical efficiency maximization. Because both \(\widehat{\psi}_{\mathrm{AIPW}}\) and \(\widehat{\psi}_{\mathrm{PPI}}\) use scores in the scaling class \(\{\lambda m(X):\lambda\in\mathbb R\}\), it is natural to choose the scaling factor data-adaptively rather than fixing it in advance. One way to do so is to minimize the estimator's empirical influence-function variance over this class: \[ \sigma_n^2(f) := \rho_n \mathbb{P}^{L}_n\left[\left\{f(X)-\widehat{\psi}(f)+\rho_n^{-1}(Y-f(X))\right\}^2\right] + (1-\rho_n)\mathbb{P}^{U}_N\left[\left\{f(\widetilde X)-\widehat{\psi}(f)\right\}^2\right]. \] That is, one chooses \(\lambda\) to minimize \(\sigma_n^2(f)\) over \(f(X)=\lambda m(X)\), a procedure known as empirical efficiency maximization rubin2008empirical,vanderlaanunified. The resulting estimator, called PPI++ by angelopoulos2023ppi, is simply an AIPW estimator with an empirically rescaled prediction rule \(\hat\lambda m(X)\), where \(\hat\lambda\) estimates the population-optimal scaling coefficient
The same principle extends beyond simple rescaling. For example, one can minimize \(\sigma_n^2(f)\) over richer classes such as \[ \{\lambda m(X)+\beta_{\mathrm{lin}}^\top X: \lambda \in \mathbb R,\ \beta_{\mathrm{lin}} \in \mathbb R^d\}, \] over monotone transformations of the score, or over a reproducing kernel Hilbert space; see Appendix (ref). More generally, the same criterion can be used in cross-validation to choose among candidate estimator classes rubin2008empirical. In particular, if many prediction models \(\{m_1(X),m_2(X),\dots\}\) are available, one could learn a weighted linear combination by minimizing the empirical variance.
Standard AIPW and prediction-powered estimators treat the prediction score \(m(X)\) as fixed. In many applications, however, an available score may be useful for ranking individuals while still being poorly calibrated, in the sense that its numerical values do not accurately reflect the outcome itself. For example, a score may correctly assign higher values to individuals with larger outcomes on average, yet still systematically understate or overstate the conditional mean. Since the efficiency of a semisupervised estimator depends on the quality of the regression adjustment, such miscalibration can leave substantial efficiency gains unrealized.
Motivated by this observation, we study a simple post-hoc strategy: calibrate \(m(X)\) using the labeled sample, then plug the calibrated score into the semisupervised mean estimator. We call the resulting approach calibrated prediction-powered inference. As we show below, this yields a simple plug-in estimator with an exact AIPW representation. Different choices of calibration class yield different calibrated PPI estimators, including the linear and isotonic procedures studied below.
Let \(m:\mathcal X\to\mathbb R\) be a black-box regression model fit using external data or a separate training sample. We post-process this score on the labeled sample by learning a map \(f_n:\mathbb R\to\mathbb R\), and define the calibrated score \[ m_n^\star := f_n \circ m. \] Our estimator averages the calibrated predictions over the pooled labeled and unlabeled samples:
Equivalently, \[ \widehat{\psi}_{\mathrm{cal}} = \rho_n \mathbb{P}^{L}_n\{m_n^\star(X)\} + (1-\rho_n)\mathbb{P}^{U}_N\{m_n^\star(\widetilde X)\}. \]
At a high level, calibration seeks to transform the original score \(m(X)\) into a new score \(m_n^\star(X)\) that is better aligned with the outcome. Ideally, \[ m_n^\star(X)\approx \mathbb{E}_{P_0}[Y\mid m_n^\star(X)], \] so that among individuals with the same predicted value, the average outcome is close to that value. Under squared error loss, calibration is also closely tied to optimal post-processing, since replacing \(m_n^\star(X)\) by \(\mathbb{E}_{P_0}[Y\mid m_n^\star(X)]\) cannot increase mean squared error. In our setting, this matters because a better regression adjustment yields a more efficient semisupervised estimator. A natural population benchmark is therefore $\mathbb{E}_{P_0}[Y\mid m(X)],$ the best predictor obtainable from the original score alone. Our goal is to move \(m(X)\) toward this benchmark through post-processing, without retraining the underlying model and without requiring that it be estimated accurately.
We learn the post-processing map by empirical risk minimization over a class \(\mathcal H\) of functions \(h:\mathbb R\to\mathbb R\):
Assume that \(\mathcal H\) contains the constant functions. Then the corresponding first-order condition implies \[ \mathbb{P}^{L}_n\!\left\{Y-m_n^\star(X)\right\}=0. \] Therefore, \(\widehat{\psi}_{\mathrm{cal}}\) is exactly the AIPW estimator \(\widehat{\psi}(m_n^\star)\) based on the post-processed score \(m_n^\star(X)\):
Important examples we will consider later include linear and isotonic calibration, corresponding to \(\mathcal H\) equal to the classes of affine and monotone functions, respectively.
So far, we have described calibration informally as a population property: the score should be interpretable as a prediction of the outcome and should admit no further improvement within a prescribed transformation class. Since the true distribution \(P_0\) is unknown, we instead impose an empirical analogue on the labeled sample. Let \(\mathcal F\) be a linear class of maps \(f:\mathbb R\to\mathbb R\) containing the constants. We say that \(m_n^\star\) is empirically calibrated over \(\mathcal F\) if
Because \(\mathcal F\) is linear, this is equivalent to the orthogonality condition
As we show next, empirical calibration yields not only the exact AIPW representation in (ref), but also a first-order equivalence to an oracle AIPW estimator based on the best population transformation of the fitted score within \(\mathcal F\).
Theorem (ref) is the key structural result. It shows that \(\widehat{\psi}_{\mathrm{cal}}\) differs from the AIPW estimator based on \(m_{0,\mathcal F}^\dagger\), the population-optimal transformation of \(m_n^\star\) within \(\mathcal F\), only through the remainder \[ (1-\rho_n)(\mathbb{P}^{U}_N-\mathbb{P}^{L}_n)\{m_n^\star-m_{0,\mathcal F}^\dagger\}. \] If \(m_n^\star\) is close to \(m_{0,\mathcal F}^\dagger\) in mean squared error, this term is higher order, so \(\widehat{\psi}_{\mathrm{cal}}\) and the oracle AIPW estimator have the same first-order asymptotic behavior. In this sense, empirical calibration makes the fitted score behave as though it had already been optimally transformed within \(\mathcal F\). It also yields a balancing-weights representation of the form \(\mathbb{P}^{L}_n\{\hat w_{\mathrm{bal}}(X)Y\}\); see Appendix (ref). We make this precise for isotonic calibration in Section (ref).
Why calibration can improve efficiency. Calibration can improve efficiency by improving the regression adjustment in the exact AIPW representation (ref). If \(f_n\) converges to a population minimizer \(g_0\in\mathcal H\), then the post-processed score \(m_n^\star=f_n\circ m\) typically converges to \[ m_0(X):=g_0\{m(X)\}, \qquad g_0\in\arg\min_{g\in\mathcal H} \mathbb{E}_{P_0}\!\left[\{Y-g(m(X))\}^2\right]. \] Thus, \(m_0\) is the best regression adjustment obtainable from the original score \(m\) using transformations in \(\mathcal H\). By Proposition (ref), when \(\mathcal H\) contains the constant functions, minimizing prediction error over the class \(\{\theta\circ m:\theta\in\mathcal H\}\) is first-order equivalent to maximizing AIPW efficiency over the same class. Consequently, the resulting AIPW estimator has asymptotic variance no larger than that of any estimator based on \(\theta\circ m\) with \(\theta\in\mathcal H\). In particular, if the identity map belongs to \(\mathcal H\), then calibration cannot do worse than using the raw score \(m\). If \(\mathcal H\) contains the oracle map \(t\mapsto \mathbb{E}_{P_0}[Y\mid m(X)=t]\), then \(m_0(X)=\mathbb{E}_{P_0}[Y\mid m(X)]\), and the resulting estimator is semiparametrically efficient in the reduced model based on the score \(m(X)\) ji2025predictions.
Full versus linear calibration. We focus on two main cases. In full calibration, the target class \(\mathcal F\) is unrestricted, so that at the population level no further transformation of the calibrated score can reduce mean squared prediction error. In practice, however, the fitted map \(f_n\) is learned over a restricted optimization class \(\mathcal H\), which serves as a regularizer. Important examples include piecewise-constant and monotone classes, corresponding to histogram and isotonic calibration; see Section (ref). In linear calibration, both \(\mathcal H\) and \(\mathcal F\) are restricted to affine maps \(t\mapsto a+bt\), corresponding to least-squares fitting of the working model \(a+b\,m(X)\); see Section (ref).
We focus on isotonic regression as a simple nonparametric post-hoc calibration method zadrozny2001obtaining,zadrozny2002transforming,niculescu2005predicting. Let \(\mathcal F_{\mathrm{iso}}\) denote the class of nondecreasing real-valued functions. We fit the calibrator using the labeled pairs \(\{(m(X_i),Y_i)\}_{i=1}^n\) by solving \[ \hat f \in \arg\min_{f \in \mathcal F_{\mathrm{iso}}} \sum_{i=1}^n \bigl\{Y_i - f(m(X_i))\bigr\}^2, \] and define the calibrated score \[ m_{n,\mathrm{iso}}^\star(X) = \hat f\{m(X)\}. \] The resulting isotonic-calibrated PPI estimator is \[ \widehat{\psi}_{\mathrm{iso}} := \frac{1}{n+N} \left\{ \sum_{i=1}^n m_{n,\mathrm{iso}}^\star(X_i) + \sum_{j=1}^N m_{n,\mathrm{iso}}^\star(\widetilde X_j) \right\}. \] Isotonic-calibrated PPI is particularly attractive because it is lightweight, tuning-free, and easy to implement with widely available software. Its population target is the monotone transformation of the prediction score \(m(X)\) that minimizes mean squared error, and therefore performs at least as well as the original score and any other monotone transformation of \(m(X)\), including positive rescalings. Thus it yields the best regression adjustment available within the class of monotone transformations of \(m(X)\).
Isotonic regression fits naturally into the \((\mathcal H,\mathcal F)\) framework with \(\mathcal H=\mathcal F=\mathcal F_{\mathrm{iso}}\), the class of nondecreasing functions. Moreover, because the fitted isotonic score is piecewise constant on the blocks learned by the pooled-adjacent-violators algorithm (PAVA) barlow1972isotonic, any further transformation of the fitted score is constant on those same blocks. In turn, isotonic calibration yields full empirical calibration.
{Implementation.} Isotonic regression is typically fit using the pooled-adjacent-violators algorithm (PAVA) barlow1972isotonic, which is fast in standard settings but can become burdensome at very large sample sizes. A scalable alternative is to use tree-based software with monotonicity constraints, such as XGBoost or LightGBM; see Appendix (ref). These implementations also allow regularization through parameters such as the number of leaves, maximum depth, and minimum leaf size.
{Histogram binning.} Histogram binning calibrates a score by partitioning its range into prespecified bins and replacing the score within each bin by a constant value zadrozny2001obtaining,zadrozny2002transforming. Thus \(\mathcal H\) is the class of piecewise-constant maps on a fixed partition of the score space gupta2020distribution,gupta2021distribution, and one may take \(\mathcal F=\mathcal H\). Like isotonic regression, histogram binning is fully empirically calibrated. The key difference is that histogram binning uses a fixed partition, whereas isotonic calibration learns the partition adaptively from the data.
When the labeled sample is small, a simple parametric calibration rule can be more stable than histogram binning or isotonic regression. We therefore consider linear calibration, which rescales and shifts the prediction score using an affine map. This is similar in spirit to Platt scaling or temperature-style recalibration platt1999probabilistic,guo2017calibration. Unlike isotonic calibration, linear calibration cannot fix general nonlinear miscalibration, but it is often more stable in small samples and is optimal within the class of affine transformations.
We fit the calibration map by least squares on the labeled data:
The resulting estimator averages the calibrated predictions over the labeled and unlabeled covariates:
Linear calibration is practically appealing, as it ensures that the empirical mean squared error of the prediction score cannot be improved by a shift or rescaling. That is, \(m_{n,\mathrm{lin}}^\star(X)\) is empirically calibrated over the class of affine transformations. This yields a family of AIPW representations.
One can also extend the linear calibration step by including additional covariates in the regression adjustment when enough labeled data are available; see Algorithm (ref).
The next result connects linear calibration to PPI++, a closely related estimator obtained by rescaling the original score via empirical efficiency maximization. In the mean-estimation setting, the two methods are first-order equivalent, though they need not coincide exactly in finite samples. The reason is that both target the same one-dimensional regression adjustment: the optimal population rescaling of \(m(X)\) by the slope \(\lambda_0\) in (ref), namely, the coefficient from the population linear regression of \(Y\) on \(m(X)\). This equivalence, however, relies on optimizing over all \(\lambda \in \mathbb{R}\). If \(\lambda\) is instead constrained to lie in \([0,(1-\rho_n)^{-1}]\), as appears to be the case in the official ppi_py implementation, then the equivalence need not hold. Our experiments also suggest that this restriction can reduce performance.
Let \(\widehat{\psi}_{++}(\widehat{\lambda}_{++})\) denote the PPI++ estimator of angelopoulos2023ppi, that is, the AIPW estimator based on the score \(\widehat{\lambda}_{++} m(X)\), where \(\widehat{\lambda}_{++}\) is chosen by empirical efficiency maximization over the class \(\{\lambda m(X): \lambda \in \mathbb{R}\}\) rubin2008empirical.
In the main exposition, we assume for simplicity that the initial black-box predictor is fixed. In practice, however, it may be trained on the labeled sample itself, so that the same labeled observations are used both to fit the predictor and to calibrate its scores. A cross-fitted implementation avoids this reuse by replacing the single prediction function with out-of-fold predictions and then fitting the calibrator to the resulting labeled scores.
This cross-fitted version is the natural DML-style implementation when one wants to reduce overfitting from using the same labeled data both to train the initial predictor and to calibrate it for inference zheng2010asymptotic,vanderLaanRose2011,DoubleML. Related procedures have been used for debiased inference in van2024stabilized,van2024automatic,rabenseifner2025causal, and a related cross-calibration variant appears in van2023causal.
We now show that isotonic-calibrated PPI can calibeat the original score---that is, improve it both as a predictor of the outcome and as a regression adjustment for semisupervised mean estimation. It also typically calibeats simpler post-processing rules, such as linear calibration.
We analyze the isotonic-calibrated estimator \(\widehat{\psi}_{\mathrm{iso}}\), establishing asymptotic linearity, asymptotic normality, and valid Wald inference. We then compare its efficiency with that of AIPW, PPI, PPI++, and linearly calibrated estimators, and identify conditions under which it attains the smallest asymptotic variance among regular estimators based on \(m(X)\) or on the full covariate vector \(X\) bickel1993efficient,van2000asymptotic. Finally, we show that isotonic calibration is first-order optimal in a natural sense: once a score has been isotonically calibrated, no further post-processing based only on the fitted score can further improve either mean squared prediction error or the leading asymptotic behavior of the resulting plug-in estimator.
Let \(f_0\) denote a population isotonic calibration map for the fixed score \(m\), defined by
Thus, \(m_0\) is the population \(L^2\)-optimal monotone transformation of the initial score \(m(X)\). Since the identity map belongs to \(\mathcal F_{\mathrm{iso}}\), \(m_0\) has mean squared error no larger than that of the original score \(m(X)\). Under the following conditions, the isotonic-calibrated estimator \(m_{n,\mathrm{iso}}^\star\) attains the classical isotonic rate in mean squared error; see, for example, chatterjee2015risk,yang2018contraction.
Condition (ref) requires only that the regression target be bounded and that the isotonic estimator remain bounded in probability. Condition (ref) allows \(Y\) to be unbounded, covering both Gaussian-like tails and somewhat heavier exponential-type tails. In particular, it holds for binary, Gaussian, Poisson, and bounded outcomes. Existing results on isotonic regression under heavy-tailed errors also suggest some robustness to violations of these conditions han2018robustness,han2019convergence.
The theorem above shows that \(\sqrt{M}(\widehat{\psi}_{\mathrm{iso}} - \psi_0)\) converges in distribution to a mean-zero normal random variable with variance \[ \sigma_0^2 := \rho_0\,\mathrm{Var}[D_{m_0}^L(X,Y)] + (1-\rho_0)\,\mathrm{Var}[D_{m_0}^U(\widetilde X)]. \] The following corollary records the resulting asymptotic normality and validity of standard Wald confidence intervals.
The next corollary records two important efficiency regimes: full semiparametric efficiency when the calibrated score recovers the full regression function, and reduced-experiment efficiency when the oracle regression depends on \(X\) only through the one-dimensional score \(m(X)\).
The main implication is that isotonic calibration weakly improves on standard monotone recalibrations, including AIPW, PPI, PPI++, and linear calibration whenever the corresponding scaling or slope coefficient is nonnegative. This improvement can be strict when the best calibration is genuinely nonlinear. The nonnegative-slope restriction is mild: if the score has the wrong sign, one can first apply a linear sign correction and then perform isotonic calibration. The corollary also clarifies the limit of what post-processing can achieve: full efficiency is possible only if the calibrated score recovers all regression information relevant to \(\psi_0\). Otherwise, some information is inevitably lost when the covariates are compressed to the scalar score \(m(X)\).
We now ask whether the isotonically calibrated score can be improved by an additional post-processing step. Since \(\widehat{\psi}_{\mathrm{iso}}\) depends on the covariates only through the one-dimensional fitted score \(m_{n,\mathrm{iso}}^\star(X)\), it is natural to study the reduced two-sample experiment \[ \mathcal E_{m_{n,\mathrm{iso}}^\star} := \Bigl( \{(m_{n,\mathrm{iso}}^\star(X_i),Y_i)\}_{i=1}^n,\, \{m_{n,\mathrm{iso}}^\star(\widetilde X_j)\}_{j=1}^N \Bigr), \] in which the full covariate vector \(X\) is replaced by the scalar score \(m_{n,\mathrm{iso}}^\star(X)\).
Within this reduced experiment, a natural benchmark is the oracle regression function \(\mathbb{E}_{P_0}[Y \mid m_{n,\mathrm{iso}}^\star(X)]\), where the conditioning is understood to be conditional on the training data used to construct \(m_{n,\mathrm{iso}}^\star\). The result below shows that no additional post-processing of the fitted score can improve the first-order performance of \(\widehat{\psi}_{\mathrm{iso}}\) relative to this benchmark.
Assumption (ref) is mild. It requires only that the population regression of \(Y\) on the initial score \(m(X)\), viewed as a one-dimensional function of the score, have bounded total variation. In particular, it allows jumps and is weaker than differentiability or Lipschitz conditions van2023causal.
Theorem (ref) implies that \(m_{n,\mathrm{iso}}^\star\) is calibrated in the sense that \(\|\widehat{m}_0-m_{n,\mathrm{iso}}^\star\|_{2,P_{0,X}}^2 = O_p(n^{-2/3})\). Theorem (ref) further shows that \(\|m_{n,\mathrm{iso}}^\star-m_0\|_{2,P_{0,X}}^2 = O_p(n^{-2/3})\), so this calibration does not asymptotically degrade predictive performance. We therefore say that \(m_{n,\mathrm{iso}}^\star\) prediction-calibeats the original predictor \(m\).
Theorem (ref) also shows that the asymptotic variance of \(\widehat{\psi}_{\mathrm{iso}}\) matches that of the oracle estimator \(\widehat{\psi}_{\mathrm{eff}}\), which uses the oracle score \(\widehat{m}_0(X):=\mathbb{E}_{P_0}[Y \mid m_{n,\mathrm{iso}}^\star(X)]\). Equivalently, within the reduced experiment based on \(m_{n,\mathrm{iso}}^\star(X)\), \(\widehat{m}_0(X)\) is the oracle post-processing that minimizes the variance of the AIPW influence function. Moreover, Theorem (ref) combined with Proposition (ref) implies that \(\widehat{\psi}_{\mathrm{iso}}\) achieves the optimal limiting variance within the class of monotone post-processings, and is therefore at least as efficient as AIPW based on the original score \(m(X)\). Hence, \(\widehat{\psi}_{\mathrm{iso}}\) functional-estimation-calibeats \(\widehat{\psi}_{\mathrm{AIPW}}\) and \(\widehat{\psi}_{\mathrm{PPI}}\).
We use the same comparison hierarchy throughout all experiments. The labeled-only estimatorserves as the safety baseline. AIPW is the primary raw-score baseline, with PPI included as a weaker comparator. PPI++ and AIPW-EM are the efficiency-rescaling competitors, while LinearCal, IsoCal, AutoCal, and related calibration rules are the proposed remedies when the score is miscalibrated. The three subsections below therefore move from a controlled mechanism check, to reproduced benchmark evidence, to a practical LLM evaluation setting.
We begin with a controlled binary-outcome simulation designed to isolate score miscalibration. Let \(S\sim N(0,1)\), define \[ \mu_0(S)=\{1+\exp(-5S)\}^{-1}, \] and draw \(Y\mid S\sim \mathrm{Bernoulli}\{\mu_0(S)\}\). To induce substantial miscalibration, we apply an affine-biased monotone distortion on the logit scale, \[ m(X)=\mathrm{clip}\left[-0.15+0.75\,\sigma\bigl\{0.8\,z(S)+0.1\,z(S)^3-1\bigr\},\,0.01,\,0.99\right], \] where \(z(S):=\mathrm{logit}\{\mu_0(S)\}\). This transformation preserves ranking information while introducing nonlinear distortion on the probability scale. We compare the labeled-only mean (Labeled-only), PPI, AIPW, PPI++, an unclipped efficiency-maximized AIPW benchmark (AIPW-EM), linear calibration (LinearCal), monotone spline calibration (MonoSpline) ramsay1988monotone,jiang2011smooth, isotonic calibration (IsoCal), and an adaptive selector (AutoCal) that chooses among AIPW and the main calibration rules by cross-validated empirical efficiency. Here, PPI++ refers to the official \texttt{ppi_py} implementation, which rescales the prediction score by one-dimensional empirical efficiency maximization and then clips the resulting coefficient to \([0,(1-\rho_n)^{-1}]\), so that the estimator shrinks PPI towards the labeled-only estimator. By contrast, \textsc{AIPW-EM} applies the same criterion without clipping the selected coefficient. The labeled sample size ranges over \(n\in\{50,100,200,400,800,1200,2400\}\), with \(N/n=1\) and \(N/n=16\). All figures and summaries are based on 500 Monte Carlo repetitions.
Figure (ref) isolates the main mechanism. All score-based AIPW-type estimators are essentially unbiased, so the key comparison is efficiency: AIPW improves on PPI, and calibration helps when the score is genuinely miscalibrated. At \(n=400\) and \(N/n=16\), the calibrated methods reduce RMSE from about \(0.0172\) for PPI to about \(0.0164\); by \(n=1200\) in the same regime, RMSE is about \(0.0087\) for AutoCal, MonoSpline, and IsoCal, compared with \(0.0092\) for PPI. The PPI++ comparison is most revealing when the unlabeled sample is large: AIPW-EM remains essentially indistinguishable from LinearCal, whereas clipped PPI++ falls back toward plain PPI. Averaging over the poorly calibrated designs with \(N/n=16\), RMSE is about \(0.0196\) for AIPW-EM and \texttt{LinearCal}, versus \(0.0208\) for PPI++, reflecting the fact that the official \texttt{ppi_py} implementation clips the optimized scaling coefficient to \([0,1]\), whereas \textsc{AIPW-EM} does not. \textsc{AutoCal} remains close to the best fixed score-based option, and \texttt{MonoSpline} provides a smooth monotone alternative that is usually close to isotonic calibration.
We evaluate the proposed calibration-based estimators on the mean-estimation benchmarks used in the original experiments of ppi2023,angelopoulos2023ppi, as provided in the ppi_py package. We use the same ppi_py datasets and labeled-sample-size grids, repeatedly split each fully labeled benchmark into labeled and unlabeled subsets, evaluate all estimators on the same split, and treat the full-sample mean as ground truth.\footnote{Because repeated sample splits of a fixed finite benchmark dataset are not i.i.d.\ draws from a superpopulation, asymptotic Wald confidence intervals can exhibit either undercoverage or overcoverage. This is not specific to our calibration extensions, but is a feature of evaluating Wald-type intervals under finite-population resampling.} The current draft summaries average over 500 random splits at each \(n\). We report the official forest, galaxies, and census_income benchmarks and focus the main text on the primary comparator set already shown in the figure: PPI, AIPW, PPI++, AIPW-EM, \texttt{LinearCal}, and \texttt{AutoCal}. Appendix (ref) gives the fuller method inventory, calibration-specific comparisons, and dataset-by-regime summary.
Figure (ref) shows the same pattern on the reproduced benchmarks. AIPW is the stronger raw-score baseline, with PPI competitive only when the unlabeled sample is extremely large. The clearest calibration gain appears in census_income, where LinearCal performs best on average, and the appendix comparison in (ref) shows that IsoCal and MonoSpline remain broadly competitive without changing the main ranking. The added efficiency-maximization baseline also sharpens the comparison with PPI++: across the 30 dataset-size cells in (ref), AIPW-EM has lower MSE than clipped PPI++ in 29 and is nearly tied with LinearCal. Averaged across all cells, the mean MSE is about \(4.75\times 10^6\) for \textsc{AIPW-EM}, \(4.75\times 10^6\) for \texttt{LinearCal}, and \(5.03\times 10^6\) for PPI++, with essentially identical coverage near \(0.919\). This mirrors the simulations: once the empirical efficiency-maximization step is left unclipped, its finite-sample behavior moves much closer to \texttt{LinearCal}. \texttt{AutoCal} serves as a practical hedge that tends to remain close to the best fixed score-based method. Appendix (ref) provides a fuller summary by dataset and regime.
We now turn to a practical LLM-evaluation setting in which we apply the same comparison hierarchy to real public model outputs, public evaluator scores, and public labels. The first track uses PPE Human Preference V1 frick2025evaluate, where the estimand for each target model is its human win rate across prompt--response comparisons, counting ties as half-wins. The second track uses a PPE correctness suite frick2025evaluate built from MMLU-Pro wang2024mmlupro, MATH hendrycksmath2021, GPQA rein2024gpqa, IFEval zhou2023instructionfollowing, and MBPP+ liu2023evalplus, where each unit consists of a sampled conflict pair from the same generator model and the estimand is the correctness probability of the first answer in the pair. In both tracks, the prompts, outputs, judgments, and evaluator scores are real; only the labeled/unlabeled split is rerandomized. Across 200 repeated splits, we compare labeled-only estimation with raw PPI and AIPW, their efficiency-maximized variants, and the calibration-based estimators from our main experiments.
\IfFileExists{assets/fig_llm_eval_main.pdf}{
}
\IfFileExists{assets/table_llm_summary.tex}{
}
Two messages stand out. First, naive use of the proxy can fail badly. Raw AIPW is again the stronger raw-score baseline, but even AIPW can perform dramatically worse than labeled-only estimation when the reward-model margin is poorly scaled. On PPE Human at \(n=100\), raw PPI and raw AIPW have MSE \(0.495\) and \(0.440\), compared with \(0.001425\) for labeled-only and \(0.001391\) for LinearCal. On PPE Correctness at \(n=100\), the corresponding MSEs are \(0.534\), \(0.353\), \(0.002010\), and \(0.001813\). These estimators remain valid first-order corrections, but the additional proxy term can inflate variance enough that labeled-only estimation is much safer.
Second, calibration makes the same evaluator scores practically useful. LinearCal is the strongest and most stable method overall, with AutoCal close behind. On PPE Human, LinearCal is best from \(n=50\) onward, improving on labeled-only from \(0.001425\) to \(0.001391\) at \(n=100\) and from \(0.000286\) to \(0.000278\) at \(n=400\), corresponding to modest label savings of about \(2.5\%\) to \(3.1\%\). On PPE Correctness, the gains are clearer: LinearCal is best at every budget, improving from \(0.002010\) to \(0.001813\) at \(n=100\) and from \(0.000248\) to \(0.000222\) at \(n=400\), corresponding to about \(7.5\%\) to \(10.5\%\) label savings. The more flexible calibration rules are also competitive once the labeled sample is moderately large. On PPE Human, AutoCal and MonoSpline remain in the \(2.5\%\) to \(2.8\%\) label-savings range from \(n=100\) to \(n=400\); on PPE Correctness, they remain in roughly the \(9\%\) to \(10\%\) range over the same budgets. \texttt{MonoSpline} therefore substantially improves on raw PPI and raw AIPW and is usually competitive with PPI++ and \textsc{AIPW-EM}, though it remains slightly less stable than \texttt{LinearCal}. The evaluator-specific appendix breakdown ((ref)) shows that the most extreme raw-proxy failures are driven by evaluators such as Skywork and Athene, whereas ArmoRM is much more benign. Calibration also improves the downstream ranking task induced by PPE Human. Raw PPI and raw AIPW recover the model ordering poorly, whereas calibrated and efficiency-maximized methods recover it well. At \(n=200\), \texttt{AutoCal} attains the highest average Spearman correlation, about \(0.905\); at \(n=400\), \texttt{LinearCal} achieves the best overall ranking quality, with Spearman correlation about \(0.941\) and top-1 identification rate about \(0.646\). We therefore view calibration not as a marginal efficiency tweak, but as the step that makes cheap public LLM evaluators safe enough to use in practice; see Appendix (ref).
We introduced Calibrated Prediction-Powered Inference, a simple framework for semisupervised mean estimation based on three steps: fit a prediction score, calibrate it on the labeled sample, and average the calibrated predictions over the pooled covariate sample. This simple plug-in procedure fits naturally within the broader literature on semiparametric debiasing and can be viewed as a semisupervised specialization of calibrated debiased machine learning van2024automatic.
Our results highlight a simple message. Calibration can improve semisupervised efficiency without retraining the original model, and isotonic calibration can calibeat the original score and simpler post-processing rules at first order. At the same time, once the score has been isotonicly calibrated, further post-processing of that fitted score brings no additional first-order gain. Empirically, this suggests a transparent and practical workflow: when the original score is already well calibrated, calibration changes little, whereas under meaningful miscalibration, even simple linear or monotone calibration can improve efficiency while preserving a transparent plug-in form.
Positioning PPI within existing semiparametric methods. More broadly, our results clarify that recent prediction-powered inference methods can be understood within the standard semiparametric, debiased machine-learning, and flexible covariate-adjustment toolkit for randomized trials and missing-at-random settings. In particular, we show that PPI++ is an AIPW estimator with empirical efficiency maximization rubin2008empirical and is first-order equivalent to linear calibration and classical prognostic-score regression adjustment hansen2008prognostic,lin2013agnostic,schuler2022increasing. It is therefore closely related to randomized-trial approaches that learn prognostic scores from larger historical, external-control, or auxiliary data sets and then use them for regression adjustment in smaller trials hansen2008prognostic,moore2009covariate,rosenblum2009using,benkeser2021improving,hojbjerre2025powering. This perspective suggests that semisupervised inference may benefit from closer integration with the broader missing-data, causal-inference, and debiased machine-learning literatures. These lines of work study closely related versions of the same core problem: how to use flexible regression adjustment learned from auxiliary data while retaining valid inference for the target parameter. Further progress may therefore come from translational work across them, especially by adapting tools developed for randomized trials and missing-at-random settings to semisupervised problems petersen2014causal,dang2023causal,ho2023current,smith2023application,ho2024examples.
Beyond mean estimation. These connections extend well beyond the semisupervised mean-estimation setting considered here. For example, when missingness affects both outcomes and covariates and may depend on observed covariates and outcomes, as well as in extensions to right-censored time-to-event outcomes and longitudinal settings, existing semiparametric methods often provide natural starting points and can frequently be adapted to these settings robins1994estimation,vanderlaanunified,bang2005doubly,moore2009increasing,rose2011targeted,van2018targeted. More generally, M-estimation and estimating-equation approaches for missing-at-random problems are well developed robins1994estimation,vanderlaanunified,DoubleML. When missingness depends on unobserved outcomes even after conditioning on observed data, however, naively pooling labeled and unlabeled data can bias estimation. There is substantial work on combining gold-standard and potentially biased auxiliary data more safely using semiparametric and machine-learning tools kallus2018removing,rosenman2025methods,dang2025experiment,van2026adaptive. Similarly, when the unlabeled sample is selected in a data-dependent way, the problem becomes closely related to adaptive and selectively sampled designs, including two-phase and informative sampling van2008construction,chow2008adaptive,rose2011targeted,malenica2021adaptive,zrnic2024active,zhang2025efficient,liu2025robust. Extending these broader missing-data and adaptive-sampling ideas to modern semisupervised inference is therefore a natural direction for future work.
Towards conditional prediction-powered inference There is already a well-developed literature on machine-learning-assisted estimation of local and conditional regression targets, including regression functions, dose-response curves, and conditional average treatment effects, which can be used to extend inference beyond marginal mean estimation rubin2006doubly,diaz2013targeted,kennedy2017non,bibaut2017data,van2018cv,westling2020causal,chernozhukov2022debiased,kennedy2023towards,foster2023orthogonal,chernozhukov2023simple,luedtke2024one,chernozhukov2024conditional,butzin2024highly,zhang2025constructing,gu2024local,sui2026prediction. In particular, kernel-regression-based bibaut2017data and isotonic-regression-based westling2020unified inference can be used for conditional PPI with AIPW-score-based pseudo-outcomes, as in van2003unified,rubin2006doubly,kennedy2017non, or more generally through influence functions and Neyman-orthogonal losses bibaut2017data,chernozhukov2024conditional. The key idea is that locally smoothed conditional targets are amenable to influence-function-based bias correction, after which inference follows from a triangular-array central limit theorem applied to estimators of a sequence of approximating parameters bibaut2017data,chernozhukov2022debiased,luedtke2024one,chernozhukov2024conditional. In the prediction-powered setting, a natural approach is to apply kernel regression to AIPW-style pseudo-outcomes for missing-data problems, following rubin2006doubly, takatsu2025debiased, and bibaut2017data,kennedy2017non,chernozhukov2024conditional.
Calibration for general loss functions. While we focus on calibration of regression functions and inference on means, the ideas here may extend to broader supervised learning problems defined through loss minimization, such as median and quantile regression rosenblum2009using,noarov2023statistical,jung2021moment,roth2022uncertain,whitehouse2024orthogonal. They also extend to continuous linear functionals of a regression function or M-estimand through calibrated DML van2024automatic,van2025automatic. More generally, when the target parameter is defined by a moment equation, as in M-estimation, one could debias or calibrate the associated augmented moment equation vanderlaanunified,DoubleML; see Appendix (ref) and ji2025predictions. A practical challenge in this setting is that the augmented loss may be nonconvex or may require separate debiasing at each parameter value, which can complicate optimization. In such cases, a more practical alternative may be a multiaccuracy-style adjustment rather than explicit augmentation or calibration, as in the Efficient Plug-in Learning framework of van2024combining, which addresses this issue for nonconvex Neyman-orthogonal loss functions foster2023orthogonal.