Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
83,991 characters · 34 sections · 72 citation commands
Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model
Modern large language models (LLMs) are tuned using human or AI feedback (RLHF/RLAIF) to better align outputs with user preferences and safety desiderata Christiano2017,Ziegler2019,Stiennon2020,Ouyang2022,Bai2022,bai2022training,nakano2021webgpt,wu2021recursively. A common setup interprets pairwise preferences as discrete choice under a latent reward, which is inferred and optimized, while constraining or penalizing the deviation from a reference model. This balances quality improvements with preservation of language abilities.
Linking preferences to rewards is usually done by assuming a particular choice model, such as Bradley-Terry (logistic link) or Thurstone (probit link), so that, given demonstrations and a reward function, the distribution of preferences becomes fully specified Rafailov2023,zhan2023provable,glaese2022improving,Ziegler2019,ibarz2018reward. A prominent example is Direct Preference Optimization (DPO, Rafailov2023), which uses the Bradley-Terry choice model. This, however, imposes a lot of structure on choice behavior, and misspecifying this link can bias inferred rewards and misalign policy optimization hong2025robustness,xu2024strong. Alternative approaches to alignment from preferences depart from structural/generative modeling of choice, focusing on loss aversion and label corruption ethayarajh2024kto,liang2024ropo,kong2024perplexity.
However, specifying a known reward-preference link is not actually necessary for a structural/generative interpretation (meaning one that assumes preferences optimize a random utility or equivalently that preferences are generated by some conditional probability distribution given demonstrations and reward). Letting the link be arbitrary and unknown gives rise to semiparametric discrete choice models, which have been studied extensively in the econometrics literature Cosslett1983,KleinSpady1993,Manski1975,Manski1985,Horowitz1992,KimPollard1990,Han1987,Sherman1993. This literature, however, focuses on reward (utility) models that are prescribed by finite-dimensional structural parameters, imposes stringent assumptions so that these structural parameters become identifiable from choice data, and develops methods and theory for estimation of and inference on these structural parameters, as they are themselves their target of interest.
In this paper, we study instead LLM alignment from preference data generated by a completely unknown link function and unknown reward function, where the target is the optimal reward-maximizing divergence-constrained policy. A key insight is that mere realizability of this optimal policy in our LLM policy class implies a so-called semiparametric single-index model for preferences, which has an arbitrary unknown link function. The policy class induces a class of indices (corresponding to reward or utility functions), which can be an arbitrary nonparametric class, and moreover the true index (corresponding to the true reward function) is generally not identifiable. We study policy learning methods and analyze bounds on the policy difference to the optimal policy. We call the resulting approach Semiparametric Preference Optimization (SPO).
Section (ref) derives the single-index formulation and the implications for policy learning. Section (ref) presents and analyzes PSPO. Section (ref) presents and analyzes OSPO. Section (ref) presents and analyzes RSPO. Section (ref) analyzes empirical calibration to the divergence budget. Section (ref) provides an empirical investigation of the algorithms, including in LLM alignment. \emph{Section (ref)} discusses possible extensions. \emph{Section (ref)} surveys related work. Full proofs appear in the appendices.
This section formalizes the policy optimizations objective, describes the preference data we observe, and shows how they jointly yield a single-index model. We define an affine-invariant metric for error in the index error and show how it controls error in the divergence-constraint-calibrated policy. We then articulate the precise identification property needed to have hope to control this metric and prove the assumption holds under a simple condition when we have at least three actions.
The target we focus on is the policy maximizing average rewards subject to a constraint on its deviation from a reference policy. This is a standard approach to align models to a new task while staying close to a pre-trained policy, preserving its language ability and preventing catastrophic forgetting. While the common divergence is Kullback-Leibler (KL), we here consider any $f$-divergence, of which KL is an example.
We let $x\in\Xcal$ denote context (e.g., user query) and $y\in\Ycal$ action (e.g., LLM response). The mean reward of taking action $y$ in context $x$ is given by the unknown reward function $r^\star(x,y)$. For simplicity we focus on a finite action space throughout the paper, $|\mathcal Y|<\infty$, while $\Xcal$ can be general Borel. Given convex $f:\RR^+\to\RR$ with $f(1)=0$, for any two probability mass functions $p,q\in\Delta^{\Ycal}$ on $\Ycal$ we define $D_f(p\|q)=\infty$ whenever $p(y)>0,q(y)=0$ for some $y$ and $D_f(p\|q)=\sum_{y:q(y)>0} q(y) f(p(y)/q(y))$ otherwise. Henceforth we will assume throughout that $f$ is twice continuously differentiable and strictly convex with derivative approaching $-\infty$ near $0$. KL is given by setting $f(u)=u\log u$. Other examples satisfying these conditions include $\alpha$-divergences (which include KL), Jensen-Shannon divergence, and the combination of KL and $\chi^2$-divergence as used in huang2024correcting.
Given a reference policy $\pi_{\rm ref}:\Xcal\to\Delta^{\Ycal}$, we are interested in the divergence-constrained reward-maximizing policy:
where the expectations are taken with respect to a context distribution $x\sim P_x$.
Combining a simple convexity argument with the results of wang2023beyond yields a closed form for $\pi^\star$.
(ref) simply requires that a pure-reward-maximizing policy is not already feasible in the divergence constraint thus rendering it irrelevant.
Toward optimizing (ref), we assume we have access to preference data consisting of $n$ tuples $(x,y_0,y_1,z)\sim P,$ with $w=(x,y_0,y_1)\in\Wcal$ drawn from some joint distribution with $x$-marginal matching $P_x$ (otherwise we can do importance sampling) and $z\in\{0,1\}$ being the indicator that $y_1$ is preferred to $y_0$. We index our data $(w_i,z_i)=(x_i,y_{i0},y_{i1},z_i)$ for $i=1,\dots,n$ and assume they are drawn iid.
We assume that preferences are drawn according to
where $\Phi^\star$ is an unknown cumulative distribution function (CDF).
The key in this paper is that we let $\Phi^\star$ be completely unknown, rather than setting it to a known function like sigmoid or the normal CDF. This is important since we only care to understand rewards so that we can optimize policies, rather than fully model the distribution of $z$.
For simplicity, we assume that $P$ is symmetric in actions (\ie, measure of a set is preserved when we replace $(x,y_0,y_1)$ by $(x,y_1,y_0)$) so that $\mathbb{E}[h(x,y_1)-h(x,y_0)]=0$ for any function $h(x,y)$.
In policy optimization we search over a policy class $\{\pi_\theta:\theta\in\Theta\}$. We emphasize that throughout, we let $\theta$ be a completely abstract parameter, not necessarily finite dimensional. In this sense, the policy class can be parametric or nonparametric, and is generally understood to be large and rich.
Let us assume realizability of this class for our learning task in (ref) in that $\pi^\star=\pi_{\theta^\star}$ for some $\theta^\star\in\Theta$. We will now show this essentially induces what is known as a single-index model ichimura1987estimation on the preference data.
Let us posit the following model for preference data: for $\theta\in\Theta$ and $\Psi$ a CDF,
Combining (ref) with (ref) and rearranging establishes that this is a valid model under policy realizability.
Specifically, (ref) can be rearranged as $r^\star(x,y)=\beta^\star h_{\theta^\star}(x,y)+\lambda^\star(x)$. Then $\lambda^\star$ is canceled by differencing the rewards of $y_1$ and $y_0$, and $\beta^\star$ is swallowed into the CDF as it is unrestricted.
(ref) is referred to as a single-index model (of binary choice) since it posits the existence of a parametric univariate composite statistic that summarizes all the relevant aspects of the inputs (here, $x,y_0,y_1$), which is often known as an index. The rest is a completely unrestricted nonparametric dependence on this univariate quantity $t^\star=t_{\theta^\star}$. We will refer to $h_\theta$ as potential functions and to $t_\theta$ as indices.
We henceforth impose (ref) so that, by (ref), $\theta^\star$, $t^\star$, and $\Psi^\star$ are well defined and characterize the data generating distribution.
Recall that, since $\theta$ is just an abstract parameter, the class of indices $$ \Tcal=\braces{ t_\theta(x,y_0,y_1)= f'\prns{\frac{\pi_\theta(y_1\mid x)}{\pi_{\rm ref}(y_1\mid x)}} - f'\prns{\frac{\pi_\theta(y_0\mid x)}{\pi_{\rm ref}(y_0\mid x)}} :\theta\in\Theta}, $$ may be parametric or nonparametric and is in general as rich as the policy class $\{\pi_\theta:\theta\in\Theta\}$. As such, we will characterize $\Tcal$ in terms of its own functional complexity, permitting rich classes and treating them generically.
While policies are the ones parameterizing the index, via the parameterization by $\theta$, not every policy is feasible in the divergence constraint. Policies are invariant to additive transforms of rewards but not scaling, as that determines the temperature or divergence. Nonetheless, once we fix the divergence constraint we obtain scaling invariance as well.
Given a parameter $\theta$, which gives a potential function $h_\theta$, let us define the corresponding $\beta$-regularized policy as $$ \pi_{\beta,\theta}(y\mid x)\;=\;\pi_{\rm ref}(y\mid x)\,(f')^{-1}\!\left(\beta^{-1}\big(h_\theta(x,y)-\lambda_{\beta,\theta}(x)\big)\right), $$ where $\lambda_{\beta,\theta}(x)$ is chosen so that $\pi_{\beta,\theta}(y\mid x)$ is a valid conditional probability mass function. With a slight abuse of notation let us further define the population-divergence-calibrated policy as $$ \pi_{\kappa,\theta}(y\mid x)\;=\; \pi_{\beta_{\kappa,\theta},\theta}(y\mid x), $$ where $\beta_{\kappa,\theta}>0$ is chosen so that $\pi_{\kappa,\theta}(y\mid x)$ satisfies $\mathbb{E}_x D_f(\pi_{\kappa,\theta}(\cdot\mid x)\|\pi_{\rm ref}(\cdot\mid x))=\kappa$ (which we know that $\pi^\star$ satisfies under (ref)). Namely, we can think of $\lambda_\beta(x)$ as being chosen so that, for any choice of $\beta$ and for any context $x$, the policy sums to one over $y$, and of $\beta_{\kappa,\theta}$ being then chosen such that the resulting policy has exactly the desired divergence.
We emphasize that here the calibration defining $\pi_{\kappa,\theta}(y\mid x)$ is done exactly on the true population distribution. We consider an empirical version in (ref), which is an actual algorithm that can be implemented on a data sample and which we show incurs an additive error. Note also that by construction, under (ref), we have $\pi^\star=\pi_{\kappa,\theta^\star}$.
Note that because of the affine transform by $\beta_{\kappa,\theta}>0,\lambda_{\kappa,\theta}(x)$, we must have that $\pi_{\kappa,\theta}$ and $\pi_{\kappa,\theta'}$ are the same whenever the indices $h_\theta$ and $h_{\theta'}$ are affine transforms of one another (with $x$-dependent location and $x$-independent scaling). Therefore, we do not actually need to learn $\theta^\star$ -- we only need to learn it up to this equivalence. Correspondingly, rather than phrase errors as $\ell_2$ distance in $\theta$ or even $L_2$ distance in $h_\theta$, we will seek to control distance up to this equivalence:
Clearly, if $\rho(h_\theta)=0$ then, under (ref), we have $\pi_{\kappa,\theta}=\pi^\star$. We would further like small $\rho$ to translate to small policy error. Before establishing this, we collect some of the basic regularity conditions we will use throughout.
The first two conditions ensure we are dealing with bounded quantities and the smoothness/curvature of objectives. The very last condition ensures that the data covers $\pi_{\rm ref}$ so that we can translate an index fitting well to the data (generated by $P$) to it inducing a small $\rho$ (defined by $\pi_{\rm ref}$). It is trivially satisfied when demonstrations are generated by drawing twice independently from $\pi_{\rm ref}$.
Equivalently, we can bound the expected total variation distance between $\pi_{\kappa,\theta}(\cdot\mid x)$ and $\pi^\star(\cdot\mid x)$. We therefore henceforth focus on convergence and rates for $\rho(h_{\hat\theta})$ for a learner $\hat\theta$ as that would imply convergence and rates for policy error.
In the previous section we argued that we need only learn $\theta^\star$ up to affine transformations of the potential function $h_{\theta^\star}$ it parameterizes, since all of those will lead to the same policy after divergence calibration. At the same time, however, under the single-index model (ref), two indices are observationally equivalent if differences therein are monotonic transformations of one another, since this transformation can be captured by the freely varying CDF $\Psi$.
Therefore, to have any hope of controlling $\rho(h_\theta)$ by making the potential $h_\theta$ explain the preference data, we need to assume that observational equivalence implies policy equivalence. We next posit this assumption and proceed to show it is in fact quite weak.
Note that this assumption is weaker than assuming that observational equivalence implies $\theta=\theta^\star$, as assumed in the econometrics literature that is interested in estimating $\theta^\star$ itself as a structural parameter (\eg, Cosslett1983,KleinSpady1993).
In fact, unlike those strong identification conditions, (ref) is quite natural and is in fact almost necessary when we have at least three actions, $|\Ycal|\geq3$. The key observation is that whenever we have $x,y_1,y_2,y_3$ with $t_{\theta^\star}(x;y_i,y_j)=m(t_{\theta}(x;y_i,y_j))$ for $(i,j)\in\{(1,2),(2,3),(1,3)\}$ then
This imposes Cauchy's functional equation (additivity) on a monotonic function. When the domain spanned by $t_{\theta}(x;y_i,y_j)$ is sufficiently rich, the only feasible solution for $m$ is a linear function Aczel1966. Note that we need $\abs{\Ycal}\geq3$ to be able to make this rich enough, else all we can possibly get is the oddness of $m$. In the application of interest, of course, $\Ycal$ is quite big.
We now summarize one simple sufficient condition for using this to guarantee (ref).
We now consider different policy learners motivated by a variety of existing semiparametric estimators of single-index models. We begin by adapting the method of Cosslett1983, which profiles the link function $\Psi$ in a maximum likelihood estimation by letting it be freely chosen to maximize the likelihood for any given parameter $\theta$.
Intuitively, profiling removes the infinite-dimensional nuisance $\Psi$ by letting the data choose the monotone function that best explains outcomes for a given index. We then show uniform convergence of the profiled objective, population identification up to affine invariance, existence of empirical maximizers, and policy consistency.
Define the binary cross-entropy given both an index $t$ and a candidate link function $\Psi$: \[ \ell(t,\Psi;w,z)= z\log \Psi\big(t(w)\big)+(1-z)\log\big(1-\Psi\big(t(w)\big). \] Recall we defined $w=(x,y_{0},y_{1})$.
If we knew $\Psi^\star$ we would plug it in and optimize the empirical average of the above likelihood over $t_\theta$ -- \eg, this is the approach of DPO using the logistic link function. Instead, since we do not know $\Psi^\star$, we can consider both $t_\theta$ and $\Psi$ as parameters to maximize the likelihood over. Equivalently, for every $t_\theta$, we optimize the link, thus profiling it out of the objective. This profiled objective is defined as follows and its maximizer is the PSPO learner:
where $\Psi\uparrow$ denotes the restriction to monotonic non-decreasing functions $\mathbb{R}\to[0,1]$.
Similar to DPO, the approach here is to align the LLM policy by maximizing the likelihood of the reward function implied by the aligned policy. Recall, the policy is wrapped inside $t_\theta(w)=f'\prns{\frac{\pi_\theta(y_1\mid x)}{\pi_{\rm ref}(y_1\mid x)}} - f'\prns{\frac{\pi_\theta(y_0\mid x)}{\pi_{\rm ref}(y_0\mid x)}}$. Here the likelihood is (ref), which profiles the link rather than assuming it is known and plugging it in.
We now establish rate-free consistency of PSPO, allowing for a nonparametric index class $\Tcal$.
Our nonparametric complexity assumption is the condition that the class of thresholded indices is $P$-Glivenko-Cantelli, which can for example be satisfied if the class of indices $\Tcal$ is VC-subgraph. The assumption that $\mathrm{Var}(\mathbb{E}[z\mid w])\neq0$ is equivalent to the true index being nontrivial, $t^\star\neq0$. And, the assumption that $\{t/\|t\|:t\in\Tcal\backslash\{0\}\}$ is $L_2$-compact is essentially a smoothness assumption so that we cannot take $t\to0$ while making it overly rough and complex at small scales. Together, these show that the PSPO induced policy converges, even if we cannot identify the index.
To optimize (ref) in practice, especially when $\pi_\theta$ is a neural network, we can use a first-order (gradient-based) method. A simple approach would be to alternate between taking a (batched stochastic) gradient step in $\theta$, taking $\Psi$ as fixed, and optimizing $\Psi$ for the current $\theta$. The former can be combined with any momentum and/or adaptive learning rate procedure. The latter is essentially a univariate monotone logistic regression and can be done using the Pool Adjacent Violators Algorithm (PAVA). This is akin to the Isotron algorithm KalaiSastry2009,KakadeKalaiKanade2011. Alternatively, differentiation through the isotonic regression can be done by implicit differentiation of PAVA KKT conditions or by training a separate network for $\Psi$ with monotonic constraints.
We can only obtain rate-free consistency for PSPO as it can generally be ill-behaved because profiling the link function can lead to a rough loss surface. This was the observation of KleinSpady1993, who instead proposed a semiparametrically efficient approach. Intuitively, rather than aim to plug in the link $\Psi_\theta$ that profiles the likelihood at $\theta$ as in PSPO, instead we should consider how perturbing $\theta$ changes the likelihood, that is, consider the score. The score has two orthogonal components: the distribution of the index and the distribution of $z$ given the index. At $\theta^\star$, the latter is just the true link $\Psi^\star$, but away from it this is just a conditional probability. Since $z$ is binary, this is just a conditional expectation, which we can estimate this conditional expectation and plug it in. In success, the behavior is as though we used the true conditional expectation. This orthogonality phenomenon is akin to Double/Debiased Machine Learning chernozhukov2024applied. In the rest of the section we formally define the learner and analyze it.
Consider the distribution of $z$ given an index $t_\theta$: $$g^\star_\theta(u)=P(z=1\mid t_\theta(w)=u). $$ This orthogonalizes perturbations to $\theta$, which determine the index, and perturbations to the distribution of $z$ given the index. These live on orthogonal components of the tangent space: we can think of the former as “angle" (corresponding to changing the distribution of the index) and the latter as “magnitude" (corresponding to the distribution of $z$ given the index).
Solving the score equal to zero is conveniently equivalent to maximizing a quasi-likelihood given by plugging in $g^\star_\theta$ as the link:
Indeed $g^\star_{\theta^\star}=\Psi^\star$ is the true link function, but away from $\theta^\star$ note that $g_{\theta}$ may not even be monotonic. Fortunately, near $\theta^\star$, the difference between the true likelihood $\ell(t_\theta,\Psi^\star;w,z)$ and the quasi-likelihood $\ell(t_\theta,g^\star_\theta;w,z)$ vanishes very quickly as we get closer to $\theta^\star$, so that, under certain regularity, the estimator that maximizes $L_n^{\rm ideal}(\theta)$ behaves as though we knew the link function exactly. For example, if the link were logistic, the first order behavior would be the same for this estimator as for the one optimizing the well-specified DPO objective. Of course, the latter would no longer even be consistent when the link is no longer logistic. The former would adapt to whatever the link may be and behave as though we knew it exactly.
Unfortunately, we do not know $g^\star_\theta$ in (ref). Fortunately, we can easily estimate $g_\theta$ as a univariate regression of $z$ on $t_\theta$. Suppose we are given an estimator $\hat g_\theta$ for each $\theta\in\Theta$. Then we can define the plug-in quasi-likelihood as follows and its maximizer as the OSPO learner:
One possible choice for $\hat g_\theta$ is a kernel regression estimator: given a dataset $\Dcal$, set
where $K$ is a kernel such as Gaussian $K(u)=\exp(-\|u\|^2/2)$ or Epanechnikov $K(u)=\max\{1-u^2,0\}$ and $b_\theta\to0$ is a bandwidth. One can choose $b_\theta=h$ to be a common bandwidth. However, we will instead use $b_\theta=h\,\hat\sigma(t_\theta(w))$ to be scaled by the empirical standard deviation of $t_\theta$ on $\Dcal$ so that $\hat g_\theta$ is scale invariant. Indeed, $g^\star_\theta$ is invariant to scale in that $g_\theta^\star(u)=g_{\theta'}^\star(\beta u)$ whenever $t_\theta=\beta t_{\theta'}$ for $\beta>0$. This is because $g_\theta^\star$ is a conditional expectation and scaling does not change the information content of the conditioning set.
The dataset $\Dcal$ can be the same $n$ datapoints used in (ref). KleinSpady1993 use a leave-one-out version where $\Dcal=\{(w_j,z_j):j\neq i\}$ is used to train the kernel regression plugged into the $i\thh$ term in (ref). That is the practical version we recommend. However, since the theoretical analysis only derives rates, we will focus the analysis on using a separate independent dataset, as splitting the data would only affect constants in the bound.
There is a small wrinkle: if $t^\star$ is a function of $t_\theta$ then (ref) cannot distinguish between them, as $g_{\theta}(t_\theta)=g_{\theta^\star}(t_\theta^\star)$ will be identical and so will the quasi-likelihood above. KleinSpady1993 get around this and achieve efficiency for $\theta^\star$ by requiring a very strong identification condition: that $g_{\theta}(t_\theta)=g_{\theta^\star}(t_\theta^\star)$ must imply $\theta=\theta^\star$. This can be unreasonably strong as it essentially means we know the sign in advance and precludes classes closed under negation ($t^\star=-t_\theta$ would imply $\theta=\theta^\star$ which would then imply $t^\star=0$). This is in contrast to PSPO's objective (ref), which restricts the profiling to monotonic nondecreasing link functions so that negating the index would not preserve the profile likelihood objective.
Focusing on identifying policies rather than underlying parameters, we will instead impose a slight strengthening of our up-to-scaling identification condition ((ref)) so that near-observational-equivalence translates to nearly pointing in the same (or opposite) directions. Quantifying the level of “near" equivalence is needed to ensure sufficient curvature in the quasi-likelihood.
Note that the left-hand side can also be written as $\mathbb{E}\bigl[\mathrm{Var}(t^\star(w)\mid t_\theta(w))\bigr]$. We can also rescale by $1/\mathrm{Var}(t^\star(w)$ to express the above as a relationship between a nonparametric $R^2$ and the linear $R^2$ (in the population). (ref) posits that if $t_\theta$ contains nearly as much information as $t^\star$, then the linear regression error needs to be small. Since $\Fcal_{\rm lin}$ permits negative coefficients, $t_\theta$ may point in the opposite direction from $t^\star$. Note that, under the (ref)'s coverage of $\pi_{\rm ref}$, (ref) implies (ref).
We now show that this means that nearly optimizing the quasi-likelihood at the population level means nearly aligning with $\pmt^\star$. This curvature in the quasi-likelihood is, in particular, needed for establishing a rate, as we do later in this section. We next introduce some additional regularity that various derivatives exist and are bounded, ensuring the quasi-likelihood is sufficiently smooth.
We focus on permitting flexibility in the parametrization and identify only policies, which are invariant to the index's scale and $x$-based location. However, (ref) still allows flipping the direction of the index while maintaining its informational content, since the acute angle will remain the same. There is a simple fix: ensure marginal sign agreement with $z$. As long as we are sufficiently well-aligned with $\pmt^\star$, even with the wrong sign, we can ensure that flipping the sign to align with $z$ will maintain error magnitude while fixing signs.
The smoothness condition is analogous (although, incomparable) to the compactness condition in (ref).
This means that it suffices to learn $t$ to align along the direction of $\pm t^\star$. We can always fix the sign afterward for any one given $t$. Notice that importantly both the minimum necessary alignment and the constants on the right-hand side above do not depend on the particular $t$. In practice, we use the same data to both train $\hat\theta^{\rm OSPO}$ and to align the sign. In theory, we simply split the data and fix the sign after learning some $\hat t$ since the rate is unchanged and this avoids having to impose more complexity assumptions on $\Tcal$ so as to be able to prove (ref) uniformly over $\Tcal$.
We now analyze the policy error incurred by the OSPO learner, given a generic plug-in $\hat g_\theta$ in terms of its own error. Crucially, we will show that the error in $\hat g_\theta$ enters into our final bound quadratically, meaning it suffices to learn $g^\star_\theta$ quite slowly and it would still not affect the final rate. In this sense, the OSPO learner is quite robust to learning $\hat g_\theta$.
We need to assume some regularity for the plug-in $\hat g_\theta$: that is has almost the same range as $g^\star_\theta$ and certain Lipschitzness locally around $t^\star$.
Note that using the scale-invariant distance $\inf_{a\in\RR}\|t^\star-at_{\theta}\|$ on the right-hand side of the first Lipschitz equation makes sense because $g^\star$ is scale invariant: for any $\theta,\theta'\in\Theta$ with $t_\theta=\beta t_{\theta'}$ for $\beta>0$, we have $g^\star_\theta(u)=g^\star_{\theta'}(\beta u)$. The same holds for the second Lipschitz equation if $\hat g$ is similarly scale invariant (such as is the case for the kernel regression in (ref)).
We also need to control the complexity of the index class $\Tcal$. We do so in terms of rates on the covering number $N_2(\epsilon,\mathcal T)$, the minimal number of square-integrable functions $t_1,\dots,t_N$ needed to obtain $\sup_{t\in\Tcal}\min_{i=1,\dots,N}\|t-t_i\|\leq\epsilon$.
This neatly packages two cases: the truly nonparametric case $p>0$ and the “simple" case $p=0$. In particular, if $\mathcal T$ were a VC-subgraph class of dimension $v$ then we would have (ref) with $p=0$ and the given $v$ and some universal constant $A$ vdVaartWellner1996. In the appendix, the proof only uses (ref) to bound a critical radius wainwright2019high, so the assumption can be supplanted by bounds thereon when available instead.
We can now state our result.
The style of our analysis is similar to the approach of foster2023orthogonal for leveraging orthogonality of loss functions to nuisances in statistical learning, but we derive all bounds (except from the empirical process bounds) from first principles to avoid too stringent conditions since we have to deal with a parameter-dependent nuisance function.
Crucially, the dependence on the error in $\hat g_\theta$ is quadratic, meaning if we learn it a rate slightly faster than $n^{-1/4}$, or even just faster than $\sqrt{\varrho_n}$, then it will not affect the final rate at all. This is incredibly lax considering that $\hat g_\theta$ is just a univariate regression. Therefore, as we show next, under very mild regularity, even kernel regression can achieve a rate fast enough that it does not affect the first order behavior of the bound above.
(ref) depended on having a plug-in estimator $\hat g$ with small uniform $L_4$ error. Any plug-in with such a guarantee can work. One proposal was the kernel regression estimator in (ref) with scale-free bandwidth $b_\theta=h\,\hat\sigma(t_\theta(w))$, which we will analyze here. We show it attains a uniform $L_\infty$ (thus also uniform $L_4$) error rate of $m^{-1/3}$ under mild conditions, where we let $m=\abs{\Dcal}$ be the size of the data set used to fit the kernel regression. Since this error enters the bound in (ref) quadratically, this means it has no influence on the rate of the bound and is eventually insignificant.
We will need some regularity conditions to establish this result. First, we impose conditions on the choice of kernel and bandwidth. These will be satisfied, \eg, if we use the Epanechnikov kernel and let $h\asymp (\log m/m)^{1/3}$.
Next, we need some regularity, including a complexity condition on $\Tcal$ that is more specific than the critical radius used in the last section.
Consequently, if we choose $h\asymp (\log m/m)^{1/3}$, we obtain a rate of $\tilde O(m^{-1/3})$. If $m\asymp n$ (\eg, evenly split the data) this contributes an insignificant $\tilde O(m^{-2/3})=o(n^{-1/2})$ term to (ref).
Given a differentiable $\hat g_\theta$ (\eg, kernel regression), we can directly apply standard first-order optimizers to (ref).
Theoretically, to obtain the aligned policy, we must split the data four ways: one quarter to learn $\hat g_\theta$ for all $\theta\in\Theta$, one quarter to construct the quasi-likelihood and fit $\hat\theta^{\rm OSPO}$ to maximize it, one quarter to learn the sign of the index using $\hat s$, and finally one quarter to learn $\hat\beta_{\hat\theta^{\rm OSPO}}$ to calibrate to the divergence constraint (see (ref)). In practice, we suggest to just reuse the whole data for each of the latter steps, since the index sign and divergence calibration are single dimensional and should be quite stable (and the latter even only depends on $x$-data).
For learning $\hat g_\theta$, we suggest using leave-one-out kernel regression so that when we evaluate it at $t_\theta(w_i)$, we fit it on everything but $(w_i,z_i)$, and only at the current iterate $\theta$ rather than for all $\theta\in\Theta$. Moreover, to accommodate first-order optimization based on batches of data, rather than use all the data, which can be prohibitive, we suggest to use a bank of recent data points of about 20$\sim$50 times the batch size. Then $\hat g_\theta$ can be computed for the current iterate and gradients be applied directly to (ref) and combine with any momentum and/or adaptive learning rate procedure.
Both PSPO and OSPO were based on maximizing something akin to a likelihood. We now consider an alternative based on optimizing a ranking objective. Han1987 proposed a max rank correlation approach for learning an identifiable linear index in a family of observation models that includes binary choice, in which case the estimator maximizes the area under the curve (AUC). Sherman1993 showed the estimated linear coefficients converge at a $n^{-1/2}$ rate and are asymptotically normally distributed. However, in the single-index binary choice setting, the asymptotic variance is generally larger than the efficiency bound derived by Cosslett1983 and achieved by KleinSpady1993, meaning the max rank correlation estimator is not efficient and does not behave asymptotically like maximum likelihood with the true link function, as if it were actually known.
Nonetheless, a ranking objective for RLHF is appealing as it involves no additional nuisances (like PSPO) while still attaining a rate (like OSPO). In this section, we study how to leverage it for our alignment problem with nonlinear index classes implied by flexible policies, characterizing policy error in terms of general function complexity. To do so, we first establish an identification margin similar to (ref), showing that small excess AUC translates to learning the index up to scale. Then we can leverage empirical risk minimization results of $U$-statistics clemenccon2008ranking to establish a bound on the excess AUC for nonparametric index classes. Chaining the two together with (ref), we obtain policy convergence.
Define the AUC of an index as
where $(w_0,z_0),(w_1,z_1)$ are two independent draws from $P$. This is another way to express the area under the receiver operating characteristic curve given by thresholding $t(w)$ to classify $z$ menon2016bipartite.
Notice this is distinct from averaging over single demonstration pairs $(x,y_0,y_1)$ how often the ranking between $h_\theta(x,y_1)$ and $h_\theta(x,y_0)$ matches the label $z$. This amounts to a 0-1 loss for the sign of $t_\theta(w)$ in classifying the label $z$, maximizing which corresponds to max-score estimators (discussed in (ref)). Importantly here we average over pairs of pairs.
In particular, taking the pairs-of-pair point of view further, notice that, since $P$ is symmetric in $y_0,y_1$, if $\Phi^\star$ were also symmetric ($\Phi^\star(u)=1-\Phi^\star(-u)$), then we can rewrite (ref) for our alignment-from-preferences setting without any conditioning:
Essentially, to get a draw from the conditional on $z=1$, take any observation $(x,y_0,y_1,z)$ and just make sure that the winning demonstration of the two per the observed label $z$ is placed in the $y_1$ position, and vice versa for $z=0$.
We next show that, in our setting, the set of maximizers of $\mathrm{AUC}(t)$ over $t\in\Tcal$ are exactly $t^\star$, up to scale.
In order to obtain rates, however, we need to show that small AUC suboptimality corresponds to being nearly equal to $t^\star$ up to scale, which correspondingly controls policy error, based on our results. We will again leverage (ref) to quantify near-observational equivalence. We will also need some additional regularity similar but distinct from (ref).
The latter part of (ref) regarding the density of $t^\star$ will be used both to establish the next result this section as well as to establish a noise (margin) condition in the next section (for which only the upper bound is needed).
Note that unlike (ref), here we can restrict to nonnegative scalings. Indeed, AUC is not invariant to flipping the sign -- that will yield the complement of the AUC. Our regularity conditions above ensure that the optimal AUC (attained by $t^\star$) is strictly above $0.5$ and that the AUC is $\frac{2}{3}$-H\"older continuous in the score function. Since AUC is invariant to positive scalings, this means that for $\iota$ small enough, the covariance of $t^\star$ and $t_\theta$ cannot be negative.
Given (ref), to align the policy to be quite close to the optimal policy, it suffices to find a policy whose implied index function attains near-optimal AUC. A reasonable approach to doing so is to maximize the empirical AUC, giving rise to the Ranking SPO (RSPO) learner:
where $\phi(u)=\indic{u>0}+\tfrac12\indic{u=0}$ and $n_1=\sum_{i=1}^n z_i$, $n_0=n-n_1$.
Under the assumption that $\Phi^\star$ is symmetric, we can alternatively use an empirical version of (ref):
Note that while (ref) and (ref) are exactly equal under the assumption that $\Phi^\star$ is symmetric, their sample versions ((ref) and (ref), respectively) are generally distinct. In practice, since the symmetric assumption is quite reasonable, we suggest to use (ref) as it essentially gets double the data for free. In theory, in order to easily leverage existing results clemenccon2008ranking and since the rates would be unaffected, we analyze (ref).
Since the AUC objective involves a step function, to appropriately control the complexity we need to consider the sets of pairs that ranked in order or anti-order. To measure the complexity of a class $\mathcal S$ of subsets $S\subset\mathcal W\times\mathcal W$, define $N_\triangle(\epsilon,\mathcal S)$ as the smallest number of subsets $S_1,\dots,S_N\subset\mathcal W\times\mathcal W$ we need to take so that for every $S\in\mathcal S$ there exists a subset $S_i$ where $\mathbb P_{w,w'\sim P^2}((w,w')\in S,(w,w')\notin S_i\vee (w,w')\notin S,(w,w')\in S_i)\leq\epsilon$, that is, the symmetric difference has measure at most $\epsilon$.
This again neatly packages two cases: the truly nonparametric case $p>0$ and the “simple" case $p=0$. In particular, if $\mathcal R_{\mathcal T}$ were a VC class of dimension $v$ then we would have (ref) with $p=0$ and the given $v$ and some universal constant $A$ vdVaartWellner1996.
We can now establish our main result for RSPO, where we permit flexible indices under (ref).
Importantly, we are able to attain policy error rates without having to estimate a nuisance parameter as in OSPO. The rates are generally incomparable, but we can sometimes obtain control on $N_\triangle(\epsilon,\mathcal R_{\mathcal T})$ via control on $N_q(\epsilon,\mathcal T)$, the minimal number of $\|\cdot\|_q$ balls needed to cover $\mathcal T$.
If we take (ref) with $\alpha=1$ (\eg, $t(w)$ has bounded densities) and assume (ref) with exponent $p$, we obtain (ref) with exponent $3/2p$. Thus, we maintain $\tilde O(1/\sqrt{n})$ rates in the “simple" case, but this can quickly deteriorate. For example, if $p=2$, OSPO would attain a rate of $\tilde O(n^{-1/4})$ compared to a much worse rate of $\tilde O(n^{-1/12})$ for RSPO. Nonetheless, the rates are indeed generally incomparable, and RSPO avoids nuisance estimation, circumventing the link function altogether by sidestepping the likelihood.
The empirical AUC objective in (ref) is somewhat difficult to optimize since $\phi$ is a step function. One approach, in the spirit of Horowitz1992, is to replace $\phi(u)$ by $F(u/h)$ where $F$ is a smooth symmetric CDF and $h\to0$ is a bandwidth. (Note that unlike the setting studied by Horowitz1992, the smoothing is not needed for statistical reasons since the population AUC objective is already sufficiently smooth, unlike the objective they study; see (ref).) This would make the objective differentiable so that we can use any off-the-shelf first-order optimizer, but the objective would still be non-convex in $t$.
A more appealing alternative is to use a surrogate loss function for bipartite ranking agarwal2014surrogate. In particular, we can obtain one from any surrogate loss for classification bartlett2006convexity by transforming the data into classification data. Let us focus on transforming (ref), as that is the variant we recommend in practice; the transformation is analogous for (ref). Create the $2n^2$ data points, where for each $i,j$, we label the features $(x_j,y_{j,1-z_j},y_{j,z_j},x_i,y_{i,z_i},y_{i,1-z_i})$ with $1$ and the features $(x_j,y_{j,z_j},y_{j,1-z_j},x_i,y_{i,1-z_i},y_{i,z_i})$ with $0$. Then (ref) is given by $2$ times the 0-1 loss of this classification problem over class scores $(w,w')\mapsto t(w)-t(w')$. Now use any convex classification surrogate loss, such as hinge or logistic. To use batched first-order optimization, rather than taking batches over the $n^2$ augmented data points, we can take batches of size $B$ from the original $n$ data points and simply consider the gradient for the corresponding $B^2$ terms, as such would still form a stochastic gradient.
After training $\hat\theta$, we must choose a temperature $\beta$ that enforces the divergence budget. So far we wrapped this into the population-calibrated policy $\pi_{\kappa,\theta}$, but this cannot be done in practice. We now discuss how to calibrate the divergence using a validation sample of contexts $x$, and we show how this leads to an additive policy error that we can simply add on top of our previous bounds on the error in $\pi_{\kappa,\hat\theta}$ for a learner $\hat\theta$.
The procedure solves a one-dimensional root-finding problem on held-out contexts. We show a uniform law of large numbers in the temperature, a root-$m$ calibration rate, and the ensuing policy error guarantee. While the results require a separate validation set, so that we can treat $\hat\theta$ as a fixed input, in practice we may calibrate on the same data (using only the contexts).
Fix a compact interval $\mathcal B=[\underline\beta,\overline\beta]\subset(0,\infty)$. Let $x_1,\dots,x_m\stackrel{\mathrm{iid}}{\sim}P_x$ and define \[ \hat\Phi_m(\beta,\theta)=\frac1m\sum_{i=1}^m D_f\!\bigl(\pi_{\beta,h_\theta}(\cdot\mid x_i)\,\big\|\,\pi_{\mathrm{ref}}(\cdot\mid x_i)\bigr), \qquad \hat\beta_\theta\in\argmin_{\beta\in\mathcal B}\bigl|\hat\Phi_m(\beta,\theta)-\kappa\bigr|. \]
Therefore, we can simply add this additional $\tilde O(1/\sqrt{m})$ term to whatever policy error we derived on the population-calibrated policy by bounding $\rho(\theta)$ via (ref). In theory, if we split the $n$ data points and use a constant fraction for calibration, we will never deteriorate the rate. In practice, we can use the same data since calibration should be rather stable and only ever uses the $x$-data.
We next study the empirical performance of the proposed methods and their robustness to arbitrary link functions. We use KL divergence ($f(u)=u\log u$) throughout. PSPO is implemented as in (ref), alternating between batched gradient steps in $\theta$ and PAVA updates to $\Psi$. OSPO is implemented as in (ref), using batched gradient steps in $\theta$ with leave-one-out kernel regression on data from recent batches to fit $\hat g_\theta$. RSPO is implemented as PoP-DPO, as in (ref), with batching over the $n$ data and taking gradients in the corresponding $(\text{batch size})^2$ loss terms. Code is available at \url{https://github.com/causalml/spo/}.
First, we consider an illustrative small synthetic setting. We simulate preference data with 20-dimensional Gaussian contexts $x$ and $\abs{\Ycal}=10$ actions. We set the reference policy to uniform and consider a policy class $\pi_\theta$ given by a feedforward network with two hidden layers. We fix some $\theta^\star$ and set $r^\star(x,y)=10 \log(\pi_{\theta^\star}(y\mid x)/\pi_{\mathrm{ref}}(y\mid x))$. Preferences are generated from an even mixture of two shifted logistic links $0.5\,\sigma(4(u-s))+0.5\,\sigma(4(u+s))$ with shift $s$ controlling misspecification ($s=0$ corresponds to logistic link). Each method learns $\pi_\theta$ from $n=1000$ preference pairs, and we repeat the experiment 1000 times. Additional detail is given in (ref).
(ref) shows the average reward (and 90% confidence intervals) at KL budget $\kappa=0.2$ as $s\in[0,1.5]$ varies and the average reward-divergence curves for $\pi_{\beta,\hat\theta}$ as $\beta$ varies, at $s=0$ and $s=1.5$. All methods perform well when logistic is well-specified ($s=0$). DPO performance deteriorates as $s$ grows, and so does RSPO due to its logistic surrogate loss, albeit providing some robustness at low misspecification. PSPO performs less well, consistent with its rough loss surface, but is robust to misspecified link, with performance roughly flat as $s$ varies. There is some light deterioration due to the lower SNR given the link is very flat around 0 when $s$ is large (rather than due to misspecification). OSPO consistently performs the best across all $s$, consistent with it being efficient no matter what the link is, whether logistic or not.
We next consider an experiment aligning the Qwen3 model yang2025qwen3 using the UltraFeedback data set cuiultrafeedback and Skywork-Reward-V2 reward model liu2025skywork. We take as our reference model Qwen3-0.6B after some light supervised fine-tuning on UltraFeedback responses to ensure format adherence. In order to be able to evaluate and compare learned policies, we take Skywork-Reward-V2-Qwen3-1.7B as the ground truth reward function. For prompts from UltraFeedback, we generate two responses from the reference policy, score them with the reward model, and take the difference, normalizing so that differences have variance 1 across the data. We generate preference observations using the same mixture of shifted logitic links as in the last section. We run each method on the preference observations alone (no reward) to train a rank-16 LoRA. We then vary $\beta$, use (ref) to generate responses from the aligned and scaled policy, and score using the reward model. Additional detail is given in (ref).
(ref) depicts the resulting average reward over 4 replications at $\kappa=0.5$ (with 90% confidence intervals for the difference to DPO). It overall appears to convey a similar story as the synthetic experiment, albeit with more noise, given only four replications. We again see all methods perform similarly with a well-specified logistic link, DPO deteriorating when the logistic link becomes more misspecified, and the SPO methods remaining robust to a non-logistic link. OSPO and PSPO seem to be noisier and possibly unstable, consistent with their more intricate training procedures. RSPO (as PoP-DPO) seems to strike a good balance of simplicity and robustness, offering a practical approach to still use logistic loss optimization while squeezing more comparisons from preference data rather than rely on a well-specified link to extrapolate, given we are interested in more than just classification.
OSPO as posited in (ref) requires we estimate $\hat g_\theta$ for each $\theta\in\Theta$, and moreover the bound in (ref) depends on its uniform convergence over $\theta\in\Theta$. One way to avoid this would be to localize the link estimate around an initial $\theta$ estimator in the style of kallus2019localized. That is, first estimate an initial $\hat\theta$ (perhaps using PSPO), then estimate $\hat\Psi$ as the univariate regression of $z$ on $t_{\hat\theta}(w)$, then maximize the plug-in quasi-likelihood $\frac{1}{n} \sum_{i=1}^n \ell(t_\theta,\hat\Psi;w_i,z_i)$.
This has the benefit that in the end what we optimize looks like a standard classification loss with a fixed link function, as in DPO. While $\hat\Psi$ may not be monotonic, we only get a better estimate via rearrangement chernozhukov2009improving, ensuring monotonicity. It may, however, not be so smooth. Additionally, it may be challenging to ensure fast enough convergence of the initial $\hat\theta$ so that we get a fast enough rate on $\hat\Psi$, since our current analysis for PSPO gives no rate. Finally, the analysis will require adapting the approach of kallus2019localized, which is developed for point estimation and inference, to orthogonal statistical learning in the vein of foster2023orthogonal. This therefore remains a promising direction of future work.
In the single-index model, the distribution of $z$ only depends on $w$ via the index $t^\star(w)$. This means that in the random utility interpretation, $z=\indic{t^\star(w)\geq\epsilon}$, the idiosyncrasy $\epsilon$ is homoskedastic, $\epsilon\indep w$. Namely, $\Psi^\star$ is the $w$-independent CDF of $\epsilon$. A strict generalization is to require only that $\mathrm{Median}(\epsilon\mid w)=0$ is a constant function of $w$. This means that $z\mid w\sim\mathrm{Bernoulli}(\Psi^\star_w(t^\star(w)))$ for an unknown $w$-dependent CDF $\Psi^\star_w$ under the only constraint $\Psi^\star_w(0)=1/2$. Note that this is not a single-index model since the distribution of $z$ would depend on $w$ not only via a single index $t^\star(w)$.
This is the model studied by Manski1975,Manski1985,KimPollard1990,Horowitz1992 for linear indices $\Tcal=\{w\mapsto\beta\tr w\}$. Indeed, in this model, any $t$ with $\indic{t(w)\geq0}=\indic{t^\star(w)\geq0}$ is observationally equivalent to $t^\star$. Therefore, it is generally impossible to identify up to scale anything much more complex than a linear index. That is, requiring that sign agreement of $t,t'$ implies $t\propto t'$ is highly restrictive. It, for example, prevents any class that admits “bump" functions, such as H\"older or Sobolev classes, since they would maintain sign while changing shape and scale. In contrast, the identification (ref) is much more lax and can admit such nonparametric classes. For example, for any atomless CDF $F_0$, any subset of $\{af:a>0,f(w)\sim F_0\}$ (such as further restricting $f$ in some nonparametric function class) satisfies (ref). Fixing the marginal distribution thus is exactly what is done with normalizing flows / invertible networks.
For learning linear indices up to scaling, KimPollard1990 establish $n^{-1/3}$ rates for the normalized coefficients that define the half-space that minimizes the binary 0-1 classification loss (the so-called max-score estimator). Horowitz1992 replaces the 0-1 step function with a smooth CDF with a vanishing bandwidth (the so-called smoothed max-score estimator) and establishes a $n^{-s/(2s+1)}$ rate when $\Psi_w^\star$ is $s$-times H\"older-continuously differentiable.
These results can be directly applied to learning indices implied by realizability via (ref) up to scale (and chained with (ref) for policy error), provided they are linear. Although identification up to scale (and therefore of divergence-constrained policies) for more complex classes is doubtful, these methods may still be practical ways to align LLMs, that is, by aligning the sign of $t_\theta(w)$ with that of $z-0.5$. In particular, the smoothed objective of Horowitz1992, while non-convex and potentially very rough, is differentiable so that first-order updates can be applied. This essentially ensures the LLM aligns (in the sense of divergence-constrained max-reward) with some reward whose differences align with the signs of true reward differences, even if the rewards are in fact different and lead to different policies -- this is of course still valuable. Notice that if instead of smoothing, we used the logistic loss as a surrogate for the 0-1 loss we would have the DPO objective.
Notice that, while max score is equivalent to maximizing the frequency that $h_\theta(x,y_z)$ is ranked higher than $h_\theta(x,y_{1-z})$, this is distinct from the ranking approach in RSPO. In RSPO we ranked pairs of pairs of demonstrations. Crucially, by doing so we probed $t^\star(w)$ at a variety of thresholds $t^\star(w')$, rather than only probing it at $0$. This ensures we were sensitive to information about all level sets, rather than just at $0$ (\ie, the sign). This is also the source of the better asymptotics seen in the economics literature on structural estimation of linear indices (\eg, $n^{-1/2}$ in Sherman1993 vs $n^{-1/3}$ in KimPollard1990 for linear indices): averaging over the many thresholds smoothes out the step function, while always thresholding at $0$ gives no opportunity for such smoothing. On the other hand, for this to work requires a single-index model (\ie, a common link function, equivalently $\epsilon\indep w$), while if all we know is $\mathrm{Median}(\epsilon\mid w)=0$ we cannot do much more than threshold at $0$ and learn only the sign. In this paper, of course, we focused on single-index models, in which case the pairs of pairs ranking appears preferable, giving both a better chance for identifying the reward shape and thus the optimal policy as well as better asymptotic behavior.
\paragraph{RLHF/RLAIF.} From the first demonstrations of reward learning from preferences Christiano2017,Ziegler2019,Stiennon2020 to instruction-following models Ouyang2022 and safety-oriented alignments Bai2022, RLHF/RLAIF is a primary ingredient in modern LLM practice. zhan2023provable prove regret bounds for offline RL from preference data. While RLHF for LLM alignment often employs KL constraints or penalties, wang2023beyond study the use of other $f$-divergences, and huang2024correcting analyze the use of $\chi^2$-divergence.
DPO Rafailov2023 avoids explicit reward modeling by recognizing the optimal policy already encodes the reward, so the aligned policy can be trained by fitting the implied reward function to the data, assuming a known logistic link function. We follow the idea of implied reward function but drop the known link function (and generalize KL to any $f$-divergence). Assuming a known link between reward and preferences is generally common to RLHF Rafailov2023,glaese2022improving,Ziegler2019,ibarz2018reward.
A variety of alignment methods depart from the structural/generative framework, where preferences are assumed to optimize a random utility or equivalently that preferences are generated by some conditional probability distribution given demonstrations and reward. ethayarajh2024kto consider loss-aversion in the spirit of the prospect theory of human decision making (as compared to utility-maximizing). kong2024perplexity,liang2024ropo consider non-stochastic corruptions to labels.
\paragraph{Econometric choice and semiparametrics.} Discrete choice modeling in econometrics was pioneered by McFadden1974, who formalized random utility models and established the multinomial logit as a tractable framework for estimating preferences and welfare from choice data. Subsequent work expanded the basic logit to allow for richer substitution patterns and unobserved heterogeneity, including nested logit McFadden1978, mixed logit and random coefficients models Berry1995,Train2009, and more general GEV formulations McFadden1981. Identification and estimation of structural parameters in the presence of price endogeneity and unobserved product characteristics led to instrumental-variables approaches such as the BLP framework BerryLevinsohnPakes1995,BerryLevinsohnPakes2004, along with related control-function and inversion-based methods Newey1999,BlundellPowell2004. Discrete choice models have also been extended to dynamic settings to study intertemporal decision-making, beginning with Rust1987 and followed by a large literature on estimation, identification, and computational methods for dynamic discrete choice HotzMiller1993,AguirregabiriaMira2002,ArcidiaconoMiller2011.
Semiparametric binary choice with an unknown link function dates to Cosslett1983, with efficient estimation first achieved by KleinSpady1993. Maximum score estimation Manski1975,Manski1985 and its smoothed variants Horowitz1992 provide robustness to link misspecification, while the cube-root asymptotics of such estimators were characterized by KimPollard1990. Closely related order-restricted and isotonic likelihood methods are developed in Barlow1972,Robertson1988. More broadly, these models fall within the class of single-index models, which extend beyond binary choice and admit alternative estimation strategies such as semiparametric least squares Ichimura1993, average derivative estimators PowellStockStoker1989, and rank-based or rank-correlation estimators Han1987,Sherman1993. This literature emphasizes identification and estimation of structural parameters under minimal distributional assumptions beyond an underlying parametric structural model.
Divergence-constrained alignment of LLMs induces a semiparametric single-index model with the policy class parametrizing the index. This significantly relaxes assuming the link function is known exactly, as in DPO. We considered policy learning by maximizing the profiled likelihood, orthogonolized quasi-likelihood, or bipartite ranking accuracy of the index implied by the policy. Our analysis gave finite-sample bounds and admitted nonparametric indices identifiable only up to scale. Empirical demonstrations showed how the SPO framework developed allows efficient alignment of LLMs to human and AI feedback in a way robust to how these are linked to rewards.