Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
43,426 characters · 14 sections · 67 citation commands
Inverse Reinforcement Learning with Just Classification and a Few Regressions
Behavioral data are abundant in robotics, economics, healthcare, and human--computer interaction. Inverse reinforcement learning (IRL) seeks to explain such behavior by recovering a reward under which the observed policy is optimal. Classical IRL often treats agents as exactly optimal ng2000algorithms,abbeel2004apprenticeship, but this deterministic view can miss the variability in real behavior. A common alternative is stochastic choice, for example through entropy regularization, which yields softmax-type policies ziebart2008maximum,ziebart2010modeling,haarnoja2017reinforcement. In maximum-entropy (MaxEnt) IRL ziebart2008maximum, closely related to dynamic discrete-choice (DDC) models with i.i.d.\ Gumbel shocks rust1987optimal, the observed policy has a softmax form induced by an unknown reward and continuation value.
Even in the softmax setting, rewards are only partially identified: different reward--value pairs can induce the same behavior policy through potential-based shaping transformations ng1999policy,cao2021identifiability,skalse2023invariance,skalse2024partial. Thus policy fit alone does not imply reward recovery. Existing methods obtain unique rewards only by adding restrictions, often implicitly: MaxEnt IRL identifies an equivalence class unless the reward class is restricted, for example to be linear ziebart2008maximum; adversarial IRL requires strong conditions for reward recovery, such as state-only rewards and deterministic transitions fu2018airl; and other neural IRL objectives depend on their chosen reward parameterization levine2011nonlinear,wulfmeier2015deep,ho2016gail, snoswell2020revisiting. This motivates normalized reward recovery, a classical idea in DDC/econometrics: impose an explicit identifying normalization and recover the unique reward--value pair satisfying it rust1987optimal,hotz1993conditional,aguirregabiria2010dynamic, geng2020deep.
Building on classical DDC for discrete actions, Deep Policy-to-\(Q\)-to-Reward (DeepPQR) geng2020deep operationalizes this idea under anchor-action normalization. It reduces reward recovery to a Bellman-type fixed point constructed from the observed behavior policy: estimate the policy, possibly using imitation learning or an IRL method such as adversarial IRL; estimate the associated \(Q\)-function with value-based offline RL tools; and recover rewards from the normalized Bellman equation. Crucially, DeepPQR uses only the policy implied by the upstream method, not its learned reward, so recovery does not require the upstream reward model to be correct. However, its identification strategy is tied to a fixed anchor action: one must specify a well-supported reference action, typically a do-nothing action, whose reward is known. This can be restrictive when actions are continuous, weakly supported, or lack a canonical reference option. Many applications instead call for more flexible normalizations, such as fixing mean rewards, using state-dependent anchors, or imposing value-based constraints.
We generalize this policy-to-\(Q\)-to-reward perspective to statewise affine normalizations, making the identifying restriction explicit and problem-dependent rather than fixed by an anchor-action convention. Given an estimated behavior policy, GenPQR solves the corresponding Bellman fixed point for \(Q\) and recovers the normalized reward directly from \(Q\). Thus, it preserves the modular appeal of DeepPQR while accommodating a broader class of normalizations. This shifts the role of normalization from an algorithm-specific anchor choice to an explicit modeling choice that can reflect the application.
Our contributions. First, we characterize reward identification in maximum-entropy IRL, equivalently the Gumbel-shock discrete-choice model. We show that behavior identifies only an equivalence class of reward--value pairs, and introduce statewise affine normalizations that select a unique representative. This generalizes the anchor-action normalization of DeepPQR geng2020deep and clarifies when exact reward recovery is possible, and when it is unnecessary for policy comparison.
Second, this characterization yields a general identification strategy and a modular recovery procedure. Under any statewise affine normalization, recovering the normalized reward reduces to estimating the behavior policy and solving a linear Bellman fixed point for an associated \(Q\)-function; the normalized reward--value pair is then obtained directly from \(Q\). This gives Generalized Policy-to-\(Q\)-to-Reward (GenPQR), which treats reward recovery as a post-processing step based on \(Q\)-evaluation after policy estimation, rather than as a specialized joint IRL objective. In the fixed-anchor neural-network setting, FQE-based GenPQR specializes to a simpler version of DeepPQR. Beyond this special case, the same reduction supports state-dependent anchors, mean-reward and value normalizations, and large or continuous action spaces through the choice of policy and \(Q\)-estimation methods.
Third, we prove finite-sample guarantees for GenPQR under general function approximation. The bounds combine with any policy estimator and any \(Q\)-function estimator; for FQE, they separate policy-estimation, Bellman-approximation, statistical, and iteration errors. Compared with DeepPQR, the theory avoids sup-norm policy-error assumptions, makes coverage explicit, does not require Bellman completeness, and is not tied to a specific neural-network architecture or training procedure.
Identifiability, shaping, and anchor-action methods. Rewards in MaxEnt IRL are only partially identified because behavior is invariant under potential-based shaping ng1999policy,fu2018airl,cao2021identifiability,skalse2023invariance, skalse2024partial. Our work is closest to DeepPQR geng2020deep, which studies anchor-action normalization and gives finite-sample guarantees for a specific neural-network procedure. We extend the anchor-action view to general affine normalizations, clarify the identification structure, simplify the recovery step (Section (ref)), and develop theory under general function approximation.
MaxEnt IRL and adversarial imitation learning. Maximum-entropy IRL fits stochastic policies induced by soft Bellman equations, often under structured reward parameterizations ziebart2008maximum,ziebart2010modeling,levine2011nonlinear, wulfmeier2015deep,zeng2022maximum. Adversarial methods such as GAIL and AIRL are effective for imitation learning and behavior-policy estimation through joint reward-policy optimization ho2016gail,fu2018airl. As noted by geng2020deep, however, they generally do not resolve reward nonidentifiability without stronger assumptions, such as state-only rewards. We instead separate behavior-policy estimation from normalized reward recovery, allowing action-dependent rewards. Thus, methods designed primarily to reproduce behavior can still serve as the first stage of a modular reward-recovery procedure.
Entropy-regularized RL and control as inference. Our analysis is also connected to entropy-regularized control and the control-as-inference perspective kappen2005linear,todorov2009efficient,levine2018rlasinf, including path-consistency learning nachum2017pcl, soft actor-critic haarnoja2018sac, and entropy-regularized offline RL haarnoja2017reinforcement,uehara2023offline. We use the same soft Bellman structure, but for the inverse problem: the behavior policy is assumed to solve an entropy-regularized control problem for an unknown reward, and the goal is to recover that reward.
Value-based offline RL. The recovery step is closely connected to value-based offline RL. Fitted \(Q\)-iteration and fitted \(Q\)-evaluation estimate Bellman fixed points by regression ernst2005tree,munos2008finite,mnih2013playing,van2025fitted, while minimax and critic-based methods relax completeness assumptions uehara2020minimax,imaizumi2021minimax,uehara2023offline,xie2020q, xie2021batch,zhan2022offline. We do not introduce a new RL update rule; instead, we show that normalized reward recovery reduces to solving a linear fixed-point equation with existing tools. Our finite-sample analysis separates first-stage policy-estimation error from second-stage value-learning error.
We consider a discounted MDP with state space \(\mathcal S\), finite or continuous action space \(\mathcal A\), transition kernel \(P\), reward \(r^\dagger:\mathcal S\times\mathcal A\to\mathbb R\), and discount \(\gamma\in[0,1)\). Let \(\pi(a\mid s)\) be the behavior policy and \(\rho\) the sampling distribution over states. We observe transitions \(\{(s_i,a_i,s_i')\}_{i=1}^n\) with \[ s_i\sim \rho,\qquad a_i\sim \pi(\cdot\mid s_i),\qquad s_i'\sim P(\cdot\mid s_i,a_i). \] Equivalently, \((s_i,a_i)\sim \nu_\pi\), where \(\nu_\pi(ds,a):=\rho(ds)\pi(a\mid s)\). Thus \(P\) is identified from the observed dynamics, whereas the reward is not. We use finite-action notation in the main text, following rust1987optimal,geng2020deep; continuous actions replace sums and softmax normalizers by integrals and Boltzmann densities with respect to a reference measure (Appendix (ref)). For any state-action function \(f\), write \[ (\mu f)(s):=\sum_a \mu(a\mid s) f(s,a). \] The goal of IRL is to recover a reward \(r\) for which \(\pi\) is optimal in the MDP \((\mathcal S,\mathcal A,P,r,\gamma)\), in an appropriate sense. We review the MaxEnt IRL setting from structural discrete-choice and maximum-entropy perspectives, then formulate the optimization problem central to our analysis.
From dynamic discrete choice to MaxEnt IRL. We adopt the dynamic discrete-choice formulation rust1987optimal,hotz1993conditional,aguirregabiria2010dynamic. At time \(t\), an agent in state \(s_t\) who takes action \(a_t\) receives utility \(r^\dagger(s_t,a_t)+\varepsilon_t(a_t)\), where \(r^\dagger\) is the unknown mean reward and \(\varepsilon_t(a)\) is an idiosyncratic shock with known distribution. Let \(V^\dagger(s)\) be the optimal ex ante value, define \(Pf(s,a):=E\{f(s')\mid s,a\}\), and set \[ Q^\dagger(s,a):=r^\dagger(s,a)+\gamma PV^\dagger(s,a), \qquad \Xi f(s):=\log\sum_a e^{f(s,a)}. \] Under i.i.d.\ Gumbel type-I extreme-value shocks, the optimal policy is softmax: \[ \pi^\dagger(a\mid s)\propto \exp\{Q^\dagger(s,a)/\tau\}, \] for temperature \(\tau>0\), and \[ Q^\dagger = r^\dagger+\gamma P\Xi Q^\dagger, \qquad V^\dagger(s)=\Xi Q^\dagger(s). \] Equivalently, with state-action continuation value \(v^\dagger:=P\Xi Q^\dagger\), \[ \pi^\dagger(a\mid s) \propto \exp\{(r^\dagger(s,a)+\gamma v^\dagger(s,a))/\tau\}, \qquad v^\dagger=P\Xi(r^\dagger+\gamma v^\dagger). \] We call the latter the soft Bellman equation. Without loss of generality, we set \(\tau=1\) and absorb the scale into \(r^\dagger\).
An equivalent perspective comes from maximum-entropy IRL ziebart2008maximum,ziebart2010modeling, where the agent maximizes expected discounted reward plus an entropy bonus for stochastic action selection. This yields the same optimal policy \(\pi^\dagger\) and soft Bellman equation, with \(V^\dagger\) and \(Q^\dagger\) the entropy-regularized value and \(Q\)-functions haarnoja2017reinforcement. Thus, despite different motivations, dynamic discrete choice and MaxEnt IRL reduce to the same mathematical object: a soft Bellman system with a softmax policy.
Partial identification and the role of normalization. Our goal is therefore to recover a reward function whose induced soft-optimal policy best matches the observed behavior, for example by minimizing the state-averaged Kullback--Leibler divergence from \(\pi(\cdot\mid s)\) to \(\pi^\star(\cdot\mid s)\). However, matching the policy does not in general identify the reward. Under a softmax policy, adding a state-dependent offset to all action values leaves the policy unchanged, so many rewards induce the same behavior policy ziebart2010modeling,cao2021identifiability. The reward is therefore only partially identified, up to an equivalence class, and selecting a unique representative requires a normalization constraint rust1987optimal,geng2020deep.
We use statewise affine normalizations, which generalize standard choices such as anchor-action, outside-option, and sum-to-zero constraints. Let \(\mu(\cdot\mid s)\) be a reference distribution over actions and \(g:\mathcal S\to\mathbb R\) a specified anchor function. We impose
or equivalently \(\mu r=g\), with \(\sup_s |g(s)|<\infty\). Thus, the \(\mu\)-average reward at each state is fixed at \(g(s)\), with \(\mu\) and \(g\) typically chosen from domain knowledge.
Such constraints are standard in economics and are often substantively meaningful hotz1993conditional,bajari2010identification. The fixed anchor-action constraint \(r(s,a^\dagger)=g(s)\) is the special case \(\mu(a\mid s)=1\{a=a^\dagger\}\) geng2020deep; when \(g\equiv0\), it reduces to the classical zero-reward normalization \(r(s,a^\dagger)=0\) used in Rust's engine-replacement model rust1987optimal. More generally, taking \(\mu(a\mid s)=1\{a=a^\dagger(s)\}\) allows state-specific anchors, where the reference action may vary with the state. DDC models often use such anchors to normalize the payoff of a no-action, status-quo, or outside option to be zero or otherwise known rust1987optimal,hotz1993conditional,aguirregabiria2010dynamic, geng2020deep. If \(\mu(a\mid s)=1/|\mathcal A|\), the constraint fixes the statewise average reward and, when \(g\equiv0\), becomes a sum-to-zero constraint kallus2016revealed.
The framework also permits data-driven choices of \((\mu,g)\). A statewise value constraint is a special case: if \(V_r^\mu\) is the value function of policy \(\mu\) under reward \(r\), then imposing \(V_r^\mu=h\) is equivalent, by the Bellman equation \(V_r^\mu=\mu r+\gamma\mu P V_r^\mu\), to taking \(g:=h-\gamma\mu P h\). For example, in medical decision problems, one may wish to normalize rewards relative to standard care using historical outcome data kallus2018removing. If an auxiliary dataset records states and realized outcomes \((s_i,y_i)\) from a population with the same reward function and behavior policy \(\pi\), then \(g(s)=E[y\mid s]=\sum_a \pi(a\mid s)r(s,a)\). Taking \(\mu=\pi\) therefore yields a natural data-driven normalization.
Main problem: Putting these pieces together, we obtain the following constrained maximum-likelihood problem for recovering \((r^\star, v^\star)\):
That is, we maximize the conditional log-likelihood over state--action functions \(r\) and \(v\) subject to the soft Bellman equation and a statewise affine normalization constraint.
The remainder of the paper shows that solving (ref) reduces to two steps: estimate \(\pi\), then solve a linear Bellman evaluation equation to recover the unique pair \((r^\star, v^\star)\). We then turn this characterization into a simple algorithm.
To expose the core structure of the problem, we first drop the normalization constraint and study the resulting relaxed optimization problem.
Relaxed problem: We remove the reward normalization in (ref) but keep soft Bellman consistency:
This problem is highly non-unique. Because the objective is invariant under potential-based transformations of the reward, the relaxed problem admits an entire equivalence class of solutions. We will exploit this invariance later to recover a solution to the normalized problem (ref) from a single convenient representative.
Without normalization, the relaxed problem admits a particularly simple solution: estimate the behavior policy \(\pi\) and set \(r(s,a)=\log \pi(a\mid s)\) and \(v(s)=0\). This pair maximizes the conditional log-likelihood and satisfies the soft Bellman equation. Indeed, $P\Xi(r+\gamma v) = 0$ since $\exp \{\Xi(r+\gamma v)(s)\} = \sum_a \exp\{r(s,a)\} = \sum_a \pi(a\mid s) = 1.$ We will use the shorthand \[ u^\star(s,a):=\log \pi(a\mid s). \]
A related observation appears in Section 4 of fu2018airl, but not as a tool for identification or estimation. Here, by contrast, it is the starting point for normalization-based identification. As shown in Section (ref), this trivial solution already suffices for policy-value comparisons.
The relaxed problem is invariant to potential-based shaping: adding a state-only potential $c:\mathcal{S} \to \mathbb{R}$ shifts all logits in the softmax by the same amount per state, leaving both feasibility and likelihood unchanged. This is the entropy-regularized analogue of reward shaping in classical RL ng1999policy and explains why the relaxed objective is flat along an affine subspace.
Related partial-identification results appear in cao2021identifiability, fu2018airl, and geng2020deep. We next show that our normalization selects a unique representative from this class.
Lemma (ref) gives one relaxed optimum, \((u^\star,0)\), and Lemma (ref) characterizes all others via potential-based transformations of the form \((r,v) = (u^\star + c - \gamma Pc, Pc)\). Since the constrained and relaxed problems have the same optimal value, solving (ref) amounts to finding the shaping function \(c\) such that the corresponding pair \((r,v)\) satisfies the desired constraint. For our normalization \(\mu r = g\), \(c\) is uniquely determined by a Bellman equation.
The next result gives the corresponding solution in terms of \(Q^\mu_{u^\star-g}\), the \(Q\)-function under reward \(u^\star-g\) and policy \(\mu\), where \(u^\star=\log \pi\). Recall that \[ (P\mu Q)(s,a):=\mathbb{E}_{s' \sim P(\cdot\mid s,a),\, a' \sim \mu(\cdot\mid s')} \bigl[Q(s',a')\bigr]. \]
This theorem is the main result of the paper. It shows that normalized reward recovery reduces to two steps: first estimate \(u^\star = \log \pi\), then solve the linear Bellman equation for \(Q^\mu_{u^\star-g}\). The normalized reward is then obtained in closed form as the advantage function \(Q^\mu_{u^\star-g} - \mu Q^\mu_{u^\star-g}\), shifted by \(g\). By Lemma (ref), every feasible representative in the shaping class induces the same \(u^\star\). Therefore, once one feasible representative is known, we can impose a different normalization without re-solving the original IRL problem.
Comparison to DeepPQR. In the anchor-action setting \(\mu(a \mid s)=1\{a=a^\dagger\}\), DeepPQR geng2020deep first estimates the behavior policy and then learns the anchored value \(W(s):=Q^\mu_{u^\star-g}(s,a^\dagger)\), which is used to reconstruct the full \(Q\)-function and hence the normalized reward. In our notation, \(Q^\mu_{u^\star-g}\) satisfies \[ Q^\mu_{u^\star-g}(s,a) = u^\star(s,a)-g(s)+\gamma \mathbb{E}\!\left[Q^\mu_{u^\star-g}(s',a^\dagger)\mid s,a\right], \] so it is fully determined by \(W\), where \[ W(s)=u^\star(s,a^\dagger)-g(s)+\gamma \mathbb{E}[W(s')\mid s,a^\dagger]. \] Thus, DeepPQR estimates the same target through the intermediate quantity \(W\). Our formulation makes this explicit by working directly with the single \(Q\)-function \(Q^\mu_{u^\star-g}\), from which both the normalized reward and the continuation value \(v^\star\) follow immediately. This removes the extra step of estimating \(PW\), yielding a simpler and more modular second stage. It also highlights a practical tradeoff: DeepPQR is tied to the anchor action \(a^\dagger\), which may be unstable when observations under \(a^\dagger\) are limited, whereas our formulation estimates the full \(Q\)-function directly and can borrow strength across actions through any suitable \(Q\)-learning method. This matches our experiments: direct estimation in GenPQR performs better when anchor actions are rare or the action space is large, whereas DeepPQR's anchor-action regression becomes unstable with rare anchors and has no direct continuous-action analogue.
For policy comparison, exact reward recovery is unnecessary once the transition kernel \(P\) and discount factor \(\gamma\) are fixed. Let \(V_r^\pi=\pi Q_r^\pi\) denote the value function under reward \(r\) and policy \(\pi\), where \(Q_r^\pi\) is the corresponding \(Q\)-function.
Thus, exact reward recovery is needed only for targets that depend on the normalization itself, such as evaluation under counterfactual transition dynamics or discount factors, or interpretation of structural features of \(r\). Otherwise, \(u^\star\), or any reward solving the relaxed problem, suffices for policy comparison. This parallels hotz1993conditional, where under Gumbel shocks value differences are identified from log-odds of observed choices.
Algorithm (ref) presents Generalized Policy-to-\(Q\)-to-Reward (GenPQR) for reward recovery. The method has two standard steps. First, we estimate the behavior policy \(\pi(a \mid s)\), equivalently \(u^\star(s,a)=\log \pi(a \mid s)\), yielding \(\hat u\). Second, we recover \((r^\star,v^\star)\) by estimating the \(Q\)-function of policy \(\mu\) under reward \(\hat u-g\), then applying the plug-in formula in Theorem (ref). Appendix (ref) gives the continuous-action version: estimate a conditional log-density, solve the same \(Q\)-fixed point with \(\mu\)-integrals, and apply the same normalized advantage formula. When \(\hat \pi(a\mid s)\) is close to \(0\), \(\hat u\) may be unstable; in practice, one may clip \(\hat \pi\) away from zero ionides2008truncated. Thus, in our setting, IRL reduces to behavior-policy estimation plus offline policy evaluation, rather than a fundamentally new estimation problem. The reward is identified only up to scale; if the scale is known, the recovered reward can be rescaled accordingly; see Appendix C.2.3 of geng2020deep.
The procedure is modular and black-box. The first stage can use any probabilistic classifier, behavior-cloning, imitation-learning, or IRL method trained on the observed data, including MaxEnt IRL and adversarial IRL. The second stage can use a range of existing offline RL methods, including temporal-difference learning tsitsiklis1996analysis, fitted \(Q\)-evaluation (FQE) munos2008finite,van2025fitted, and minimax or saddle-point \(Q\)-learning methods uehara2020minimax,imaizumi2021minimax,xie2020q,xie2021batch. This separation lets each stage leverage existing methods and remain compatible with techniques for handling misspecification or distribution shift fujimoto2019off,agarwal2021deep,chen2019information,foster2021offline. Moreover, given any feasible reward--value pair under one normalization, GenPQR can recover the corresponding reward under another normalization without re-solving the original IRL problem.
Algorithm (ref) instantiates the second stage of GenPQR using FQE, thereby generalizing DeepPQR geng2020deep. It requires only standard tools for policy estimation and regression. Moreover, the regression step in Algorithm (ref) need not be solved exactly at each iteration: one may instead parametrize \(Q\) by a neural network and take one or a few stochastic-gradient steps per iteration, as in deep fitted \(Q\)-learning or boosting riedmiller2005neural,tosatto2017boosted. The expectation under \(\mu\) can also be approximated by Monte Carlo sampling to avoid explicit integration.
We now turn to finite-sample guarantees for $\hat r = \hat Q -\mu \hat Q + g$ obtained via Algorithm (ref). The following result shows how errors in the estimated log-policy \(\hat u\) and anchored \(Q\)-function \(\hat Q\) propagate to the recovered reward. We measure errors in the behavior norm \(L^2(\nu_\pi)\), but the Bellman operator is most naturally analyzed in \(L^2(d_\mu)\), where \(d_\mu\) is any stationary state distribution under \((\mu,P)\) patterson2022generalized,van2025fitted, that is, \[ d_\mu = d_\mu P_\mu, \qquad P_\mu(s' \mid s) := \int_{\mathcal A} P(s' \mid s,a)\,\mu(a\mid s)\,da. \] Recalling that \(\nu_\pi := \rho \otimes \pi\), we write \[ \|f\|_{2,\mathrm{beh}} := \Bigl(\mathbb{E}_{(s,a)\sim \nu_\pi}[f(s,a)^2]\Bigr)^{1/2}, \qquad \|f\|_\infty := \operatorname*{ess\,sup}_{s,a} |f(s,a)|. \]
We impose a coverage assumption requiring the observed system \((\rho,\pi,P)\) to provide sufficient support relative to the normalization policy \(\mu\) under the same dynamics \(P\): \(\pi\) must cover \(\mu\), the behavior state distribution \(\rho\) must cover the stationary distribution \(d_\mu\), and \(d_\mu\) must cover the one-step state distribution \(\nu_\pi P\).
The coverage constant \(C_{\mathrm{cov}}\) is analogous to standard coverage and concentrability coefficients in offline RL xie2022role,zhan2022offline. It equals \(1\) under stationary sampling from \((P,\mu)\). Since the normalization policy \(\mu\) is typically chosen with knowledge of the behavior policy \(\pi\), this condition is often less restrictive here than in standard offline policy evaluation. For example, in the anchor-action case, one would not anchor on an action that is rarely or never observed.
The reward-estimation error decomposes into a \(Q\)-estimation term, \(\|Q^\mu_{\hat u-g}-\hat Q\|_{2,\mathrm{beh}}\), and a policy-estimation term, \(\|\hat u-u^\star\|_{2,\mathrm{beh}}\). This makes the bound modular: any guarantees for \(Q\)-learning and policy estimation translate directly into a reward-recovery guarantee, with the latter scaled by \((1-\gamma)^{-1}\) and the coverage coefficients in Assumption (ref). Under sample splitting foster2023orthogonal, for example, existing results apply directly to \(\hat u\) and \(\hat Q\), including bounds for FQE munos2008finite,van2025fitted and minimax \(Q\)-learning uehara2020minimax. In the next section, we apply this template to derive finite-sample bounds for PQR with FQE.
We now specialize Theorem (ref) to the case in which \(\hat Q\) is obtained by running \(K\) steps of FQE over a regression class \(\mathcal F\), initialized at \(\hat Q^{(0)}\), as in Algorithm (ref). To state the resulting finite-sample bound, we assume PAC-style generalization guarantees for the policy-estimation and regression steps.
Lower bounds on \(\hat \pi\) and \(\pi^\star\) ensures that KL divergence controls \(\|\hat u-u^\star\|_{2,\mathrm{beh}}\). For maximum likelihood estimation over a class \(\mathcal U\), one typically has \(\rho_\pi(n,\delta)\lesssim r_{\mathcal U}(n)+\sqrt{\log(1/\delta)/n},\) where \(r_{\mathcal U}(n)\) is a local complexity measure of \(\mathcal U\) (e.g., VC dimension) geer2000empirical,wainwright2019high.
We require each regression step to achieve small excess risk relative to the best approximation in \(\mathcal F\) to the Bellman target \(\mathcal T_{\hat u}^\mu(\hat Q^{(k-1)})\), where \(\mathcal T_u^\mu f:=u-g+\gamma P\mu f\). Define \[ \mathrm{reg}(Q\mid u,Q') := \|\mathcal T_u^\mu(Q')-Q\|_{2,\mathrm{beh}}^2 - \inf_{f\in\mathcal F}\|\mathcal T_u^\mu(Q')-f\|_{2,\mathrm{beh}}^2. \]
Such bounds for nuisance-dependent regression targets follow from foster2023orthogonal,van2026researcher. Under a fresh-sample analysis munos2008finite, one typically has \(\rho_{Q}(n,\delta)\lesssim r_{\mathcal F}(n/K)+\sqrt{\log(1/\delta)/n}\), where \(r_{\mathcal F}(n/K)\) is the local Rademacher critical radius of \(\mathcal F\); see Appendix (ref). Similar bounds hold for weakly dependent trajectory data under mixing conditions yu1994rates,mohri2010stability.
We now state our main result for GenPQR with FQE under approximate Bellman-completeness. To account for misspecification in the regression steps, we introduce the inherent Bellman error of the regression class munos2008finite: \[ \varepsilon_{\mathcal F} := \sup_{f \in \mathcal F}\inf_{g \in \mathcal F} \|\mathcal T_{u^\star}^\mu f-g\|_{\infty}. \] This quantity is zero when \(\mathcal F\) is Bellman complete, that is, when \(\mathcal T_{u^\star}^\mu f \in \mathcal F\) for every \(f \in \mathcal F\) chen2019information. In that case, each regression step in Algorithm (ref) is correctly specified, since its population target lies in \(\mathcal F\). We define the standard concentrability coefficient \[ C_{\mathrm{conc}} := \sup_{m\ge 0} \left\| \frac{d\bigl(\nu_\pi (P\mu)^m\bigr)}{d\nu_\pi} \right\|_\infty < \infty, \] which is finite under Assumption (ref) with $C_{\mathrm{conc}} \le C_{\mathrm{cov}}$ by Lemma (ref) in Appendix (ref).
Proof sketch. Apply Theorem (ref) and bound \(\|Q^\mu_{\hat u-g}-\hat Q\|_{2,\mathrm{beh}}\) by the inexact Picard argument of munos2008finite. The only change is that inherent Bellman error is measured for \(\mathcal T^\mu_{u^\star}\), not the data-dependent \(\mathcal T^\mu_{\hat u}\); Lemma (ref) shows this suffices up to policy-estimation error. Approximate Bellman completeness could be relaxed using suitable weighting van2025fitted or minimax formulations uehara2023offline.
Discussion. The bound matches the usual FQE structure munos2008finite,van2025fitted, with additional terms due to estimation of \(u^\star\). The three terms respectively capture finite-iteration error, approximation and statistical error from the fitted Bellman updates, and the IRL-specific error from estimating \(u^\star\). The last term can be large when the behavior policy is nearly deterministic, for example in low-temperature softmax regimes, because KL divergence controls \(L^2\) log-policy error only up to a factor of \(\underline p^{-2}\). In particular, if \(\hat u\) and \(\hat Q\) are learned by ERM over parametric classes \(\mathcal U\) and \(\mathcal F\) with pseudo-dimensions \(d_{\mathcal U}\) and \(d_{\mathcal F}\), and \(K \asymp \log n\), then one typically obtains, up to \(O(\sqrt{\log \log n / n})\) terms, \[ \|\hat r-r^\star\|_{2,\mathrm{beh}} \lesssim \frac{\sqrt{C_{\mu/\pi}C_{\mathrm{cov}}}}{1-\gamma} \left( \varepsilon_{\mathcal F} + \sqrt{\frac{d_{\mathcal F}}{n}} + \frac{1}{\underline p^{2}}\sqrt{\frac{d_{\mathcal U}}{n}} \right). \]
Comparison to DeepPQR. Theorem 2 of geng2020deep gives a related FQE bound for anchor-action reward recovery under a specific neural-network architecture. Our result is more modular: it is not tied to a particular function class, allows approximate Bellman completeness for \(\mathcal T^\mu_{u^\star}\), makes coverage explicit, and replaces sup-norm policy-error control with Kullback--Leibler or \(L^2\)-type control, which is natural for likelihood-based policy estimators, multiclass classification, MaxEnt IRL ziebart2008maximum, and adversarial IRL fu2018airl,snoswell2020revisiting,ke2020imitation, foster2024behavior.
We adapt DeepPQR's synthetic study to isolate identification rather than imitation performance, using its infinite-horizon environment with continuous states, five actions, deterministic transitions, and anchor-action normalization \(g(s)=0\). We compare DeepPQR and GenPQR under matched policy estimation, and study how policy and \(Q\)-estimation choices affect modular reward recovery. The key distinction is the effective anchor sample: DeepPQR estimates its anchor \(Q\)-function only on anchor-action transitions, whereas GenPQR uses the full sample for \(Q\)-evaluation. We report reward MSE, reward correlation, held-out policy negative log-likelihood, and runtime over \(100\) seeds with \(95\%\) confidence intervals; details are in Appendix (ref).
Matched comparison to DeepPQR. Both methods use the same AIRL policy estimate and neural downstream approximation, so differences reflect the identification step (Section (ref)). We vary trajectory count and anchor frequency, which determine DeepPQR's effective anchor sample size. Figure (ref) summarizes three regimes; Appendix (ref) reports similar behavior with more actions.
\noindentEstimator choices. At \(1000\) trajectories with a well-supported anchor action, we compare AIRL versus behavior cloning for policy estimation, neural versus boosted FQE, and standard reward-recovery baselines. Figure (ref) in Appendix (ref) shows that GenPQR remains effective across estimator choices, with substantial runtime variation across implementations.
\noindentDiscussion. The matched comparison isolates the statistical cost of anchor-subset identification; the estimator-choice study shows that GenPQR remains effective under practical policy and \(Q\)-estimation choices. We provide concluding remarks in Appendix (ref).