The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
43,426 characters
Inverse Reinforcement Learning with Just Classification and a Few Regressions
\maketitle
\begin{abstract}
Inverse reinforcement learning (IRL) aims to infer rewards from observed
behavior, but rewards are not identified from the policy alone: many
reward--value pairs can rationalize the same actions. Meaningful reward
recovery therefore requires a normalization, yet existing normalized IRL methods
often rely on anchor-action restrictions or specialized neural architectures.
We study reward recovery in the maximum-entropy, or Gumbel-shock, model under a
broad class of statewise affine normalizations, with anchor-action constraints
as a special case. This yields \emph{Generalized Policy-to-\(Q\)-to-Reward}
(GenPQR), a modular procedure that estimates the behavior policy, evaluates its
soft \(Q\)-function through the Bellman equation, and recovers the normalized
reward. Both stages can be implemented with off-the-shelf classification and
regression methods. We prove modular finite-sample guarantees under general function approximation,
with separate policy-estimation and \(Q\)-estimation errors. As a concrete instantiation, we study GenPQR with fitted \(Q\)-evaluation,
reducing IRL to policy estimation followed by regression. Experiments show that
GenPQR matches or improves reward recovery relative to DeepPQR while remaining
simpler and more modular. Compared with DeepPQR, our theory goes beyond anchor
actions, accommodates large and continuous action spaces, makes coverage
requirements explicit, and is not tied to a specific neural-network architecture
or training procedure.
\end{abstract}
\section{Introduction}
\label{sec:motivation}
Behavioral data are abundant in robotics, economics, healthcare, and
human--computer interaction. Inverse reinforcement learning (IRL) seeks to
explain such behavior by recovering a reward under which the observed policy is
optimal. Classical IRL often treats agents as exactly optimal
\citep{ng2000algorithms,abbeel2004apprenticeship}, but this deterministic view
can miss the variability in real behavior. A common alternative is stochastic
choice, for example through entropy regularization, which yields softmax-type
policies
\citep{ziebart2008maximum,ziebart2010modeling,haarnoja2017reinforcement}. In
maximum-entropy (MaxEnt) IRL \citep{ziebart2008maximum}, closely related to
dynamic discrete-choice (DDC) models with i.i.d.\ Gumbel shocks
\citep{rust1987optimal}, the observed policy has a softmax form induced by an
unknown reward and continuation value.
Even in the softmax setting, rewards are only partially identified: different
reward--value pairs can induce the same behavior policy through potential-based
shaping transformations
\citep{ng1999policy,cao2021identifiability,skalse2023invariance,skalse2024partial}.
Thus policy fit alone does not imply reward recovery. Existing methods obtain
unique rewards only by adding restrictions, often implicitly: MaxEnt IRL
identifies an equivalence class unless the reward class is restricted, for
example to be linear \citep{ziebart2008maximum}; adversarial IRL requires
strong conditions for reward recovery, such as state-only rewards and
deterministic transitions \citep{fu2018airl}; and other neural IRL objectives
depend on their chosen reward parameterization
\citep{levine2011nonlinear,wulfmeier2015deep,ho2016gail,
snoswell2020revisiting}. This motivates normalized reward recovery, a classical idea in DDC/econometrics:
impose an explicit identifying normalization and recover the unique
reward--value pair satisfying it
\citep{rust1987optimal,hotz1993conditional,aguirregabiria2010dynamic,
geng2020deep}.
Building on classical DDC for discrete actions, Deep Policy-to-\(Q\)-to-Reward
(DeepPQR) \citep{geng2020deep} operationalizes this idea under anchor-action
normalization. It reduces reward recovery to a Bellman-type fixed point
constructed from the observed behavior policy: estimate the policy, possibly
using imitation learning or an IRL method such as adversarial IRL; estimate the
associated \(Q\)-function with value-based offline RL tools; and recover rewards
from the normalized Bellman equation. Crucially, DeepPQR uses only the policy
implied by the upstream method, not its learned reward, so recovery does not
require the upstream reward model to be correct. However, its identification
strategy is tied to a fixed anchor action: one must specify a well-supported
reference action, typically a do-nothing action, whose reward is known. This can
be restrictive when actions are continuous, weakly supported, or lack a
canonical reference option. Many applications instead call for more flexible
normalizations, such as fixing mean rewards, using state-dependent anchors, or
imposing value-based constraints.
We generalize this policy-to-\(Q\)-to-reward perspective to statewise affine
normalizations, making the identifying restriction explicit and
problem-dependent rather than fixed by an anchor-action convention. Given an
estimated behavior policy, GenPQR solves the corresponding Bellman fixed point
for \(Q\) and recovers the normalized reward directly from \(Q\). Thus, it
preserves the modular appeal of DeepPQR while accommodating a broader class of
normalizations. This shifts the role of normalization from an algorithm-specific
anchor choice to an explicit modeling choice that can reflect the application.
\textbf{Our contributions.}
First, we characterize reward identification in maximum-entropy IRL,
equivalently the Gumbel-shock discrete-choice model. We show that behavior
identifies only an equivalence class of reward--value pairs, and introduce
statewise affine normalizations that select a unique representative. This
generalizes the anchor-action normalization of DeepPQR \citep{geng2020deep} and
clarifies when exact reward recovery is possible, and when it is unnecessary
for policy comparison.
Second, this characterization yields a general identification strategy and a
modular recovery procedure. Under any statewise affine normalization, recovering
the normalized reward reduces to estimating the behavior policy and solving a
linear Bellman fixed point for an associated \(Q\)-function; the normalized
reward--value pair is then obtained directly from \(Q\). This gives
\emph{Generalized Policy-to-\(Q\)-to-Reward} (GenPQR), which treats reward
recovery as a post-processing step based on \(Q\)-evaluation after policy
estimation, rather than as a specialized joint IRL objective. In the
fixed-anchor neural-network setting, FQE-based GenPQR specializes to a simpler
version of DeepPQR. Beyond this special case, the same reduction supports
state-dependent anchors, mean-reward and value normalizations, and large or
continuous action spaces through the choice of policy and \(Q\)-estimation
methods.
Third, we prove finite-sample guarantees for GenPQR under general function
approximation. The bounds combine with any policy estimator and any
\(Q\)-function estimator; for FQE, they separate policy-estimation,
Bellman-approximation, statistical, and iteration errors. Compared with
DeepPQR, the theory avoids sup-norm policy-error assumptions, makes coverage
explicit, does not require Bellman completeness, and is not tied to a specific
neural-network architecture or training procedure.
\subsection{Related Work}
\textbf{Identifiability, shaping, and anchor-action methods.}
Rewards in MaxEnt IRL are only partially identified because behavior is
invariant under potential-based shaping
\citep{ng1999policy,fu2018airl,cao2021identifiability,skalse2023invariance,
skalse2024partial}. Our work is closest to DeepPQR \citep{geng2020deep}, which
studies anchor-action normalization and gives finite-sample guarantees for a
specific neural-network procedure. We extend the anchor-action view to general
affine normalizations, clarify the identification structure, simplify the
recovery step (Section~\ref{sec:solve-normalization}), and develop theory under
general function approximation.
\textbf{MaxEnt IRL and adversarial imitation learning.}
Maximum-entropy IRL fits stochastic policies induced by soft Bellman equations,
often under structured reward parameterizations
\citep{ziebart2008maximum,ziebart2010modeling,levine2011nonlinear,
wulfmeier2015deep,zeng2022maximum}. Adversarial methods such as GAIL and AIRL
are effective for imitation learning and behavior-policy estimation through
joint reward-policy optimization \citep{ho2016gail,fu2018airl}. As noted by
\citet{geng2020deep}, however, they generally do not resolve reward
nonidentifiability without stronger assumptions, such as state-only rewards. We
instead separate behavior-policy estimation from normalized reward recovery,
allowing action-dependent rewards. Thus, methods designed primarily to reproduce
behavior can still serve as the first stage of a modular reward-recovery
procedure.
\textbf{Entropy-regularized RL and control as inference.}
Our analysis is also connected to entropy-regularized control and the
control-as-inference perspective
\citep{kappen2005linear,todorov2009efficient,levine2018rlasinf}, including
path-consistency learning \citep{nachum2017pcl}, soft actor-critic
\citep{haarnoja2018sac}, and entropy-regularized offline RL
\citep{haarnoja2017reinforcement,uehara2023offline}. We use the same soft
Bellman structure, but for the inverse problem: the behavior policy is assumed
to solve an entropy-regularized control problem for an unknown reward, and the
goal is to recover that reward.
\textbf{Value-based offline RL.}
The recovery step is closely connected to value-based offline RL. Fitted
\(Q\)-iteration and fitted \(Q\)-evaluation estimate Bellman fixed points by
regression \citep{ernst2005tree,munos2008finite,mnih2013playing,van2025fitted},
while minimax and critic-based methods relax completeness assumptions
\citep{uehara2020minimax,imaizumi2021minimax,uehara2023offline,xie2020q,
xie2021batch,zhan2022offline}. We do not introduce a new RL update rule;
instead, we show that normalized reward recovery reduces to solving a linear
fixed-point equation with existing tools. Our finite-sample analysis separates
first-stage policy-estimation error from second-stage value-learning error.
\section{Problem Setup}
\label{sec:background}
We consider a discounted MDP with state space \(\mathcal S\), finite or
continuous action space \(\mathcal A\), transition kernel \(P\), reward
\(r^\dagger:\mathcal S\times\mathcal A\to\mathbb R\), and discount
\(\gamma\in[0,1)\). Let \(\pi(a\mid s)\) be the behavior policy and \(\rho\)
the sampling distribution over states. We observe transitions
\(\{(s_i,a_i,s_i')\}_{i=1}^n\) with
\[
s_i\sim \rho,\qquad
a_i\sim \pi(\cdot\mid s_i),\qquad
s_i'\sim P(\cdot\mid s_i,a_i).
\]
Equivalently, \((s_i,a_i)\sim \nu_\pi\), where
\(\nu_\pi(ds,a):=\rho(ds)\pi(a\mid s)\). Thus \(P\) is identified from the
observed dynamics, whereas the reward is not. We use finite-action notation in
the main text, following \citet{rust1987optimal,geng2020deep}; continuous
actions replace sums and softmax normalizers by integrals and Boltzmann
densities with respect to a reference measure
(Appendix~\ref{app:continuous-actions}). For any state-action function \(f\),
write
\[
(\mu f)(s):=\sum_a \mu(a\mid s) f(s,a).
\]
The \textbf{goal} of IRL is to recover a reward \(r\) for which \(\pi\) is
optimal in the MDP \((\mathcal S,\mathcal A,P,r,\gamma)\), in an appropriate
sense. We review the MaxEnt IRL setting from structural discrete-choice and
maximum-entropy perspectives, then formulate the optimization problem central to
our analysis.
\textbf{From dynamic discrete choice to MaxEnt IRL.}
We adopt the dynamic discrete-choice formulation
\citep{rust1987optimal,hotz1993conditional,aguirregabiria2010dynamic}. At time
\(t\), an agent in state \(s_t\) who takes action \(a_t\) receives utility
\(r^\dagger(s_t,a_t)+\varepsilon_t(a_t)\), where \(r^\dagger\) is the unknown
mean reward and \(\varepsilon_t(a)\) is an idiosyncratic shock with known
distribution. Let \(V^\dagger(s)\) be the optimal ex ante value, define
\(Pf(s,a):=E\{f(s')\mid s,a\}\), and set
\[
Q^\dagger(s,a):=r^\dagger(s,a)+\gamma PV^\dagger(s,a),
\qquad
\Xi f(s):=\log\sum_a e^{f(s,a)}.
\]
Under i.i.d.\ Gumbel type-I extreme-value shocks, the optimal policy is softmax:
\[
\pi^\dagger(a\mid s)\propto \exp\{Q^\dagger(s,a)/\tau\},
\]
for temperature \(\tau>0\), and
\[
Q^\dagger = r^\dagger+\gamma P\Xi Q^\dagger,
\qquad
V^\dagger(s)=\Xi Q^\dagger(s).
\]
Equivalently, with state-action continuation value
\(v^\dagger:=P\Xi Q^\dagger\),
\[
\pi^\dagger(a\mid s)
\propto
\exp\{(r^\dagger(s,a)+\gamma v^\dagger(s,a))/\tau\},
\qquad
v^\dagger=P\Xi(r^\dagger+\gamma v^\dagger).
\]
We call the latter the soft Bellman equation. Without loss of generality, we
set \(\tau=1\) and absorb the scale into \(r^\dagger\).
An equivalent perspective comes from maximum-entropy IRL
\citep{ziebart2008maximum,ziebart2010modeling}, where the agent maximizes
expected discounted reward plus an entropy bonus for stochastic action
selection. This yields the same optimal policy \(\pi^\dagger\) and soft Bellman
equation, with \(V^\dagger\) and \(Q^\dagger\) the entropy-regularized value
and \(Q\)-functions \citep{haarnoja2017reinforcement}. Thus, despite different
motivations, dynamic discrete choice and MaxEnt IRL reduce to the same
mathematical object: a soft Bellman system with a softmax policy.
\textbf{Partial identification and the role of normalization.}
Our goal is therefore to recover a reward function whose induced soft-optimal
policy best matches the observed behavior, for example by minimizing the
state-averaged Kullback--Leibler divergence from \(\pi(\cdot\mid s)\) to
\(\pi^\star(\cdot\mid s)\). However, matching the policy does not in general
identify the reward. Under a softmax policy, adding a state-dependent offset to
all action values leaves the policy unchanged, so many rewards induce the same
behavior policy
\citep{ziebart2010modeling,cao2021identifiability}. The reward is therefore
only \emph{partially identified}, up to an equivalence class, and selecting a
unique representative requires a normalization constraint \citep{rust1987optimal,geng2020deep}.
We use \textbf{statewise affine normalizations}, which generalize standard
choices such as anchor-action, outside-option, and sum-to-zero constraints. Let
\(\mu(\cdot\mid s)\) be a reference distribution over actions and
\(g:\mathcal S\to\mathbb R\) a specified anchor function. We impose
\begin{equation}
\sum_a \mu(a\mid s) r(s,a)=g(s)
\qquad \text{for all } s,
\end{equation}
or equivalently \(\mu r=g\), with \(\sup_s |g(s)|<\infty\). Thus, the
\(\mu\)-average reward at each state is fixed at \(g(s)\), with \(\mu\) and
\(g\) typically chosen from domain knowledge.
Such constraints are standard in economics and are often substantively
meaningful \citep{hotz1993conditional,bajari2010identification}. The fixed
anchor-action constraint \(r(s,a^\dagger)=g(s)\) is the special case
\(\mu(a\mid s)=1\{a=a^\dagger\}\) \citep{geng2020deep}; when \(g\equiv0\), it
reduces to the classical zero-reward normalization \(r(s,a^\dagger)=0\) used in
Rust's engine-replacement model \citep{rust1987optimal}. More generally, taking
\(\mu(a\mid s)=1\{a=a^\dagger(s)\}\) allows state-specific anchors, where the
reference action may vary with the state. DDC models often use such anchors to
normalize the payoff of a no-action, status-quo, or outside option to be zero
or otherwise known
\citep{rust1987optimal,hotz1993conditional,aguirregabiria2010dynamic,
geng2020deep}. If \(\mu(a\mid s)=1/|\mathcal A|\), the constraint fixes the
statewise average reward and, when \(g\equiv0\), becomes a sum-to-zero
constraint \citep{kallus2016revealed}.
The framework also permits data-driven choices of \((\mu,g)\). A statewise
value constraint is a special case: if \(V_r^\mu\) is the value function of
policy \(\mu\) under reward \(r\), then imposing \(V_r^\mu=h\) is equivalent,
by the Bellman equation \(V_r^\mu=\mu r+\gamma\mu P V_r^\mu\), to taking
\(g:=h-\gamma\mu P h\). For example, in medical decision problems, one may wish
to normalize rewards relative to standard care using historical outcome data
\citep{kallus2018removing}. If an auxiliary dataset records states and realized
outcomes \((s_i,y_i)\) from a population with the same reward function and
behavior policy \(\pi\), then
\(g(s)=E[y\mid s]=\sum_a \pi(a\mid s)r(s,a)\). Taking \(\mu=\pi\) therefore
yields a natural data-driven normalization.
\textbf{Main problem:} Putting these pieces together, we obtain the following
constrained maximum-likelihood problem for recovering \((r^\star, v^\star)\):
\begin{equation}
\label{eq:main-irl}
\begin{aligned}
\arg\max_{r,v}\quad
& \mathbb{E}_{(s,a) \sim \nu_{\pi}}\!\left[
r(s,a) + \gamma v(s,a) - \exp\{\Xi(r+\gamma v)(s)\}
\right] \\
\text{s.t.}\quad
& v = P\Xi(r+\gamma v), \qquad\qquad\text{\color{gray}(soft Bellman)} \\
& \mu r = g.
\qquad\text{\color{gray}(affine normalization)}
\end{aligned}
\end{equation}
That is, we maximize the conditional log-likelihood over state--action
functions \(r\) and \(v\) subject to the soft Bellman equation and a statewise
affine normalization constraint.
The remainder of the paper shows that solving \eqref{eq:main-irl} reduces to two steps: estimate \(\pi\), then solve a linear Bellman evaluation equation to recover the unique pair \((r^\star, v^\star)\). We then turn this characterization into a simple algorithm.
\section{From partial to point identification}
\label{sec:char}
To expose the core structure of the problem, we first drop the normalization constraint and study the resulting relaxed optimization problem.
\textbf{Relaxed problem:} We remove the reward normalization in
\eqref{eq:main-irl} but keep soft Bellman consistency:
\begin{equation}
\label{eq:relaxed-irl}
\begin{aligned}
\arg\max_{r,v}\quad
& \mathbb{E}_{(s,a) \sim \nu_{\pi}}\!\left[
r(s,a) + \gamma v(s,a) - \exp\{\Xi(r+\gamma v)(s)\}
\right] \\
\text{s.t.}\quad
& v = P\Xi(r+\gamma v).
\qquad {\color{gray}(\text{soft Bellman})}
\end{aligned}
\end{equation}
This problem is highly non-unique. Because the objective is invariant under
potential-based transformations of the reward, the relaxed problem admits an
entire equivalence class of solutions. We will exploit this invariance later to
recover a solution to the normalized problem \eqref{eq:main-irl} from a single
convenient representative.
\subsection{Behavior cloning solves the relaxed problem}
\label{sec:easy-solution}
Without normalization, the relaxed problem admits a particularly simple
solution: estimate the behavior policy \(\pi\) and set
\(r(s,a)=\log \pi(a\mid s)\) and \(v(s)=0\). This pair maximizes the
conditional log-likelihood and satisfies the soft Bellman equation. Indeed, $P\Xi(r+\gamma v) = 0$ since $\exp \{\Xi(r+\gamma v)(s)\} = \sum_a \exp\{r(s,a)\}
=
\sum_a \pi(a\mid s)
=
1.$
We will use the shorthand
\[
u^\star(s,a):=\log \pi(a\mid s).
\]
\begin{lemma}[Trivial optimum of the relaxed problem]
\label{lem:trivial}
The pair \((r,v) := (u^\star,0)\) solves \eqref{eq:relaxed-irl}.
\end{lemma}
A related observation appears in Section~4 of \citet{fu2018airl}, but not as a
tool for identification or estimation. Here, by contrast, it is the starting
point for normalization-based identification. As shown in
Section~\ref{sec:aside}, this trivial solution already suffices for policy-value
comparisons.
\subsection{An invariance among solutions}
\label{sec:invariance}
The relaxed problem is invariant to \emph{potential-based shaping}: adding a state-only potential $c:\mathcal{S} \to \mathbb{R}$ shifts all logits in the softmax by the same amount per state, leaving both feasibility and likelihood unchanged. This is the entropy-regularized analogue of reward shaping in classical RL \citep{ng1999policy} and explains why the relaxed objective is flat along an affine subspace.
\begin{lemma}[Potential-based shaping invariance]
\label{lem:shaping}
Let \((r,v)\) be feasible for \eqref{eq:relaxed-irl}, and let
\(c:\mathcal S\to\mathbb R\) be arbitrary. Define
\(\tilde r = r + c - \gamma Pc\) and \(\tilde v = v + Pc\).
Then \((\tilde r,\tilde v)\) is also feasible for \eqref{eq:relaxed-irl} and
attains the same objective value as \((r,v)\). In particular, the induced
log-policy
\(u^\star(s,a)=r(s,a)+\gamma v(s,a)-\Xi(r+\gamma v)(s)\) is unchanged.
\end{lemma}
Related partial-identification results appear in
\citet[Theorem~1]{cao2021identifiability}, \citet{fu2018airl}, and
\citet[Lemma~2]{geng2020deep}. We next show that our normalization
selects a unique representative from this class.
\subsection{Solving the original normalized problem}
\label{sec:solve-normalization}
Lemma~\ref{lem:trivial} gives one relaxed optimum, \((u^\star,0)\), and
Lemma~\ref{lem:shaping} characterizes all others via potential-based
transformations of the form \((r,v) = (u^\star + c - \gamma Pc, Pc)\). Since
the constrained and relaxed problems have the same optimal value, solving
\eqref{eq:main-irl} amounts to finding the shaping function \(c\) such that
the corresponding pair \((r,v)\) satisfies the desired constraint. For our
normalization \(\mu r = g\), \(c\) is uniquely determined by a Bellman
equation.
The next result gives the corresponding solution in terms of
\(Q^\mu_{u^\star-g}\), the \(Q\)-function under reward \(u^\star-g\) and policy
\(\mu\), where \(u^\star=\log \pi\). Recall that
\[
(P\mu Q)(s,a):=\mathbb{E}_{s' \sim P(\cdot\mid s,a),\, a' \sim \mu(\cdot\mid s')}
\bigl[Q(s',a')\bigr].
\]
\begin{theorem}[IRL via a Bellman equation]
\label{thm:unique}
Let \(Q^\mu_{u^\star-g}\) be the unique bounded solution to
\[
Q^\mu_{u^\star-g}(s,a) = u^\star(s,a)-g(s) + \gamma (P \mu Q^\mu_{u^\star-g})(s,a).
\]
Then \eqref{eq:main-irl} admits a unique optimal solution \((r^\star,v^\star)\), given by
\begin{align*}
r^\star(s,a)
&= Q^\mu_{u^\star-g}(s,a)
- (\mu Q^\mu_{u^\star-g})(s)
+ g(s), \\
v^\star(s,a)
&= \frac{1}{\gamma}\bigl(
u^\star(s,a) - g(s) - Q^\mu_{u^\star-g}(s,a)
\bigr).
\end{align*}
\end{theorem}
This theorem is the main result of the paper. It shows that normalized reward
recovery reduces to two steps: first estimate \(u^\star = \log \pi\), then solve
the linear Bellman equation for \(Q^\mu_{u^\star-g}\). The normalized reward is then obtained in closed form as the advantage function
\(Q^\mu_{u^\star-g} - \mu Q^\mu_{u^\star-g}\), shifted by \(g\). By Lemma~\ref{lem:shaping}, every feasible
representative in the shaping class induces the same \(u^\star\). Therefore,
once one feasible representative is known, we can impose a different
normalization without re-solving the original IRL problem.
\textbf{Comparison to DeepPQR.} In the anchor-action setting
\(\mu(a \mid s)=1\{a=a^\dagger\}\), DeepPQR \citep{geng2020deep} first
estimates the behavior policy and then learns the anchored value
\(W(s):=Q^\mu_{u^\star-g}(s,a^\dagger)\), which is used to reconstruct the full
\(Q\)-function and hence the normalized reward. In our notation,
\(Q^\mu_{u^\star-g}\) satisfies
\[
Q^\mu_{u^\star-g}(s,a)
=
u^\star(s,a)-g(s)+\gamma \mathbb{E}\!\left[Q^\mu_{u^\star-g}(s',a^\dagger)\mid s,a\right],
\]
so it is fully determined by \(W\), where
\[
W(s)=u^\star(s,a^\dagger)-g(s)+\gamma \mathbb{E}[W(s')\mid s,a^\dagger].
\]
Thus, DeepPQR estimates the same target through the intermediate quantity
\(W\). Our formulation makes this explicit by working directly with the single
\(Q\)-function \(Q^\mu_{u^\star-g}\), from which both the normalized reward and
the continuation value \(v^\star\) follow immediately. This removes the extra
step of estimating \(PW\), yielding a simpler and more modular second stage.
It also highlights a practical tradeoff: DeepPQR is tied to the anchor action
\(a^\dagger\), which may be unstable when observations under \(a^\dagger\) are
limited, whereas our formulation estimates the full \(Q\)-function directly and
can borrow strength across actions through any suitable \(Q\)-learning method.
This matches our experiments: direct estimation in GenPQR performs better when
anchor actions are rare or the action space is large, whereas DeepPQR's
anchor-action regression becomes unstable with rare anchors and has no direct
continuous-action analogue.
\subsection{Behavior cloning suffices for policy comparison}
\label{sec:aside}
For policy comparison, exact reward recovery is unnecessary once the transition
kernel \(P\) and discount factor \(\gamma\) are fixed. Let
\(V_r^\pi=\pi Q_r^\pi\) denote the value function under reward \(r\) and policy
\(\pi\), where \(Q_r^\pi\) is the corresponding \(Q\)-function.
\begin{theorem}[Identification of policy value differences]
\label{thm:identification}
Let \((r,v)\) solve \eqref{eq:relaxed-irl}, for example \((u^\star,0)\). Then,
for any two policies \(\pi_1,\pi_2\), $V_{r^\star}^{\pi_1}(s)-V_{r^\star}^{\pi_2}(s)
=
V_r^{\pi_1}(s)-V_r^{\pi_2}(s).$
\end{theorem}
Thus, exact reward recovery is needed only for targets that depend on the
normalization itself, such as evaluation under counterfactual transition
dynamics or discount factors, or interpretation of structural features of
\(r\). Otherwise, \(u^\star\), or any reward solving the relaxed problem,
suffices for policy comparison. This parallels \citet{hotz1993conditional},
where under Gumbel shocks value differences are identified from log-odds of
observed choices.
\section{A generic algorithm}
\label{sec:genalg}
Algorithm~\ref{alg:generic-irl} presents Generalized
Policy-to-\(Q\)-to-Reward (GenPQR) for reward recovery. The method has two
standard steps. First, we estimate the behavior policy \(\pi(a \mid s)\),
equivalently \(u^\star(s,a)=\log \pi(a \mid s)\), yielding \(\hat u\).
Second, we recover \((r^\star,v^\star)\) by estimating the \(Q\)-function of
policy \(\mu\) under reward \(\hat u-g\), then applying the plug-in formula in
Theorem~\ref{thm:identification}. Appendix~\ref{app:continuous-actions} gives
the continuous-action version: estimate a conditional log-density, solve the
same \(Q\)-fixed point with \(\mu\)-integrals, and apply the same normalized
advantage formula. When \(\hat \pi(a\mid s)\) is
close to \(0\), \(\hat u\) may be unstable;
in practice, one may clip \(\hat \pi\) away from zero
\citep{ionides2008truncated}. Thus, in our setting, IRL reduces to
behavior-policy estimation plus offline policy evaluation, rather than a
fundamentally new estimation problem. The reward is identified only up to scale; if the scale is known, the recovered reward can be rescaled accordingly; see Appendix C.2.3 of \citet{geng2020deep}.
The procedure is modular and black-box. The first stage can use any
probabilistic classifier, behavior-cloning, imitation-learning, or IRL method
trained on the observed data, including MaxEnt IRL and adversarial IRL. The
second stage can use a range of existing offline RL methods, including
temporal-difference learning \citep{tsitsiklis1996analysis}, fitted
\(Q\)-evaluation (FQE) \citep{munos2008finite,van2025fitted}, and minimax or
saddle-point \(Q\)-learning methods
\citep{uehara2020minimax,imaizumi2021minimax,xie2020q,xie2021batch}. This
separation lets each stage leverage existing methods and remain compatible with
techniques for handling misspecification or distribution shift
\citep{fujimoto2019off,agarwal2021deep,chen2019information,foster2021offline}.
Moreover, given any feasible reward--value pair under one normalization,
GenPQR can recover the corresponding reward under another normalization without
re-solving the original IRL problem.
\begin{figure}[t]
\centering
\footnotesize
\begin{minipage}[t]{0.48\linewidth}
\begin{algorithm}[H]
\caption{\textsc{Generalized Policy-to-\(Q\)-to-Reward (GenPQR)}}
\label{alg:generic-irl}
\begin{algorithmic}[1]
\INPUT Transitions \(\{(s_i,a_i,s_i')\}_{i=1}^n\), normalization
\(\mu(a \mid s)\), anchor \(g(s)\), discount \(\gamma\)
\STATE \textbf{Policy estimation:} fit
\(\hat u(s,a)\approx \log \pi(a\mid s)\) using classification, behavior
cloning, or IRL
\STATE \textbf{\(Q\)-evaluation:} solve
\[
\hat Q(s,a)\approx \hat u(s,a)-g(s)+\gamma (P\mu \hat Q)(s,a)
\]
using any approximate dynamic-programming or policy-evaluation method
\OUTPUT \textbf{Reward:}
\[
\hat r(s,a)=\hat Q(s,a)-\sum_{a'}\mu(a'\mid s)\hat Q(s,a')+g(s)
\]
\end{algorithmic}
\end{algorithm}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\linewidth}
\begin{algorithm}[H]
\caption{\textsc{Fitted $Q$ Evaluation} for \(Q^\mu_{\hat u-g}\)}
\label{alg:simple-irl}
\begin{algorithmic}[1]
\INPUT Transitions \(\{(s_i,a_i,s_i')\}_{i=1}^n\), log-policy \(\hat u\),
normalization \(\mu(a\mid s)\), anchor \(g(s)\), discount \(\gamma\), class
\(\mathcal{F}\), iterations \(K\)
\STATE Initialize \(\hat Q^{(0)}(s,a)\gets 0\)
\FOR{\(k=1,\ldots,K\)}
\STATE For each \(i\), set
\[
y_i \gets \hat u(s_i,a_i)-g(s_i)+\gamma\sum_{a'}\mu(a'\mid s_i')\,
\hat Q^{(k-1)}(s_i',a')
\]
\STATE Fit \(\hat Q^{(k)} \in \mathcal{F}\) by regressing \(y_i\) on
\((s_i,a_i)\)
\ENDFOR
\OUTPUT \(\hat Q \gets \hat Q^{(K)}\)
\end{algorithmic}
\end{algorithm}
\end{minipage}
\end{figure}
Algorithm~\ref{alg:simple-irl} instantiates the second stage of GenPQR using
FQE, thereby generalizing DeepPQR \citep{geng2020deep}. It requires only
standard tools for policy estimation and regression. Moreover, the regression
step in Algorithm~\ref{alg:simple-irl} need not be solved exactly at each
iteration: one may instead parametrize \(Q\) by a neural network and take one
or a few stochastic-gradient steps per iteration, as in deep fitted
\(Q\)-learning or boosting \citep{riedmiller2005neural,tosatto2017boosted}. The expectation under
\(\mu\) can also be approximated by Monte Carlo sampling to avoid explicit
integration.
\section{Theoretical Guarantees}
\label{sec:theory}
\subsection{A generic and modular deterministic bound}
We now turn to finite-sample guarantees for $\hat r = \hat Q -\mu \hat Q + g$ obtained via Algorithm \ref{alg:generic-irl}. The following result shows how errors
in the estimated log-policy \(\hat u\) and anchored \(Q\)-function \(\hat Q\)
propagate to the recovered reward. We measure errors in the behavior norm \(L^2(\nu_\pi)\), but the Bellman
operator is most naturally analyzed in \(L^2(d_\mu)\), where \(d_\mu\) is any
stationary state distribution under \((\mu,P)\)
\citep{patterson2022generalized,van2025fitted}, that is,
\[
d_\mu = d_\mu P_\mu,
\qquad
P_\mu(s' \mid s) := \int_{\mathcal A} P(s' \mid s,a)\,\mu(a\mid s)\,da.
\]
Recalling that \(\nu_\pi := \rho \otimes \pi\), we write
\[
\|f\|_{2,\mathrm{beh}}
:=
\Bigl(\mathbb{E}_{(s,a)\sim \nu_\pi}[f(s,a)^2]\Bigr)^{1/2},
\qquad
\|f\|_\infty
:=
\operatorname*{ess\,sup}_{s,a} |f(s,a)|.
\]
We impose a coverage assumption requiring the observed system
\((\rho,\pi,P)\) to provide sufficient support relative to the normalization
policy \(\mu\) under the same dynamics \(P\): \(\pi\) must cover \(\mu\), the
behavior state distribution \(\rho\) must cover the stationary distribution
\(d_\mu\), and \(d_\mu\) must cover the one-step state distribution
\(\nu_\pi P\).
\begin{assumption}[Policy mismatch, stationary-state coverage, and one-step state coverage]
\label{cond::coverage}
Assume
\[
C_{\mathrm{cov}}
:=
C_{\nu_\pi P/d_\mu}\,C_{d_\mu/\rho}\,C_{\mu/\pi}
< \infty,
\]
where
\[
C_{\mu/\pi}
:=
\sup_{s,a}\frac{\mu(a\mid s)}{\pi(a\mid s)},
\quad
C_{d_\mu/\rho}
:=
\left\|\frac{d d_\mu}{d\rho}\right\|_{\infty},
\quad
C_{\nu_\pi P/d_\mu}
:=
\left\|\frac{d(\nu_\pi P)}{d d_\mu}\right\|_{\infty}.
\]
\end{assumption}
The coverage constant \(C_{\mathrm{cov}}\) is analogous to standard coverage
and concentrability coefficients in offline RL
\citep{xie2022role,zhan2022offline}. It equals \(1\) under stationary sampling
from \((P,\mu)\). Since the normalization policy \(\mu\) is typically chosen
with knowledge of the behavior policy \(\pi\), this condition is often less
restrictive here than in standard offline policy evaluation. For example, in
the anchor-action case, one would not anchor on an action that is rarely or
never observed.
\begin{theorem}[Reward recovery bound]
\label{thm:reward-recovery}
Under Assumption~\ref{cond::coverage},
\[
\|r^\star-\hat r\|_{2,\mathrm{beh}}
\le
\bigl(1+\sqrt{C_{\mu/\pi}}\bigr)
\left\{
\|Q^\mu_{\hat u-g}-\hat Q\|_{2,\mathrm{beh}}
+
\left(
1+\frac{\gamma\sqrt{C_{\mathrm{cov}}}}{1-\gamma}
\right)
\|u^\star-\hat u\|_{2,\mathrm{beh}}
\right\}.
\]
\end{theorem}
The reward-estimation error decomposes into a \(Q\)-estimation term,
\(\|Q^\mu_{\hat u-g}-\hat Q\|_{2,\mathrm{beh}}\), and a policy-estimation
term, \(\|\hat u-u^\star\|_{2,\mathrm{beh}}\). This makes the bound modular:
any guarantees for \(Q\)-learning and policy estimation translate directly into
a reward-recovery guarantee, with the latter scaled by \((1-\gamma)^{-1}\) and
the coverage coefficients in Assumption~\ref{cond::coverage}. Under sample
splitting \citep{foster2023orthogonal}, for example, existing results apply
directly to \(\hat u\) and \(\hat Q\), including bounds for FQE
\citep{munos2008finite,van2025fitted} and minimax \(Q\)-learning
\citep{uehara2020minimax}. In the next section, we apply this
template to derive finite-sample bounds for PQR with FQE.
\subsection{Generalization conditions for GenPQR with FQE}
\label{sec:theory2}
We now specialize Theorem~\ref{thm:reward-recovery} to the case in which
\(\hat Q\) is obtained by running \(K\) steps of FQE over a regression class
\(\mathcal F\), initialized at \(\hat Q^{(0)}\), as in
Algorithm~\ref{alg:simple-irl}. To state the resulting finite-sample bound, we
assume PAC-style generalization guarantees for the policy-estimation and
regression steps.
\begin{assumption}[Policy generalization]
\label{assump:policy-gen}
There exists \(\underline p>0\) such that, for all \(\delta\in(0,0.5)\), with
probability at least \(1-\delta\),
\[
\min\{\pi(a\mid s),\hat\pi(a\mid s)\}\ge \underline p
\quad\text{for all }(s,a),
\qquad
\left\{
\mathbb{E}_{s\sim\rho}\!\left[
\mathrm{KL}\!\left(\pi(\cdot\mid s)\,\|\,\hat\pi(\cdot\mid s)\right)
\right]
\right\}^{1/2}
\le
\rho_\pi(n,\delta).
\]
\end{assumption}
Lower bounds on \(\hat \pi\) and \(\pi^\star\) ensures that KL divergence controls \(\|\hat u-u^\star\|_{2,\mathrm{beh}}\). For maximum likelihood estimation over a class \(\mathcal U\), one typically has \(\rho_\pi(n,\delta)\lesssim r_{\mathcal U}(n)+\sqrt{\log(1/\delta)/n},\) where \(r_{\mathcal U}(n)\) is a local complexity measure of \(\mathcal U\) (e.g., VC dimension) \citep{geer2000empirical,wainwright2019high}.
We require each regression step to achieve small excess risk relative to the best approximation in \(\mathcal F\) to the Bellman target \(\mathcal T_{\hat u}^\mu(\hat Q^{(k-1)})\), where \(\mathcal T_u^\mu f:=u-g+\gamma P\mu f\). Define
\[
\mathrm{reg}(Q\mid u,Q')
:=
\|\mathcal T_u^\mu(Q')-Q\|_{2,\mathrm{beh}}^2
-
\inf_{f\in\mathcal F}\|\mathcal T_u^\mu(Q')-f\|_{2,\mathrm{beh}}^2.
\]
\begin{assumption}[One-step regression generalization]
\label{assump:q-gen}
$\sup_{f \in \mathcal{F}} \|f\|_{\infty} < \infty $ and for all \(k\in[K]\) and \(\delta\in(0,0.5)\), with probability at least
\(1-\delta\), $\{\mathrm{reg}(\hat Q^{(k)}\mid \hat u,\hat Q^{(k-1)})\}^{1/2}
\le
\rho_{Q}(n,\delta).$
\end{assumption}
Such bounds for nuisance-dependent regression targets follow from
\citet{foster2023orthogonal,van2026researcher}. Under a fresh-sample analysis
\citep{munos2008finite}, one typically has
\(\rho_{Q}(n,\delta)\lesssim r_{\mathcal F}(n/K)+\sqrt{\log(1/\delta)/n}\),
where \(r_{\mathcal F}(n/K)\) is the local Rademacher critical radius of
\(\mathcal F\); see Appendix~\ref{appendix::LSE}. Similar bounds hold for
weakly dependent trajectory data under mixing conditions
\citep{yu1994rates,mohri2010stability}.
\subsection{Finite-sample bound for GenPQR with FQE}
\label{sec:theory3}
We now state our main result for GenPQR with FQE under approximate Bellman-completeness. To account for misspecification in the regression steps, we introduce the
\emph{inherent Bellman error} of the regression class \citep{munos2008finite}:
\[
\varepsilon_{\mathcal F}
:=
\sup_{f \in \mathcal F}\inf_{g \in \mathcal F}
\|\mathcal T_{u^\star}^\mu f-g\|_{\infty}.
\]
This quantity is zero when \(\mathcal F\) is Bellman complete, that is, when
\(\mathcal T_{u^\star}^\mu f \in \mathcal F\) for every \(f \in \mathcal F\)
\citep{chen2019information}. In that case, each regression step in
Algorithm~\ref{alg:simple-irl} is correctly specified, since its population
target lies in \(\mathcal F\). We define the standard concentrability coefficient
\[
C_{\mathrm{conc}}
:=
\sup_{m\ge 0}
\left\|
\frac{d\bigl(\nu_\pi (P\mu)^m\bigr)}{d\nu_\pi}
\right\|_\infty < \infty,
\]
which is finite under Assumption~\ref{cond::coverage} with $C_{\mathrm{conc}}
\le C_{\mathrm{cov}}$ by Lemma~\ref{lem:coverage-implies-conc} in Appendix \ref{appendix:lemmas}.
\begin{theorem}[Finite-sample bound for GenPQR with FQE]
\label{thm:pqr-fqe}
Assume Assumptions~\ref{cond::coverage}, \ref{assump:policy-gen}, and
\ref{assump:q-gen}.
Then with probability at least \(1-\delta\),
\begin{align*}
\|r^\star-\hat r\|_{2,\mathrm{beh}}
&\le
\bigl(1+\sqrt{C_{\mu/\pi}}\bigr)
\Bigg\{
\sqrt{C_{\mathrm{conc}}}\,
\gamma^K \|\hat Q^{(0)}-Q^\mu_{\hat u-g}\|_{\infty}
+
\frac{\sqrt{C_{\mathrm{conc}}}}{1-\gamma}
\Bigl(
\varepsilon_{\mathcal F}
+\rho_Q\bigl(n,\tfrac{\delta}{2K}\bigr)
\Bigr)
\\
&\qquad\qquad+
\frac{\sqrt{2}}{\underline p^{2}}\left(
1
+\frac{\sqrt{C_{\mathrm{conc}}}}{1-\gamma}
+\frac{\gamma\sqrt{C_{\mathrm{cov}}}}{1-\gamma}
\right)
\rho_\pi\bigl(n,\tfrac{\delta}{2}\bigr)
\Bigg\}.
\end{align*}
\end{theorem}
\textbf{Proof sketch.}
Apply Theorem~\ref{thm:reward-recovery} and bound
\(\|Q^\mu_{\hat u-g}-\hat Q\|_{2,\mathrm{beh}}\) by the inexact Picard argument
of \citet{munos2008finite}. The only change is that inherent Bellman error is
measured for \(\mathcal T^\mu_{u^\star}\), not the data-dependent
\(\mathcal T^\mu_{\hat u}\); Lemma~\ref{lemma::periter} shows this suffices up
to policy-estimation error. Approximate Bellman completeness could be
relaxed using suitable weighting \citep{van2025fitted} or minimax formulations
\citep{uehara2023offline}.
\textbf{Discussion.}
The bound matches the usual FQE structure
\citep{munos2008finite,van2025fitted}, with additional terms due to estimation
of \(u^\star\). The three terms respectively capture finite-iteration error,
approximation and statistical error from the fitted Bellman updates, and the
IRL-specific error from estimating \(u^\star\). The last term can be large when
the behavior policy is nearly deterministic, for example in low-temperature
softmax regimes, because KL divergence controls \(L^2\) log-policy error only
up to a factor of \(\underline p^{-2}\). In particular, if \(\hat u\) and
\(\hat Q\) are learned by ERM over parametric classes \(\mathcal U\) and
\(\mathcal F\) with pseudo-dimensions \(d_{\mathcal U}\) and
\(d_{\mathcal F}\), and \(K \asymp \log n\), then one typically obtains, up to
\(O(\sqrt{\log \log n / n})\) terms,
\[
\|\hat r-r^\star\|_{2,\mathrm{beh}}
\lesssim
\frac{\sqrt{C_{\mu/\pi}C_{\mathrm{cov}}}}{1-\gamma}
\left(
\varepsilon_{\mathcal F}
+
\sqrt{\frac{d_{\mathcal F}}{n}}
+
\frac{1}{\underline p^{2}}\sqrt{\frac{d_{\mathcal U}}{n}}
\right).
\]
\textbf{Comparison to DeepPQR.}
Theorem~2 of \citet{geng2020deep} gives a related FQE bound for anchor-action
reward recovery under a specific neural-network architecture. Our result is
more modular: it is not tied to a particular function class, allows approximate
Bellman completeness for \(\mathcal T^\mu_{u^\star}\), makes coverage explicit,
and replaces sup-norm policy-error control with Kullback--Leibler or
\(L^2\)-type control, which is natural for likelihood-based policy estimators,
multiclass classification, MaxEnt IRL \citep{ziebart2008maximum}, and
adversarial IRL \citep{fu2018airl,snoswell2020revisiting,ke2020imitation,
foster2024behavior}.
\section{Experimental investigation}
\label{sec:experiments}
We adapt DeepPQR's synthetic study to isolate identification rather than
imitation performance, using its infinite-horizon environment with continuous
states, five actions, deterministic transitions, and anchor-action
normalization \(g(s)=0\). We compare DeepPQR and GenPQR under matched policy
estimation, and study how policy and \(Q\)-estimation choices affect modular
reward recovery. The key distinction is the effective anchor sample: DeepPQR
estimates its anchor \(Q\)-function only on anchor-action transitions, whereas
GenPQR uses the full sample for \(Q\)-evaluation. We report reward MSE,
reward correlation, held-out policy negative log-likelihood, and runtime over
\(100\) seeds with \(95\%\) confidence intervals; details are in
Appendix~\ref{app:simulation-details}.
\begin{figure*}[htb]
\centering
\begin{minipage}[c]{0.71\textwidth}
\centering
\scriptsize
\resizebox{\linewidth}{!}{
\begin{tabular}{llcccc}
\toprule
Setting & Method & Anchor ct. (frac.) & MSE $\downarrow$ & Corr. $\uparrow$ & Time $\downarrow$ \\
\midrule
200 / rare & DeepPQR & 311 (0.16) & 2.60 $\pm$ 0.15 & 0.46 $\pm$ 0.03 & 7.33 $\pm$ 0.10 \\
200 / rare & GenPQR & 311 (0.16) & \textbf{0.91 $\pm$ 0.11} & \textbf{0.66 $\pm$ 0.03} & \textbf{5.22 $\pm$ 0.08} \\
\midrule
1000 / rare & DeepPQR & 1559 (0.16) & 1.48 $\pm$ 0.10 & 0.61 $\pm$ 0.03 & 38.84 $\pm$ 0.60 \\
1000 / rare & GenPQR & 1559 (0.16) & \textbf{0.76 $\pm$ 0.08} & \textbf{0.73 $\pm$ 0.02} & \textbf{27.76 $\pm$ 0.42} \\
\midrule
2500 / common & DeepPQR & 6280 (0.25) & 0.70 $\pm$ 0.07 & 0.73 $\pm$ 0.02 & 92.82 $\pm$ 1.28 \\
2500 / common & GenPQR & 6280 (0.25) & \textbf{0.60 $\pm$ 0.06} & \textbf{0.73 $\pm$ 0.02} & \textbf{64.09 $\pm$ 0.90} \\
\bottomrule
\end{tabular}
}
\end{minipage}
\hfill
\begin{minipage}[c]{0.25\textwidth}
\centering
\includegraphics[width=\linewidth]{exp1_matched_curve.png}
\end{minipage}
\caption{Matched DeepPQR vs.\ GenPQR comparison. Both use the same AIRL policy
estimate and neural FQE. Entries are mean \(\pm\) 95\% CI over 100 seeds.
Anchor count is the number of anchor-action transitions used by DeepPQR's
anchor-\(Q\) step.}
\label{fig:exp1_combo}
\end{figure*}
\textbf{Matched comparison to DeepPQR.}
Both methods use the same AIRL policy estimate and neural downstream
approximation, so differences reflect the identification step
(Section~\ref{sec:solve-normalization}). We vary trajectory count and anchor
frequency, which determine DeepPQR's effective anchor sample size.
Figure~\ref{fig:exp1_combo} summarizes three regimes; Appendix
\ref{app:additional-experiments} reports similar behavior with more actions.
\vspace{0.35em}
\noindent\textbf{Estimator choices.}
At \(1000\) trajectories with a well-supported anchor action, we compare AIRL
versus behavior cloning for policy estimation, neural versus boosted FQE, and
standard reward-recovery baselines. Figure~\ref{fig:exp2_combo} in
Appendix~\ref{app:fighigher} shows that GenPQR remains effective across
estimator choices, with substantial runtime variation across implementations.
\vspace{0.35em}
\noindent\textbf{Discussion.}
The matched comparison isolates the statistical cost of anchor-subset
identification; the estimator-choice study shows that GenPQR remains effective
under practical policy and \(Q\)-estimation choices. We provide concluding remarks in Appendix \ref{app:additional-conclusion}.
\bibliographystyle{plainnat}
\bibliography{references}