Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
139,127 characters · 39 sections · 72 citation commands
A Lecture Note on Offline RL and IRL Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
In the forward reinforcement-learning problem, the reward function is a fixed, known input. The world handed us $r$ and asked us to learn $Q^\star$. In a great many real-world problems, however, $r$ is exactly what we want to learn, and the closest thing we have to it is a log of decisions made by someone who knew what they were doing.
In each case the data is offline: a fixed log of state-action-(next-state) tuples from one or more experts, and the reward itself is never directly observed levine2020offline.
Two literatures have been chipping away at this problem for decades, under almost identical probabilistic structure but completely different vocabularies:
Our first order of business (Section (ref)) is to prove these are the same model. Sections (ref)--(ref) then survey the methods each tradition developed, and Section (ref) presents the unified ERM treatment of kang2025empirical that fixes the issues both traditions inherited.
This section sets up the model, defines the soft Bellman operator, proves its contraction, and, most importantly, proves the equivalence between the DDC and entropy-regularized IRL formulations. Readers familiar with standard MDP notation can skim the MDP setup, but should not skip Section (ref) (the Gumbel-max identity) or Section (ref) (the equivalence theorem).
We consider an infinite-horizon discounted MDP $\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, \nu_0, r, \beta)$ where $\mathcal{S}$ is a (possibly continuous) state space, $\mathcal{A}$ is a finite action space, $P : \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$ is the transition kernel, $\nu_0 \in \Delta(\mathcal{S})$ is the initial distribution, $r : \mathcal{S} \times \mathcal{A} \to [-R_{\max}, R_{\max}]$ is the (unknown) reward, and $\beta \in (0, 1)$ is the discount. A stationary Markov policy is a measurable map $\pi : \mathcal{S} \to \Delta(\mathcal{A})$ with $\pi(a \mid s)$ the probability of $a$ at $s$. We write $\mathbb{P}_{\nu_0, \pi}$ and $\mathbb{E}_{\nu_0, \pi}$ for the distribution and expectation over the resulting trajectory $\tau = (s_0, a_0, s_1, a_1, \ldots)$.
The discounted state-action occupancy of $\pi$ is
a probability measure on $\mathcal{S} \times \mathcal{A}$. The offline dataset is
sampled iid from $d^{\pi^\star}$ for a single expert policy $\pi^\star$. Note what is missing: we do not observe rewards in $\mathcal{D}$: only states, actions, and next states. That omission is precisely what makes the problem “inverse,” and it is the canonical setup of both DDC and IRL.
Let $\mathcal{Q} := \{Q : \mathcal{S} \times \mathcal{A} \to \mathbb{R}\;\text{measurable, bounded}\}$. The sup-norm is $\|Q\|_\infty := \sup_{s, a} |Q(s,a)|$. For $\rho \in \Delta(\mathcal{S} \times \mathcal{A})$ we write $\|Q\|_\rho^2 := \mathbb{E}_{(s,a) \sim \rho}[Q(s,a)^2]$ and $\langle Q_1, Q_2 \rangle_\rho := \mathbb{E}_\rho[Q_1 Q_2]$, so $L^2(\rho)$ is a Hilbert space. For a closed linear subspace $\mathcal{F} \subseteq L^2(\rho)$, the $L^2(\rho)$-projection is the unique element $\Pi_{\mathcal{F}, \rho} Q \in \mathcal{F}$ satisfying $\|Q-\Pi_{\mathcal{F}, \rho}Q\|_\rho = \inf_{f \in \mathcal{F}}\|Q-f\|_\rho$. This restriction to closed linear subspaces matters: projections onto arbitrary closed nonconvex sets may fail to be unique. For closed linear subspaces, $\Pi_{\mathcal{F}, \rho}$ is a non-expansion: $\|\Pi Q_1 - \Pi Q_2\|_\rho \le \|Q_1 - Q_2\|_\rho$ for all $Q_1, Q_2$. We will need this fact, both the projection itself and its non-expansion property, when analyzing temporal-difference methods in Section (ref).
The DDC setup follows the Type 1 Extreme Value (T1EV) derivation in kang2025empirical. At each period the agent observes the deterministic flow payoff $r(s_t,a_t)$ plus an idiosyncratic alternative-specific shock $\epsilon_{t,a_t}$. We use the unit-scale Gumbel distribution with location parameter $\delta$:
where $\gamma_E$ is the Euler--Mascheroni constant. The mean-zero T1EV normalization used for the baseline DDC--MaxEnt equivalence is
The DDC agent maximizes
where the expectation averages over the Markov chain and the idiosyncratic shocks. The policy is induced by choosing the alternative with the largest shock-augmented choice-specific value. The derivation in Section (ref) shows that, under the mean-zero normalization $\delta=-\gamma_E$, this DDC problem implies
If $\delta\neq-\gamma_E$, the same derivation gives the same softmax rule but adds the constant $\beta(\delta+\gamma_E)$ to the right side of the $Q$ Bellman equation.
The MaxEnt-IRL setup in kang2025empirical instead treats the agent as choosing a stochastic stationary policy and explicitly rewards entropy. With regularization coefficient $\lambda$, the forward problem is
where $\operatorname{H}(\pi(\cdot\mid s)):=-\sum_a\pi(a\mid s)\log\pi(a\mid s)$. Throughout the main text, and in the equivalence theorem below, we set
which is the entropy scale corresponding to unit-scale T1EV shocks. Section (ref) derives the same softmax rule (ref) and soft Bellman equation (ref) from this entropy-regularized problem.
The log-sum-exp is a smooth surrogate for $\max$, sandwiched between it and a $\log|\mathcal{A}|$ slack: $\max_{a'} Q(s', a') \le \operatorname{LSE}(Q(s', \cdot)) \le \max_{a'} Q(s', a') + \log|\mathcal{A}|$. The two facts below are what we need to treat $\mathcal{T}_{\mathrm{soft}}$ as well-behaved as the ordinary hard Bellman operator.
By Banach's fixed-point theorem on $(\mathcal{Q}, \|\cdot\|_\infty)$, $\mathcal{T}_{\mathrm{soft}}$ has a unique fixed point $Q^\star$ and soft value iteration $Q_{k+1} := \mathcal{T}_{\mathrm{soft}} Q_k$ converges geometrically: $\|Q_k - Q^\star\|_\infty \le \beta^k \|Q_0 - Q^\star\|_\infty$. So far this is identical to the hard-Bellman story; the action is in what comes next.
\paragraph{MaxEnt-IRL derivation.} Define the value and choice-specific value functions by
At a fixed state $s$, the Bellman recursion is the finite-dimensional entropy-regularized choice problem
By Lemma (ref),
Substituting (ref) into the definition of $Q^\star$ gives the choice-specific Bellman equation
The paper also records the equivalent continuation representation. Since (ref) implies
Equation (ref) can be written as
This is the analogue of Equation (A5) in the paper.
\paragraph{DDC derivation.} Let $\epsilon_{t,a}\overset{\mathrm{i.i.d.}}{\sim}\operatorname{Gumbel}(\delta,1)$ across actions and dates. The shock-realized value and its shock average are
and the DDC choice-specific value is
Thus
For each action $a$, $Q^\star(s,a)+\epsilon_a$ is $\operatorname{Gumbel}(Q^\star(s,a)+\delta,1)$. Lemma (ref) gives
Substituting (ref) into (ref) yields
Under the mean-zero T1EV normalization $\delta=-\gamma_E$, this becomes exactly (ref). This is Equation (3) in the paper.
The same DDC Bellman equation can also be written in the policy-continuation form used in the paper. Conditional on the selected next action $a'$, Lemma (ref) gives
Therefore
When $\delta=-\gamma_E$, Equation (ref) is identical to the MaxEnt-IRL continuation representation (ref).
Given $\mathcal{D} = \{(s_i, a_i, s_i')\}_{i=1}^N$, the sample soft Bellman operator is
For each fixed $(s, a)$, $\mathbb{E}_{s' \sim P(\cdot \mid s, a)}[\hat{\mathcal{T}}_{\mathrm{soft}} Q(s, a, s')] = \mathcal{T}_{\mathrm{soft}} Q(s, a)$: one sample is unbiased for the operator's value. Squaring breaks this. Define
and write $V_Q(s') := \operatorname{LSE}(Q(s', \cdot))$.
The discrepancy $\beta^2 \operatorname{Var}_{s' \sim P(\cdot \mid s, a)}(V_Q(s'))$ is the double-sampling bias. It is a $Q$-dependent population quantity that does not vanish with more data, so its gradient is nonzero at $Q^\star$. Minimizing the empirical squared TD residual is not the same as minimizing the population Bellman error. Two independent next-state samples from the same $(s, a)$ would suffice to break the bias, but offline RL never gives us two next-states at the same $(s, a)$. Section (ref) fixes this by deriving the bi-conjugate correction explicitly.
With the equivalence and the sampling subtlety in hand, we proceed to identification (Section (ref)), the classical DDC methods (Section (ref)), the IRL methods (Section (ref)), and the unified ERM treatment (Section (ref)).
Before discussing how to estimate the reward, we have to confront a more basic question: is the reward identifiable from offline expert data at all?
Without further structure, the answer is no. This is not a quirk of any particular estimator; it is a fact about the model. We make it precise, then describe the two normalizations, one from econometrics and one from machine learning, that resolve it, and assess what each delivers rust1994structural,magnac2002identifying,cao2021identifiability.
Suppose two reward functions $r$ and $r'$ produce, via the soft Bellman equation, the same optimal policy $\pi^\star$ and the same trajectory distribution under $P$. Then the offline expert data $\mathcal{D}$ has identical distribution under $(r, \pi^\star)$ and $(r', \pi^\star)$, and no estimator based on $\mathcal{D}$ alone can possibly distinguish them. Worse, the indeterminacy is large: the soft-Bellman fixed point is invariant under state-only potential-based shaping in the sense of ng1999policy, and we can write down explicitly an entire family of rewards that all produce the same expert behavior. This non-identification was established in the DDC framework by magnac2002identifying (their Proposition 1); we restate it in the entropy-regularized language for self-containedness.
magnac2002identifying proved that, in the DDC framework, this is essentially the only non-identification: any two per-period utilities producing the same choice probabilities under the same $(P, \beta)$ differ by a state-only potential. cao2021identifiability gave the corresponding result explicitly in the entropy-regularized MDP language. To make the result fully concrete:
Lemma (ref) establishes the “if” direction; the “only if” is the content of magnac2002identifying and cao2021identifiability. The structural takeaway is the following:
The DDC and modern-IRL literatures take very different roads at exactly this fork. We examine both.
The classical DDC route is to declare the value of $r$ at one designated action per state magnac2002identifying.
In the Rust bus-engine application, the anchor action is “replace the engine”; its payoff (the cost of a new engine) is part of the model specification, not something to be estimated from data. The anchor action is allowed to depend on $s$.
A small but important point: the anchor-action assumption is a normalization, not a restriction. It fixes the location of the reward without restricting which preferences over $(s, a)$ pairs the model can represent. It is the standard identification device in structural econometrics for DDC rust1994structural,magnac2002identifying,arcidiacono2011conditional,arcidiacono2011practical.
The machine-learning tradition takes a different fork at the identification problem of Lemma (ref). The DDC route (Section (ref)) fixes the shaping potential by fiat: declare one action's reward known at every state. Adversarial IRL fu2017learning instead tries to learn the split between reward and potential, the state-dependent term whose Bellman difference can change the numerical reward without changing behavior, by building the potential-based shaping invariance of ng1999policy directly into the structure of a discriminator. This subsection asks how much that buys.
The guiding distinction is simple. From transitions $(s,a,s')$, expert behavior directly reveals which actions look better relative to other actions at the same state; it does not directly reveal how much of that attractiveness comes from immediate reward rather than continuation value. AIRL is an attempt to learn this split: one part should be the transferable reward, and the other part should be the shaping/value correction. The question is whether the split is actually identified.
fu2017learning build on the MaxEnt-IRL framework of finn2016connection and the generative-adversarial framework of goodfellow2020generative,ho2016generative. AIRL introduces a binary classifier $D$, called the discriminator. Given a transition $x=(s,a,s')$, $D(x)$ is trained to be large on expert transitions and small on transitions generated by the current policy $\pi$. Before writing $D$ itself, AIRL specifies the reward-like score that will enter the classifier. That score is constrained to the potential-shaped form
where $g$ is intended as the transferable reward and $h$ as the shaping potential, which is equivalently the soft value term that should telescope away. In words, AIRL writes the expert-like score as \[ \text{reward part} + \text{future potential} - \text{current potential}. \] The hope is that the reward part $g$ survives transfer, while the potential terms only correct for where the transition starts and ends. We write the state-only reward-head version here because it is the version under which AIRL's disentanglement claim can be true; if $g$ is allowed to depend on $(s,a)$, the same ambiguity returns immediately. In the terminology of cao2021identifiability, “state-only” is an action-independent reward restriction. It is a real identifying restriction: recovering such a reward also needs decomposability conditions on how actions move between states, not only the adversarial objective.
This potential-shaped quantity is only a score. To turn such a score into a classifier, the adversarial-IRL derivation starts from the standard GAN/Bayes-classifier density-ratio identity. For a binary classifier trained to classify expert samples against current-policy samples, the Bayes classifier with equal class priors has
Thus the classifier logit is a log density ratio. The next step is to specify the two trajectory densities whose ratio will be used. Write a finite trajectory as $\tau=(s_0,a_0,s_1,a_1,\ldots,s_{H-1},a_{H-1},s_H)$. The density-ratio motivation used in GAN-guided cost learning and AIRL starts from an energy model over trajectories: keep the environment part of the trajectory probability, $\nu_0(s_0)\prod_t P(s_{t+1}\mid s_t,a_t)$, and tilt it toward trajectories whose accumulated learned score $\sum_t f(s_t,a_t,s_{t+1})$ is large. Exponential tilting gives the expert-side density
whereas the current policy generates trajectories according to
Taking the log-ratio cancels the common initial-state and transition factors:
By the Bayes log-density-ratio identity (ref), the corresponding trajectory classifier has log-odds
where \(C_f\) absorbs the trajectory-energy normalizer and any class-prior constant. This trajectory classifier is the GAN-GCL starting point.
\paragraph{From trajectory odds to the local AIRL discriminator.} AIRL fu2017learning now makes a one-step localization of the trajectory-level odds relation. Since the trajectory logit in (ref) is a sum of per-transition terms, the natural local contribution of a transition $(s,a,s')$ is \[ f(s,a,s')-\log\pi(a\mid s). \] AIRL therefore defines the transition-level discriminator by the odds equation
Equivalently,
How can we justify this localization? The clean way is not to view it as literally marginalizing the trajectory Bayes classifier down to one transition. A true marginal classifier for a single transition would have to account for the full distribution of times and histories that can produce the same $(s,a,s')$. The justification is more local: because the trajectory logit in (ref) is additive, the term $f(s,a,s')-\log\pi(a\mid s)$ is the one-transition contribution suggested by that logit. The lemma below records why this is the right surrogate for AIRL: it gives the local discriminator gradient used in the adversarial update and the entropy-regularized reward used in the policy update.
Thus the disappearance of the trajectory-level $\sum_t$ is the answer to a localization question, not a new cancellation in the likelihood ratio: AIRL keeps the one-transition contribution because it has the right discriminator-gradient and policy-objective identities, not because the full trajectory Bayes classifier literally collapses to one step.
This is also the right place to keep one interpretive caveat in view. The trajectory-energy model above and its corresponding localization is not the same object as the dynamic soft-control model in Section (ref). It is myopic in the specific sense emphasized by cao2021identifiability: the trajectory likelihood is tilted by accumulated one-step scores \(f(s_t,a_t,s_{t+1})\), whereas a dynamically optimal policy evaluates an action through immediate reward plus the continuation value induced by the transition kernel. In the dynamic model, \[ \log \pi^\star(a\mid s) = Q^\star(s,a)-V^\star(s) = r(s,a)+\beta\,\mathbb{E}[V^\star(s')\mid s,a]-V^\star(s). \] So the local AIRL score should first be read as a policy-rationalizing advantage-like object. It becomes a reward only after a separate decomposition argument explains which part is immediate payoff and which part is value shaping.
Specializing the generic local discriminator (ref) to the structured AIRL score $f=f_{g,h}$ from (ref) gives
The $\pi(a\mid s)$ term has an intuitive role: a generated transition may contain action $a$ simply because the current policy already takes $a$ often at state $s$. AIRL therefore asks whether the expert-like score $\exp f_{g,h}(s,a,s')$ is large relative to the current policy's own probability of producing that action. This is why the discriminator is policy-dependent, written $D_{g,h}^{\pi}$.
In the trained model $g = g_\theta$ and $h = h_\phi$ are parametrized, with $f_{\theta,\phi}:=f_{g_\theta,h_\phi}$ and $D_{\theta,\phi}^{\pi} := D_{g_\theta,h_\phi}^{\pi}$. We write $f_{g,h}$ when reasoning about identification at the level of functions, and $f_{\theta,\phi}$ when reasoning about the optimization. Also, in the adversarial objectives below, the notation $(s,a,s')\sim d^\pi$ means $(s,a)\sim d^\pi$ followed by $s'\sim P(\cdot\mid s,a)$; equivalently it is the transition-augmented discounted occupancy.
The two training updates now follow from the two roles of this classifier.
Putting these two derived objectives together, AIRL alternates:
so the policy maximizes entropy-regularized return under the implicit reward $f_{\theta,\phi}$. The modeling bet is that (ref) splits the soft advantage cleanly into a reward part and a shaping part. The bet can pay off only if that split is unique; the rest of this subsection is about when it is.
We are finally ready to discuss the identification logic of AIRL. We discuss that:
\paragraph{The saddle identifies the advantage, not the reward.} The next statement is conditional and population-level. It assumes (a) the policy player has reached the expert soft-optimal policy, so $d^\pi = d^{\pi^\star}$ on the support under discussion, and (b) the discriminator class can realize its Bayes optimum there. These assumptions say that expert and generated transitions are truly indistinguishable to the classifier, and that the classifier class is rich enough to express the corresponding optimum. If either fails, the conclusions below do not apply: the logit would describe the current policy rather than the expert, or the AIRL class would not contain the Bayes-optimal discriminator.
Under (a)--(b), the best discriminator cannot tell the two sources apart, so $D^\star \equiv \tfrac12$ on the matched support. Since the policy has matched the expert, the $\pi$ appearing in (ref) is now $\pi^\star$. Hence $D_{\theta,\phi}^{\pi^\star} = \tfrac12$ exactly when $\exp f_{\theta,\phi} = \pi^\star(a \mid s)$, which implies
So an attained, realizable saddle pins down the soft advantage $A^\star$. Two consequences are worth stating plainly.
This is not new information. In population, $A^\star(s, a) = \log \pi^\star(a \mid s)$ is just the expert's log conditional choice probability. Hotz--Miller inversion (ref) already extracts the same object through the log-probability differences $\log \pi^\star(a \mid s) - \log \pi^\star(a_s \mid s)$. This leads to the following Remark (ref).
This is the key interpretive point. The advantage says how attractive action $a$ is relative to the baseline value of state $s$. It does not say whether that attractiveness came from high immediate reward, a favorable next-state distribution, or a potential-shaping term. Behavior may tell us that action $a$ is preferred to action $b$ at state $s$, but it does not by itself say which part is immediate payoff and which part is future continuation value. So the equality $f_{\theta,\phi}=A^\star$ is not yet reward recovery; it is only the object that reward recovery must decompose.
Exact equality may be unreachable in stochastic environments. The target $A^\star(s, a)$ does not depend on the realized next state $s'$, whereas the AIRL class $g(s) + \beta h(s') - h(s)$ generally does. So in a stochastic environment the pointwise equality (ref) can fail within the parametrized class, before any reward question is even posed. We return to this in the discussion of restriction (iii) below; until then, transitions are deterministic.
\paragraph{From advantage to reward: decomposable deterministic dynamics.} Assume now $s' = T(s, a)$ deterministic. Reward recovery still does not follow for free. If the AIRL reward head is allowed to be a state-action function, then for any bounded $h : \mathcal{S} \to \mathbb{R}$ the choice
satisfies $g_h(s, a) + \beta h(T(s, a)) - h(s) = \log \pi^\star(a \mid s)$ exactly, so infinitely many decompositions of the same saddle logit coexist. Thus writing $f=g+\beta h'-h$ does not, by itself, tell us which piece is the reward. AIRL needs extra structure that makes the split unique.
The relevant property is not that next states identify current states; it is almost the opposite; enough overlap in one-step reachability that a next-state-only term cannot hide inside a sum of a current-state term and a next-state term. Define the one-step successor and predecessor sets
In a standard MDP $\mathcal{S}_{-} = \mathcal{S}$, but $\mathcal{S}_{+} = \mathcal{S}$ only if every state has a one-step in-neighbor. Build an undirected graph on $\mathcal{S}_{+}$, placing an edge between successors $x$ and $y$ whenever they share a predecessor (some $s$ and actions $a, b$ with $P(x \mid s, a) > 0$ and $P(y \mid s, b) > 0$) and close the relation transitively. The transition kernel is decomposable on the relevant state space (the Fu--Luo--Levine condition) if this successor graph is connected and, for any claim of equality on all of $\mathcal{S}$, additionally $\mathcal{S}_{+} = \mathcal{S}_{-} = \mathcal{S}$.
The useful consequence is the following separation lemma. The domain qualification matters: the equation only sees $v$ at states that appear as successors, and only sees $u$ at states that appear as predecessors.
\paragraph{All four restrictions are load-bearing.} Proposition (ref) is sharp: each substantive restriction removes a different obstruction. Here the state-only restriction combines assumptions (i)--(ii), so the four restrictions are state-only structure, deterministic dynamics, decomposable coverage, and an attained exact saddle.
(i)--(ii) State-only rewards. If the reward head may depend on $(s, a)$, the shaping family of Lemma (ref) comes straight back. For any bounded potential $\Phi : \mathcal{S} \to \mathbb{R}$, set $h = V^\star + \Phi$ and
Then $g_\Phi(s, a) + \beta\, \mathbb{E}[h(s') \mid s, a] - h(s) = A^\star(s, a)$ identically. In deterministic dynamics this expectation is just evaluation at $T(s,a)$; in stochastic dynamics it is the population version of the same ambiguity. Either way, every potential-shaped reward produces the same soft advantage, so AIRL leaves state-action rewards unidentified beyond the baseline shaping equivalence.
(iii) Deterministic dynamics. Determinism matters because then the next state $T(s,a)$ is fixed once $(s,a)$ is fixed, so the AIRL term $h(T(s,a))$ can line up pointwise with the Bellman equation. The proof used exactly this identity: $A^\star(s, a) = r(s) + \beta V^\star(T(s, a)) - V^\star(s)$. Under stochastic transitions, the Bellman identity carries a conditional expectation, while AIRL's realized term $h(s')$ depends on one sampled next state. The most one can ask is the population moment
Even granting (ref), reward recovery is not automatic. With a state-only reward and state-only heads, the soft Bellman equation turns (ref) into $g(s) + \beta \mathbb{E}[h(s') \mid s, a] - h(s) = r(s) + \beta \mathbb{E}[V^\star(s') \mid s, a] - V^\star(s)$; with $\Delta := h - V^\star$ and $\delta := g - r$ this rearranges to
The left-hand side is independent of $a$, so $\mathbb{E}[\Delta(s') \mid s, a]$ must be independent of $a$ as well. This forces $\Delta$ to be locally, and then globally, constant only if the transition family is rich enough that the condition “all action-conditioned expectations of $\Delta$ agree” implies $\Delta$ is constant: a completeness condition on $P$. Mere dependence of $P(\cdot \mid s, a)$ on $a$ is not enough, since two next-state distributions can share the expectation of a non-constant $\Delta$. Under the completeness condition one recovers $h = V^\star + c$ and $g = r + (1 - \beta)c$ as before; without it, (ref) leaves residual shaping ambiguity.
This distinction clarifies the role of transition-aware AIRL variants. zhan2024model tries to address the stochastic-realizability obstruction: replacing AIRL's realized-next-state term \(h(s')\) by a model-based approximation to \(\mathbb{E}[h(s')\mid s,a]\), equivalently a transition-aware shaping term such as \(R(s,a)+\beta\mathbb{E}_{\hat T}[\varphi(s')\mid s,a]-\varphi(s)\), makes the logit compatible with the stochastic Bellman identity up to learned-transition error. That repair does not by itself supply the completeness or normalization condition needed for identification. After the expectation is inserted, the remaining equation is still (ref), so reward recovery still requires that the only potential differences \(\Delta\) whose action-conditioned transition expectations agree are constants.
(iv) Connectivity and coverage. If the successor graph splits into several components, the separation argument forces $\Delta$ constant only within each component, and different per-component constants give genuinely different decompositions consistent with the same saddle. States that never appear as one-step successors leave $h$ unconstrained there; states that never appear as predecessors leave $g$ unconstrained. Deterministic systems with little branching, or with disconnected recurrent classes, routinely fail the condition.
(v) Exact saddle. If the policy player has not matched the expert occupancy, or if the discriminator class cannot realize its Bayes optimum, the equality $f_{g,h}=A^\star$ need not hold on the expert support. The learned logit then reflects the current policy/discriminator game rather than the expert's soft advantage, so there is no exact advantage identity to decompose into reward plus shaping terms.
\paragraph{Relation to anchor actions.} The bottom line is that AIRL is elegant because it builds potential shaping into the discriminator, but that removes the baseline identification problem only under assumptions such as state-dependent rewards and deterministic transitions. At the perfect saddle, it recovers the expert's soft advantage; reward recovery requires additional structure that makes the advantage decomposition unique.
To summarize:
For the remainder of these notes, we go back to the anchor action assumption Assumption (ref): it identifies state-action rewards directly, on the expert support, without requiring state-only rewards, deterministic dynamics, or a decomposable transition graph.
We now survey the classical estimation routes that grew out of the Magnac--Thesmar identification: Rust's nested fixed-point algorithm (Section (ref)), the conditional-choice-probability approach of Hotz--Miller (Section (ref)), and the Adusumilli--Eckardt temporal-difference route (Section (ref)), which itself contains two methods: linear semi-gradient TD and approximate value iteration. Each route gets one of the following ingredients right, but pays a price elsewhere:
Section (ref) is then about how we can address this issue.
The starting point is common to all. By Theorem (ref), $\pi^\star$ pins down $Q^\star$ once an anchor action is fixed. Operationally, this requires solving two coupled conditions (the softmax MLE condition and the soft Bellman equation at the anchor action) that we record now.
Notice what the NLL gives us and what it does not. The softmax $\hat p_Q$ is invariant under state-only shifts $Q(s, a) \mapsto Q(s, a) + c(s)$ (Lemma (ref)). So fitting $\pi^\star$ alone pins down $Q$ only up to such shifts, exactly the shaping ambiguity of Lemma (ref). The Bellman condition is what locks down those shifts. The classical methods differ in how they enforce it.
rust1987optimal proposed to parametrize the reward $r_\theta$ in a low-dimensional class and treat the Bellman equation as an inner-loop constraint:
Convergence of the inner loop is guaranteed by Lemma (ref); consistency of the outer MLE is the classical result of rust1987optimal. This is the right thing to do, statistically; the problem is doing it.
\paragraph{The computational obstruction.} The inner soft value iteration is over $\mathcal{S} \times \mathcal{A}$. For each iteration, the LSE
must be evaluated at every $s'$ the outer expectation might query. Discretizing $\mathcal{S}$ to $|\mathcal{S}|$ points and running value iteration costs $\Theta(|\mathcal{S}| \cdot |\mathcal{A}|)$ per iteration to convergence in $\Theta(\log(1/\varepsilon)/(1-\beta))$ iterations, and the inner loop sits inside the outer loop over $\theta$. When the underlying state has intrinsic dimension $d$, fixed-grid discretization to resolution $h$ produces $|\mathcal{S}| = \Theta(h^{-d})$, so the inner cost scales as $h^{-d}$: the curse of dimensionality of dynamic programming. Workarounds (sieve approximations kristensen2021solving,arcidiacono2013approximating, recursive partitioning barzegary2022recursive, state aggregation geng2023data, neural-network approximations norets2012estimation) push the boundary but do not change the scaling.
\paragraph{The transition kernel is also needed.} There is a second problem hidden inside the first. The inner soft value iteration requires the expectation $\mathbb{E}_{s' \sim P(\cdot \mid s, a)}[\operatorname{LSE}(Q(s', \cdot))]$, which presumes either (i) closed-form access to $P$ (rare outside textbook problems), or (ii) a separately estimated $\hat P$. In option (ii), error in $\hat P$ propagates through the fixed-point computation; rust1994structural requires $P$ to be known a priori or consistently estimable in a first stage, a non-trivial demand in high dimensions. A constrained-optimization reformulation (MPEC) that avoids the nested loop in low dimensions is given by su2012constrained: treat both $\theta$ and the value function as optimization variables and minimize the likelihood subject to the Bellman equation as a constraint. MPEC still requires $P$.
hotz1993conditional attack the inner-loop bottleneck differently. Their starting point is the observation underpinning (ref): the differences of $Q^\star$ across actions at the same state are already pinned down by the observed log choice-probability ratios. Identity (ref) rearranges to
and the linear fixed-point equation (ref) that follows is a Bellman-like equation for $V^\star$ alone, with reward $-\log \pi^\star(a_s \mid \cdot)$. Once $V^\star$ is in hand, $Q^\star(s, a)$ follows from (ref) and (ref), and $r$ follows from the soft Bellman equation (ref).
\paragraph{Forward simulation.} The expectation in (ref) can be approximated by forward Monte Carlo under the anchor-conditioned kernel: simulate trajectories of length $T$ starting at $s$, set the action at each simulated state $s_t$ to the anchor action $a_{s_t}$, draw $s_{t+1}$ from $\hat P(\cdot \mid s_t,a_{s_t})$, and accumulate the per-step pseudo-reward $-\log \hat\pi^\star(a_{s_t} \mid s_t)$. Drawing actions from $\hat\pi^\star$ would solve a different Bellman equation; the anchor equation (ref) specifically uses $P(\cdot\mid s,a_s)$. This sidesteps the high-dimensional fixed-point computation of NFXP entirely. Recursive variants of CCP aguirregabiria2002swapping,aguirregabiria2007sequential interleave $\pi^\star$ updates and value iteration.
\paragraph{The statistical obstruction.} We bought the computational savings of CCP at a price, and it is a steep one. Forward simulation requires estimating $\hat P$ in high dimensions, then propagating its error over $T = \Theta(1/(1-\beta))$ steps. With a continuous or high-dimensional state space, $\hat P$ is itself subject to a curse of dimensionality: a nonparametric kernel density estimator of $P(s' \mid s, a)$ has rate $N^{-c/d}$ for some smoothness-dependent $c$. The statistical complexity blows up exactly where NFXP's computational complexity did.
\paragraph{Recovery of $r$.} Once $V^\star$ and $Q^\star$ are recovered, the reward is read off (ref). This expression requires another $P$-expectation; in continuous states it is again subject to transition-estimation error.
\paragraph{The summary.} NFXP and CCP are duals. NFXP fits $r$ in an outer loop and solves the Bellman fixed point in the inner. CCP fits $\pi^\star$ first and exploits Hotz--Miller inversion to avoid the inner fixed point but needs $P$ everywhere. Both require either knowing or estimating $P$, and both impose either computational ($|\mathcal{S}| \cdot |\mathcal{A}|$ DP table) or statistical ($\hat P$ in $d$ dimensions) burdens scaling exponentially in $d$. Neither produces a single differentiable objective whose minimizer is $Q^\star$.
The natural next thought is: can we avoid the transition kernel by using temporal-difference updates based directly on observed transitions? adusumilli2019temporal propose exactly this. More precisely, they propose two transition-density-free TD-style approaches for the recursive terms that enter CCP estimation:
In the full DDC pseudo-likelihood these recursive terms are the continuation-value objects induced by Hotz--Miller inversion. In our anchor-action notation they collapse to the auxiliary value function in (ref). Thus the same analysis covers both Adusumilli--Eckardt methods: one is the semi-gradient form of TD, and the other is fitted/approximate value iteration. The only twist is that we are now bootstrapping the auxiliary value function (ref), not the original value function of an MDP.
Fix a feature map $\phi : \mathcal{S} \to \mathbb{R}^d$ and let $\Phi$ denote the $|\mathcal{S}| \times d$ feature matrix (in the finite-state case; the general case is analogous in $L^2$). Parametrize $V_\theta(s) := \phi(s)^\top \theta$, $\theta \in \mathbb{R}^d$. The hypothesis class is the linear span $\mathcal{F} := \{V : V(s) = \phi(s)^\top \theta,\;\theta \in \mathbb{R}^d\}$, a closed subspace of bounded functions on $\mathcal{S}$.
Fix a distribution $\rho \in \Delta(\mathcal{S})$ (the data distribution) with associated $L^2(\rho)$-inner product $\langle f, g \rangle_\rho := \mathbb{E}_{s \sim \rho}[f(s) g(s)]$ and norm $\|f\|_\rho^2 := \langle f, f\rangle_\rho$. The $L^2(\rho)$-projection onto $\mathcal{F}$ is
which is linear in $V$ and a non-expansion in $\|\cdot\|_\rho$.
Let $\mathcal{T}^{\pi_b}$ denote the (linear) Bellman expectation operator under a behavior policy $\pi_b$ with corresponding kernel $P^{\pi_b}(s' \mid s) = \sum_a \pi_b(a \mid s) P(s' \mid s, a)$:
For the TD-DDC application below, $\pi_b$ will be the anchor-conditioned chain $\pi_b(a \mid s) = \mathbf{1}[a = a_s]$ and the per-state “reward” will be $-\log \pi^\star(a_s \mid s)$.
The unique fixed point $V_{\mathcal{F}, \rho}^\star$ of $\Pi_{\mathcal{F}, \rho}\mathcal{T}^{\pi_b}$ is the projected Bellman fixed point. It solves the projected Bellman equation $V_{\mathcal{F}, \rho} = \Pi_{\mathcal{F}, \rho}\mathcal{T}^{\pi_b} V_{\mathcal{F}, \rho}$; in matrix form, $\theta^\star$ solves $A\theta = b$ with
When $\rho$ is the stationary distribution of $P^{\pi_b}$, tsitsiklis1996analysis show that $A$ is positive definite. Linear TD$(0)$, whose update direction has expectation $b - A\theta$, then converges to $A^{-1} b$ with rate determined by the smallest eigenvalue of $A$.
The on-policy stability of Lemma (ref) relies critically on $\rho$ being the stationary distribution of $P^{\pi_b}$. The combined presence of (i) linear (or more general) function approximation, (ii) bootstrapping, and (iii) off-policy data is known as the deadly triad sutton2018reinforcement,van2018deep. Here is its TD-DDC face.
This is not a pathological construction. wang2021instabilities document the same phenomenon with neural-network function approximation, where it persists even with mild representation pre-training. And critically for us: the deadly triad applies in particular to the DDC application below. The offline DDC dataset is generated by the expert acting according to $\pi^\star$ under all actions, so its state marginal is $d^{\pi^\star}_S$. The TD-DDC update, however, bootstraps along the anchor-conditioned chain whose kernel is $P(\cdot \mid \cdot, a_s)$, a deterministic “policy” that always picks the state-specific anchor. After restricting to anchor transitions, the effective sampling distribution is $\rho_A(s) \propto \pi^\star(a_s \mid s)\, d^{\pi^\star}_S(s)$, which is in general not the stationary distribution of $P(\cdot \mid \cdot, a_s)$. The on-policy contraction of Lemma (ref) therefore does not apply to this update, and Example (ref)'s pathology is on the table.
The second Adusumilli--Eckardt method is not the same as the linear semi-gradient recursion above. It is Approximate Value Iteration: freeze the current approximation inside the continuation term, then solve a new regression problem for the next approximation. In the linear case this takes the form
and iterate. The inner problem is a well-conditioned least-squares problem whenever $\Phi^\top D_\rho \Phi \succ 0$; its closed form is
Viewed as a map $\theta^- \mapsto \theta^+$, this is exactly fitted value iteration (FVI): $V_{\theta^+} = \Pi_{\mathcal{F}, \rho} \mathcal{T}^{\pi_b} V_{\theta^-}$. In a nonparametric or sieve implementation, the same map is computed by replacing the linear least-squares problem with the corresponding regression learner over the chosen function class. Thus, in these notes, Adusumilli--Eckardt's AVI method is the DDC specialization of the standard fitted-value-iteration operator.
adusumilli2019temporal apply this machinery in the DDC setting with the substitutions of Section (ref): $r^{\pi_b}(s) := -\log \pi^\star(a_s \mid s)$ and $\pi_b(a \mid s) := \mathbf{1}[a = a_s]$. Let
be the set of observed transitions in which the expert took the anchor action, and write $\mathcal{D}_A := \{(s_i, s_i') : i \in I_A\}$ for the corresponding sub-dataset with state marginal $\rho_A$.
\paragraph{Method 1: linear semi-gradient TD.} The first method updates the coefficient vector using the current parameter on both sides of the TD error:
sampled from transitions $(s_t, a_t = a_{s_t}, s_{t+1})$ in $\mathcal{D}$.
\paragraph{Method 2: approximate value iteration.} The second method freezes the previous value approximation. Starting from $V_0 \in \mathcal{F}$, define pseudo-outcomes for $i \in I_A$ by
and compute
When $\mathcal{F}$ is linear, $V_k(s)=\phi(s)^\top\theta_k$ and (ref) is exactly (ref); when $\mathcal{F}$ is a sieve or nonparametric class, (ref) is the nonparametric AVI regression. Either way, the population operator is $V_{k+1}=\Pi_{\mathcal{F},\rho_A}\mathcal{T}^{\pi_b}V_k$.
\paragraph{Replaying the analysis in the DDC setting.} The two methods have different failure modes.
\paragraph{The honest scorecard.} Both Adusumilli--Eckardt methods avoid the explicit transition kernel that CCP/Hotz--Miller needed: they use observed $(s_t, s_{t+1})$ pairs in place of an explicit $\mathbb{E}_{P(\cdot \mid s, a)}$. That is a genuine improvement over forward simulation. But the price is not the same for the two methods. Tallying:
None produces a single unbiased objective whose global minimum is the true $Q^\star$, computable with mild conditioning assumptions on the parametrization and without specifying $P$.
A reader familiar with deep RL might wonder whether the linear semi-gradient instability of Method 1 can be cured by introducing a target network, exactly as in the forward-RL setting. The answer is the same here as there, for the same reason: target networks change which method you are running, not where it converges. We make this precise, then explain why it leaves the underlying issue intact.
\paragraph{What the proposition says, and what it doesn't.} Target stabilization moves Method 1 onto Method 2's orbit; it does not change Method 2's destination. The projected-fixed-point obstruction of Corollary (ref) is unaffected: even with perfect target stabilization and idealized inner optimization, the limit (when it exists) is the projected fixed point $V_{\mathcal{F}, \rho_A}^\star$. That limit equals the true auxiliary $V^\star$ only under additional conditions such as realizability of $V^\star$ in $\mathcal{F}$ and uniqueness/stability of the projected fixed point. These conditions are not innocuous for neural function classes in continuous-state, off-policy settings; they are exactly the kind of structural assumptions the deadly-triad analysis (Section (ref)) warns us we cannot expect for free.
In particular, three caveats are worth recording:
The methodological takeaway is the following:
This rules out the most familiar deep-RL fix and points to the same conclusion: the right escape route in offline IRL/DDC is to abandon bootstrapping entirely in favor of direct (bias-corrected) Bellman-residual minimization. That is the gap Section (ref) closes.
We now turn to the ML/IRL literature. Four families dominate it: adversarial IRL, occupancy matching, IQ-Learn, and offline maximum-likelihood IRL. For each we (a) write out the actual training objective, (b) state precisely what it identifies and what it does not, and (c) compare to the anchor-action route. A general survey of imitation learning is given by zare2024survey.
A preview of where the section lands: each modern method makes a different bet against the same identification ambiguity (Lemma (ref)), and each pays for the bet somewhere: in the form of reward identified, in stability, in compatibility with offline data, or in what the method actually optimizes.
Section (ref) introduced AIRL's language and formulation: the potential-shaped discriminator (ref) and the alternating saddle update (ref). We return to AIRL here as a modern IRL method, focusing on what that formulation delivers computationally and what it still cannot identify.
\paragraph{What the saddle delivers.} The textbook saddle calculation needs two idealizations made explicit. First, the policy player must have converged to the expert, so $d^\pi = d^{\pi^\star}$ on the relevant support. Second, the AIRL discriminator class must be able to realize its Bayes optimum. Under both, $D^\star \equiv \tfrac12$ on the matched support, and since (ref) gives $D = \tfrac12$ exactly when $\exp f_{\theta, \phi} = \pi(a \mid s)$, the saddle pins down
As emphasized in Section (ref), this is an advantage identity, not yet a reward identity, and it carries no information beyond the expert's conditional choice probabilities, the same object Hotz--Miller inversion (ref) consumes through log-probability differences. The discriminator's real burden is therefore not to discover the advantage, but to split it into a reward part and a shaping part. In stochastic environments even the equality above is delicate: the right-hand side $A^\star(s, a)$ is independent of $s'$ while the AIRL class $g(s) + \beta h(s') - h(s)$ is not, so exact $D \equiv \tfrac12$ within the parametrized class may be unattainable. Section (ref) treats this carefully through the population moment (ref).
\paragraph{Identification limitations.} The self-contained analysis of Section (ref) (Proposition (ref)) shows precisely what extra structure makes reward recovery valid, and what goes wrong otherwise. The limitations are structural rather than merely computational:
The anchor-action route gives strictly more for these notes: it identifies state-action $r$ on the expert support without requiring the reward to be state-only and without relying on the AIRL decomposition (cf.\ Theorem (ref) and Theorem (ref)).
\paragraph{Optimization limitations.} On top of this identification ceiling, AIRL inherits the well-documented instabilities of GAN training: the alternating minimax updates on $(\theta, \phi)$ and $\pi$ can fail to converge, exhibit mode collapse, or settle at a non-optimal saddle. And AIRL is fundamentally an online method: the discriminator update (ref) requires sampling from the current policy $\pi$, which is exactly what offline IRL forbids.
ho2016generative introduced Generative Adversarial Imitation Learning (GAIL). The starting point is the occupancy-matching objective
where $D_{\mathrm{JS}}$ is the Jensen--Shannon divergence. The variational identity behind GAIL is the following. For any two distributions $p, q$ on a common space $\mathcal{X}$, the GAN value at the inner $D$-optimum is
with optimum attained at $D^\star = p / (p + q)$ goodfellow2020generative. The derivation is direct: writing the integrand at a point as $V(D) := p \log D + q \log(1 - D)$ and setting $\partial V/\partial D = 0$ gives $D^\star = p/(p+q)$, and substituting back recovers $-\log 4 + 2D_{\mathrm{JS}}(p \| q)$ via $D_{\mathrm{JS}}(p\|q) = \tfrac{1}{2}\operatorname{KL}(p \| (p+q)/2) + \tfrac{1}{2}\operatorname{KL}(q \| (p+q)/2)$. So minimizing the JS over $\pi$ is equivalent (up to additive constants) to the saddle
GAIL solves (ref) with a GAN-style optimizer over a parametrized $D_\phi$ and $\pi_\theta$, using on-policy policy-gradient methods such as PPO schulman2017proximal,raffin2021stable for the $\pi_\theta$ update. Related state-marginal-matching variants restrict $D$ to depend only on $s$ ni2021f.
\paragraph{What GAIL is and isn't.} GAIL is fundamentally an imitation algorithm: its goal is to make $d^{\pi_\theta}$ close to $d^{\pi^\star}$ in JS divergence, and it succeeds at this on many tasks. But the crucial point is that it is not a reward-recovery algorithm. At the saddle, the discriminator satisfies
which is a log Radon--Nikodym derivative of expert vs.\ current-policy occupancies, not a reward. There is no way to extract $r$ from $D^\star$ without further identification assumptions of exactly the kind we already discussed (Theorem (ref) and the anchor-action assumption).
\paragraph{Bellman consistency is not built in.} The limitation can be seen directly from a one-line Bellman-flow identity. Fix any policy $\pi$, reward $r$, and candidate state-action value function $Q$. Define the policy Bellman operator
Using the normalized discounted occupancy
we have
The first line is just telescoping: the second term equals $(1-\beta)\sum_{t\ge 0}\beta^{t+1}\mathbb{E}[Q(s_{t+1},a_{t+1})]$, so subtracting it from $(1-\beta)\sum_{t\ge 0}\beta^t\mathbb{E}[Q(s_t,a_t)]$ leaves only the $t=0$ term. The second line subtracts the reward.
Equation (ref) is a single weighted average of Bellman residuals under the occupancy $d^\pi$. Matching $d^\pi$ to $d^{\pi^\star}$ can at most match such occupancy-weighted averages. It does not imply the pointwise Bellman equation
Residuals can be positive on one part of the support and negative on another while their $d^\pi$-average remains zero. Thus occupancy matching places a distributional constraint on where the policy goes; it does not place an equation-of-motion constraint on a candidate $Q$.
This also explains why a reward cannot be recovered from GAIL by first extracting some $Q$ and then applying the Bellman rearrangement $r=Q-\beta\mathbb{E}[V_Q(s')\mid s,a]$. At a perfect GAIL match the discriminator itself is uninformative ($D^\star=1/2$), and any separately fitted $Q$ is unconstrained by the pointwise Bellman residual unless that residual is explicitly included in the objective. The anchor-action and ERM routes of Section (ref) and Section (ref) do exactly that: they enforce the Bellman equation at the anchor action rather than hoping it follows from occupancy matching.
\paragraph{Offline limitations.} GAIL's policy update is on-policy: it requires sampling from $\pi_\theta$ to compute $\mathbb{E}_{d^{\pi_\theta}}[\log(1 - D)]$. An offline variant, ValueDice kostrikov2019imitation, re-derives the objective using off-policy distribution-ratio estimation via the Fenchel dual of the $f$-divergence: instead of sampling from $\pi_\theta$, it estimates the ratio $d^{\pi_\theta}/d^{\pi^\star}$ from offline data and substitutes it into the GAIL objective. The empirical performance of these off-policy corrections is mixed; li2022rethinkingvaluedice show that improvements over a strong behavior-cloning baseline can be entirely attributed to architectural and regularization differences, not the off-policy correction itself. The deeper issue is the curse of horizon all over again: trajectory importance sampling for $T$-step corrections has variance $\Theta((\rho_{\max})^T)$ where $\rho_{\max}$ is the per-step importance ratio bound and $T \sim 1/(1-\beta)$ is the effective horizon, exponentially bad in the horizon for any non-trivial policy shift, as standard importance-sampling calculations show.
garg2021iq propose IQ-Learn (Inverse soft-Q Learning), which sidesteps both forward simulation and adversarial training by changing variables from $(r, \pi)$ to $(Q, \pi)$. The key identity is the soft Bellman equation (ref), rearranged:
Given any $Q$, equation (ref) defines a reward $r_Q$ for which $Q$ satisfies the soft Bellman equation by construction. IQ-Learn parametrizes $Q_\theta$ directly and maximizes
where $\psi$ is a strongly convex regularizer (the chi-squared $\psi(x) = x^2/4$ in the simplest version) and $\lambda > 0$. To see why the first two terms behave as an entropy-regularized return gap, use the Bellman-flow identity: for any function $V$ and any policy $\pi$,
which is the discounted-occupancy form of the telescoping sum $\sum_t \beta^t(V(s_t) - \beta V(s_{t+1}))$. Applying (ref) with $V = V_Q$ and $\pi = \pi^\star$, and using the definition of $r_Q$,
The RHS is exactly $\mathbb{E}_{(s, a) \sim d^{\pi^\star}}[\log \hat p_Q(a \mid s)] = -\mathbb{E}_{s \sim d^{\pi^\star}_S}\!\bigl[\operatorname{KL}(\pi^\star(\cdot|s)\,\|\,\hat p_Q(\cdot|s))\bigr] - \mathbb{E}_{s \sim d^{\pi^\star}_S}[\operatorname{H}(\pi^\star(\cdot|s))]$ by the same decomposition as in Lemma (ref). So maximizing the first two terms of $\mathcal{J}$ in $Q$ is exactly minimizing the KL between the softmax of $Q$ and the expert policy, i.e., it fits $Q$ to a value function whose softmax matches $\pi^\star$, hence to the soft Bellman equation of $r_Q$ by construction. The chi-squared term keeps $r_Q$ from blowing up.
\paragraph{What IQ-Learn does and does not pin down.} The first two terms of (ref) do not fix the state-only shaping ambiguity. Let $c:\mathcal{S}\to\mathbb{R}$ be bounded and define $Q_c(s,a):=Q(s,a)+c(s)$. Then
The change in the reward term is, by the Bellman-flow identity (ref),
The initial-value term changes by exactly the same amount with the opposite sign:
Therefore the first two terms of IQ-Learn are invariant under every bounded state-only shift $c$, not merely under a scalar normalization. The regularizer $-\lambda\mathbb{E}[\psi(r_Q)]$ is the only part of (ref) that selects a representative within the potential-shaped equivalence class, and that selection is a penalty preference rather than statistical identification of the ground-truth reward. Thus IQ-Learn can fit a softmax policy and define a Bellman-consistent reward by construction, but without an additional normalization such as the anchor action it does not identify a unique state-action reward. This is materially weaker than the anchor-action identification of Theorem (ref), which fixes the shaping function rather than choosing a regularizer-preferred representative.
zeng2023understanding propose offline ML-IRL. Stripped of vocabulary, the method is essentially a parametric, neural re-skinning of Hotz--Miller / CCP (Section (ref)) with two modifications: (a) the closed-form auxiliary inversion of Section (ref) is replaced by an iterated forward-soft-RL inner loop with an explicit reward parametrization $r_\theta$, driven by an outer softmax-MLE objective; and (b) a conservatism penalty, analogous to Conservative Q-Learning (CQL) of kumar2020conservative, is added inside the soft-RL inner loop to handle distribution shift. Related conservative offline IRL methods appear in yue2023clare.
It is worth making the structural parallel to Hotz--Miller explicit, because it is the right way to understand what the method does and does not buy. Both methods (i) estimate $\hat P$ from $\mathcal{D}$, and (ii) alternate between a forward step that produces a value function from $\hat P$ together with the current reward (or current value) and a fitting step that updates the reward (or the choice probabilities). CCP performs the forward step in closed form by solving the auxiliary linear fixed-point equation (ref) and reading off $r$ from (ref); Zeng et al.\ instead solve a parametric soft-RL problem at each outer step. The substrate is the same; only the parametrization and the objective differ.
The procedure is a doubly-nested alternation. At outer iteration $k$ with current reward parameter $\theta_k$:
The conservatism penalty is the one substantive new ingredient, and it is genuinely useful: it handles the unvisited states that vanilla CCP could not, by discouraging the inner-loop value from extrapolating wildly outside the support of $\mathcal{D}$. Everything else inherits CCP's structure, as well as CCP's limitations:
zeng2023understanding prove finite-time guarantees of the form $\|\widehat r - r\|_2^2 \le \mathcal{O}(N^{-1/2}) + \text{(model error)}$ under linear-MDP assumptions and bounded transition-model error, but the analysis does not extend directly to neural function approximation without further structure.
The honest takeaway is that offline ML-IRL inherits Hotz--Miller's strengths and weaknesses essentially unchanged; conservatism is orthogonal to the underlying estimator and could in principle be bolted onto any method in this section, including the ERM framework of Section (ref).
\paragraph{Where modern IRL leaves us.} None of the methods reviewed in this section gives all of the following at once:
The next section gives one route through this gap under explicit anchor, empirical Jacobian-conditioning/PL, realizability, and stability assumptions.
We finally reach the ERM framework of kang2025empirical. The easiest way to understand the method is not to start with the minimax objective. Instead, start from the two restrictions that have appeared throughout the notes:
The ERM estimator is simply the one-shot version of these two restrictions. It parametrizes $Q$ directly, rather than parametrizing $r$ and repeatedly solving the forward Bellman equation. This is the step that removes the nested $r\mapsto Q$ optimization that classical DDC methods require.
Throughout this section, $a_s$ denotes the anchor action at state $s$, and \[ r_A(s):=r(s,a_s) \] denotes its known reward. The normalized presentation often sets $r_A(s)=0$ for all $s$; the formulation below allows any known anchor payoff, which is the notation used in the paper.
The build-up is as follows. Section (ref) rewrites the classical identification equations as losses. Section (ref) combines them into the population expected risk. Section (ref) explains why the empirical problem cannot be obtained by naively replacing the Bellman expectation by one next-state sample. Section (ref) introduces the conditional-mean correction that removes that bias. Sections (ref) and (ref) state the resulting GLADIUS algorithm and its conditional convergence guarantees.
Recall the two soft optimality equations. If \[ V_Q(s):=\operatorname{LSE}(Q(s,\cdot))=\log\sum_{b\in\mathcal{A}}\exp(Q(s,b)), \qquad \hat p_Q(a\mid s):={\exp(Q(s,a))\over\sum_{b\in\mathcal{A}}\exp(Q(s,b))}, \] then the expert policy and the optimal $Q$ satisfy
The first equation is what likelihood-based estimation uses. The second equation is what dynamic programming, CCP, and Bellman-residual methods try to enforce.
Under the anchor-action assumption, the key Magnac--Thesmar observation is that we do not need the Bellman equation for every action in order to identify $Q^\star$. The likelihood condition (ref) identifies all within-state differences $Q^\star(s,a)-Q^\star(s,b)$. It is only missing one scalar level per state. The anchor-action Bellman equation supplies exactly that missing scalar:
This is the bridge from classical DDC to ERM. Rust's nested fixed-point method enforces the full Bellman equation after parametrizing $r$; Magnac--Thesmar identification says that, once the anchor is fixed, the likelihood plus the anchor-action Bellman equation is enough. Therefore, if we parametrize $Q$ directly, the two identifying restrictions can be expressed as two loss-minimization problems.
The first loss is the negative log-likelihood,
By Lemma (ref), its population minimizers are precisely the $Q$ functions whose softmax equals the expert policy on the expert-covered states.
The second loss is the anchor Bellman-error loss. For any candidate $Q$, define the anchor residual
Then $Q$ satisfies the anchor Bellman equation exactly when $\Delta_A(Q)(s)=0$. Because the expert softmax puts positive probability on every action at each covered state, this is equivalently the minimizer set of
This is the main conceptual simplification: the Bellman part contains no unknown reward term, because it is evaluated only at the anchor action where $r_A(s)$ is known.
Combining the two losses gives the population's expected risk
This is the population ERM-DDC/IRL objective. It has the flow that the classical methods suggest, but without the classical nesting:
The NLL term has no unknown reward in it. The anchor Bellman term also has no unknown reward in it. Thus, at the population level, the inverse problem has become a direct optimization over $Q$.
The expected risk (ref) is still a population object. The hard term is \[ \mathbb{E}[V_Q(s')\mid s,a_s], \] which is an expectation over the transition kernel. If the transition kernel were known, one could evaluate this term directly. In the offline setting, however, we only observe samples $(s,a,s')$.
The first tempting move is to replace the conditional expectation by the one observed next state. For a general state-action pair, define the sampled Bellman operator
and define the sampled TD residual
At anchor actions, $r(s,a)$ is the known quantity $r_A(s)$. The sampled operator is unbiased for the Bellman operator: \[ \mathbb{E}[\widehat{\mathcal{T}}Q(s,a,s')\mid s,a]=(\mathcal{T} Q)(s,a). \] But the squared TD loss is not unbiased for the squared Bellman error. The variance decomposition gives
The second term is not a harmless constant: it depends on $Q$. Therefore, minimizing squared TD error generally does not minimize squared Bellman error. This is the double-sampling problem in this setting. It disappears under deterministic transitions, but not under stochastic transitions.
The correction is to subtract the conditional variance term in (ref) without estimating the transition kernel. The variance has the least-squares representation
Promote the scalar minimizer to a function $\zeta(s,a)$. Combining (ref) and (ref) yields the corrected Bellman-error identity
The maximization appears because subtracting the best least-squares error is the same as maximizing the negative squared error.
Plugging the corrected identity into the anchor-weighted expected risk gives the population minimax version of ERM-DDC/IRL:
The anchor indicator is important. The Bellman-identification term is imposed at the anchor action because that is where the reward is known. The auxiliary function $\zeta$, however, has an additional role: at the solution, \[ \zeta^\star_{Q^\star}(s,a)=\mathbb{E}[V_{Q^\star}(s')\mid s,a], \] so it is exactly the continuation-value object needed to recover rewards for all supported state-action pairs:
For this reason, the paper estimates $\zeta$ over observed state-action pairs, not only at anchor observations. The anchor-weighted corrected Bellman term identifies $Q^\star$; the all-pair conditional-mean regression makes reward recovery transition-estimation-free.
Replacing the population expectations in (ref) and (ref) by empirical averages gives the finite-sample objective. For parametrizations $Q_\theta$ and $\zeta_\phi$, define the auxiliary regression loss
For fixed $Q_\theta$, descending this loss estimates the conditional mean $\mathbb{E}[V_{Q_\theta}(s')\mid s,a]$ on the observed support. Given a current auxiliary function, the empirical $Q$ objective is
Equivalently, one may view the profiled empirical risk as \[ \widehat g_N(\theta):=\widehat f_N(\theta,\widehat\phi(\theta)), \qquad \widehat\phi(\theta)\in\operatorname*{arg\,min}_\phi \widehat F_\zeta(\phi;\theta). \] This is the empirical-risk version of the expected-risk problem above.
GLADIUS solves the empirical problem by alternating the two natural gradient steps. The $\zeta$ step is a regression step: fit $\zeta_\phi(s,a)$ to $V_{Q_\theta}(s')$. The $Q$ step is a descent step on the NLL plus the anchor-weighted corrected Bellman term.
The language “ascent--descent” comes from the minimax display (ref): maximizing the negative quadratic correction in $\zeta$ is equivalent to descending the positive regression loss (ref). The practical algorithm uses the regression form because it is clearer and numerically standard.
The objective in (ref) is generally nonconvex in $\theta$. The paper's convergence argument therefore relies on Polyak--{\L}ojasiewicz geometry, not convexity. A differentiable function $f$ satisfies a PL inequality with constant $c>0$ on a set $U$ if
PL is weaker than convexity but still rules out spurious stationary points on the relevant set.
For the minimax objective, the relevant geometry is two-sided. The profiled $Q$ objective must satisfy PL in $\theta$, while the auxiliary regression problem must satisfy the corresponding PL condition in $\phi$ for the ascent side (equivalently, the regression loss is PL for descent).
Before stating the conditioning assumption, it is useful to say what kind of condition it is. We are not assuming directly that the nonconvex parameter-space objective is PL. Instead, the argument first proves PL-type inequalities in the finite empirical output coordinates, and then transfers them to the parameters through the Jacobians of the output maps \[ \theta\mapsto (Q_\theta(z))_{z\in Z_Q}, \qquad \phi\mapsto (\zeta_\phi(\bar z))_{\bar z\in Z_\zeta}. \] This transfer condition is checkable for common parametrizations. For linear models it is exactly a full-row-rank empirical feature condition. For sufficiently wide smooth neural networks, the same type of lower bound is obtained in the usual NTK/lazy-training regime on any fixed finite empirical evaluation set. We state the assumption first and then record these two verification cases as lemmas.
The main takeaway is now straightforward. ERM-DDC/IRL is a transition-estimation-free minimax estimator for $Q^\star$. It is built by translating the two classical identifying equations into losses, correcting the empirical Bellman term for double-sampling bias, and recovering rewards from the same learned conditional-mean correction $\widehat\zeta$.
The classical DDC literature gave us identification through the anchor-action argument, plus computational templates such as NFXP and CCP. The modern IRL literature gave us scalable objectives, but often without pointwise Bellman identification of the reward. The ERM framework of kang2025empirical combines the useful pieces from both sides: use the likelihood term for within-state action-value differences, use the anchor Bellman equation for the state-wise level, use the bias-corrected TD identity to avoid transition-kernel estimation, and solve the resulting objective by alternating gradients. This is why the section should be read as a build-up from classical DDC to a one-shot gradient-based estimator, rather than as an isolated minimax formula.