Siliang Zeng, Mingyi Hong, Alfredo Garcia
arXiv 4 Oct 2022 · Machine Learning · publishedOperations Research (2024) · 3 citations (OpenAlex)
arXiv:2210.01282 · PDF · DOI · OpenAlex · Extracted main text
We consider the task of estimating a structural model of dynamic decisions by a human agent based upon the observable history of implemented actions and visited states. This problem has an inherent nested structure: in the inner problem, an optimal policy for a given reward function is identified while in the outer problem, a measure of fit is maximized. Several approaches have been proposed to alleviate the computational burden of this nested-loop structure, but these methods still suffer from high complexity when the state space is either discrete with large cardinality or continuous in high dimensions. Other approaches in the inverse reinforcement learning (IRL) literature emphasize policy estimation at the expense of reduced reward estimation accuracy. In this paper we propose a single-loop estimation algorithm with finite time guarantees that is equipped to deal with high-dimensional state spaces without compromising reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm converges to a stationary solution with a finite-time guarantee. Further, if the reward is parameterized linearly, we show that the algorithm approximates the maximum likelihood estimator sublinearly. Finally, by using robotics control problems in MuJoCo and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.
appendix boundary found by appendix_titled_section at “Appendix” · 48% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | T. Ni, H. Sikchi, Y. Wang, T. Gupta, L. Lee, and B. Eysenbach, “f-ir… (2020) f-irl: Inverse reinforcement learning via state marginal matching | 1.000 | 7 | 5 | 100% |
| 2 | D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon, “Iq-learn:… (2021) Iq-learn: Inverse soft-q learning for imitation | 1.000 | 7 | 4 | 100% |
| 3 | V. J. Hotz and R. A. Miller, “Conditional choice probabilities and t… (1993) Conditional choice probabilities and the estimation of dynamic models | 1.000 | 6 | 4 | 100% |
| 4 | V. J. Hotz, R. A. Miller, S. Sanders, and J. Smith, “A simulation es… (1994) A simulation estimator for dynamic models of discrete choice | 0.928 | 4 | 3 | 100% |
| 5 | J. Rust, “Optimal replacement of GMC bus engines: An empirical model… (1987) Optimal replacement of GMC bus engines: An empirical model of Harold Zurcher | 0.928 | 4 | 3 | 100% |
| 6 | T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic:… (2018) Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor | 0.928 | 4 | 3 | 100% |
| 7 | S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi, “Fast global converge… (2021) Fast global convergence of natural policy gradient methods with entropy regularization | 0.894 | 7 | 5 | 71% |
| 8 | J. Rust, “Structural estimation of Markov decision processes,” Handb… (1994) Structural estimation of Markov decision processes | 0.811 | 4 | 2 | 100% |
| 9 | B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey et al., “Maximum… (2008) Maximum entropy inverse reinforcement learning | 0.737 | 4 | 2 | 75% |
| 10 | T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learn… (2017) Reinforcement learning with deep energy-based policies | 0.737 | 3 | 3 | 67% |
Showing the top 10 of 50 scored citations.