arXiv 18 Aug 2020 · Mathematics — Optimization · 2 citations (OpenAlex)
arXiv:2008.07820 · PDF · DOI · OpenAlex · Extracted main text
We study the relation between different Markov Decision Process (MDP) frameworks in the machine learning and econometrics literatures, including the standard MDP, the entropy and general regularized MDP, and stochastic MDP, where the latter is based on the assumption that the reward function is stochastic and follows a given distribution. We show that the entropy-regularized MDP is equivalent to a stochastic MDP model, and is strictly subsumed by the general regularized MDP. Moreover, we propose a distributional stochastic MDP framework by assuming that the distribution of the reward function is ambiguous. We further show that the distributional stochastic MDP is equivalent to the regularized MDP, in the sense that they always yield the same optimal policies. We also provide a connection between stochastic/regularized MDP and constrained MDP. Our work gives a unified view on several important MDP frameworks, which would lead new ways to interpret the (entropy/general) regularized MDP frameworks through the lens of stochastic rewards and vice-versa. Given the recent popularity of regularized MDP in (deep) reinforcement learning, our work brings new understandings of how such algorithmic schemes work and suggest ideas to develop new ones.
appendix boundary found by appendix_command · 50% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Brian D Ziebart, J Andrew Bagnell, and Anind K Dey (2010) Modeling interaction via the principle of maximum causal entropy | 1.000 | 7 | 4 | 100% |
| 2 | Rust John (1988) Maximum likelihood estimation of discrete control processes | 1.000 | 6 | 3 | 100% |
| 3 | Matthieu Geist, Bruno Scherrer, and Olivier Pietquin (2019) A theory of regularized markov decision processes | 0.928 | 5 | 3 | 80% |
| 4 | John Rust (1987) Optimal replacement of gmc bus engines: An empirical model of harold zurcher | 0.928 | 5 | 3 | 80% |
| 5 | Karthik Natarajan, Miao Song, and Chung-Piaw Teo (2009) Persistency model and its applications in choice modeling | 0.874 | 5 | 2 | 100% |
| 6 | Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos… (1806) Maximum a posteriori policy optimisation | 0.843 | 3 | 3 | 100% |
| 7 | Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine (2017) Reinforcement learning with deep energy-based policies | 0.737 | 3 | 3 | 67% |
| 8 | Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven B… (1812) Relative entropy regularized policy iteration | 0.644 | 2 | 2 | 100% |
| 9 | Eitan Altman (1999) Constrained Markov decision processes, volume 7 | 0.644 | 2 | 2 | 100% |
| 10 | Mohammad Gheshlaghi Azar, Vicenc Gómez, and Hilbert J Kappen (2012) Dynamic policy programming | 0.644 | 2 | 2 | 100% |
Showing the top 10 of 32 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | Constrained Recursive Logit for Route Choice Analysis | 0.644 | 2 | 2 |