Paper Collection of Hierarchical Reinforcement Learning

This is a collection of research and review papers of hierarchicial reinforcement learning (HRL). Several multi-goal reinforcement learning research papers are also listed here due to high correlation. All the papers are sorted by time. Any suggestions and pull requests are more than welcome.

The sharing principle of these references here is for research. If any authors do not want their paper to be listed here, please feel free to contact Jiarui Jin (Email: jinjiarui97 [AT] gmail.com).

Review Papers

• A Survey on Intrinsic Motivation in Reinforcement Learning by Arthur Aubret, Laetitia Matignon, Salima Hassas. arXiv, 2019.
• Temporal abstraction in reinforcement learning by Doina Precup, Richard S. Sutton, University of Massachusetts Amherst, 2000.
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning by Richard S. Sutton, Doina Precup, Satinder Singh, Artificial Intelligence, 1999.
• Recent Advances in Hierarchical Reinforcement Learning by Andrew G. Barto, Sridhar Mahadevan, Discrete Event Dynamic Systems: Theory and Applications, 1999.

Research Papers

Goal-Oriented RL

• Planning with Goal-Conditioned Policies by Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, Sergey Levine. NeurIPS, 2019.
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning by Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine. arXiv, 2019.

Feudal Learning

• Learning Hierarchical Teaching in Cooperative Multiagent Reinforcement Learning by Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, Jonathan P. How. arXiv, 2019.
• Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information by Arjun Sharma, Mohit Sharma, Nicholas Rhinehart, Kris M. Kitani. ICLR, 2019.
• Mind-Aware Multi-Agent Management Reinforcement Learning by Tianmin Shu, Yuandong Tian. ICLR Workshop, 2019.
• Data-Efficient Hierarchical Reinforcement Learning by Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine. NeurIPS, 2018.
• Hierarchical Imitation and Reinforcement Learning by Hoang M. Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, Hal Daumé III. ICML, 2018.
• Meta Learning Shared Hierarchies by Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman. ICLR, 2018.
• Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning by Saurabh Kumar, Pararth Shah, Dilek Hakkani-Tür, Larry Heck. NeurIPS Workshop, 2017.
• FeUdal Networks for Hierarchical Reinforcement Learning by Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu. ICML, 2017.

Options Framework

• Dynamics-Aware Unsupervised Discovery of Skills by Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman. arXiv. 2019.
• Successor Options: An Option Discovery Framework for Reinforcement Learning by Rahul Ramesh, Manan Tomar, Balaraman Ravindran. IJCAI, 2019.
• The Option-Critic Architecture by Pierre-Luc Bacon, Jean Harb, Doina Precup. AAAI, 2017.
• Combining intrinsic motivation and hierarchical reinforcement learning by Maria K. Eckstein, Anne GE Collins. NeurIPS Workshop, 2017.
• Importance Sampled Option-Critic for More Sample Efficient Reinforcement Learning by Karan Goel, Emma Brunskill. NeurIPS Workshop, 2017.
• Optimal Hierarchical Policy Extraction From Noisy Imperfect Demonstrations by Karan Goel, Tong Mu, Emma Brunskil. NeurIPS Workshop, 2017.
• When waiting is not an option: Learning options with a deliberation cost by Jean Harb, Pierre-Luc Bacon, Martin Klissarov, Doina Precup. arXiv, 2017.
• Toward Good Abstractions for Lifelong Learning by David Abel, Dilip Arumugam, Lucas Lehnert, Michael Littman. NeurIPS Workshop, 2017.
• Landmark Options Via Reflection (LOVR) in Multi-task Lifelong Reinforcement Learning by Nicholas Denis, Maia Fraser. NeurIPS Workshop, 2017.
• Learning with Options that Terminate Off-Policy by Anna Harutyunyan, Peter Vrancx, Pierre-Luc Bacon, Doina Precup, Ann Nowe. NeurIPS Workshop, 2017.
• Universal Option Models by Hengshuai Yao, Csaba Szepesvari, Rich Sutton, Joseph Modayil. NeurIPS, 2014.

Value Function

• Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces by Haotian Fu, Hongyao Tang, Jianye Hao, Zihan Lei, Yingfeng Chen, Changjie Fan. IJCAI, 2019.
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning by Cedric Colas, Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, Pierre-Yves Oudeyer. ICML, 2019.
• Learning Multi-level Hierarchies with Hindsight by Andrew Levy, Andrew Levy, Robert Platt, Kate Saenko. ICLR, 2019.
• Hierarchical Imitation and Reinforcement Learning by Hoang M. Le, Nan Jiang, Alekh Agarwal, Miroslav Dud´ık, Yisong Yue, Hal Daume III. ICML, 2018.
• Hindsight Experience Replay by Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba. NeurIPS, 2017.
• A Demon Control Architecture with Off-Policy Learning and Flexible Behavior Policy by Shangtong Zhang, Richard Sutton. NeurIPS Workshop, 2017.
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation by Tejas D. Kullkarni, Karthik R. Narasimhan, Ardavan Saeedi and Joshua B. Tenenbaum. NeurIPS, 2016.
• Universal Value Function Approximators by Tom Schaul, Dan Horgan, Karol Gregor, David Silver. ICML, 2015.
• Hierarchical Reinforcement Learning with the Maxq Value Function Decomposition by Thomas G Dietterich. Journal of Artificial Intelligence Research, 2000.

Representaion Learning

• Hierarchical Decision Making by Generating and Following Natural Language Instructions by Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, Mike Lewis. NeurIPS, 2019.
• Language as an Abstraction for Hierarchical Reinforcement Learning by Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn. NeurIPS, 2019.
• Hierarchical Policy Learning is Sensitive to Goal Space Design by Zach Dwiel, Madhavun Candadai, Mariano Phielipp, Arjun K. Bansal. ICLR Workshop, 2019.
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning by Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine. ICLR, 2019.
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization by Takayuki Osa, Voot Tangkaratt, Masashi Sugiyama. ICLR, 2019.
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings by John D. Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, Sergey Levine. ICML, 2018.
• Stochastic Neural Networks for Hierarchical Reinforcement Learning by Carlos Florensa, Yan Duan, Pieter Abbeel. ICLR, 2017.

Entropy-Based Methods

• Maximum Entropy-Regularized Multi-Goal Reinforcement Learning by Rui Zhao, Xudong Sun, Volker Tresp. ICML, 2019.
• Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies by Kenneth Marino, Abhinav Gupta, Rob Fergus, Arthur Szlam. ICLR, 2019.
• Latent Space Policies for Hierarchical Reinforcement Learning by Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbee, Sergey Levine. ICML, 2018.

Exploration

• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning ? by Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, Sergey Levine. NeurIPS Workshop, 2019.
• Exploration via Hindsight Goal Generation by Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, Jian Peng. NeurIPS, 2019.
• Exploring Hierarchy-Aware Inverse Reinforcement Learning by Chris Cundy, Daniel Filan. ICML Workshop, 2018.
• Automatic Goal Generation for Reinforcement Learning Agents by Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel. ICML, 2018.

Application Papers

• [Multi-task] Gradient Surgery for Multi-Task Learning by Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn. arXiv, 2020.
• [Multi-agent] CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning by Jiachen Yang, Alireza Nakhaei, David Isele, Kikuo Fujimura, Hongyuan Zha. ICLR 2020.
• [Robot] Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation by Suraj Nair, Chelsea Finn. ICLR, 2020.
• [Adaptation] Sub-policy Adaptation for Hierarchical Reinforcement Learning by Alexander C. Li, Carlos Florensa, Ignasi Clavera, Pieter Abbeel. ICLR, 2020.
• [Fairness] Learning Fairness in Multi-Agent Systems by Jiechuan Jiang, Zongqing Lu. NeurIPS, 2019.
• [Environment-aware] Playing FPS Games With Environment-Aware Hierarchical Reinforcement Learning by Shihong Song, Jiayi Weng, Hang Su, Dong Yan, Haosheng Zou, Jun Zhu. IJCAI, 2019.
• [Multi-task] MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies by Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, Sergey Levine. arXiv, 2019.
• [E-commerce] Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning by Ryuichi Takanobu, Tao Zhuang, Minlie Huang, Jun Feng, Haihong Tang, Bo Zheng. WWW, 2019.
• [Recommender-system] Hierarchical Reinforcement Learning for Course Recommendation in MOOCs by Jing Zhang, Bowen Hao, Bo Chen, Cuiping Li, Hong Chen, Jimeng Sun. AAAI, 2019.
• [Relation-extraction] A Hierarchical Framework for Relation Extraction with Reinforcement Learning by Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, Minlie Huang. AAAI, 2019.
• [Multi-agent] Hierarchical Deep Multiagent Reinforcement Learning by Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Changjie Fan, Li Wang. AAAI 2019.
• [Communication] Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning by Sanjeevan Ahilan, Peter Dayan. arXiv, 2019.
• [Ride-hailing] CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms by Jiarui Jin, Ming Zhou, Weinan Zhang, Minne Li, Zilong Guo, Zhiwei Qin, Yan Jiao, Xiaocheng Tang, Chenxi Wang, Jun Wang, Guobin Wu, Jieping Ye. arXiv, 2019.
• [Text Generation] Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation by Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Wu, Jianfeng Wang, Xiaodong He. AAAI, 2019.
• [Multi-task] Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning by Tianmin Shu, Caiming Xiong, Richard Socher. ICLR, 2018.
• [Robot] Neural Modular Control for Embodied Question Answering by Abhishek Das, Georgia Gkioxari, Stefan Lee1 Devi Parikh, Dhruv Batra. CoRL, 2018.
• [Robot] Learning by Playing – Solving Sparse Reward Tasks from Scratch by Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, Tobias Springenberg. ICML, 2018.
• [Multi-task] Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning by Tianmin Shu, Caiming Xiong, Richard Socher. ICLR, 2018.
• [Subtask] Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies by Sungryull Sohn, Junhyuk Oh, Honglak Lee. NeurIPS, 2018.
• [Text-generation] Long Text Generation via Adversarial Training with Leaked Information by Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, Jun Wang. AAAI, 2018.