Hierarchical Average Reward Policy Gradient Algorithms

Dharmavaram, A and Riemer, M and Bhatnagar, S (2020) Hierarchical Average Reward Policy Gradient Algorithms. In: 34th AAAI Conference on Artificial Intelligence, AAAI 2020, 7-12 Feb 2020, New York, pp. 13777-13778.

PDF AAAI_2020.pdf - Published Version Restricted to Registered users only Download (822kB) | Request a copy

Abstract

Option-critic learning is a general-purpose reinforcement learning (RL) framework that aims to address the issue of long term credit assignment by leveraging temporal abstractions. However, when dealing with extended timescales, discounting future rewards can lead to incorrect credit assignments. In this work, we address this issue by extending the hierarchical option-critic policy gradient theorem for the average reward criterion. Our proposed framework aims to maximize the long-term reward obtained in the steady-state of the Markov chain defined by the agent's policy. Furthermore, we use an ordinary differential equation based approach for our convergence analysis and prove that the parameters of the intra-option policies, termination functions, and value functions, converge to their corresponding optimal values, with probability one. Finally, we illustrate the competitive advantage of learning options, in the average reward setting, on a grid-world environment with sparse rewards. © 2020 The Twenty-Fifth AAAI/SIGAI Doctoral Consortium (AAAI-20). All Rights Reserved.

Item Type:	Conference Paper
Publication:	AAAI 2020 - 34th AAAI Conference on Artificial Intelligence
Publisher:	AAAI press
Additional Information:	The copyright for this article belongs to Association for the Advancement of Artificial Intelligence
Keywords:	Competition; Markov chains; Ordinary differential equations, Average reward criteria; Competitive advantage; Convergence analysis; Credit assignment; Optimal values; Policy gradient; Temporal abstraction; Value functions, Reinforcement learning
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	19 Aug 2021 07:08
Last Modified:	19 Aug 2021 07:08
URI:	http://eprints.iisc.ac.in/id/eprint/69288

Actions (login required)

View Item