An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

Bhatnagar, Shalabh and Lakshmanan, K (2012) An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes. In: JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS, 153 (3). pp. 688-708.

PDF MAT_STA_153-3_688-708_2012.pdf - Published Version Restricted to Registered users only Download (891kB) | Request a copy

Abstract

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

Item Type:	Journal Article
Publication:	JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS
Publisher:	Springer Link
Additional Information:	Copy right for this article belongs to Springer Science
Keywords:	ASYNCHRONOUS STOCHASTIC APPROXIMATIONS;NETWORKS
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	17 Jul 2012 11:22
Last Modified:	17 Jul 2012 11:22
URI:	http://eprints.iisc.ac.in/id/eprint/44623

Actions (login required)

View Item