Reinforcement learning algorithm for non-stationary environments

Padakandla, S and Prabuchandran, KJ and Bhatnagar, S (2020) Reinforcement learning algorithm for non-stationary environments. In: Applied Intelligence, 50 (11). pp. 3590-3606.

PDF APP_INT_50_11_3590-3606_2020.pdf - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

Reinforcement learning (RL) methods learn optimal decisions in the presence of a stationary environment. However, the stationary assumption on the environment is very restrictive. In many real world problems like traffic signal control, robotic applications, etc., one often encounters situations with non-stationary environments, and in these scenarios, RL methods yield sub-optimal decisions. In this paper, we thus consider the problem of developing RL methods that obtain optimal decisions in a non-stationary environment. The goal of this problem is to maximize the long-term discounted reward accrued when the underlying model of the environment changes over time. To achieve this, we first adapt a change point algorithm to detect change in the statistics of the environment and then develop an RL algorithm that maximizes the long-run reward accrued. We illustrate that our change point method detects change in the model of the environment effectively and thus facilitates the RL algorithm in maximizing the long-run reward. We further validate the effectiveness of the proposed solution on non-stationary random Markov decision processes, a sensor energy management problem, and a traffic signal control problem. © 2020, Springer Science+Business Media, LLC, part of Springer Nature.

Item Type:	Journal Article
Publication:	Applied Intelligence
Publisher:	Springer
Additional Information:	copy right for this article belongs to Springer
Keywords:	Learning algorithms; Markov processes; Traffic signals, Environment change; Management problems; Markov Decision Processes; Non-stationary environment; Reinforcement learning method; Robotic applications; Stationary environments; Traffic signal control, Reinforcement learning
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	01 Dec 2020 09:23
Last Modified:	01 Dec 2020 09:23
URI:	http://eprints.iisc.ac.in/id/eprint/65932

Actions (login required)

View Item