Efficient energy management in smart grids with finite horizon Q-learning

VP, V and Bhatnagar, S (2024) Efficient energy management in smart grids with finite horizon Q-learning. In: Sustainable Energy, Grids and Networks, 38 .

PDF sus_ene_gri_net_38_2024.pdf - Published Version Restricted to Registered users only Download (2MB) | Request a copy

Abstract

Efficient energy distribution in smart grids is an important problem driven by the need to manage increasing power consumption across the globe. This problem has been studied in the past using different frameworks including Markov Decision Processes (MDP) and Reinforcement Learning. However, existing algorithms in Reinforcement Learning Theory largely deal with infinite horizon decision making. On the other hand, smart grid problems are inherently finite horizon in nature and so can best be addressed using Finite Horizon algorithms. We therefore analyze the smart grid setup using a finite horizon MDP model and develop, for the first time, a finite horizon version of the popular Q-learning algorithm. We observe that our algorithm shows good empirical performance on the smart grid setting. We also theoretically analyze the full convergence of the algorithm by proving both the stability and the convergence of the same. Our analysis of stability and convergence of finite horizon Q-learning is based entirely on the ordinary differential equations (O.D.E) method. Apart from smart grids, we additionally demonstrate the performance of our algorithm on a setting of random MDPs indicating that the algorithm is more generally applicable and can be studied on other settings in the future. © 2024 Elsevier Ltd

Item Type:	Journal Article
Publication:	Sustainable Energy, Grids and Networks
Publisher:	Elsevier Ltd
Additional Information:	The copyright for this article belongs to Elsevier Ltd.
Keywords:	Behavioral research; Decision making; Dynamic programming; Electric power distribution; Energy efficiency; Energy management; Energy management systems; Learning algorithms; Markov processes; Ordinary differential equations; Smart power grids, Finite horizon Q-learning; Finite horizon setting; Finite horizons; Markov Decision Processes; Microgrid; Optimal controls; Q-learning; Reinforcement learnings; Smart grid, Reinforcement learning
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	04 Mar 2024 05:34
Last Modified:	04 Mar 2024 05:34
URI:	https://eprints.iisc.ac.in/id/eprint/84127

Actions (login required)

View Item