Barrier Functions Inspired Reward Shaping for Reinforcement Learning

Nilaksh, . and Ranjan, A and Agrawal, S and Jain, A and Jagtap, P and Kolathaya, S (2024) Barrier Functions Inspired Reward Shaping for Reinforcement Learning. In: 2024 IEEE International Conference on Robotics and Automation, ICRA 2024, 13 May 2024 through 17 May 2024, Yokohama, pp. 10807-10813.

PDF pro_ieee_int_con_rob_aut_2024.pdf - Published Version Restricted to Registered users only Download (6MB) | Request a copy

Abstract

Reinforcement Learning (RL) has progressed from simple control tasks to complex real-world challenges with large state spaces. While RL excels in these tasks, training time remains a limitation. Reward shaping is a popular solution, but existing methods often rely on value functions, which face scalability issues. This paper presents a novel safety-oriented reward-shaping framework inspired by barrier functions, offering simplicity and ease of implementation across various environments and tasks. To evaluate the effectiveness of the proposed reward formulations, we conduct simulation experiments on CartPole, Ant, and Humanoid environments, along with real-world deployment on the Unitree Go1 quadruped robot. Our results demonstrate that our method leads to 1.4-2.8 times faster convergence and as low as 50-60 actuation effort compared to the vanilla reward. In a sim-to-real experiment with the Go1 robot, we demonstrated better control and dynamics of the bot with our reward framework. We have open-sourced our code at https://github.com/Safe-RL-IISc/barrier-shaping. © 2024 IEEE.

Item Type:	Conference Paper
Publication:	Proceedings - IEEE International Conference on Robotics and Automation
Publisher:	Institute of Electrical and Electronics Engineers Inc.
Additional Information:	The copyright for this article belongs to publisher.
Keywords:	Adversarial machine learning; Anthropomorphic robots; Bot (Internet); Contrastive Learning; Multipurpose robots, Barriers functions; Control task; Excel; Real-world; Reinforcement learnings; Simple++; State-space; Task trainings; Training time; Value functions, Reinforcement learning
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation Division of Interdisciplinary Sciences > Robert Bosch Centre for Cyber Physical Systems
Date Deposited:	12 Sep 2024 06:30
Last Modified:	12 Sep 2024 06:30
URI:	http://eprints.iisc.ac.in/id/eprint/86143

Actions (login required)

View Item