Mishra, UA and Samineni, SR and Goel, P and Kunjeti, C and Lodha, H and Singh, A and Sagi, A and Bhatnagar, S and Kolathaya, S (2022) Dynamic Mirror Descent based Model Predictive Control for Accelerating Robot Learning. In: 39th IEEE International Conference on Robotics and Automation, ICRA 2022, 23 - 27 May 2022, Philadelphia, pp. 1631-1637.
|
PDF
IEEE-ICRA 2022_1631-1637_2022.pdf - Published Version Download (1MB) | Preview |
Abstract
Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL. Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL. In particular, two loops are proposed, where the Dynamic Mirror Descent based Model Predictive Control (DMD-MPC) is used as the inner loop Mb-RL to obtain an optimal sequence of actions. These actions are in turn used to significantly accelerate the outer loop Mf-RL. We show that our formulation is generic for a broad class of MPC based policies and objectives, and includes some of the well-known Mb-Mf approaches. We finally introduce a new algorithm: Mirror-Descent Model Predictive RL (M-DeMoRL), which uses Cross-Entropy Method (CEM) with elite fractions for the inner loop. Our experiments show faster convergence of the proposed hierarchical approach on benchmark MuJoCo tasks. We also demonstrate hardware training for trajectory tracking in a 2R leg, and hardware transfer for robust walking in a quadruped. We show that the inner-loop Mb-RL significantly decreases the number of training iterations required in the hardware setting, thereby validating the proposed approach.
| Item Type: | Conference Paper |
|---|---|
| Publication: | Proceedings - IEEE International Conference on Robotics and Automation |
| Publisher: | Institute of Electrical and Electronics Engineers Inc. |
| Additional Information: | The copyright for this article belongs to the Authors. |
| Keywords: | Learning systems; Mirrors; Model predictive control; Predictive control systems, Asymptotic performance; Inner loops; Loop models; Model free; Model-based OPC; Model-based reinforcement learning; Model-predictive control; Reinforcement learning algorithms; Reinforcement learning approach; Reinforcement learnings, Reinforcement learning |
| Department/Centre: | Division of Electrical Sciences > Computer Science & Automation |
| Date Deposited: | 15 Sep 2022 05:13 |
| Last Modified: | 15 Sep 2022 05:13 |
| URI: | https://eprints.iisc.ac.in/id/eprint/76464 |
Actions (login required)
 |
View Item |
