Convergence of Momentum-based Distributed Stochastic Approximation with RL Applications

Naskar, A and Thoppe, G (2023) Convergence of Momentum-based Distributed Stochastic Approximation with RL Applications. In: UNSPECIFIED, pp. 178-179.

PDF 9th_Ind_con_con_2023. pdf - Published Version Restricted to Registered users only Download (266kB) | Request a copy

Abstract

We develop a novel proof strategy for deriving almost sure convergence of momentum-based distributed stochas-Tic approximation (DSA) schemes. Popular momentum-based schemes such as Polyak's heavy-ball and Nesterov's Accelerated SGD can be analyzed using our template. Our technique enables us to do away with three restrictive assumptions of existing approaches. One, we do not need the communication matrix to be doubly stochastic. Two, we do not need the noise to be uniformly bounded. Lastly, our approach can handle cases where there are multiple or non-point attractors. As an application, we use our technique to derive convergence for momentum-based extensions of the multi-Agent TD(O) algorithm, where the above restrictive assumptions do not hold. © 2023 IEEE.

Item Type:	Conference Paper
Publication:	2023 9th Indian Control Conference, ICC 2023 - Proceedings
Publisher:	Institute of Electrical and Electronics Engineers Inc.
Additional Information:	The copyright for this article belongs to Institute of Electrical and Electronics Engineers Inc.
Keywords:	Multi agent systems; Stochastic systems, Almost sure convergence; Approximation scheme; Communication matrixes; Doubly stochastic; Multi agent; O algorithm; Proof strategy; Stochastic approximations; Uniformly bounded, Momentum
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	16 May 2024 05:22
Last Modified:	16 May 2024 05:22
URI:	https://eprints.iisc.ac.in/id/eprint/84511

Actions (login required)

View Item