Analysis of Gradient Descent Methods With Nondiminishing Bounded Errors

Ramaswamy, Arunselvan and Bhatnagar, Shalabh (2018) Analysis of Gradient Descent Methods With Nondiminishing Bounded Errors. In: IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 63 (5). pp. 1465-1471.

PDF Ieee_Tra_Aut_Can_68-5_1465_2018.pdf - Published Version Restricted to Registered users only Download (433kB) | Request a copy

Abstract

The main aim of this paper is to provide an analysis of gradient descent (GD) algorithms with gradient errors that do not necessarily vanish, asymptotically. In particular, sufficient conditions are presented for both stability (almost sure boundedness of the iterates) and convergence of GD with bounded (possibly) nondiminishing gradient errors. In addition to ensuring stability, such an algorithm is shown to converge to a small neighborhood of the minimum set, which depends on the gradient errors. It is worth noting that the main result of this paper can be used to show that GD with asymptotically vanishing errors indeed converges to the minimum set. The results presented herein are not only more general when compared to previous results, but our analysis of GD with errors is new to the literature to the best of our knowledge. Our work extends the contributions of Mangasarian and Solodov, Bertsekas and Tsitsiklis, and Tadic and Doucet. Using our framework, a simple yet effective implementation of GD using simultaneous perturbation stochastic approximations, with constant sensitivity parameters, is presented. Another important improvement over many previous results is that there are no ``additional'' restrictions imposed on the step sizes. In machine learning applications where step sizes are related to learning rates, our assumptions, unlike those of other papers, do not affect these learning rates. Finally, we present experimental results to validate our theory.

Item Type:	Journal Article
Publication:	IEEE TRANSACTIONS ON AUTOMATIC CONTROL
Publisher:	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 445 HOES LANE, PISCATAWAY, NJ 08855-4141 USA
Additional Information:	Copy right for this article belong to IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 445 HOES LANE, PISCATAWAY, NJ 08855-4141 USA
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	16 May 2018 15:57
Last Modified:	16 May 2018 15:57
URI:	http://eprints.iisc.ac.in/id/eprint/59860

Actions (login required)

View Item