Gowgi, P and Garani, SS (2020) Hessian-based Bounds on Learning Rate for Gradient Descent Algorithms. In: 2020 International Joint Conference on Neural Networks, IJCNN 2020, 19-24 Jul 2020, Glasgow; United Kingdom.
![]() |
PDF
IJCNN_2020.pdf - Published Version Restricted to Registered users only Download (1MB) | Request a copy |
Abstract
Learning rate is a crucial parameter governing the convergence rate of any learning algorithm. Most of the learning algorithms based on stochastic gradient descent (SGD) method depend on heuristic choice of learning rate. In this paper, we derive bounds on the learning rate of SGD based adaptive learning algorithms by analyzing the largest eigenvalue of the Hessian matrix from first principles. The proposed approach is analytical. To illustrate the efficacy of the analytical approach, we considered several high-dimensional data sets and compared the rate of convergence of error for the neural gas algorithm and showed that the proposed bounds on learning rate result in a faster rate of convergence than AdaDec, Adam, and AdaDelta approaches which require hyper-parameter tuning. © 2020 IEEE.
Item Type: | Conference Paper |
---|---|
Publication: | Proceedings of the International Joint Conference on Neural Networks |
Publisher: | Institute of Electrical and Electronics Engineers Inc |
Additional Information: | The copyright of this paper belongs to Institute of Electrical and Electronics Engineers Inc |
Keywords: | Approximation theory; Clustering algorithms; Eigenvalues and eigenfunctions; Gradient methods; Heuristic algorithms; Heuristic methods; Matrix algebra; Neural networks; Parameter estimation; Stochastic systems, Adaptive learning algorithm; Analytical approach; Gradient descent algorithms; High dimensional data; Largest eigenvalues; Neural gas algorithms; Rate of convergence; Stochastic gradient descent, Learning algorithms |
Department/Centre: | Division of Electrical Sciences > Electronic Systems Engineering (Formerly Centre for Electronic Design & Technology) |
Date Deposited: | 22 Dec 2021 10:31 |
Last Modified: | 22 Dec 2021 10:31 |
URI: | http://eprints.iisc.ac.in/id/eprint/67405 |
Actions (login required)
![]() |
View Item |