Does Momentum Help in Stochastic Optimization? A Sample Complexity Analysis

Ganesh, S and Deb, R and Thoppe, G and Budhiraja, A (2023) Does Momentum Help in Stochastic Optimization? A Sample Complexity Analysis. In: 39th Conference on Uncertainty in Artificial Intelligence, UAI 2023, 31 July - 4 August 2023, Pittsburgh, pp. 602-612.

PDF UAI-2023_216_602-612_2023.pdf - Published Version Restricted to Registered users only Download (886kB) | Request a copy

Abstract

Stochastic Heavy Ball (SHB) and Nesterov's Accelerated Stochastic Gradient (ASG) are popular momentum methods in optimization. While the benefits of these acceleration ideas in deterministic settings are well understood, their advantages in stochastic optimization are unclear. Several works have recently claimed that SHB and ASG always help in stochastic optimization. Our work shows that i.) these claims are either flawed or one-sided (e.g., consider only the bias term but not the variance), and ii.) when both these terms are accounted for, SHB and ASG do not always help. Specifically, for any quadratic optimization, we obtain a lower bound on the sample complexity of SHB and ASG, accounting for both bias and variance, and show that the vanilla SGD can achieve the same bound. © UAI 2023. All rights reserved.

Item Type:	Conference Paper
Publication:	Proceedings of Machine Learning Research
Publisher:	ML Research Press
Additional Information:	The copyright for this article belongs to the ML Research Press.
Keywords:	Quadratic programming, Complexity analysis; Deterministics; Low bound; Momentum method; Optimisations; Quadratic optimization; Sample complexity; Stochastic gradient; Stochastic optimizations; Stochastics, Stochastic systems
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	28 Oct 2023 09:56
Last Modified:	28 Oct 2023 09:56
URI:	https://eprints.iisc.ac.in/id/eprint/83149

Actions (login required)

View Item