Decentralized Stochastic Projection-Free Learning with Compressed Push-Sum

Francis, R and Chepuri, SP (2023) Decentralized Stochastic Projection-Free Learning with Compressed Push-Sum. In: UNSPECIFIED.

PDF Iee_int_wor_mac_lea_sig_pro_mls_2023 - Published Version Restricted to Registered users only Download (277kB) | Request a copy

Abstract

We consider decentralized stochastic learning methods with data being distributed among multiple nodes. The nodes communicate gradients with their connected neighbors. To reduce communication overhead, several compression techniques have been proposed for solving unconstrained optimization problems in a decentralized setting. In this paper, we focus on decentralized stochastic learning with convex constraints. Specifically, we propose a novel communication-efficient decentralized stochastic Frank-Wolfe algorithm to solve finite-sum constrained minimization problems by communicating only a compressed version of the gradient. The proposed method guarantees a convergence rate of about O(n-1/2k-1/3) for convex objectives, and for non-convex objectives, it guarantees a convergence rate of about O(n-1/2k-2/9), where n is the number of nodes in the network. We empirically validate the efficacy of the proposed algorithm in terms of communication overhead and suboptimality gap on several benchmark machine learning tasks. © 2023 IEEE.

Item Type:	Conference Paper
Publication:	IEEE International Workshop on Machine Learning for Signal Processing, MLSP
Publisher:	IEEE Computer Society
Additional Information:	The copyright for this article belongs to IEEE Computer Society.
Keywords:	Constrained optimization; Learning systems; Machine learning, Communication overheads; Compression techniques; Convergence rates; Convex objectives; Decentralised; Learning methods; Multiple nodes; Stochastic learning; Stochastics; Unconstrained optimization problems, Stochastic systems
Department/Centre:	Division of Electrical Sciences > Electrical Communication Engineering
Date Deposited:	04 Mar 2024 09:13
Last Modified:	04 Mar 2024 09:13
URI:	https://eprints.iisc.ac.in/id/eprint/84302

Actions (login required)

View Item