Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data

Rangwani, H and Aithal, SK and Mishra, M and Venkatesh Babu, R (2022) Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data. In: 36th Conference on Neural Information Processing Systems, NeurIPS 2022, 28 - 9 December 2022, New Orleans.

PDF neurips_2022.pdf - Published Version Restricted to Registered users only Download (917kB) | Request a copy

Abstract

Real-world datasets exhibit imbalances of varying types and degrees. Several techniques based on re-weighting and margin adjustment of loss are often used to enhance the performance of neural networks, particularly on minority classes. In this work, we analyze the class-imbalanced learning problem by examining the loss landscape of neural networks trained with re-weighting and margin based techniques. Specifically, we examine the spectral density of Hessian of class-wise loss, through which we observe that the network weights converges to a saddle point in the loss landscapes of minority classes. Following this observation, we also find that optimization methods designed to escape from saddle points can be effectively used to improve generalization on minority classes. We further theoretically and empirically demonstrate that Sharpness-Aware Minimization (SAM), a recent technique that encourages convergence to a flat minima, can be effectively used to escape saddle points for minority classes. Using SAM results in a 6.2% increase in accuracy on the minority classes over the state-of-the-art Vector Scaling Loss, leading to an overall average increase of 4% across imbalanced datasets. The code is available at https://github.com/val-iisc/Saddle-LongTail.

Item Type:	Conference Paper
Publication:	Advances in Neural Information Processing Systems
Publisher:	Neural information processing systems foundation
Additional Information:	The copyright for this article belongs to Neural information processing systems foundation.
Department/Centre:	Division of Interdisciplinary Sciences > Computational and Data Sciences
Date Deposited:	20 Jul 2023 09:57
Last Modified:	20 Jul 2023 09:57
URI:	https://eprints.iisc.ac.in/id/eprint/82498

Actions (login required)

View Item