Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting

Agrawal, P and Ganapathy, S (2020) Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting. In: IEEE/ACM Transactions on Audio Speech and Language Processing, 28 . pp. 2823-2836.

PDF iee_tra_aud_spe_lan_pro_28_2823-2836_2020.pdf - Published Version Restricted to Registered users only Download (5MB) | Request a copy

Abstract

The learning of interpretable representations from raw data presents significant challenges for time series data like speech. In this work, we propose a relevance weighting scheme that allows the interpretation of the speech representations during the forward propagation of the model itself. The relevance weighting is achieved using a sub-network approach that performs the task of feature selection. A relevance sub-network, applied on the output of first layer of a convolutional neural network model operating on raw speech signals, acts as an acoustic filterbank (FB) layer with relevance weighting. A similar relevance sub-network applied on the second convolutional layer performs modulation filterbank learning with relevance weighting. The full acoustic model consisting of relevance sub-networks, convolutional layers and feed-forward layers is trained for a speech recognition task on noisy and reverberant speech in the Aurora-4, CHiME-3 and VOiCES datasets. The proposed representation learning framework is also applied for the task of sound classification in the UrbanSound8K dataset. A detailed analysis of the relevance weights learned by the model reveals that the relevance weights capture information regarding the underlying speech/audio content. In addition, speech recognition and sound classification experiments reveal that the incorporation of relevance weighting in the neural network architecture improves the performance significantly. © 2014 IEEE.

Item Type:	Journal Article
Publication:	IEEE/ACM Transactions on Audio Speech and Language Processing
Publisher:	IEEE
Additional Information:	The copyright of this article belongs to IEEE
Keywords:	Backpropagation; Classification (of information); Convolution; Convolutional neural networks; Filter banks; Modulation; Multilayer neural networks; Network architecture; Network layers; Reverberation; Speech; Speech communication, Forward propagation; Interpretable representation; Learning frameworks; Modulation filterbank; Relevance weights; Sound classification; Time-series data; Weighting scheme, Speech recognition
Department/Centre:	Division of Electrical Sciences > Electrical Engineering
Date Deposited:	15 Mar 2021 06:39
Last Modified:	15 Mar 2021 06:39
URI:	http://eprints.iisc.ac.in/id/eprint/67438

Actions (login required)

View Item