ePrints@IIScePrints@IISc Home | About | Browse | Latest Additions | Advanced Search | Contact | Help

Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting

Agrawal, P and Ganapathy, S (2020) Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting. In: IEEE/ACM Transactions on Audio Speech and Language Processing, 28 . pp. 2823-2836.

[img] PDF
iee_tra_aud_spe_lan_pro_28_2823-2836_2020.pdf - Published Version
Restricted to Registered users only

Download (5MB) | Request a copy
Official URL: https://dx.doi.org/ 10.1109/TASLP.2020.3030489


The learning of interpretable representations from raw data presents significant challenges for time series data like speech. In this work, we propose a relevance weighting scheme that allows the interpretation of the speech representations during the forward propagation of the model itself. The relevance weighting is achieved using a sub-network approach that performs the task of feature selection. A relevance sub-network, applied on the output of first layer of a convolutional neural network model operating on raw speech signals, acts as an acoustic filterbank (FB) layer with relevance weighting. A similar relevance sub-network applied on the second convolutional layer performs modulation filterbank learning with relevance weighting. The full acoustic model consisting of relevance sub-networks, convolutional layers and feed-forward layers is trained for a speech recognition task on noisy and reverberant speech in the Aurora-4, CHiME-3 and VOiCES datasets. The proposed representation learning framework is also applied for the task of sound classification in the UrbanSound8K dataset. A detailed analysis of the relevance weights learned by the model reveals that the relevance weights capture information regarding the underlying speech/audio content. In addition, speech recognition and sound classification experiments reveal that the incorporation of relevance weighting in the neural network architecture improves the performance significantly. © 2014 IEEE.

Item Type: Journal Article
Publication: IEEE/ACM Transactions on Audio Speech and Language Processing
Publisher: IEEE
Additional Information: The copyright of this article belongs to IEEE
Keywords: Backpropagation; Classification (of information); Convolution; Convolutional neural networks; Filter banks; Modulation; Multilayer neural networks; Network architecture; Network layers; Reverberation; Speech; Speech communication, Forward propagation; Interpretable representation; Learning frameworks; Modulation filterbank; Relevance weights; Sound classification; Time-series data; Weighting scheme, Speech recognition
Department/Centre: Division of Electrical Sciences > Electrical Engineering
Date Deposited: 15 Mar 2021 06:39
Last Modified: 15 Mar 2021 06:39
URI: http://eprints.iisc.ac.in/id/eprint/67438

Actions (login required)

View Item View Item