Second Language Transfer Learning in Humans and Machines Using Image Supervision

Praveen, K and Gupta, A and Soman, A and Ganapathy, S (2019) Second Language Transfer Learning in Humans and Machines Using Image Supervision. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019, 15-18, December 2019, Singapore, pp. 1040-1047.

PDF iee_aut_spe_rec_und_wor_1040-1047_2019.pdf - Published Version Restricted to Registered users only Download (2MB) | Request a copy

Abstract

In the task of language learning, humans exhibit remarkable ability to learn new words from a foreign language with very few instances of image supervision. The question therefore is whether such transfer learning efficiency can be simulated in machines. In this paper, we propose a deep semantic model for transfer learning words from a foreign language (Japanese) using image supervision. The proposed model is a deep audio-visual correspondence network that uses a proxy based triplet loss. The model is trained with large dataset of multi-modal speech/image input in the native language (English). Then, a subset of the model parameters of the audio network are transfer learned to the foreign language words using proxy vectors from the image modality. Using the proxy based learning approach, we show that the proposed machine model achieves transfer learning performance for an image retrieval task which is comparable to the human performance. We also present an analysis that contrasts the errors made by humans and machines in this task. © 2019 IEEE.

Item Type:	Conference Paper
Publication:	2019 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019 - Proceedings
Publisher:	Institute of Electrical and Electronics Engineers Inc.
Additional Information:	cited By 0; Conference of 2019 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019 ; Conference Date: 15 December 2019 Through 18 December 2019; Conference Code:157953
Keywords:	Image retrieval; Large dataset; Learning systems; Semantics; Speech recognition, Distance Metric Learning; Document Retrieval; Human performance; Human-machine; Learning approach; Learning efficiency; Learning performance; Multi-modal learning, Transfer learning
Department/Centre:	Division of Electrical Sciences > Electrical Engineering
Date Deposited:	17 Aug 2020 10:09
Last Modified:	17 Aug 2020 10:09
URI:	http://eprints.iisc.ac.in/id/eprint/65000

Actions (login required)

View Item