Semi-Supervised Cross-Modal Retrieval with Label Prediction

Mandal, D and Rao, P and Biswas, S (2020) Semi-Supervised Cross-Modal Retrieval with Label Prediction. In: IEEE Transactions on Multimedia, 22 (9). pp. 2345-2353.

PDF iee_tra_mul_22-09_2345-2353_2020.pdf - Published Version Restricted to Registered users only Download (3MB) | Request a copy

Abstract

Cross-modal retrieval tasks with image-text, audio-image, etc. are gaining increasing importance due to an abundance of data from multiple modalities. In general, supervised approaches give significant improvement over their unsupervised counterparts at the additional cost of labeling or annotation of the training data. Recently, semi-supervised methods are becoming popular as they provide an elegant framework to balance the conflicting requirement of labeling cost and accuracy. In this work, we propose a novel deep semi-supervised framework, which can seamlessly handle both labeled as well as unlabeled data. The network has two important components: (a) first, the labels for the unlabeled portion of the training data are predicted using the label prediction component, and then (b) a common representation for both the modalities is learned for performing cross-modal retrieval. The two parts of the network are trained sequentially one after the other. Extensive experiments on three benchmark datasets, Wiki, Pascal VOC, and NUS-WIDE demonstrate that the proposed framework outperforms the state-of-the-art for both supervised and semi-supervised settings. © 1999-2012 IEEE.

Item Type:	Journal Article
Publication:	IEEE Transactions on Multimedia
Publisher:	Institute of Electrical and Electronics Engineers Inc.
Additional Information:	The copyright of this article belongs to Institute of Electrical and Electronics Engineers Inc.
Keywords:	Multimedia systems; Signal processing, Additional costs; Benchmark datasets; Label predictions; Multiple modalities; Semi-supervised; Semi-supervised method; State of the art; Unlabeled data, Semi-supervised learning
Department/Centre:	Division of Electrical Sciences > Electrical Engineering
Date Deposited:	24 Sep 2020 06:30
Last Modified:	24 Sep 2020 06:30
URI:	http://eprints.iisc.ac.in/id/eprint/66519

Actions (login required)

View Item