Generalized Zero-Shot Cross-Modal Retrieval

Dutta, Titir and Biswas, Soma (2019) Generalized Zero-Shot Cross-Modal Retrieval. In: IEEE TRANSACTIONS ON IMAGE PROCESSING, 28 (12). pp. 5953-5962.

PDF iee_tra_ima_pro_28-12_5953_2019.pdf - Published Version Restricted to Registered users only Download (2MB) | Request a copy

Abstract

Cross-modal retrieval is an important research area due to its wide range of applications, and several algorithms have been proposed to address this task. We feel that it is the right time to take a step hack and analyze the current status of research in this area. As new object classes are continuously being discovered over time, it is necessary to design algorithms that can generalize to data from previously unseen classes. Towards that goal, our first contribution is to establish protocols for generalized zero-shot cross-modal retrieval and analyze the generalization ability of the standard cross-modal algorithms. Second, we propose a semantic-aware ranking algorithm that can be used as an add-on to any existing cross-modal approach to improve its performance on both seen and unseen classes. Finally, we propose a modification of the standard evaluation metric (MAP for single-label data and NUCG for multi-label data), which we feel is a more intuitive measure of the cross-modal retrieval performance. Extensive experiments on two single-label and three multi-label crass-modal datasets show the effectiveness of the proposed approach.

Item Type:	Journal Article
Publication:	IEEE TRANSACTIONS ON IMAGE PROCESSING
Publisher:	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
Additional Information:	copyright for this article belongs to IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
Keywords:	Cross-modal retrieval; zero-shot learning; attributes; evaluation metric
Department/Centre:	Division of Electrical Sciences > Electrical Engineering
Date Deposited:	17 Oct 2019 10:02
Last Modified:	17 Oct 2019 10:02
URI:	http://eprints.iisc.ac.in/id/eprint/63663

Actions (login required)

View Item