Cross-modal retrieval in challenging scenarios using attributes

Dutta, Titir and Biswas, Soma (2019) Cross-modal retrieval in challenging scenarios using attributes. In: PATTERN RECOGNITION LETTERS, 125 . pp. 618-624.

PDF pat_rec_let_125_618_2019.pdf - Published Version Restricted to Registered users only Download (933kB) | Request a copy

Abstract

Cross-modal retrieval is an important field of research today because of the abundance of multi-media data. In this work, we attempt to address two challenging scenarios that we may encounter in real-life cross-modal retrieval, but which are relatively unexplored in literature. First, due to the ever-increasing number of new categories of data, cross-modal algorithms should be able to generalize to categories which it has not seen during training. Second, the data that is available during testing may be degraded (for example, it has low resolution or noise) as compared to those available during training. Here, we evaluate how these adverse conditions affect the performance of the state-of-the-art cross-modal approaches. We also propose a unified framework that can handle all these diverse and challenging scenarios without any modification. In the proposed approach, the data from different modalities are projected into a common semantic preserving latent space in which semantic relations as given by the classname embeddings (attributes) are preserved. Extensive experiments on diverse cross-modal data including image-text, RGB-depth and comparison with the state-of-the-art approaches show the usefulness of the proposed approach for these challenging scenarios.

Item Type:	Journal Article
Publication:	PATTERN RECOGNITION LETTERS
Publisher:	ELSEVIER
Additional Information:	Copyright to the article is owned by ELSEVIER
Keywords:	Cross-modal retrieval; Attributes; Unseen query; Low-resolution data
Department/Centre:	Division of Electrical Sciences > Electrical Engineering
Date Deposited:	20 Sep 2019 10:16
Last Modified:	20 Sep 2019 10:16
URI:	http://eprints.iisc.ac.in/id/eprint/63610

Actions (login required)

View Item