Cross-specificity: modelling data semantics for cross-modal matching and retrieval

Verma, Y and Jha, A and Jawahar, CV (2018) Cross-specificity: modelling data semantics for cross-modal matching and retrieval. In: International Journal of Multimedia Information Retrieval, 7 (2). pp. 139-146.

PDF int_jou_mul_inf_ret_7-2_139-146_2018.pdf - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

While dealing with multi-modal data such as pairs of images and text, though individual samples may demonstrate inherent heterogeneity in their content, they are usually coupled with each other based on some higher-level concepts such as their categories. This shared information can be useful in measuring semantics of samples across modalities in a relative manner. In this paper, we investigate the problem of analysing the degree of specificity in the semantic content of a sample in one modality with respect to semantically similar samples in another modality. Samples that have high similarity with semantically similar samples from another modality are considered to be specific, while others are considered to be relatively ambiguous. To model this property, we propose a novel notion of “cross-specificity”. We present two mechanisms to measure cross-specificity: one based on human judgement and other based on an automated approach. We analyse different aspects of cross-specificity and demonstrate its utility in cross-modal retrieval task. Experiments show that though conceptually simple, it can benefit several existing cross-modal retrieval techniques and provide significant boost in their performance.

Item Type:	Journal Article
Publication:	International Journal of Multimedia Information Retrieval
Publisher:	Springer London
Additional Information:	The copyright for this article belongs to the Springer London.
Keywords:	Cross-media analysis; Cross-modal retrieval; Semantic matching
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	10 Aug 2022 11:48
Last Modified:	10 Aug 2022 11:48
URI:	https://eprints.iisc.ac.in/id/eprint/75537

Actions (login required)

View Item