ePrints@IIScePrints@IISc Home | About | Browse | Latest Additions | Advanced Search | Contact | Help

Towards Semantic Visual Representation: Augmenting Image Representation with Natural Language Descriptors

Mopuri, Konda Reddy and Babu, Venkatesh R (2016) Towards Semantic Visual Representation: Augmenting Image Representation with Natural Language Descriptors. In: 10th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), DEC 18-22, 2016, Indian Inst Technol, Guwahati, INDIA.

[img] PDF
2016_I_C_C_V_G_I_P.pdf - Published Version
Restricted to Registered users only

Download (615kB) | Request a copy
Official URL: http://dx.doi.org/10.1145/3009977.3010010


Learning image representations has been an interesting and challenging problem. When users upload images to photo sharing websites, they often provide multiple textual tags for ease of reference. These tags can reveal significant information about the content of the image such as the objects present in the image or the action that is taking place. Approaches have been proposed to extract additional information from these tags in order to augment the visual cues and build a multi-modal image representation. However, the existing approaches do not pay much attention to the semantic meaning of the tags while they encode. In this work, we attempt to enrich the image representation with the tag encodings that leverage their semantics. Our approach utilizes neural network based natural language descriptors to represent the tag information. By complementing the visual features learned by convnets, our approach results in an efficient multi-modal image representation. Experimental evaluation suggests that our approach results in a better multi-modal image representation by exploiting the two data modalities for classification on benchmark datasets.

Item Type: Conference Proceedings
Additional Information: Copy right for this article belongs to theCopy right for this article belongs to the
Department/Centre: Division of Interdisciplinary Sciences > Supercomputer Education & Research Centre
Date Deposited: 15 Jul 2017 07:38
Last Modified: 15 Jul 2017 07:38
URI: http://eprints.iisc.ac.in/id/eprint/57428

Actions (login required)

View Item View Item