Variational methods for Conditional Multimodal Deep Learning

Pandey, Gaurav and Dukkipati, Ambedkar (2017) Variational methods for Conditional Multimodal Deep Learning. In: International Joint Conference on Neural Networks (IJCNN), MAY 14-19, 2017, Anchorage, AK, pp. 308-315.

PDF Int_Joi_Con_Net_308_2017.pdf - Published Version Restricted to Registered users only Download (801kB) | Request a copy

Abstract

In this paper, we address the problem of conditional modality learning, whereby one is interested in generating one modality given the other. While it is straightforward to learn a joint distribution over multiple modalities using a deep multimodal architecture, we observe that such models are not very effective at conditional generation. Hence, we address the problem by learning conditional distributions between the modalities. We use variational methods for maximizing the corresponding conditional log-likelihood. The resultant deep model, which we refer to as conditional multimodal autoencoder (CMMA), forces the latent representation obtained from a single modality alone to be `close' to the joint representation obtained from multiple modalities. We use the proposed model to generate faces from attributes. We show that the faces generated from attributes using the proposed model are qualitatively and quantitatively more representative of the attributes from which they were generated, than those obtained by other deep generative models. We also propose a secondary task, whereby the existing faces are modified by modifying the corresponding attributes. We observe that the modifications in face introduced by the proposed model are representative of the corresponding modifications in attributes.

Item Type:	Conference Proceedings
Series.:	IEEE International Joint Conference on Neural Networks (IJCNN)
Publisher:	IEEE, 345 E 47TH ST, NEW YORK, NY 10017 USA
Additional Information:	Copy right for this article belong to IEEE, 345 E 47TH ST, NEW YORK, NY 10017 USA
Department/Centre:	Division of Interdisciplinary Sciences > Supercomputer Education & Research Centre
Date Deposited:	13 Apr 2018 19:56
Last Modified:	13 Apr 2018 19:56
URI:	http://eprints.iisc.ac.in/id/eprint/59553

Actions (login required)

View Item