Generalized Zero-Shot Learning using Multimodal Variational Auto-Encoder with Semantic Concepts

With the ever-increasing amount of data, the central challenge in multimodal\nlearning involves limitations of labelled samples. For the task of\nclassification, techniques such as meta-learning, zero-shot learning, and\nfew-shot learning showcase the ability to learn information about novel classes\nbased on prior knowledge. Recent techniques try to learn a cross-modal mapping\nbetween the semantic space and the image space. However, they tend to ignore\nthe local and global semantic knowledge. To overcome this problem, we propose a\nMultimodal Variational Auto-Encoder (M-VAE) which can learn the shared latent\nspace of image features and the semantic space. In our approach we concatenate\nmultimodal data to a single embedding before passing it to the VAE for learning\nthe latent space. We propose the use of a multi-modal loss during the\nreconstruction of the feature embedding through the decoder. Our approach is\ncapable to correlating modalities and exploit the local and global semantic\nknowledge for novel sample predictions. Our experimental results using a MLP\nclassifier on four benchmark datasets show that our proposed model outperforms\nthe current state-of-the-art approaches for generalized zero-shot learning.\n