Augmenting Visual Question Answering with Semantic Frame Information in a Multitask Learning Approach

Visual Question Answering (VQA) concerns providing answers to Natural\nLanguage questions about images. Several deep neural network approaches have\nbeen proposed to model the task in an end-to-end fashion. Whereas the task is\ngrounded in visual processing, if the question focuses on events described by\nverbs, the language understanding component becomes crucial. Our hypothesis is\nthat models should be aware of verb semantics, as expressed via semantic role\nlabels, argument types, and/or frame elements. Unfortunately, no VQA dataset\nexists that includes verb semantic information. Our first contribution is a new\nVQA dataset (imSituVQA) that we built by taking advantage of the imSitu\nannotations. The imSitu dataset consists of images manually labeled with\nsemantic frame elements, mostly taken from FrameNet. Second, we propose a\nmultitask CNN-LSTM VQA model that learns to classify the answers as well as the\nsemantic frame elements. Our experiments show that semantic frame element\nclassification helps the VQA system avoid inconsistent responses and improves\nperformance.\n