A Novel Attention-based Aggregation Function to Combine Vision and Language

The joint understanding of vision and language has been recently gaining a\nlot of attention in both the Computer Vision and Natural Language Processing\ncommunities, with the emergence of tasks such as image captioning, image-text\nmatching, and visual question answering. As both images and text can be encoded\nas sets or sequences of elements -- like regions and words -- proper reduction\nfunctions are needed to transform a set of encoded elements into a single\nresponse, like a classification or similarity score. In this paper, we propose\na novel fully-attentive reduction method for vision and language. Specifically,\nour approach computes a set of scores for each element of each modality\nemploying a novel variant of cross-attention, and performs a learnable and\ncross-modal reduction, which can be used for both classification and ranking.\nWe test our approach on image-text matching and visual question answering,\nbuilding fair comparisons with other reduction choices, on both COCO and VQA\n2.0 datasets. Experimentally, we demonstrate that our approach leads to a\nperformance increase on both tasks. Further, we conduct ablation studies to\nvalidate the role of each component of the approach.\n