Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering

We introduce an evaluation methodology for visual question answering (VQA) to\nbetter diagnose cases of shortcut learning. These cases happen when a model\nexploits spurious statistical regularities to produce correct answers but does\nnot actually deploy the desired behavior. There is a need to identify possible\nshortcuts in a dataset and assess their use before deploying a model in the\nreal world. The research community in VQA has focused exclusively on\nquestion-based shortcuts, where a model might, for example, answer "What is the\ncolor of the sky" with "blue" by relying mostly on the question-conditional\ntraining prior and give little weight to visual evidence. We go a step further\nand consider multimodal shortcuts that involve both questions and images. We\nfirst identify potential shortcuts in the popular VQA v2 training set by mining\ntrivial predictive rules such as co-occurrences of words and visual elements.\nWe then introduce VQA-CounterExamples (VQA-CE), an evaluation protocol based on\nour subset of CounterExamples i.e. image-question-answer triplets where our\nrules lead to incorrect answers. We use this new evaluation in a large-scale\nstudy of existing approaches for VQA. We demonstrate that even state-of-the-art\nmodels perform poorly and that existing techniques to reduce biases are largely\nineffective in this context. Our findings suggest that past work on\nquestion-based biases in VQA has only addressed one facet of a complex issue.\nThe code for our method is available at\nhttps://github.com/cdancette/detect-shortcuts.\n