Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA

Recent works have shown that supervised models often exploit data artifacts\nto achieve good test scores while their performance severely degrades on\nsamples outside their training distribution. Contrast sets (Gardneret al.,\n2020) quantify this phenomenon by perturbing test samples in a minimal way such\nthat the output label is modified. While most contrast sets were created\nmanually, requiring intensive annotation effort, we present a novel method\nwhich leverages rich semantic input representation to automatically generate\ncontrast sets for the visual question answering task. Our method computes the\nanswer of perturbed questions, thus vastly reducing annotation cost and\nenabling thorough evaluation of models' performance on various semantic aspects\n(e.g., spatial or relational reasoning). We demonstrate the effectiveness of\nour approach on the GQA dataset and its semantic scene graph image\nrepresentation. We find that, despite GQA's compositionality and carefully\nbalanced label distribution, two high-performing models drop 13-17% in accuracy\ncompared to the original test set. Finally, we show that our automatic\nperturbation can be applied to the training set to mitigate the degradation in\nperformance, opening the door to more robust models.\n