Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers

Pretrained vision-and-language BERTs aim to learn representations that\ncombine information from both modalities. We propose a diagnostic method based\non cross-modal input ablation to assess the extent to which these models\nactually integrate cross-modal information. This method involves ablating\ninputs from one modality, either entirely or selectively based on cross-modal\ngrounding alignments, and evaluating the model prediction performance on the\nother modality. Model performance is measured by modality-specific tasks that\nmirror the model pretraining objectives (e.g. masked language modelling for\ntext). Models that have learned to construct cross-modal representations using\nboth modalities are expected to perform worse when inputs are missing from a\nmodality. We find that recently proposed models have much greater relative\ndifficulty predicting text when visual information is ablated, compared to\npredicting visual object categories when text is ablated, indicating that these\nmodels are not symmetrically cross-modal.\n