Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer

Pre-trained language models are still far from human performance in tasks\nthat need understanding of properties (e.g. appearance, measurable quantity)\nand affordances of everyday objects in the real world since the text lacks such\ninformation due to reporting bias. In this work, we study whether integrating\nvisual knowledge into a language model can fill the gap. We investigate two\ntypes of knowledge transfer: (1) text knowledge transfer using image captions\nthat may contain enriched visual knowledge and (2) cross-modal knowledge\ntransfer using both images and captions with vision-language training\nobjectives. On 5 downstream tasks that may need visual knowledge to solve the\nproblem, we perform extensive empirical comparisons over the presented\nobjectives. Our experiments show that visual knowledge transfer can improve\nperformance in both low-resource and fully supervised settings.\n