Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models

We investigate the use of multimodal information contained in images as an\neffective method for enhancing the commonsense of Transformer models for text\ngeneration. We perform experiments using BART and T5 on concept-to-text\ngeneration, specifically the task of generative commonsense reasoning, or\nCommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text\nGeneration. VisCTG involves captioning images representing appropriate everyday\nscenarios, and using these captions to enrich and steer the generation process.\nComprehensive evaluation and analysis demonstrate that VisCTG noticeably\nimproves model performance while successfully addressing several issues of the\nbaseline generations, including poor commonsense, fluency, and specificity.\n