Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs

Natural language rationales could provide intuitive, higher-level\nexplanations that are easily understandable by humans, complementing the more\nbroadly studied lower-level explanations based on gradients or attention\nweights. We present the first study focused on generating natural language\nrationales across several complex visual reasoning tasks: visual commonsense\nreasoning, visual-textual entailment, and visual question answering. The key\nchallenge of accurate rationalization is comprehensive image understanding at\nall levels: not just their explicit content at the pixel level, but their\ncontextual contents at the semantic and pragmatic levels. We present\nRationale^VT Transformer, an integrated model that learns to generate free-text\nrationales by combining pretrained language models with object recognition,\ngrounded visual semantic frames, and visual commonsense graphs. Our experiments\nshow that the base pretrained language model benefits from visual adaptation\nand that free-text rationalization is a promising research direction to\ncomplement model interpretability for complex visual-textual reasoning tasks.\n