MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

As humans, we navigate a multimodal world, building a holistic understanding\nfrom all our senses. We introduce MERLOT Reserve, a model that represents\nvideos jointly over time -- through a new training objective that learns from\naudio, subtitles, and video frames. Given a video, we replace snippets of text\nand audio with a MASK token; the model learns by choosing the correct\nmasked-out snippet. Our objective learns faster than alternatives, and performs\nwell at scale: we pretrain on 20 million YouTube videos.\n Empirical results show that MERLOT Reserve learns strong multimodal\nrepresentations. When finetuned, it sets state-of-the-art on Visual Commonsense\nReasoning (VCR), TVQA, and Kinetics-600; outperforming prior work by 5%, 7%,\nand 1.5% respectively. Ablations show that these tasks benefit from audio\npretraining -- even VCR, a QA task centered around images (without sound).\nMoreover, our objective enables out-of-the-box prediction, revealing strong\nmultimodal commonsense understanding. In a fully zero-shot setting, our model\nobtains competitive results on four video tasks, even outperforming supervised\napproaches on the recently proposed Situated Reasoning (STAR) benchmark.\n We analyze why audio enables better vision-language representations,\nsuggesting significant opportunities for future research. We conclude by\ndiscussing ethical and societal implications of multimodal pretraining.\n