Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions

When people observe events, they are able to abstract key information and\nbuild concise summaries of what is happening. These summaries include\ncontextual and semantic information describing the important high-level details\n(what, where, who and how) of the observed event and exclude background\ninformation that is deemed unimportant to the observer. With this in mind, the\ndescriptions people generate for videos of different dynamic events can greatly\nimprove our understanding of the key information of interest in each video.\nThese descriptions can be captured in captions that provide expanded attributes\nfor video labeling (e.g. actions/objects/scenes/sentiment/etc.) while allowing\nus to gain new insight into what people find important or necessary to\nsummarize specific events. Existing caption datasets for video understanding\nare either small in scale or restricted to a specific domain. To address this,\nwe present the Spoken Moments (S-MiT) dataset of 500k spoken captions each\nattributed to a unique short video depicting a broad range of different events.\nWe collect our descriptions using audio recordings to ensure that they remain\nas natural and concise as possible while allowing us to scale the size of a\nlarge classification dataset. In order to utilize our proposed dataset, we\npresent a novel Adaptive Mean Margin (AMM) approach to contrastive learning and\nevaluate our models on video/caption retrieval on multiple datasets. We show\nthat our AMM approach consistently improves our results and that models trained\non our Spoken Moments dataset generalize better than those trained on other\nvideo-caption datasets.\n