COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning

Despite the recent attention to DeepFakes, one of the most prevalent ways to\nmislead audiences on social media is the use of unaltered images in a new but\nfalse context. To address these challenges and support fact-checkers, we\npropose a new method that automatically detects out-of-context image and text\npairs. Our key insight is to leverage the grounding of image with text to\ndistinguish out-of-context scenarios that cannot be disambiguated with language\nalone. We propose a self-supervised training strategy where we only need a set\nof captioned images. At train time, our method learns to selectively align\nindividual objects in an image with textual claims, without explicit\nsupervision. At test time, we check if both captions correspond to the same\nobject(s) in the image but are semantically different, which allows us to make\nfairly accurate out-of-context predictions. Our method achieves 85%\nout-of-context detection accuracy. To facilitate benchmarking of this task, we\ncreate a large-scale dataset of 200K images with 450K textual captions from a\nvariety of news websites, blogs, and social media posts. The dataset and source\ncode is publicly available at\nhttps://shivangi-aneja.github.io/projects/cosmos/.\n