Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

The availability of large-scale image captioning and visual question\nanswering datasets has contributed significantly to recent successes in\nvision-and-language pre-training. However, these datasets are often collected\nwith overrestrictive requirements inherited from their original target tasks\n(e.g., image caption generation), which limit the resulting dataset scale and\ndiversity. We take a step further in pushing the limits of vision-and-language\npre-training data by relaxing the data collection pipeline used in Conceptual\nCaptions 3M (CC3M) [Sharma et al. 2018] and introduce the Conceptual 12M\n(CC12M), a dataset with 12 million image-text pairs specifically meant to be\nused for vision-and-language pre-training. We perform an analysis of this\ndataset and benchmark its effectiveness against CC3M on multiple downstream\ntasks with an emphasis on long-tail visual recognition. Our results clearly\nillustrate the benefit of scaling up pre-training data for vision-and-language\ntasks, as indicated by the new state-of-the-art results on both the nocaps and\nConceptual Captions benchmarks.\n