Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Contrastive Language-Image Pretraining (CLIP) has emerged as a novel paradigm\nto learn visual models from language supervision. While researchers continue to\npush the frontier of CLIP, reproducing these works remains challenging. This is\nbecause researchers do not choose consistent training recipes and even use\ndifferent data, hampering the fair comparison between different methods. In\nthis work, we propose CLIP-benchmark, a first attempt to evaluate, analyze, and\nbenchmark CLIP and its variants. We conduct a comprehensive analysis of three\nkey factors: data, supervision, and model architecture. We find considerable\nintuitive or counter-intuitive insights: (1). Data quality has a significant\nimpact on performance. (2). Certain supervision has different effects for\nConvolutional Networks (ConvNets) and Vision Transformers (ViT). Applying more\nproper supervision can effectively improve the performance of CLIP. (3).\nCurtailing the text encoder reduces the training cost but not much affect the\nfinal performance. Moreover, we further combine DeCLIP with FILIP, bringing us\nthe strongest variant DeFILIP. The CLIP-benchmark would be released at:\nhttps://github.com/Sense-GVT/DeCLIP for future CLIP research.\n