Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations

We investigate a strategy for improving the efficiency of contrastive\nlearning of visual representations by leveraging a small amount of supervised\ninformation during pre-training. We propose a semi-supervised loss, SuNCEt,\nbased on noise-contrastive estimation and neighbourhood component analysis,\nthat aims to distinguish examples of different classes in addition to the\nself-supervised instance-wise pretext tasks. On ImageNet, we find that SuNCEt\ncan be used to match the semi-supervised learning accuracy of previous\ncontrastive approaches while using less than half the amount of pre-training\nand compute. Our main insight is that leveraging even a small amount of labeled\ndata during pre-training, and not only during fine-tuning, provides an\nimportant signal that can significantly accelerate contrastive learning of\nvisual representations. Our code is available online at\ngithub.com/facebookresearch/suncet.\n