A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives

Modern natural language processing (NLP) methods employ self-supervised\npretraining objectives such as masked language modeling to boost the\nperformance of various application tasks. These pretraining methods are\nfrequently extended with recurrence, adversarial or linguistic property\nmasking, and more recently with contrastive learning objectives. Contrastive\nself-supervised training objectives enabled recent successes in image\nrepresentation pretraining by learning to contrast input-input pairs of\naugmented images as either similar or dissimilar. However, in NLP, automated\ncreation of text input augmentations is still very challenging because a single\ntoken can invert the meaning of a sentence. For this reason, some contrastive\nNLP pretraining methods contrast over input-label pairs, rather than over\ninput-input pairs, using methods from Metric Learning and Energy Based Models.\nIn this survey, we summarize recent self-supervised and supervised contrastive\nNLP pretraining methods and describe where they are used to improve language\nmodeling, few or zero-shot learning, pretraining data-efficiency and specific\nNLP end-tasks. We introduce key contrastive learning concepts with lessons\nlearned from prior research and structure works by applications and cross-field\nrelations. Finally, we point to open challenges and future directions for\ncontrastive NLP to encourage bringing contrastive NLP pretraining closer to\nrecent successes in image representation pretraining.\n