Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding

Contrastive learning is emerging as a powerful technique for extracting\nknowledge from unlabeled data. This technique requires a balanced mixture of\ntwo ingredients: positive (similar) and negative (dissimilar) samples. This is\ntypically achieved by maintaining a queue of negative samples during training.\nPrior works in the area typically uses a fixed-length negative sample queue,\nbut how the negative sample size affects the model performance remains unclear.\nThe opaque impact of the number of negative samples on performance when\nemploying contrastive learning aroused our in-depth exploration. This paper\npresents a momentum contrastive learning model with negative sample queue for\nsentence embedding, namely MoCoSE. We add the prediction layer to the online\nbranch to make the model asymmetric and together with EMA update mechanism of\nthe target branch to prevent the model from collapsing. We define a maximum\ntraceable distance metric, through which we learn to what extent the text\ncontrastive learning benefits from the historical information of negative\nsamples. Our experiments find that the best results are obtained when the\nmaximum traceable distance is at a certain range, demonstrating that there is\nan optimal range of historical information for a negative sample queue. We\nevaluate the proposed unsupervised MoCoSE on the semantic text similarity (STS)\ntask and obtain an average Spearman's correlation of $77.27\\%$. Source code is\navailable at https://github.com/xbdxwyh/mocose.\n