Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

There are two approaches for pairwise sentence scoring: Cross-encoders, which\nperform full-attention over the input pair, and Bi-encoders, which map each\ninput independently to a dense vector space. While cross-encoders often achieve\nhigher performance, they are too slow for many practical use cases.\nBi-encoders, on the other hand, require substantial training data and\nfine-tuning over the target task to achieve competitive performance. We present\na simple yet efficient data augmentation strategy called Augmented SBERT, where\nwe use the cross-encoder to label a larger set of input pairs to augment the\ntraining data for the bi-encoder. We show that, in this process, selecting the\nsentence pairs is non-trivial and crucial for the success of the method. We\nevaluate our approach on multiple tasks (in-domain) as well as on a domain\nadaptation task. Augmented SBERT achieves an improvement of up to 6 points for\nin-domain and of up to 37 points for domain adaptation tasks compared to the\noriginal bi-encoder performance.\n