Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation

How to learn highly compact yet effective sentence representation?\nPre-trained language models have been effective in many NLP tasks. However,\nthese models are often huge and produce large sentence embeddings. Moreover,\nthere is a big performance gap between large and small models. In this paper,\nwe propose Homomorphic Projective Distillation (HPD) to learn compressed\nsentence embeddings. Our method augments a small Transformer encoder model with\nlearnable projection layers to produce compact representations while mimicking\na large pre-trained language model to retain the sentence representation\nquality. We evaluate our method with different model sizes on both semantic\ntextual similarity (STS) and semantic retrieval (SR) tasks. Experiments show\nthat our method achieves 2.7-4.5 points performance gain on STS tasks compared\nwith previous best representations of the same size. In SR tasks, our method\nimproves retrieval speed (8.2$\\times$) and memory usage (8.0$\\times$) compared\nwith state-of-the-art large models.\n