NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application

Pre-trained language models (PLMs) like BERT have made great progress in NLP.\nNews articles usually contain rich textual information, and PLMs have the\npotentials to enhance news text modeling for various intelligent news\napplications like news recommendation and retrieval. However, most existing\nPLMs are in huge size with hundreds of millions of parameters. Many online news\napplications need to serve millions of users with low latency tolerance, which\nposes huge challenges to incorporating PLMs in these scenarios. Knowledge\ndistillation techniques can compress a large PLM into a much smaller one and\nmeanwhile keeps good performance. However, existing language models are\npre-trained and distilled on general corpus like Wikipedia, which has some gaps\nwith the news domain and may be suboptimal for news intelligence. In this\npaper, we propose NewsBERT, which can distill PLMs for efficient and effective\nnews intelligence. In our approach, we design a teacher-student joint learning\nand distillation framework to collaboratively learn both teacher and student\nmodels, where the student model can learn from the learning experience of the\nteacher model. In addition, we propose a momentum distillation method by\nincorporating the gradients of teacher model into the update of student model\nto better transfer useful knowledge learned by the teacher model. Extensive\nexperiments on two real-world datasets with three tasks show that NewsBERT can\neffectively improve the model performance in various intelligent news\napplications with much smaller models.\n