Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation

Most dominant neural machine translation (NMT) models are restricted to make\npredictions only according to the local context of preceding words in a\nleft-to-right manner. Although many previous studies try to incorporate global\ninformation into NMT models, there still exist limitations on how to\neffectively exploit bidirectional global context. In this paper, we propose a\nConfidence Based Bidirectional Global Context Aware (CBBGCA) training framework\nfor NMT, where the NMT model is jointly trained with an auxiliary conditional\nmasked language model (CMLM). The training consists of two stages: (1)\nmulti-task joint training; (2) confidence based knowledge distillation. At the\nfirst stage, by sharing encoder parameters, the NMT model is additionally\nsupervised by the signal from the CMLM decoder that contains bidirectional\nglobal contexts. Moreover, at the second stage, using the CMLM as teacher, we\nfurther pertinently incorporate bidirectional global context to the NMT model\non its unconfidently-predicted target words via knowledge distillation.\nExperimental results show that our proposed CBBGCA training framework\nsignificantly improves the NMT model by +1.02, +1.30 and +0.57 BLEU scores on\nthree large-scale translation datasets, namely WMT'14 English-to-German, WMT'19\nChinese-to-English and WMT'14 English-to-French, respectively.\n