Fully Synthetic Data Improves Neural Machine Translation with Knowledge Distillation

This paper explores augmenting monolingual data for knowledge distillation in\nneural machine translation. Source language monolingual text can be\nincorporated as a forward translation. Interestingly, we find the best way to\nincorporate target language monolingual text is to translate it to the source\nlanguage and round-trip translate it back to the target language, resulting in\na fully synthetic corpus. We find that combining monolingual data from both\nsource and target languages yields better performance than a corpus twice as\nlarge only in one language. Moreover, experiments reveal that the improvement\ndepends upon the provenance of the test set. If the test set was originally in\nthe source language (with the target side written by translators), then forward\ntranslating source monolingual data matters. If the test set was originally in\nthe target language (with the source written by translators), then\nincorporating target monolingual data matters.\n