Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT

Using a language model (LM) pretrained on two languages with large\nmonolingual data in order to initialize an unsupervised neural machine\ntranslation (UNMT) system yields state-of-the-art results. When limited data is\navailable for one language, however, this method leads to poor translations. We\npresent an effective approach that reuses an LM that is pretrained only on the\nhigh-resource language. The monolingual LM is fine-tuned on both languages and\nis then used to initialize a UNMT model. To reuse the pretrained LM, we have to\nmodify its predefined vocabulary, to account for the new language. We therefore\npropose a novel vocabulary extension method. Our approach, RE-LM, outperforms a\ncompetitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk)\nand English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four\ntranslation directions.\n