Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation

Successful methods for unsupervised neural machine translation (UNMT) employ\ncrosslingual pretraining via self-supervision, often in the form of a masked\nlanguage modeling or a sequence generation task, which requires the model to\nalign the lexical- and high-level representations of the two languages. While\ncross-lingual pretraining works for similar languages with abundant corpora, it\nperforms poorly in low-resource and distant languages. Previous research has\nshown that this is because the representations are not sufficiently aligned. In\nthis paper, we enhance the bilingual masked language model pretraining with\nlexical-level information by using type-level cross-lingual subword embeddings.\nEmpirical results demonstrate improved performance both on UNMT (up to 4.5\nBLEU) and bilingual lexicon induction using our method compared to a UNMT\nbaseline.\n