Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation

Back-translation is a critical component of Unsupervised Neural Machine\nTranslation (UNMT), which generates pseudo parallel data from target\nmonolingual data. A UNMT model is trained on the pseudo parallel data with\ntranslated source, and translates natural source sentences in inference. The\nsource discrepancy between training and inference hinders the translation\nperformance of UNMT models. By carefully designing experiments, we identify two\nrepresentative characteristics of the data gap in source: (1) style gap (i.e.,\ntranslated vs. natural text style) that leads to poor generalization\ncapability; (2) content gap that induces the model to produce hallucination\ncontent biased towards the target language. To narrow the data gap, we propose\nan online self-training approach, which simultaneously uses the pseudo parallel\ndata {natural source, translated target} to mimic the inference scenario.\nExperimental results on several widely-used language pairs show that our\napproach outperforms two strong baselines (XLM and MASS) by remedying the style\nand content gaps.\n