Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation

Scarcity of parallel sentence-pairs poses a significant hurdle for training\nhigh-quality Neural Machine Translation (NMT) models in bilingually\nlow-resource scenarios. A standard approach is transfer learning, which\ninvolves taking a model trained on a high-resource language-pair and\nfine-tuning it on the data of the low-resource MT condition of interest.\nHowever, it is not clear generally which high-resource language-pair offers the\nbest transfer learning for the target MT setting. Furthermore, different\ntransferred models may have complementary semantic and/or syntactic strengths,\nhence using only one model may be sub-optimal. In this paper, we tackle this\nproblem using knowledge distillation, where we propose to distill the knowledge\nof ensemble of teacher models to a single student model. As the quality of\nthese teacher models varies, we propose an effective adaptive knowledge\ndistillation approach to dynamically adjust the contribution of the teacher\nmodels during the distillation process. Experiments on transferring from a\ncollection of six language pairs from IWSLT to five low-resource language-pairs\nfrom TED Talks demonstrate the effectiveness of our approach, achieving up to\n+0.9 BLEU score improvement compared to strong baselines.\n