NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21

This paper provides an overview of NVIDIA NeMo's neural machine translation\nsystems for the constrained data track of the WMT21 News and Biomedical Shared\nTranslation Tasks. Our news task submissions for English-German (En-De) and\nEnglish-Russian (En-Ru) are built on top of a baseline transformer-based\nsequence-to-sequence model. Specifically, we use a combination of 1) checkpoint\naveraging 2) model scaling 3) data augmentation with backtranslation and\nknowledge distillation from right-to-left factorized models 4) finetuning on\ntest sets from previous years 5) model ensembling 6) shallow fusion decoding\nwith transformer language models and 7) noisy channel re-ranking. Additionally,\nour biomedical task submission for English-Russian uses a biomedically biased\nvocabulary and is trained from scratch on news task data, medically relevant\ntext curated from the news task dataset, and biomedical data provided by the\nshared task. Our news system achieves a sacreBLEU score of 39.5 on the WMT'20\nEn-De test set outperforming the best submission from last year's task of 38.8.\nOur biomedical task Ru-En and En-Ru systems reach BLEU scores of 43.8 and 40.3\nrespectively on the WMT'20 Biomedical Task Test set, outperforming the previous\nyear's best submissions.\n