Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation

Neural networks tend to gradually forget the previously learned knowledge\nwhen learning multiple tasks sequentially from dynamic data distributions. This\nproblem is called \\textit{catastrophic forgetting}, which is a fundamental\nchallenge in the continual learning of neural networks. In this work, we\nobserve that catastrophic forgetting not only occurs in continual learning but\nalso affects the traditional static training. Neural networks, especially\nneural machine translation models, suffer from catastrophic forgetting even if\nthey learn from a static training set. To be specific, the final model pays\nimbalanced attention to training samples, where recently exposed samples\nattract more attention than earlier samples. The underlying cause is that\ntraining samples do not get balanced training in each model update, so we name\nthis problem \\textit{imbalanced training}. To alleviate this problem, we\npropose Complementary Online Knowledge Distillation (COKD), which uses\ndynamically updated teacher models trained on specific data orders to\niteratively provide complementary knowledge to the student model. Experimental\nresults on multiple machine translation tasks show that our method successfully\nalleviates the problem of imbalanced training and achieves substantial\nimprovements over strong baseline systems.\n