Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation

Self-supervised neural machine translation (SSNMT) jointly learns to identify\nand select suitable training data from comparable (rather than parallel)\ncorpora and to translate, in a way that the two tasks support each other in a\nvirtuous circle. In this study, we provide an in-depth analysis of the sampling\nchoices the SSNMT model makes during training. We show how, without it having\nbeen told to do so, the model self-selects samples of increasing (i) complexity\nand (ii) task-relevance in combination with (iii) performing a denoising\ncurriculum. We observe that the dynamics of the mutual-supervision signals of\nboth system internal representation types are vital for the extraction and\ntranslation performance. We show that in terms of the Gunning-Fog Readability\nindex, SSNMT starts extracting and learning from Wikipedia data suitable for\nhigh school students and quickly moves towards content suitable for first year\nundergraduate students.\n