English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too

Intermediate-task training---fine-tuning a pretrained model on an\nintermediate task before fine-tuning again on the target task---often improves\nmodel performance substantially on language understanding tasks in monolingual\nEnglish settings. We investigate whether English intermediate-task training is\nstill helpful on non-English target tasks. Using nine intermediate\nlanguage-understanding tasks, we evaluate intermediate-task transfer in a\nzero-shot cross-lingual setting on the XTREME benchmark. We see large\nimprovements from intermediate training on the BUCC and Tatoeba sentence\nretrieval tasks and moderate improvements on question-answering target tasks.\nMNLI, SQuAD and HellaSwag achieve the best overall results as intermediate\ntasks, while multi-task intermediate offers small additional improvements.\nUsing our best intermediate-task models for each target task, we obtain a 5.4\npoint improvement over XLM-R Large on the XTREME benchmark, setting the state\nof the art as of June 2020. We also investigate continuing multilingual MLM\nduring intermediate-task training and using machine-translated\nintermediate-task data, but neither consistently outperforms simply performing\nEnglish intermediate-task training.\n