XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

This paper presents XLS-R, a large-scale model for cross-lingual speech\nrepresentation learning based on wav2vec 2.0. We train models with up to 2B\nparameters on nearly half a million hours of publicly available speech audio in\n128 languages, an order of magnitude more public data than the largest known\nprior work. Our evaluation covers a wide range of tasks, domains, data regimes\nand languages, both high and low-resource. On the CoVoST-2 speech translation\nbenchmark, we improve the previous state of the art by an average of 7.4 BLEU\nover 21 translation directions into English. For speech recognition, XLS-R\nimproves over the best known prior work on BABEL, MLS, CommonVoice as well as\nVoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets\na new state of the art on VoxLingua107 language identification. Moreover, we\nshow that with sufficient model size, cross-lingual pretraining can outperform\nEnglish-only pretraining when translating English speech into other languages,\na setting which favors monolingual pretraining. We hope XLS-R can help to\nimprove speech processing tasks for many more languages of the world.\n