On the ability of monolingual models to learn language-agnostic representations

Pretrained multilingual models have become a de facto default approach for\nzero-shot cross-lingual transfer. Previous work has shown that these models are\nable to achieve cross-lingual representations when pretrained on two or more\nlanguages with shared parameters. In this work, we provide evidence that a\nmodel can achieve language-agnostic representations even when pretrained on a\nsingle language. That is, we find that monolingual models pretrained and\nfinetuned on different languages achieve competitive performance compared to\nthe ones that use the same target language. Surprisingly, the models show a\nsimilar performance on a same task regardless of the pretraining language. For\nexample, models pretrained on distant languages such as German and Portuguese\nperform similarly on English tasks.\n