How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

In this work, we provide a systematic and comprehensive empirical comparison\nof pretrained multilingual language models versus their monolingual\ncounterparts with regard to their monolingual task performance. We study a set\nof nine typologically diverse languages with readily available pretrained\nmonolingual models on a set of five diverse monolingual downstream tasks. We\nfirst aim to establish, via fair and controlled comparisons, if a gap between\nthe multilingual and the corresponding monolingual representation of that\nlanguage exists, and subsequently investigate the reason for any performance\ndifference. To disentangle conflating factors, we train new monolingual models\non the same data, with monolingually and multilingually trained tokenizers. We\nfind that while the pretraining data size is an important factor, a designated\nmonolingual tokenizer plays an equally important role in the downstream\nperformance. Our results show that languages that are adequately represented in\nthe multilingual model's vocabulary exhibit negligible performance decreases\nover their monolingual counterparts. We further find that replacing the\noriginal multilingual tokenizer with the specialized monolingual tokenizer\nimproves the downstream performance of the multilingual model for almost every\ntask and language.\n