Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan

Multilingual language models have been a crucial breakthrough as they\nconsiderably reduce the need of data for under-resourced languages.\nNevertheless, the superiority of language-specific models has already been\nproven for languages having access to large amounts of data. In this work, we\nfocus on Catalan with the aim to explore to what extent a medium-sized\nmonolingual language model is competitive with state-of-the-art large\nmultilingual models. For this, we: (1) build a clean, high-quality textual\nCatalan corpus (CaText), the largest to date (but only a fraction of the usual\nsize of the previous work in monolingual language models), (2) train a\nTransformer-based language model for Catalan (BERTa), and (3) devise a thorough\nevaluation in a diversity of settings, comprising a complete array of\ndownstream tasks, namely, Part of Speech Tagging, Named Entity Recognition and\nClassification, Text Classification, Question Answering, and Semantic Textual\nSimilarity, with most of the corresponding datasets being created ex novo. The\nresult is a new benchmark, the Catalan Language Understanding Benchmark (CLUB),\nwhich we publish as an open resource, together with the clean textual corpus,\nthe language model, and the cleaning pipeline. Using state-of-the-art\nmultilingual models and a monolingual model trained only on Wikipedia as\nbaselines, we consistently observe the superiority of our model across tasks\nand settings.\n