Generative Pre-trained Transformers (GPTs) have recently been scaled to\nunprecedented sizes in the history of machine learning. These models, solely\ntrained on the language modeling objective, have been shown to exhibit\noutstanding few-shot learning capabilities in a number of different tasks.\nNevertheless, aside from anecdotal experiences, little is known regarding their\nmultilingual capabilities, given the fact that the pre-training corpus is\nalmost entirely composed of English text. In this work, we investigate the\nmultilingual skills of GPT-3, focusing on one language that barely appears in\nthe pre-training corpus, Catalan, which makes the results especially\nmeaningful; we assume that our results may be relevant for other languages as\nwell. We find that the model shows an outstanding performance, particularly in\ngenerative tasks, with predictable limitations mostly in language understanding\ntasks but still with remarkable results given the zero-shot scenario. We\ninvestigate its potential and limits in extractive question-answering and\nnatural language generation, as well as the effect of scale in terms of model\nsize.\n