Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

Multilingual transformer models like mBERT and XLM-RoBERTa have obtained\ngreat improvements for many NLP tasks on a variety of languages. However,\nrecent works also showed that results from high-resource languages could not be\neasily transferred to realistic, low-resource scenarios. In this work, we study\ntrends in performance for different amounts of available resources for the\nthree African languages Hausa, isiXhosa and Yor\\`ub\\'a on both NER and topic\nclassification. We show that in combination with transfer learning or distant\nsupervision, these models can achieve with as little as 10 or 100 labeled\nsentences the same performance as baselines with much more supervised training\ndata. However, we also find settings where this does not hold. Our discussions\nand additional experiments on assumptions such as time and hardware\nrestrictions highlight challenges and opportunities in low-resource learning.\n