The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding

While recent benchmarks have spurred a lot of new work on improving the\ngeneralization of pretrained multilingual language models on multilingual\ntasks, techniques to improve code-switched natural language understanding tasks\nhave been far less explored. In this work, we propose the use of bilingual\nintermediate pretraining as a reliable technique to derive large and consistent\nperformance gains on three different NLP tasks using code-switched text. We\nachieve substantial absolute improvements of 7.87%, 20.15%, and 10.99%, on the\nmean accuracies and F1 scores over previous state-of-the-art systems for\nHindi-English Natural Language Inference (NLI), Question Answering (QA) tasks,\nand Spanish-English Sentiment Analysis (SA) respectively. We show consistent\nperformance gains on four different code-switched language-pairs\n(Hindi-English, Spanish-English, Tamil-English and Malayalam-English) for SA.\nWe also present a code-switched masked language modelling (MLM) pretraining\ntechnique that consistently benefits SA compared to standard MLM pretraining\nusing real code-switched text.\n