Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

Self-supervised pre-training of transformer models has revolutionized NLP\napplications. Such pre-training with language modeling objectives provides a\nuseful initial point for parameters that generalize well to new tasks with\nfine-tuning. However, fine-tuning is still data inefficient -- when there are\nfew labeled examples, accuracy can be low. Data efficiency can be improved by\noptimizing pre-training directly for future fine-tuning with few examples; this\ncan be treated as a meta-learning problem. However, standard meta-learning\ntechniques require many training tasks in order to generalize; unfortunately,\nfinding a diverse set of such supervised tasks is usually difficult. This paper\nproposes a self-supervised approach to generate a large, rich, meta-learning\ntask distribution from unlabeled text. This is achieved using a cloze-style\nobjective, but creating separate multi-class classification tasks by gathering\ntokens-to-be blanked from among only a handful of vocabulary terms. This yields\nas many unique meta-training tasks as the number of subsets of vocabulary\nterms. We meta-train a transformer model on this distribution of tasks using a\nrecent meta-learning framework. On 17 NLP tasks, we show that this\nmeta-training leads to better few-shot generalization than language-model\npre-training followed by finetuning. Furthermore, we show how the\nself-supervised tasks can be combined with supervised tasks for meta-learning,\nproviding substantial accuracy gains over previous supervised meta-learning.\n