It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

When scaled to hundreds of billions of parameters, pretrained language models\nsuch as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance.\nHowever, enormous amounts of compute are required for training and applying\nsuch big models, resulting in a large carbon footprint and making it difficult\nfor researchers and practitioners to use them. We show that performance similar\nto GPT-3 can be obtained with language models that are much "greener" in that\ntheir parameter count is several orders of magnitude smaller. This is achieved\nby converting textual inputs into cloze questions that contain a task\ndescription, combined with gradient-based optimization; exploiting unlabeled\ndata gives further improvements. We identify key factors required for\nsuccessful natural language understanding with small language models.\n