Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning

Recently, leveraging pre-trained Transformer based language models in down\nstream, task specific models has advanced state of the art results in natural\nlanguage understanding tasks. However, only a little research has explored the\nsuitability of this approach in low resource settings with less than 1,000\ntraining data points. In this work, we explore fine-tuning methods of BERT -- a\npre-trained Transformer based language model -- by utilizing pool-based active\nlearning to speed up training while keeping the cost of labeling new data\nconstant. Our experimental results on the GLUE data set show an advantage in\nmodel performance by maximizing the approximate knowledge gain of the model\nwhen querying from the pool of unlabeled data. Finally, we demonstrate and\nanalyze the benefits of freezing layers of the language model during\nfine-tuning to reduce the number of trainable parameters, making it more\nsuitable for low-resource settings.\n