Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Annotating training data for sequence tagging of texts is usually very\ntime-consuming. Recent advances in transfer learning for natural language\nprocessing in conjunction with active learning open the possibility to\nsignificantly reduce the necessary annotation budget. We are the first to\nthoroughly investigate this powerful combination for the sequence tagging task.\nWe conduct an extensive empirical study of various Bayesian uncertainty\nestimation methods and Monte Carlo dropout options for deep pre-trained models\nin the active learning framework and find the best combinations for different\ntypes of models. Besides, we also demonstrate that to acquire instances during\nactive learning, a full-size Transformer can be substituted with a distilled\nversion, which yields better computational performance and reduces obstacles\nfor applying deep active learning in practice.\n