Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

Fine-tuning pretrained contextual word embedding models to supervised\ndownstream tasks has become commonplace in natural language processing. This\nprocess, however, is often brittle: even with the same hyperparameter values,\ndistinct random seeds can lead to substantially different results. To better\nunderstand this phenomenon, we experiment with four datasets from the GLUE\nbenchmark, fine-tuning BERT hundreds of times on each while varying only the\nrandom seeds. We find substantial performance increases compared to previously\nreported results, and we quantify how the performance of the best-found model\nvaries as a function of the number of fine-tuning trials. Further, we examine\ntwo factors influenced by the choice of random seed: weight initialization and\ntraining data order. We find that both contribute comparably to the variance of\nout-of-sample performance, and that some weight initializations perform well\nacross all tasks explored. On small datasets, we observe that many fine-tuning\ntrials diverge part of the way through training, and we offer best practices\nfor practitioners to stop training less promising runs early. We publicly\nrelease all of our experimental data, including training and validation scores\nfor 2,100 trials, to encourage further analysis of training dynamics during\nfine-tuning.\n