To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

While most previous work has focused on different pretraining objectives and\narchitectures for transfer learning, we ask how to best adapt the pretrained\nmodel to a given target task. We focus on the two most common forms of\nadaptation, feature extraction (where the pretrained weights are frozen), and\ndirectly fine-tuning the pretrained model. Our empirical results across diverse\nNLP tasks with two state-of-the-art models show that the relative performance\nof fine-tuning vs. feature extraction depends on the similarity of the\npretraining and target tasks. We explore possible explanations for this finding\nand provide a set of adaptation guidelines for the NLP practitioner.\n