Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Although pretrained language models can be fine-tuned to produce\nstate-of-the-art results for a very wide range of language understanding tasks,\nthe dynamics of this process are not well understood, especially in the low\ndata regime. Why can we use relatively vanilla gradient descent algorithms\n(e.g., without strong regularization) to tune a model with hundreds of millions\nof parameters on datasets with only hundreds or thousands of labeled examples?\nIn this paper, we argue that analyzing fine-tuning through the lens of\nintrinsic dimension provides us with empirical and theoretical intuitions to\nexplain this remarkable phenomenon. We empirically show that common pre-trained\nmodels have a very low intrinsic dimension; in other words, there exists a low\ndimension reparameterization that is as effective for fine-tuning as the full\nparameter space. For example, by optimizing only 200 trainable parameters\nrandomly projected back into the full space, we can tune a RoBERTa model to\nachieve 90\\% of the full parameter performance levels on MRPC. Furthermore, we\nempirically show that pre-training implicitly minimizes intrinsic dimension\nand, perhaps surprisingly, larger models tend to have lower intrinsic dimension\nafter a fixed number of pre-training updates, at least in part explaining their\nextreme effectiveness. Lastly, we connect intrinsic dimensionality with low\ndimensional task representations and compression based generalization bounds to\nprovide intrinsic-dimension-based generalization bounds that are independent of\nthe full parameter count.\n