A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

The rapid recent progress in machine learning (ML) has raised a number of\nscientific questions that challenge the longstanding dogma of the field. One of\nthe most important riddles is the good empirical generalization of\noverparameterized models. Overparameterized models are excessively complex with\nrespect to the size of the training dataset, which results in them perfectly\nfitting (i.e., interpolating) the training data, which is usually noisy. Such\ninterpolation of noisy data is traditionally associated with detrimental\noverfitting, and yet a wide range of interpolating models -- from simple linear\nmodels to deep neural networks -- have recently been observed to generalize\nextremely well on fresh test data. Indeed, the recently discovered double\ndescent phenomenon has revealed that highly overparameterized models often\nimprove over the best underparameterized model in test performance.\n Understanding learning in this overparameterized regime requires new theory\nand foundational empirical studies, even for the simplest case of the linear\nmodel. The underpinnings of this understanding have been laid in very recent\nanalyses of overparameterized linear regression and related statistical\nlearning tasks, which resulted in precise analytic characterizations of double\ndescent. This paper provides a succinct overview of this emerging theory of\noverparameterized ML (henceforth abbreviated as TOPML) that explains these\nrecent findings through a statistical signal processing perspective. We\nemphasize the unique aspects that define the TOPML research area as a subfield\nof modern ML theory and outline interesting open questions that remain.\n