Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously

Generalization is a central problem in Machine Learning. Most prediction\nmethods require careful calibration of hyperparameters carried out on a\nhold-out \\textit{validation} dataset to achieve generalization. The main goal\nof this paper is to present a novel approach based on a new measure of risk\nthat allows us to develop novel fully automatic procedures for generalization.\nWe illustrate the pertinence of this new framework in the regression problem.\nThe main advantages of this new approach are: (i) it can simultaneously train\nthe model and perform regularization in a single run of a gradient-based\noptimizer on all available data without any previous hyperparameter tuning;\n(ii) this framework can tackle several additional objectives simultaneously\n(correlation, sparsity,...) $via$ the introduction of regularization\nparameters. Noticeably, our approach transforms hyperparameter tuning as well\nas feature selection (a combinatorial discrete optimization problem) into a\ncontinuous optimization problem that is solvable via classical gradient-based\nmethods ; (iii) the computational complexity of our methods is $O(npK)$ where\n$n,p,K$ denote respectively the number of observations, features and iterations\nof the gradient descent algorithm. We observe in our experiments a\nsignificantly smaller runtime for our methods as compared to benchmark methods\nfor equivalent prediction score. Our procedures are implemented in PyTorch\n(code is available for replication).\n