Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks

This paper proposes a new mean-field framework for over-parameterized deep\nneural networks (DNNs), which can be used to analyze neural network training.\nIn this framework, a DNN is represented by probability measures and functions\nover its features (that is, the function values of the hidden units over the\ntraining data) in the continuous limit, instead of the neural network\nparameters as most existing studies have done. This new representation\novercomes the degenerate situation where all the hidden units essentially have\nonly one meaningful hidden unit in each middle layer, and further leads to a\nsimpler representation of DNNs, for which the training objective can be\nreformulated as a convex optimization problem via suitable re-parameterization.\nMoreover, we construct a non-linear dynamics called neural feature flow, which\ncaptures the evolution of an over-parameterized DNN trained by Gradient\nDescent. We illustrate the framework via the standard DNN and the Residual\nNetwork (Res-Net) architectures. Furthermore, we show, for Res-Net, when the\nneural feature flow process converges, it reaches a global minimal solution\nunder suitable conditions. Our analysis leads to the first global convergence\nproof for over-parameterized neural network training with more than $3$ layers\nin the mean-field regime.\n