Deep Learning via Neural Energy Descent

This paper proposes a method called Neural Energy Descent (NED) via neural network (NN) evolution equations for a wide class of deep learning (DL) problems. We demonstrate that the training process of DL can be reformulated as the evolution of the network parameters governed by a partial differential equation (PDE) where the steady-state solution yields a solution for DL. This equation corresponds to a gradient descent flow of a variational problem and hence the proposed PDE solves an energy minimization problem to obtain a global minimizer of DL. This gives a novel interpretation and solution for DL optimization. The computational complexity of the proposed energy descent method can be enhanced by randomly sampling the spatial domain of the PDE. Numerical examples are provided to demonstrate the numerical advantages of NED over stochastic gradient descent (SGD) method and Adam.