Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

We study the dynamics of optimization and the generalization properties of\none-hidden layer neural networks with quadratic activation function in the\nover-parametrized regime where the layer width $m$ is larger than the input\ndimension $d$.\n We consider a teacher-student scenario where the teacher has the same\nstructure as the student with a hidden layer of smaller width $m^*\\le m$.\n We describe how the empirical loss landscape is affected by the number $n$ of\ndata samples and the width $m^*$ of the teacher network. In particular we\ndetermine how the probability that there be no spurious minima on the empirical\nloss depends on $n$, $d$, and $m^*$, thereby establishing conditions under\nwhich the neural network can in principle recover the teacher.\n We also show that under the same conditions gradient descent dynamics on the\nempirical loss converges and leads to small generalization error, i.e. it\nenables recovery in practice.\n Finally we characterize the time-convergence rate of gradient descent in the\nlimit of a large number of samples.\n These results are confirmed by numerical experiments.\n