Global Convergence of Sobolev Training for Overparameterized Neural Networks

Sobolev loss is used when training a network to approximate the values and\nderivatives of a target function at a prescribed set of input points. Recent\nworks have demonstrated its successful applications in various tasks such as\ndistillation or synthetic gradient prediction. In this work we prove that an\noverparameterized two-layer relu neural network trained on the Sobolev loss\nwith gradient flow from random initialization can fit any given function values\nand any given directional derivatives, under a separation condition on the\ninput data.\n