On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting

Deep learning empirically achieves high performance in many applications, but\nits training dynamics has not been fully understood theoretically. In this\npaper, we explore theoretical analysis on training two-layer ReLU neural\nnetworks in a teacher-student regression model, in which a student network\nlearns an unknown teacher network through its outputs. We show that with a\nspecific regularization and sufficient over-parameterization, the student\nnetwork can identify the parameters of the teacher network with high\nprobability via gradient descent with a norm dependent stepsize even though the\nobjective function is highly non-convex. The key theoretical tool is the\nmeasure representation of the neural networks and a novel application of a dual\ncertificate argument for sparse estimation on a measure space. We analyze the\nglobal minima and global convergence property in the measure space.\n