A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions

Gradient descent (GD) type optimization methods are the standard instrument\nto train artificial neural networks (ANNs) with rectified linear unit (ReLU)\nactivation. Despite the great success of GD type optimization methods in\nnumerical simulations for the training of ANNs with ReLU activation, it remains\n- even in the simplest situation of the plain vanilla GD optimization method\nwith random initializations and ANNs with one hidden layer - an open problem to\nprove (or disprove) the conjecture that the risk of the GD optimization method\nconverges in the training of such ANNs to zero as the width of the ANNs, the\nnumber of independent random initializations, and the number of GD steps\nincrease to infinity. In this article we prove this conjecture in the situation\nwhere the probability distribution of the input data is equivalent to the\ncontinuous uniform distribution on a compact interval, where the probability\ndistributions for the random initializations of the ANN parameters are standard\nnormal distributions, and where the target function under consideration is\ncontinuous and piecewise affine linear. Roughly speaking, the key ingredients\nin our mathematical convergence analysis are (i) to prove that suitable sets of\nglobal minima of the risk functions are \\emph{twice continuously differentiable\nsubmanifolds of the ANN parameter spaces}, (ii) to prove that the Hessians of\nthe risk functions on these sets of global minima satisfy an appropriate\n\\emph{maximal rank condition}, and, thereafter, (iii) to apply the machinery in\n[Fehrman, B., Gess, B., Jentzen, A., Convergence rates for the stochastic\ngradient descent method for non-convex objective functions. J. Mach. Learn.\nRes. 21(136): 1--48, 2020] to establish convergence of the GD optimization\nmethod with random initializations.\n