When equipped with efficient optimization algorithms, the over-parameterized\nneural networks have demonstrated high level of performance even though the\nloss function is non-convex and non-smooth. While many works have been focusing\non understanding the loss dynamics by training neural networks with the\ngradient descent (GD), in this work, we consider a broad class of optimization\nalgorithms that are commonly used in practice. For example, we show from a\ndynamical system perspective that the Heavy Ball (HB) method can converge to\nglobal minimum on mean squared error (MSE) at a linear rate (similar to GD);\nhowever, the Nesterov accelerated gradient descent (NAG) may only converges to\nglobal minimum sublinearly.\n Our results rely on the connection between neural tangent kernel (NTK) and\nfinite over-parameterized neural networks with ReLU activation, which leads to\nanalyzing the limiting ordinary differential equations (ODE) for optimization\nalgorithms. We show that, optimizing the non-convex loss over the weights\ncorresponds to optimizing some strongly convex loss over the prediction error.\nAs a consequence, we can leverage the classical convex optimization theory to\nunderstand the convergence behavior of neural networks. We believe our approach\ncan also be extended to other optimization algorithms and network\narchitectures.\n