Convergence of continuous-time stochastic gradient descent with applications to deep neural networks

We study a continuous-time approximation of the stochastic gradient descent process for minimizing the population expected loss in learning problems. The main results establish general sufficient conditions for the convergence, extending the results of Chatterjee (2022) established for (nonstochastic) gradient descent. We show how the main result can be applied to the case of overparametrized neural network training.

cs.LG math.OC stat.ML Computer Science Mathematics

Paper

References (18)

01Stochastic Differential Equations models for Least-Squares Stochastic Gradient DescentAdrien Schertzer, Loucas Pillaud-Vivien2024 · arXiv.org · 2 citations In Library

02Convergence of stochastic gradient descent under a local Lajasiewicz condition for deep neural networksJing An, Jianfeng Lu2023 · arXiv.org · 6 citations In Library

03Type-II Saddles and Probabilistic Stability of Stochastic Gradient DescentLiu Ziyin, Botao Li, Tomer Galanti et al.2023 · 8 citations In Library

04From Gradient Flow on Population Loss to Learning with Stochastic Gradient DescentSatyen Kale, Jason D. Lee, Chris De Sa et al.2022 · Neural Information Processing Systems · 5 citations In Library

05Convergence of gradient descent for deep neural networksS. Chatterjee2022 · arXiv.org · 33 citations In Library

06Convergence of gradient descent for learning linear neural networksGabin Maxime Nguegnang, Holger Rauhut, Ulrich Terstiege2021 · Advances in Continuous and Discrete Models · 30 citations In Library

07Higher Order Generalization Error for First Order Discretization of Langevin DiffusionMufan Bill Li, M. Gazeau2021 · arXiv.org · 1 citation In Library

08Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU NetworksQuynh N. Nguyen, Marco Mondelli, Guido Montúfar2020 · International Conference on Machine Learning · 102 citations In Library

09Probability theoryDušan Teodorović, M. Nikolić2020 · Quantitative Methods in Transportation · 2.5k citations

10Loss landscapes and optimization in over-parameterized non-linear systems and neural networksChaoyue Liu, Libin Zhu, Mikhail Belkin2020 · Applied and Computational Harmonic Analysis · 359 citations In Library

11Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizersB. Bah, Holger Rauhut, Ulrich Terstiege et al.2019 · Information and Inference A Journal of the IMA · 86 citations In Library

12Stochastic Gradient and Langevin ProcessesXiang Cheng, Dong Yin, P. Bartlett et al.2019 · International Conference on Machine Learning · 55 citations

Scroll for more · 6 remaining

Similar papers

Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networksJing An, Jianfeng Lu2023 · arXiv (Cornell University) · 6 citations In Library

Convergence Analysis of Stochastic Gradient Descent with Adaptive Learning Rates: A Mathematical FrameworkSohail Ahmed Memon*, Imtiaz Ahmed Shar, Ghulam Muhammad2025 · Physical Education Health and Social Sciences In Library

Non-convergence of stochastic gradient descent in the training of deep neural networksPatrick Cheridito, Arnulf Jentzen, Florian Rossmannek2021 · Repository for Publications and Research Data (ETH Zurich) In Library

Non-convergence of stochastic gradient descent in the training of deep neural networksPatrick Cheridito, Arnulf Jentzen, Florian Rossmannek2020 · Journal of Complexity In Library

Non-convergence of stochastic gradient descent in the training of deep neural networksPatrick Cheridito, Arnulf Jentzen, Florian Rossmannek2020 · Journal of Complexity · 44 citations In Library

Analysis of Stochastic Gradient Descent in Continuous TimeJ. Latz2020 · Statistics and computing · 51 citations In Library

Novel Convergence Results of Adaptive Stochastic Gradient DescentsTao Sun, Linbo Qiao, Qing Liao et al.2020 · IEEE Transactions on Image Processing · 17 citations In Library

Convergence of Stochastic Gradient Descent under a Local Łojasiewicz Condition for Deep Neural NetworksJing An Jing An, Jianfeng Lu2025 · Journal of Machine Learning In Library

Analysis of stochastic gradient descent in continuous timeJonas Latz2021 · Statistics and Computing In Library

Stochastic Gradient Descent in Continuous Time: A Central Limit TheoremJustin Sirignano, Konstantinos Spiliopoulos2020 · Stochastic Systems In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal