Library

Training Linear Neural Networks: Non-Local Convergence and Complexity\n Results

Linear networks provide valuable insights into the workings of neural\nnetworks in general. This paper identifies conditions under which the gradient\nflow provably trains a linear network, in spite of the non-strict saddle points\npresent in the optimization landscape. This paper also provides the\ncomputational complexity of training linear networks with gradient flow. To\nachieve these results, this work develops a machinery to provably identify the\nstable set of gradient flow, which then enables us to improve over the state of\nthe art in the literature of linear networks (Bah et al., 2019;Arora et al.,\n2018a). Crucially, our results appear to be the first to break away from the\nlazy training regime which has dominated the literature of neural networks.\nThis work requires the network to have a layer with one neuron, which subsumes\nthe networks with a scalar output, but extending the results of this\ntheoretical work to all linear networks remains a challenging open problem.\n

Neural Networks and ApplicationsMachine Learning and ELMStochastic Gradient Optimization TechniquesComputer ScienceMathematics
Source

Paper

References (80)

01Singular Value DecompositionK. Kanatani2021 · Encyclopedia of Social Network Analysis and Mining · 281 citations
02Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear NetworksWei Hu, Lechao Xiao, Jeffrey Pennington2020 · International Conference on Learning Representations · 139 citations In Library
03How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?Zixiang Chen, Yuan Cao, Difan Zou et al.2019 · International Conference on Learning Representations · 138 citations In Library
04Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizersB. Bah, Holger Rauhut, Ulrich Terstiege et al.2019 · Information and Inference A Journal of the IMA · 86 citations In Library
05Pure and Spurious Critical Points: a Geometric Study of Linear NetworksMatthew Trager, Kathlén Kohn, Joan Bruna2019 · International Conference on Learning Representations · 45 citations In Library
06TRAINABILITY OF ReLU NETWORKS AND DATA-DEPENDENT INITIALIZATIONYeonjong Shin, G. Karniadakis2019 · Journal of Machine Learning for Modeling and Computing · 30 citations
07Trainability and Data-dependent Initialization of Over-parameterized ReLU Neural NetworksYeonjong Shin, G. Karniadakis2019 · arXiv.org · 9 citations In Library
08Implicit Regularization in Deep Matrix FactorizationSanjeev Arora, Nadav Cohen, Wei Hu et al.2019 · Neural Information Processing Systems · 639 citations In Library
09Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural NetworksYuan Cao, Quanquan Gu2019 · Neural Information Processing Systems · 434 citations In Library
10Fast Convergence of Natural Gradient Descent for Overparameterized Neural NetworksGuodong Zhang, James Martens, R. Grosse2019 · Neural Information Processing Systems · 139 citations In Library
11On Learning Over-parameterized Neural Networks: A Functional Approximation ProspectiveLili Su, Pengkun Yang2019 · Neural Information Processing Systems · 62 citations In Library
12On the Power and Limitations of Random Features for Understanding Neural NetworksGilad Yehudai, Ohad Shamir2019 · Neural Information Processing Systems · 196 citations In Library

Scroll for more · 38 remaining

Similar papers

A Non-local Convergence Analysis of Gradient Flow for Deep Linear NetworksKun Chen, Dachao Lin, Zhihua Zhang2026 · IEEE Transactions on Pattern Analysis and Machine Intelligence · 0 citations In Library
Global Convergence Analysis of Deep Linear Networks with A One-neuron LayerKun Chen, Dachao Lin, Zhihua Zhang2022 · arXiv (Cornell University) In Library
Gradient Flow Convergence Guarantee for General Neural Network ArchitecturesYash Jakhmola2025 · arXiv.org · 0 citations In Library
A Unifying View on Implicit Bias in Training Linear Neural NetworksChulhee Yun, Shankar Krishnan, Hossein Mobahi2020 · arXiv (Cornell University) · 95 citations In Library
Convergence of gradient descent for learning linear neural networksGabin Maxime Nguegnang, Holger Rauhut, Ulrich Terstiege2024 · Advances in Continuous and Discrete Models In Library
Convergence of gradient descent for learning linear neural networksGabin Maxime Nguegnang, Holger Rauhut, Ulrich Terstiege2021 · arXiv (Cornell University) · 30 citations In Library
Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizersBubacarr Bah, Holger Rauhut, Ulrich Terstiege et al.2021 · Information and Inference A Journal of the IMA In Library
Convergence of gradient based training for linear Graph Neural NetworksDhiraj Patel, Anton Savostianov, Michael T. Schaub2025 · arXiv (Cornell University) · 1 citation In Library
Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizersB. Bah, Holger Rauhut, Ulrich Terstiege et al.2019 · Information and Inference A Journal of the IMA · 85 citations In Library
A geometric approach of gradient descent algorithms in linear neural networksYacine Chitour, Laboratoire des Signaux et Systèmes, CentraleSupélec, Université Paris-Saclay, France, Zhenyu Liao et al.2022 · Mathematical Control and Related Fields In Library
© 2026 NYSGPT2525 LLC