Gradient directions and relative inexactness in optimization and machine learning

In this paper, we investigate the influence of noise giving an estimate of the gradient having a acute angle with the original. Noise amplitude has a relative model. The work offers both theoretical calculations and theorems, as well as experimental results. Classic machine learning problems were chosen as experiments -- linear and logistic regression, computer vision and natural language processing.

Computer Science Mathematics

View graph Source

Paper

References (18)

01Intermediate Gradient Methods with Relative InexactnessNikita Kornilov, Eduard A. Gorbunov, M. Alkousa et al.2023 · Journal of Optimization Theory and Applications · 8 citations

02First Order Methods with Markovian Noise: from Acceleration to Variational InequalitiesA. Beznosikov, Sergey Samsonov, Marina Sheshukova et al.2023 · Neural Information Processing Systems · 25 citations In Library

03Accelerated gradient methods with absolute and relative noise in the gradientA. Vasin, A. Gasnikov, P. Dvurechensky et al.2021 · Optim. Methods Softw. · 18 citations In Library

04Minibatch vs Local SGD for Heterogeneous Distributed LearningBlake E. Woodworth, Kumar Kshitij Patel, N. Srebro2020 · Neural Information Processing Systems · 228 citations In Library

05Gradient-Free Methods with Inexact Oracle for Convex-Concave Stochastic Saddle-Point ProblemA. Beznosikov, Abdurakhmon Sadiev, A. Gasnikov2020 · Mathematical Optimization Theory and Operations Research · 25 citations In Library

06A frequency-domain analysis of inexact gradient methodsOran Gannot2019 · Mathematical programming · 31 citations In Library

07PyTorch: An Imperative Style, High-Performance Deep Learning LibraryAdam Paszke, Sam Gross, Francisco Massa et al.2019 · Neural Information Processing Systems · 54k citations In Library

08Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated PerceptronSharan Vaswani, F. Bach, Mark W. Schmidt2018 · International Conference on Artificial Intelligence and Statistics · 347 citations In Library

09Accelerated methods for weakly-quasi-convex optimization problemsSergey Guminov, A. Gasnikov, Ilya A. Kuruzov2017 · Computational Management Science · 24 citations

10Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex OptimizationPan Xu, Jinghui Chen, Quanquan Gu2017 · Neural Information Processing Systems · 238 citations In Library

11Stochastic Intermediate Gradient Method for Convex Problems with Stochastic Inexact OracleP. Dvurechensky, A. Gasnikov2016 · Journal of Optimization Theory and Applications · 88 citations

12Deep Residual Learning for Image RecognitionKaiming He, X. Zhang, Shaoqing Ren et al.2015 · Computer Vision and Pattern Recognition · 235k citations In Library

Scroll for more · 6 remaining

Similar papers

Accelerated gradient methods with absolute and relative noise in the gradientVasin Artem, Alexander Gasnikov, Pavel Dvurechensky et al.2021 · arXiv (Cornell University) In Library

From inexact optimization to learning via gradient concentrationBernhard Stankewitz, Nicole Mücke, Lorenzo Rosasco2021 · arXiv (Cornell University) In Library

Lower and upper bounds of the convergence rate of gradient methods with composite noise in gradientA.S. Vasin, Alexander Gasnikov2026 · arXiv (Cornell University) In Library

GRADIENT-BASED METHODS FOR NOISY QUADRATIC OPTIMIZATIONHuỳnh Trâm Bùi, Lam Thuan Le, Ba Dat Tran et al.2026 · Tạp chí Khoa học In Library

Accelerated and nonaccelerated stochastic gradient descent with inexact modelDarina Dvinskikh, Alexander Tyurin, Alexander Gasnikov et al.2020 · arXiv (Cornell University) · 0 citations In Library

Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise ChannelsHao Wang, Rui Gao, Flávio P. Calmon2021 · arXiv (Cornell University) · 21 citations In Library

Global Convergence of Noisy Gradient DescentX. Qin, Xin Xu, Xiaopeng Luo2022 · IEEE International Conference on Systems, Man and Cybernetics · 2 citations In Library

Modified Gauss-Newton Algorithms under NoiseKrishna Pillutla, Vincent Roulet, Sham M. Kakade et al.2023 In Library

Modified Gauss-Newton Algorithms under NoiseKrishna Pillutla, Vincent Roulet, Sham M. Kakade et al.2023 · arXiv (Cornell University) In Library

Hyperparameter optimization with approximate gradientFabian Pedregosa2016 · International Conference on Machine Learning · 532 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal