Stepsize Hedging: an Alternative Mechanism for Accelerating Gradient Descent

Can gradient descent be accelerated by just choosing better stepsizes? Surprisingly, the answer is yes. This short expository article provides an accessible introduction to this phenomenon of stepsize hedging.

Mathematics Computer Science

View graph Source

Paper

References (52)

01Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low DimensionM. Crawshaw, Mingrui Liu2026 · arXiv.org · 1 citation In Library

02Generalization of Silver Stepsize Schedule to Stochastic OptimizationLuwei Bai, Yang Zeng, Baoyu Zhou2025 · 1 citation In Library

03Min-Max Optimization Is Strictly Easier Than Variational InequalitiesHenry Shugart, Jason M. Altschuler2025 · arXiv.org · 2 citations In Library

04On the convergence rate of the Douglas-Rachford splitting algorithmHadi Abbaszadehpeivasti, M. Zamani2025 · 1 citation

05Learning Acceleration Algorithms for Fast Parametric Convex Optimization with Certified RobustnessRajiv Sambharya, Jinho Bok, Nikolai Matni et al.2025 · 5 citations

06Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?S. Meng, Baptiste Goujaud, Antonio Orvieto et al.2025 · arXiv.org · 4 citations In Library

07Optimized methods for composite optimization: a reduction perspectiveJinho Bok, Jason M. Altschuler2025 · Mathematical programming · 4 citations

08Constant Stepsize Local GD for Logistic Regression: Acceleration by InstabilityM. Crawshaw, Blake Woodworth, Mingrui Liu2025 · International Conference on Machine Learning · 2 citations In Library

09Empirical and computer-aided robustness analysis of long-step and accelerated methods in smooth convex optimizationPierre Vernimmen, Franccois Glineur2025 · arXiv.org · 4 citations In Library

10Acceleration via silver step-size on Riemannian manifolds with applications to Wasserstein spaceJiyoung Park, A. Roy, Jonathan W. Siegel et al.2025 · 3 citations In Library

11Large Stepsizes Accelerate Gradient Descent for Regularized Logistic RegressionJingfeng Wu, Pierre Marion, Peter L. Bartlett2025 · Neural Information Processing Systems · 7 citations In Library

12Negative Stepsizes Make Gradient-Descent-Ascent ConvergeHenry Shugart, Jason M. Altschuler2025 · arXiv.org · 9 citations In Library

Scroll for more · 38 remaining

Similar papers

Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize ScheduleJason M. Altschuler, Pablo A. Parrilo2023 · arXiv (Cornell University) In Library

Accelerating Proximal Gradient Descent via Silver StepsizesJinho Bok, Jason M. Altschuler2024 · Annual Conference Computational Learning Theory · 9 citations In Library

Learning to Accelerate by the Methods of Step-size PlanningHengshuai Yao2022 · arXiv (Cornell University) In Library

Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize ScalingYu-Guan Hsieh, Franck Iutzeler, Jérôme Malick et al.2020 · arXiv (Cornell University) · 80 citations In Library

Provably Faster Gradient Descent via Long StepsBenjamin Grimmer2023 · arXiv (Cornell University) · 55 citations In Library

Improved Gradient Descent Optimization Using Adaptive Step-SizingDivya Sharma, Vikram Sethi2023 In Library

On Kahan's automatic step-size control and an anadromic gradient descent iterationZirui Gu, Kexin Jin, Yifeng Meng et al.2023 · Optimization In Library

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long StepsBenjamin Grimmer, Kevin Shu, Alex L. Wang2024 · INFORMS Journal on Optimization · 26 citations In Library

Accelerated Gradient Descent via Long StepsBenjamin Grimmer, Kevin Shu, Alex L. Wang2023 · arXiv (Cornell University) · 25 citations In Library

Negative Stepsizes Make Gradient-Descent-Ascent ConvergeHenry Shugart, Jason M. Altschuler2025 · arXiv (Cornell University) · 9 citations In Library

Patents near this paper

SYSTEM AND METHOD FOR INCREASING EFFICIENCY OF GRADIENT DESCENT WHILE TRAINING MACHINE-LEARNING MODELSUS 11,126,893 · INTUIT, INC. · 2021 SYSTEM AND METHOD FOR INCREASING EFFICIENCY OF GRADIENT DESCENT WHILE TRAINING MACHINE-LEARNING MODELSUS 11,763,151 · APPLIED MATERIALS, INC. · 2023 SYSTEMS AND METHODS USING ANGLE-BASED STOCHASTIC GRADIENT DESCENTUS 11,347,997 · THE FLORIDA INTERNATIONAL UNIVERSITY BOARD OF TRUSTEES · 2022 ASYNCHRONOUS STOCHASTIC GRADIENT DESCENTUS 10,628,740 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2020 ACCELERATING DEEP NEURAL NETWORK TRAINING WITH INCONSISTENT STOCHASTIC GRADIENT DESCENTUS 10,572,800 · NEC LABORATORIES AMERICA, INC. · 2020

Source of record

arXiv