Stochastic Learning Rate Optimization in the Stochastic Approximation and Online Learning Settings

In this work, multiplicative stochasticity is applied to the learning rate of\nstochastic optimization algorithms, giving rise to stochastic learning-rate\nschemes. In-expectation theoretical convergence results of Stochastic Gradient\nDescent equipped with this novel stochastic learning rate scheme under the\nstochastic setting, as well as convergence results under the online\noptimization settings are provided. Empirical results consider the case of an\nadaptively uniformly distributed multiplicative stochasticity and include not\nonly Stochastic Gradient Descent, but also other popular algorithms equipped\nwith a stochastic learning rate. They demonstrate noticeable optimization\nperformance gains, with respect to their deterministic-learning-rate versions.\n