Resolving learning rates adaptively by locating Stochastic Non-Negative Associated Gradient Projection Points using line searches

Learning rates in stochastic neural network training are currently determined\na priori to training, using expensive manual or automated iterative tuning.\nThis study proposes gradient-only line searches to resolve the learning rate\nfor neural network training algorithms. Stochastic sub-sampling during training\ndecreases computational cost and allows the optimization algorithms to progress\nover local minima. However, it also results in discontinuous cost functions.\nMinimization line searches are not effective in this context, as they use a\nvanishing derivative (first order optimality condition), which often do not\nexist in a discontinuous cost function and therefore converge to\ndiscontinuities as opposed to minima from the data trends. Instead, we base\ncandidate solutions along a search direction purely on gradient information, in\nparticular by a directional derivative sign change from negative to positive (a\nNon-negative Associative Gradient Projection Point (NN- GPP)). Only considering\na sign change from negative to positive always indicates a minimum, thus\nNN-GPPs contain second order information. Conversely, a vanishing gradient is\npurely a first order condition, which may indicate a minimum, maximum or saddle\npoint. This insight allows the learning rate of an algorithm to be reliably\nresolved as the step size along a search direction, increasing convergence\nperformance and eliminating an otherwise expensive hyperparameter.\n