Adaptive scaling of the learning rate by second order automatic differentiation

In the context of the optimization of Deep Neural Networks, we propose to\nrescale the learning rate using a new technique of automatic differentiation.\nThis technique relies on the computation of the {\\em curvature}, a second order\ninformation whose computational complexity is in between the computation of the\ngradient and the one of the Hessian-vector product. If (1C,1M) represents\nrespectively the computational time and memory footprint of the gradient\nmethod, the new technique increase the overall cost to either (1.5C,2M) or\n(2C,1M). This rescaling has the appealing characteristic of having a natural\ninterpretation, it allows the practitioner to choose between exploration of the\nparameters set and convergence of the algorithm. The rescaling is adaptive, it\ndepends on the data and on the direction of descent. The numerical experiments\nhighlight the different exploration/convergence regimes.\n