Following early work on Hessian-free methods for deep learning, we study a\nstochastic generalized Gauss-Newton method (SGN) for training DNNs. SGN is a\nsecond-order optimization method, with efficient iterations, that we\ndemonstrate to often require substantially fewer iterations than standard SGD\nto converge. As the name suggests, SGN uses a Gauss-Newton approximation for\nthe Hessian matrix, and, in order to compute an approximate search direction,\nrelies on the conjugate gradient method combined with forward and reverse\nautomatic differentiation. Despite the success of SGD and its first-order\nvariants, and despite Hessian-free methods based on the Gauss-Newton Hessian\napproximation having been already theoretically proposed as practical methods\nfor training DNNs, we believe that SGN has a lot of undiscovered and yet not\nfully displayed potential in big mini-batch scenarios. For this setting, we\ndemonstrate that SGN does not only substantially improve over SGD in terms of\nthe number of iterations, but also in terms of runtime. This is made possible\nby an efficient, easy-to-use and flexible implementation of SGN we propose in\nthe Theano deep learning platform, which, unlike Tensorflow and Pytorch,\nsupports forward automatic differentiation. This enables researchers to further\nstudy and improve this promising optimization technique and hopefully\nreconsider stochastic second-order methods as competitive optimization\ntechniques for training DNNs; we also hope that the promise of SGN may lead to\nforward automatic differentiation being added to Tensorflow or Pytorch. Our\nresults also show that in big mini-batch scenarios SGN is more robust than SGD\nwith respect to its hyperparameters (we never had to tune its step-size for our\nbenchmarks!), which eases the expensive process of hyperparameter tuning that\nis instead crucial for the performance of first-order methods.\n