A Neural Network model with Bidirectional Whitening

We present here a new model and algorithm which performs an efficient Natural gradient descent for multilayer perceptrons. Natural gradient descent was originally proposed from a point of view of information geometry, and it performs the steepest descent updates on manifolds in a Riemannian space. In particular, we extend an approach taken by the “Whitened Neural Networks” model. We make the whitening process not only in the feed-forward direction as in the original model, but also in the back-propagation phase. Its efficacy is shown by an application of this “Bidirectional Whitened Neural Networks” model to a handwritten character recognition data (MNIST data).