Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization

Machine learning is predicated on the concept of generalization: a model\nachieving low error on a sufficiently large training set should also perform\nwell on novel samples from the same distribution. We show that both data\nwhitening and second order optimization can harm or entirely prevent\ngeneralization. In general, model training harnesses information contained in\nthe sample-sample second moment matrix of a dataset. For a general class of\nmodels, namely models with a fully connected first layer, we prove that the\ninformation contained in this matrix is the only information which can be used\nto generalize. Models trained using whitened data, or with certain second order\noptimization schemes, have less access to this information, resulting in\nreduced or nonexistent generalization ability. We experimentally verify these\npredictions for several architectures, and further demonstrate that\ngeneralization continues to be harmed even when theoretical requirements are\nrelaxed. However, we also show experimentally that regularized second order\noptimization can provide a practical tradeoff, where training is accelerated\nbut less information is lost, and generalization can in some circumstances even\nimprove.\n