Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale

Coherent Gradients (CGH) is a recently proposed hypothesis to explain why\nover-parameterized neural networks trained with gradient descent generalize\nwell even though they have sufficient capacity to memorize the training set.\nThe key insight of CGH is that, since the overall gradient for a single step of\nSGD is the sum of the per-example gradients, it is strongest in directions that\nreduce the loss on multiple examples if such directions exist. In this paper,\nwe validate CGH on ResNet, Inception, and VGG models on ImageNet. Since the\ntechniques presented in the original paper do not scale beyond toy models and\ndatasets, we propose new methods. By posing the problem of suppressing weak\ngradient directions as a problem of robust mean estimation, we develop a\ncoordinate-based median of means approach. We present two versions of this\nalgorithm, M3, which partitions a mini-batch into 3 groups and computes the\nmedian, and a more efficient version RM3, which reuses gradients from previous\ntwo time steps to compute the median. Since they suppress weak gradient\ndirections without requiring per-example gradients, they can be used to train\nmodels at scale. Experimentally, we find that they indeed greatly reduce\noverfitting (and memorization) and thus provide the first convincing evidence\nthat CGH holds at scale. We also propose a new test of CGH that does not depend\non adding noise to training labels or on suppressing weak gradient directions.\nUsing the intuition behind CGH, we posit that the examples learned early in the\ntraining process (i.e., "easy" examples) are precisely those that have more in\ncommon with other training examples. Therefore, as per CGH, the easy examples\nshould generalize better amongst themselves than the hard examples amongst\nthemselves. We validate this hypothesis with detailed experiments, and believe\nthat it provides further orthogonal evidence for CGH.\n