Spectral Bias in Practice: The Role of Function Frequency in Generalization

Despite their ability to represent highly expressive functions, deep learning\nmodels seem to find simple solutions that generalize surprisingly well.\nSpectral bias -- the tendency of neural networks to prioritize learning low\nfrequency functions -- is one possible explanation for this phenomenon, but so\nfar spectral bias has primarily been observed in theoretical models and\nsimplified experiments. In this work, we propose methodologies for measuring\nspectral bias in modern image classification networks on CIFAR-10 and ImageNet.\nWe find that these networks indeed exhibit spectral bias, and that\ninterventions that improve test accuracy on CIFAR-10 tend to produce learned\nfunctions that have higher frequencies overall but lower frequencies in the\nvicinity of examples from each class. This trend holds across variation in\ntraining time, model architecture, number of training examples, data\naugmentation, and self-distillation. We also explore the connections between\nfunction frequency and image frequency and find that spectral bias is sensitive\nto the low frequencies prevalent in natural images. On ImageNet, we find that\nlearned function frequency also varies with internal class diversity, with\nhigher frequencies on more diverse classes. Our work enables measuring and\nultimately influencing the spectral behavior of neural networks used for image\nclassification, and is a step towards understanding why deep models generalize\nwell.\n