We derive analytical expressions for the generalization performance of kernel\nregression as a function of the number of training samples using theoretical\nmethods from Gaussian processes and statistical physics. Our expressions apply\nto wide neural networks due to an equivalence between training them and kernel\nregression with the Neural Tangent Kernel (NTK). By computing the decomposition\nof the total generalization error due to different spectral components of the\nkernel, we identify a new spectral principle: as the size of the training set\ngrows, kernel machines and neural networks fit successively higher spectral\nmodes of the target function. When data are sampled from a uniform distribution\non a high-dimensional hypersphere, dot product kernels, including NTK, exhibit\nlearning stages where different frequency modes of the target function are\nlearned. We verify our theory with simulations on synthetic data and MNIST\ndataset.\n