Classical learning theory suggests that the optimal generalization\nperformance of a machine learning model should occur at an intermediate model\ncomplexity, with simpler models exhibiting high bias and more complex models\nexhibiting high variance of the predictive function. However, such a simple\ntrade-off does not adequately describe deep learning models that simultaneously\nattain low bias and variance in the heavily overparameterized regime. A primary\nobstacle in explaining this behavior is that deep learning algorithms typically\ninvolve multiple sources of randomness whose individual contributions are not\nvisible in the total variance. To enable fine-grained analysis, we describe an\ninterpretable, symmetric decomposition of the variance into terms associated\nwith the randomness from sampling, initialization, and the labels. Moreover, we\ncompute the high-dimensional asymptotic behavior of this decomposition for\nrandom feature kernel regression, and analyze the strikingly rich phenomenology\nthat arises. We find that the bias decreases monotonically with the network\nwidth, but the variance terms exhibit non-monotonic behavior and can diverge at\nthe interpolation boundary, even in the absence of label noise. The divergence\nis caused by the \\emph{interaction} between sampling and initialization and can\ntherefore be eliminated by marginalizing over samples (i.e. bagging) \\emph{or}\nover the initial parameters (i.e. ensemble learning).\n