Summary
This work studies the generalization behavior of overfitting methods in terms of the smoothness of the estimator, showing that only non-smooth estimators can interpolate benignly. They give a discussion of this result in the context of NTKs and their corresponding infinite-width architectures.
Strengths
Originality and quality: this work seems to shed some new light on a well-studied problem. I'm not very familiar with background results on asymptotic risk w.r.t. function smoothness, so it's difficult for me to assess the first several theorem statements. I'm quite familiar with the latter results on NTKs, and to my knowledge this observation about spikiness in activation functions is new, and I find it well-explained.
Clarity: the paper seems pretty clear and well-written.
Significance: the math here is nice, and it seems like this might lead to some more clarity regarding exotic activation functions.
Weaknesses
On significance: it's sort of unclear to me how this matters, or where we go from here. For example: the results of Mallinar et al. suggest that early stopping in nets (or including a ridge parameter in KRR, as shown by prior work too) is enough to make fitting consistent. These questions like "how do we design our model so it overfits benignly?" seem like they're actually missing some motivation -- why do we need to do that? Can't we just use a ridge parameter or optimal stopping?
The results around spike-inducing kernels seem quite hacky. The fact that the spike has to get smaller with dataset size makes it clear that this is just effectively adding a ridge. (The authors acknowledge this, but it's still not a surprising conclusion in my view.) It's also not useful -- the authors do not try to train their Hermite-polynomial nets, and (based on my own attempts to do such things) I suspect this may be because they're unstable, even at moderately large width!
As stated above, I'm unfamiliar with prior art around function smoothness and overfitting behavior, so there's a fair bit of uncertainty in my assessment here.
Questions
The authors argue that the key object to consider is estimator smoothness, not input dimension. Given that there exists a body of results that do find that input dimension is important, is it clear how to reconcile the author's new results with these old ones? For example, do the smoothness requirements get more relaxed as dimension increases?
Note: I think an activation fn corresponding to the Gaussian NNGP kernel can be written down in closed form. (I do not know if this is true for the NTK, and would be curious to know if it can.)
Note: a high-frequency sinusoidal activation function can also give you a spiky kernel!
Note for future research: this smooth + spiky kernel thing seems like a hack. However, the results of Mallinar et al. suggest that if you could write down any kernel with an eigendecay like $\lambda_i \sim i^{-1} \log^{-\alpha} i$ for $\alpha > 1$, you'd get a consistent estimator without the need to scale down the kernel. It seems plausible to me that by taking a sort of exponent-equals-zero limit of Bietti and Bach's analysis, you could construct a kernel that actually gives you this decay (even on the unit circle), and it'd be consistent while being (a) a continuous function and (b) not needing to be scaled with dataset size. Could be an interesting thing to look into.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.