Summary
This article presents a theoretical study of Sobolev Training, an alternative to least-squares fitting that replaces an $\mathcal{L}^2$ fitting error with $\mathcal{H}^p$ fitting error, for some $p > 0$. This method fits the target function by not only matching its values on the training data, but also matching higher order derivatives, leading to improved convergence speed.
To understand this effect, the authors study a toy model for training neural networks. They focus on one layer, single node RELU and RELU^2 networks, and they study the dynamics of model parameters under gradient flow-based minimization of the Sobolev loss. The Jacobian of the Sobolev loss has additional terms which are positive semidefinite when $w_0$ is sufficiently close to $w^*$, leading to (potentially) improved convergence speeds. These additional terms are not surprising, since the Sobolev loss can be thought of as a sum of $\mathcal{L}^2$ loss and additional terms measuring the $\mathcal{L}^2$ mismatch of gradients, which improve the convergence speed of gradient flow.
Finally, to approximate Sobolev Training in practice, the authors propose to use Chebyshev spectral differentiation, a popular numerical differentiation method. Across a variety of toy experiments, Sobolev Training is shown to accelerate convergence of GD and SGD.
Strengths
- Simple idea and analysis: the idea behind Sobolev training is clearly presented and easy to understand. The authors identify a simple and straightforward theoretical explanation for empirical observations that Sobolev training improves convergence speed.
- Clear and correct proofs: the proofs in this work are presented clearly and they are correct to the best of my knowledge.
Weaknesses
- Unfair/vacuous comparison: it is unfair to compare the convergence of gradient flow on $\mathcal{H}(w)$ and on $\mathcal{L}(w)$, since $\mathcal{H}(w)$ has extra terms that are added in an unnormalized way. To understand why this is unfair, consider another (trivial) way to accelerate the convergence speed of gradient flow, by rescaling the loss. Set $\tilde{\mathcal{L}}(w) = \mathcal{L}(w) + \mathcal{L}(w)$, then in the notation of Theorem 1, $\partial \tilde{V} / \partial t = -2 (w - w^*)^T \nabla_w \mathcal{L} \leq 2 \partial V / \partial t < 0$, which is already a significantly stronger 'acceleration' than Theorems 2 and 3 because it converges twice as fast (whereas Theorem 2 and 3 show only convergence that is not any slower than Theorem 1). Simply adding extra convex terms to the loss isn't sufficient to argue an acceleration phenomena, any more than rescaling the loss would be. It would be more fair to average the terms in the Sobolev loss, or to rescale so that $\mathcal{L}(0) = \mathcal{H}(0)$, but then Theorems 2 and 3 might not hold.
- Synthetic/toy experiments: the experimental evaluation in this work is entirely focused training toy objectives. While the Sobolev loss is shown to improve convergence on these objectives, it's not clear whether this benefit extends to actual datasets.
Questions
In Figure 3, Sobolev training seems to induce training instability in the form of large loss spikes during late training. Does the Sobolev training loss have higher variance over random initializations of GD? It would be helpful to add error bars to Figures 3, 4, and 5.
Rating
3: reject, not good enough
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.