Re: Additional Clarifications
> I am curious why did the authors turn off all the optimizations to achieve SOTA accuracies ? These optimizations like LR decay, data augmentation are standard and are widely used.
We do this for several reasons:
- First, our paper's aim is *not* to compete with methods/papers/systems whose goal is to achieve SOTA generalization accuracies on selected benchmarks - we agree that in such work such optimizations should be used. Our work is of a completely different variety : our paper is not about generalization at all. Our work isolates an open theoretical question (can error feedback provably work with small minibatches?) and proposes an algorithmic fix (use of momentum), and theory which conclusively shows that this trick works as advertised, in the class of smooth nonconvex functions.
- Second, our experiments are designed to test the predictive power of our theory. Heuristics such as data augmentation and LR decay are orthogonal considerations which are entirely irrelevant in our study. They are important as far as actual generalization performance of various optimizers is concerned, but as explained above, this is not the subject of our paper. For this reason, if we included these heuristics in our experiments, it would actually make the experiments and conclusions one can draw from them more confusing.
> "A theorem is worth a thousand experiments" and "Our paper is not a systems/software work, where comparison using run time would be appropriate and expected". Unfortunately I as a reviewer strongly disagree with these statements. However, I understand the authors have a different opinion.
Yes, we are of a different opinion. We strongly believe that theory and empirics have equal value in ML research. One feeds into the other and vice versa. We believe that the ML field needs to stand on both its feet (theory and empirics) to advance and to be truly useful. State of the art empirics typically stands on the shoulders of strong theory, and uses additional tricks, heuristics and ideas to push things further. Such tricks are then studied by theoreticians, uncovering their robustness or brittleness, improving and modifying them, or replacing them with more theoretically well grounded tricks. The actual interplay between theory and practice is much more complicated and intricate than this, of course.
Once you have the belief that theory and empirics have equal intrinsic value (and we actually believe this is what the community ideally/hopefully *should* believe), then it becomes clear that the community should be able to equally appreciate strong theory and empirical works. Fortunately, as the record of papers accepted to NeurIPS in the past clearly shows, this is the case. We believe it is in fact dangerous to use a double standard. For example,
- We believe that a strong practical/empirical work should stand on its own, and be perfectly acceptable to NeurIPS without the requirement that it contains any theory whatsoever. Of course, some theory is welcome, and can make that paper even stronger (say an 8/9/10), but it should not be a requirement for acceptance.
- Likewise, a strong theory work should stand on its own, and be perfectly acceptable to NeurIPS without the requirement that it contains any experiments whatsoever. Of course, some empirics is welcome, and can make that paper even stronger (say an 8/9/10), but it should not be a requirement for acceptance.
It seems to us you do not subscribe to this philosophy.
> I personally believe to have a meaningful impact both theoretical and experimental validation should be provided on real metrics.
This can't possibly be the way to evaluate NeurIPS papers since otherwise no pure empirical and pure theory work would ever get published - and there are many examples of immensely influential works in these categories.
> For example - Work like Powersgd (Vogels et al. ) which end up having meaningful impact on improving accuracy had extensive experiments.
This paper is not a theory paper - it does not include a single theorem in the main body of the paper. The one theorem in the appendix is minor, and not central to the paper. This is a very good example of an empirical work which we believe should be accepted to a top venue. In the same manner, pure theory papers with perhaps just one experiment in the appendix should also be perfectly acceptable to a top conference. As an example, consider the 2012 NeurIPS paper by Nicolas Roux, Mark Schmidt and Francis Bach: A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets. This work is of a theoretical nature just like ours, isolating an important theory problem and proposing a solution. The experiments are designed to test the theory. Yet, this work won the Lagrange Prize in Continuous Optimization, and had an enormous impact.