Dear Authors,
Thank you for your replies.
> We note in the paper/Appendix that we ran we could afford to. CD-GraB converges faster than D-RR, which the existing results show. We are academics, not an industry lab. We can run LeNet for longer, but saved our training budget for other tasks.
I completely understand the challenge of limited resources. However, there exists many prior works from academia applying larger models such as ResNet on CIFAR10. I point out that there has been cases where proposed optimization methods fail to reach the high accuracy obtained by random reshuffling GD even though they might be faster in the beginning of training. Therefore, plots showing random reshuffling is outperformed by CD-GraB can become misleading especially for readers not familiar with the expected state of the art results. This is why I find the evidence provided here very insufficient to support practical use of CD-GraB.
> Based on the intended scope of our contributions (i.e., primarily theory, with empirical implications to be explored in future systems work), the fairest comparison for our experiments is to the original GraB paper (also a theory-focused paper). We used this prior work as the basis for designing our experiments.
In the whole paper, the argument has been always about working both in theory **and** in practice. If this is not the case, this should be clarified and the reason why the optimizer is not practice-ready explained.
> We additionally include results for a small GPT-2 on WikiText-103 in the Appendix. We note in the conclusion why we did not focus on transformers. Models like LLMs today are finetuned typically for a really small number of epochs. The benefits of CD-GraB (i.e., getting the theory-backed benefits of achieving the herding bound) only come in the multi-epoch setting, and we cannot afford the pretraining costs of a LLM for a lot of epochs.
Thank you. Please add a reference to these experiments in the main text as well.
> Even so, we demonstrate the benefits of CD-GraB on other ML tasks, and believe that this is an important contribution. It is worth investing the effort in future ML systems research to build and stress-test a full implementation of an order server at scale. Without the present work, we would not have been able to know that such an investment was worth it. Such a system necessarily is the realm of future work.
If I understand correctly, this means that there exist a bottleneck or an obstacle in scaling up CD-GraB which needs a full implementation. This is what I was asking for in the limitation section of my original review. What are these limitations? It does not seem to be the memory based on your explanation. Assuming off-loading to CPU is not needed, what other significant implementation is needed or what part of the algorithm needs to be improved to make it suitable for large scale experiments?
> Single node pair-balance
Please do add the experiments if you have them. If I am understanding correctly, replacing the ordering method can be beneficial even in the single node setting since it removes the need to store all past gradients. It would be also important to assess whether using this method instead of the original method is less performant in finding the right order or not (which might be hidden in the multi node settings due to incompatibility of the original method with that setting). In general, unless needed, changing only one hyper-parameter (algorithm) instead of both (algorithm and number of workers) allows for better comparison.
> Learning rate schedule: We used standard settings for the tasks that we investigated, and for the LSTM on WikiText-2 experiment we still used learning rate scheduler. Investigating how to tune the learning rate schedule in response to our improved convergence is the realm for future theory work.
I am not sure what is the standard schedule for LeNet on CIFAR10. A decay in learning rate usually shows itself in a spike in accuracy but I could not find this in the plots. Can you please describe the standard schedule you mentioned?
**Regarding the memory**: Thank you for the explanation. I did not understand what is exactly reported in the table. What is meant by overhead? What is the increase in full memory usage (percentage for example)?
**Overall comment and suggestion**: My main objection currently is that the evidence presented is not convincing enough for using this method directly in practice which I initially considered to be part of the claim. Still the idea behind the paper is novel and as the authors said the method can open doors for further improvements. My suggestion is that the authors would add a Limitation section clearly discussing all the limitations including the gap between the accuracies on CIFAR10 and state of the art, possible bottlenecks if any (discussed above), etc. to avoid any confusion. Assuming this is possible (please confirm), I have updated my score.
Thank you.