We thank the reviewer for the positive comments on the value of DropCompute, and for the additional feedback. Regarding the remaining concerns:
**1a. Is it a limitation that DropCompute requires large batch sizes?**
Large batch size is not the main constraint, but the number of micro-batches used (as the reviewer pointed out in the previous comment). We focused on applying DropCompute on workloads, such as LLMs pretraining, that already require large batch sizes, and enough micro-batches (>16, as we explained before). In addition, we believe that the trend in training larger and larger LLMs is leading towards an increased number of micro batches ― a trend that can only further help DropCompute. For example, we mentioned GPT3 which had 24 microbatches (in our rebuttal), and we think this number typically increases in even larger models, such as the 160 microbatches in *[Smith et al. “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”]*; note the authors there also explicitly mention gradient accumulation as a desired method in LLM training. Therefore, we do not believe this is a serious limitation.
**1b. Are longer sentence sizes an issue for DropCompute?**
No. Like the reviewer mentioned, LLMs are using longer sentence sizes in pretraining. However, we are not sure why this is a problem. In fact, this seems to benefit DropCopmute: longer sentence sizes lead to smaller micro-batches (because of memory constraints) which lead to more micro-batches per worker.
**2. Does the DropCompute speedup we showed come for 'free’ (i.e. without accuracy degradation)?**
Yes. There might be some misunderstanding regarding how we define the speedup versus the results in Table 1, so we would like to clarify it.
Effective speedup (as defined in equation (6)) already considers the dropped samples. For example, if the drop rate is 10% and the effective speedup is 10%, it means that we have gone through ~11% more samples (while dropping 10% of all samples), in 10% less time. To compensate for the dropped samples and competitive runtime performance, we suggested doing more training steps (e.g., lines 275-277: “...achieving the same training loss as the baseline might requires additional training steps, however, it leads to a notable reduction in overall training time.”). We found this to be a hassle-free method in our experiments, as we illustrated in Figure 5. There, we trained with DropCompute (with 6.7% drop rate) until reaching the baseline training loss (instead of training a fixed budget of steps). This required 3% more steps, but took 13% less time.
In Table 1, however, we report the model accuracy without adding more training steps (which leads to a further increased speedup, higher than the “effective speedup”). Therefore, we expected Table 1 to have accuracy losses. The point of Table 1 (and Figure 9 in the appendix) was to show that the actual speedup can be higher than the effective speedup, since we do not even have to add more steps, unless drop rate > 10% (we note we did not observe experiments with drop rate > 7%, even in our most extreme cases, presented in the PDF attached to the global response part of the rebuttal). If we were to add more iterations to the training sessions used to produce in Table 1 (to compensate for the drop rate), we would have reached a similar accuracy as the baseline (as we meant to say on lines 251-252, we apologize if this was not very clear).
We hope that this answers any remaining concerns. Please let us know if there are any other concerns.