Response to Reviewer yLRG
We sincerely thank the reviewer for their time and constructive feedback.
**Response to weaknesses:**
1. We further confirm our findings on the larger and more complex datasets: Tiny ImageNet (see Figure 2d) and ImageNet (see Figure 15 in the appendix).
2. So far, LRR has only been proposed as a variant of IMP, which is the most common pruning baseline for its simplicity and high accuracy. It was therefore restricted to magnitude as a pruning criterion. Nevertheless, we have transferred the same principle to random pruning (see Figure 2. b,c and Figure 13 in the appendix), for which our insights hold as well. To extend the analysis, we have further added experiments for iterative Synflow and iterative SNIP in Fig. 16. The superiority of LRR can be generalized to all these different criteria. Also, our additional experiments with a VGG16 architecture on CIFAR10 in Figure 17 in the appendix are in line with these results.
3. In Figure 8 and 9 in the appendix, we provide a comparison on the effect of different learning rate schedules and highlight the benefits of using warmup epochs at the beginning of every training cycle to improve performance of both IMP and LRR.
Minor: We have updated the draft with the corrected typo and added the references.
**Response to questions:**
1. Experimental insights:
- The main insight we draw from Figure 4 is that LRR is able to improve parameter optimization for different sparse masks, like the mask found by IMP (IMP mask). This finding actually holds for multiple datasets and learning settings. LRR with the IMP mask achieves a significantly higher performance than IMP on CIFAR10. If we modify LRR with BN parameter rewinding (red line) this holds also true on CIFAR100. Furthermore, note that we can draw similar conclusions for Tiny ImageNet (Fig. 2d), ImageNet (Fig. 15), different pruning criteria (Fig. 16), and VGG networks (Fig. 17).
- Fig 7(c, d) additionally indicates that LRR depends significantly on the learned sign information, as sign perturbations can greatly hurt its performance. Yet, it is robust to magnitude rewinding (IMP reset amplitude in Fig 7). LRR signs and masks seem to contain sufficient information to achieve the high LRR performance. This conclusion is supported by the results on all studied datasets.
2. The blue line in Fig 11 (Fig 12 in the updated draft) refers to experiments, where we fix several aspects that have been learned by LRR. Concretely, we train an architecture with LRR mask and initial parameters that have the same signs as the learned LRR parameters. The only part that is still learned in the iterative scheme is the parameter magnitude, which starts from the initial IMP magnitude. The fact that this blue line achieves a similar performance as LRR suggests that the high LRR performance can primarily be attributed to the learned signs and the mask.
3. Figure 12 (13 in the updated draft) plots the difference between the total number of signs flipped from the initialization. It shows that LRR flips many more signs in comparison with IMP. We have removed the conclusion in the caption, as not only this figure but the collection of different experiments provide evidence for the stated hypothesis.