Find A Winning Sign: Sign Is All We Need to Win the Lottery

We thank the reviewer for the encouraging comments and constructive feedback. We are especially pleased that the reviewer described our work as pioneering and recognized the significance of identifying a winning ticket with any random initialization. We have done our best to address the concerns raised, as detailed below. ## Weakness 1: Experiments on ImageNet We agree that MobileNetV2 and MLP-Mixer are lightweight models and may not be sufficient to fully demonstrate the generalization of our method. As the reviewer suggested, we have evaluated our method using ResNet-50 on the ImageNet dataset. Table 1 in the revised manuscript shows that transferring the signed mask obtained using our method to a random initialization is more effective than the case of learning rate rewinding [1] and achieves performance comparable to that of a dense network. These results suggest that our conclusion holds even for this larger-scale evaluation. Additionally, as the reviewers Kwwx and q2c9 noted, the overall performance of both MobileNetV2 and MLP-Mixer is relatively low in the original manuscript. To achieve full convergence, we increased the number of training epochs (from 120 to 300) and we achieved approximately 70% and 60% test accuracy for MobileNetV2 and MLP-Mixer before pruning, respectively, and confirmed that the network had fully converged. Using this setup, we reproduced the results in Table 1 and confirmed that the same trends hold true. We have revised the results accordingly in the revised manuscript. ## Weakness 2: Dependence on normalization layers > *This paper assumes that networks always include normalization layers and its parameters is necessary, which could limit its generalization.* Although there are few deep learning models without normalization layers, investigating several networks without them seems necessary to strengthen the validity of our work. As the reviewer suggested, we conducted experiments using LeNet and VGG16 without normalization layers on CIFAR-10 and CIFAR-100 datasets, respectively. The results below show that as the sparsity level increases, training a randomly initialized network with the LRR-driven signed mask (third column) leads to performance far below that of the LRR solution (second column) and, in some cases, results in a network that is not trainable (the last row in VGG16 experiments). However, we expected that the LRR-driven signed mask could transfer the generalization potential (generalization capacity achievable after training) of the LRR subnetwork since there is no adverse impact of the normalization layer parameters as discussed in Section 3.2. We find that these results are worth investigating in our future work. Thank you for your suggestion. **CIFAR-10 with LeNet** |Remaining parameter ratio|$\mathcal{A}(\boldsymbol{\theta}_T^\text{LRR} \odot \boldsymbol{m}_T^\text{LRR})$ (%)|$\mathcal{A}(\text{abs}(\boldsymbol{\theta}_\text{init}) \odot \boldsymbol{s}_T^\text{LRR})$ (%)| |:---:|:---:|:---:| |0.11|65.81|64.94| |0.085|65.6|64.63| |0.064|65.53|64.32| |0.055|64.72|61.58| |0.044|64.55|61.52| |0.035|63.89|60.83| **CIFAR-100 with VGG16** |Remaining parameter ratio|$\mathcal{A}(\boldsymbol{\theta}_T^\text{LRR} \odot \boldsymbol{m}_T^\text{LRR})$ (%)|$\mathcal{A}(\text{abs}(\boldsymbol{\theta}_\text{init}) \odot \boldsymbol{s}_T^\text{LRR})$ (%)| |:---:|:---:|:---:| |0.64|69.34|69.21| |0.52|69.21|69.11| |0.41|69.41|68.94| |0.33|69.12|66.13| |0.27|69.09|1| ## Weakness 3: Dependence on training multiple iterations > *However, this work still requires the learning rate rewinding to obtain the signed mask with multiple ($T$) iterations.* Obtaining a signed mask from a pre-trained model is equivalent to pruning a network in one shot, which significantly damages the network and makes recovering its original performance challenging. We agree that iterative pruning is computationally heavy and reducing the cost is a valuable research direction. However, iterative pruning is still essential to preserve the original performance, especially when aiming to find a winning ticket that is defined at initialization. We would like to emphasize that when we say, 'an effective signed mask is ALL WE NEED to win the lottery,' we mean that the signed mask obtained through our method enables **any** initialized network to perform comparably to a dense network after training. ## Weakness 4: Several incorrect notations Thank you for pointing out our mistake. We have corrected the typo at Line 184 in the revised manuscript. Additionally, $\boldsymbol{\theta}_0$ denotes the initialized parameters at the beginning of iterative magnitude pruning or learning rate rewinding. This is different from $\boldsymbol{\theta}_{init}$, which refers to arbitrary random initialization. [1] Renda etal, ‘Comparing Rewinding and Fine-tuning in Neural Network Pruning’, in ICLR2020.

We appreciate the reviewer’s positive comments on the presentation quality, the simplicity of our method, and the experimental results that demonstrate its efficacy. The reviewer has expressed concerns regarding the novelty of our work and raised several questions about the experiments. We hope the following feedback addresses these concerns and questions. ## Weakness 1: Novelty > *I find that the main novelty of the proposed approach is the insights into the normalization issues and the introduced AWS variant. In this context, I do not consider the contribution to be of the highest novelty.* We would like to highlight that existing works are either constrained to ad-hoc initialization and small-scale settings like iterative magnitude pruning (IMP) [1] or focus on finding sparse networks with trained weights, which do not represent true winning tickets [2,3]. In contrast, our work is the first to demonstrate that a signed mask, accounting for the effect of normalization layers, enables **any randomly initialized network** to generalize comparably to a dense network, validated at **relatively large scales**. We believe that our work is pioneering, as the reviewer q2c9 noted, and will serve as a significant advancement in the field of the lottery ticket hypothesis. ## Question 1: Issues with the ImageNet experiments >*Why did the authors choose only the MobileNetV2 and MLP-Mixer for the Imagenet setting? Were there any issues with the other architectures?* We were initially limited to evaluating our method only on MobileNetV2 and MLP-Mixer due to computing resource constraints. However, we have since overcome this issue using cloud computing services and now present additional results on ResNet50, a larger-scale architecture, in Table 1 of the revised manuscript. ## Question 2: Details on Figure 3 > *Can the authors provide a corresponding table (similar to Table 1) for the CIFAR-100 and Tiny-ImageNet experiments? What are the sparsity levels for these experiments? Do these correspond to the remaining parameter ratio in Fig. 3?* Please note that Figure 3 provides more information than Table 1, including the performance of LRR/AWS solution and randomly initialized networks with signed masks obtained through LRR/AWS across 15–20 sparsity levels. The 'remaining parameter ratio' in Figure 3 corresponds to 1−sparsity. In the revised manuscript, we have unified the terminology to avoid confusion by replacing ‘Sparsity’ with ‘Remaining params.’ in Table 1. Thanks for your feedback. ## Question 3: Different sparsity in ImageNet > *What is the behavior of the proposed approach in different sparsity levels in the case of ImageNet?* In the revised manuscript, we have added results on different sparsity levels for ImageNet experiments in Table 2. The results demonstrate that across various sparsity levels, training a randomly initialized network with a signed mask obtained through our method outperforms that with a LRR-driven signed mask and performs comparably to the AWS solution. Also, a randomly initialized network with a signed mask obtained through our method can achieve performance comparable to the dense network after training. These results demonstrate that our claim holds true across various sparsity levels in the ImageNet experiments. [1] Frankle etal, ‘The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks’, in ICLR2019. [2] Frankle etal, ‘Linear Mode Connectivity and the Lottery Ticket Hypothesis’, in ICML2020. [3] Renda etal, ‘Comparing Rewinding and Fine-tuning in Neural Network Pruning’, in ICLR2020.

We appreciate the reviewer's positive comments on our method and its applicability and effectiveness. The reviewer's main concern appears to be the generalization ability and scalability of our method to additional models and tasks. We hope to address these concerns through our responses below. ## Weakness 1: Generalization to more models and tasks > *It is not clear whether the proposed method can be generalized to more model architectures and tasks.* Currently, finding a winning ticket is a challenging and exploratory early-stage research field, where many recent works are still limited to relatively simple models and datasets using the iterative magnitude pruning (IMP) [1]. For instance, the effectiveness of IMP has been demonstrated in LeNet or 6-layer CNN with MNIST and CIFAR-10 datasets. Also, just like other research fields at early stage, the works in this field have performed experiments on classification. Although some works deals with other tasks like object recognition [2], they do not focus on a true winning ticket and instead rely on rewinding to trained weights. In that sense, we want to emphasize that the success of our method with ResNet-50, MobileNetv2, ShuffleNet, EfficientNet, and MLP-Mixer on CIFAR-100, Tiny-ImageNet, and ImageNet datasets extends the lottery ticket hypothesis to relatively larger scales and represents one step forward toward achieving the goal of the lottery ticket hypothesis. To further validate our method's generalization, we additionally provide the experimental results on ImageNet with ResNet-50, which is a larger-scale experiment, in Table 1 of the revised manuscript. Similarly to the results on MobileNetV2 and MLP-Mixer in Table 1, we observe that our claim still holds in such a larger-scale setting. ## Weakness 2: Scalability of the proposed method > *The need to interpolate normalization parameters during training may bring challenges when scaling up the proposed method* If we understand correctly, this concern pertains to the computational overhead introduced by our method. Compared to the original learning rate rewinding [3], our approach simply involves randomly and linearly interpolating normalization parameters with their initialization. Specifically, we randomly sample the interpolation ratio $\alpha$ and obtain $\alpha\cdot\boldsymbol{\psi} + (1-\alpha)\cdot\boldsymbol{\psi}_0$ where $\boldsymbol{\psi}$ and $\boldsymbol{\psi}_0$ indicate trainable and initialized normalization parameters, respectively. The interpolated parameters are then used during the forward pass. As a result, our method induces almost no computational overhead. [1] Frankle etal, ‘The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks’, in ICLR2019. [2] Girish etal, ‘The Lottery Ticket Hypothesis for Object Recognition’, in CVPR2021. [3] Renda etal, ‘Comparing Rewinding and Fine-tuning in Neural Network Pruning’, in ICLR2020.