Towards Higher Ranks via Adversarial Weight Pruning

Summary

This paper proposes a novel Rank-based PruninG (RPG) method for network pruning that maintains the ranks of sparse weights in an adversarial manner, leading to high-rank topology and improved performance. The proposed method is evaluated on various datasets and tasks, including image classification, object detection, and semantic segmentation, and compared to state-of-the-art pruning methods. The experimental results show that the proposed RPG method outperforms the existing methods in terms of accuracy and efficiency. The paper also provides insights into the importance of rank preservation in network pruning and the potential of adversarial training for improving the performance of pruned networks.

Strengths

+ This paper is well-written and organized, with clear explanations of the proposed method and experimental results. The authors provide detailed insights into the importance of rank preservation in network pruning and the potential of adversarial training for improving the performance of pruned networks. + This paper proposes a novel Rank-based PruninG (RPG) method that maintains the ranks of sparse weights in an adversarial manner, leading to high-rank topology and improved performance. This approach is original and creative, as it combines the ideas of rank preservation and adversarial training to address the limitations of existing pruning methods. + This paper provides a comprehensive evaluation of the proposed RPG method on various datasets and tasks, including image classification, object detection, and semantic segmentation. The experimental results show that the proposed method outperforms the existing methods in terms of accuracy and efficiency, which demonstrates the quality and effectiveness of the proposed approach. Overall, this paper has a high-quality and significant contribution to the field of network pruning, with original ideas, rigorous evaluation, clear explanations, and practical implications.

Weaknesses

- One potential weakness of the proposed Rank-based Pruning (RPG) method is that it may only achieve better performance than existing methods when the sparsity rate is high. As mentioned in the paper, the RPG method outperforms existing methods such as WoodFisher, PowerPropagation, and AC/DC at sparsity rates of 90%, 95%, and 98%. However, at lower sparsity rates such as 80%, the RPG method is slightly lower than WoodFisher in terms of ImageNet accuracy. This suggests that the RPG method may not be as effective at lower sparsity rates, and may require higher sparsity rates to achieve better performance than existing methods. This could be a limitation for some applications where lower sparsity rates are preferred due to memory or computational constraints. - The proposed method can only be applied on weight pruning, which is slower than filter pruning when their pruning rates are same.

Questions

This paper mentions that the proposed method is based on the rank of the weights, but it would be helpful to have more details on how exactly the rank is determined and how it affects the pruning process. Additionally, are there any limitations or potential drawbacks to using rank-based pruning compared to other pruning methods that use different criteria for selecting weights to prune?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This work propose a novel objective for performing element-wise pruning of DNN model. The work identifies the loss of weight rank as the key factor influencing the performance of DNN when pruned to high sparsity. This issue is tackled by including a rank loss in the pruning criteria, so that weight elements contributing to the weight matrix rank are preserved. Experiment results show the proposed method outperform previous work under high sparsity.

Strengths

1. The motivation of preserving high rank in element-wise pruning is novel, and it motivates the proposed method well 2. The proposed method is well formulated and is technically sound 3. Adequate ablation study and extensive experiments are conducted, showing promising results

Weaknesses

One major concern of this paper is whether aiming for higher element-wise sparsity is useful for the deployment of efficient DNN model. As mentioned in the limitation, element-wise sparsity is not well supported on GPU. Even on CPU, as shown in Tab.3 only 2x speedup can be achieved with 95% sparsity, at the cost of 3% accuracy drop. While a 50% structural sparsity can lead to 4x FLOPs reduction, potentially 3x speedup, with less accuracy drop. This would indicate that aiming towards a high-rank element-wise sparsity is not as useful as directly having structural sparsity. I would suggest the author to provide a comparison of accuracy-speed tradeoff of the proposed method and previous structural pruning results, to show the importance of the proposed method. Minor issue: It would be better to have a pseudo code for the pruning procedure performed in Sec. 2.5

Questions

See weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The limitation is adequately discussed. No potential negative social impact is observed.

Summary

This paper proposes a new weight pruning method for compressing Convolutional Neural Networks (CNNs) called Rank-based PruninG (RPG). The RPG method consists of two steps: first, the low-rank approximation error for the weight matrices is minimized using singular value decomposition, and second, the weight matrices are pushed away from their low-rank approximation to maximize their distance. The authors demonstrate that the RPG method outperforms other state-of-the-art methods in terms of accuracy and compression rate on various datasets and tasks.

Strengths

Originality: Pruning weights while maintaining the ranks of sparse weights in an adversarial manner is original and has not been proposed before, which is different from other pruning methods that focus on removing individual weights or neurons. Quality: The experimental results show that the RPG method achieves higher accuracy and compression rate than the previous baselines. The authors also provide insights into the mechanism of the RPG method and its impact on the network structure, which enhances the quality of the paper. Clarity: The paper is well-written and easy to understand, which enhances its clarity. The authors provide clear explanations of the proposed method and its implementation. They also provide detailed experimental results and analysis. Significance: When the pruning rate is high, traditional pruning methods can lead to a structured pattern in the remaining weights, which limits their performance. The proposed rank-based pruning method maintains the ranks of sparse weights in an adversarial manner, which ensures that the pruned network retains its structure and performance even at high pruning rates.

Weaknesses

1) The adversarial optimization involves a min-max problem that requires additional computation and optimization steps, which can increase the training time and complexity. Similarly, SVD is a computationally expensive operation that requires matrix factorization and singular value decomposition, which can also increase the training cost. Moreover, the proposed method requires the computation of low-rank approximations and the search for the best k-rank approximation, which can further increase the training cost. These additional computations and operations can make the proposed method less practical for large-scale networks or real-time applications. 2) Singular value decomposition (SVD) is used in the proposed method to compute the rank and low-rank approximations of weight matrices. In contrast, Canonical Polyadic Decomposition (CPD) and Tucker Decomposition are other methods for decomposing tensors into lower-rank components. It is better to explain why choosing SVD to estimate the rank of weights.

Questions

Please refer to the weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

This paper proposes a novel unstructured pruning method, trying to maximize the matrix rank while trying to remove as many model weights as possible. The paper first demonstrates the phenomenon that unstructured pruning may degrades to structured pruning at large sparsity ratios, which is closely related to the fact that the pruned weight matrices become low-rank matrices after many weights are set to zero. Thus, the objective of the proposed method is trying to on the one hand, minimize the task-related loss, and on the other, maximize the rank of the pruned weight matrices, which forms a min-max problem. This min-max problem is then integrated into model pruning via a matrix rank-boosting regularization term. With the gradual pruning framework, the proposed method (RPG) is examined empirically on CIFAR-10, ImageNet, COCO datasets with CNNs and ViTs model architectures. The results show the effectiveness of the RPG.

Strengths

This paper studies the unstructured pruning from the perspective of rank maintenance, which is very novel. The authors made very informative and helpful illustrations to help the readers understand this paper without difficulty. Therefore, the presentation is also very good. Extensive experiments were conducted on different tasks, and the results look good. In summary, this work is a very good attempt to connect model pruning with weight ranks in a novel perspective.

Weaknesses

I listed some weakness of this work from different perspectives. I will consider to raise my score if they are addressed properly. 1. [Motivations] The motivations of this paper by "removing the structuring patterns in the unstructured pruning" do not seem very direct to me. In general, currently the results of the unstructured pruning can not be directly used for hardware acceleration. Therefore, one important research direction is to generate structured mask from the unstructured pruning results, such as [ICML22] Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets. Therefore, I wonder if the motivation of this paper will make the unstructured pruning less adaptable to hardwares? 2. [Presentations] I would suggest the authors to remove some discussions on SVD, which are very basic knowledge in linear algebra. In contrast, more important information in the Appendix can be brought back to the main paper. 3. [Method] It is a bit vague to me, how the gradients are dealt with when Eq. (2.6) is treated as the regularization term. Is $\text{Trunc}(U\Sigma V^T)$ treated as a constant value during back-propagation? Also, I suggest the authors explicitly write down the $\mathcal{L}(\text{task})$ as well, which shows how the sparsity of the model weights are imposed. 4. [Method] In Eq. (2.6), the model weights are denoted by one variable $\mathbf{W}$. However, as we know the model weights may contain many layers and many types (convolutional kernels, fully connected layers, etc.) Do the authors sum them up in implementation using the same coefficient? Or there are different weights assigned to different layers. It is not clear in the paper regarding this point. 5. [Experiments] The baselines used in Tab. 1 and Tab. 2 are not consistent. Are their any specific reasons? The same issue is also spotted in Tab. 5, where in the last row block the method "SViTE" is missing, but the authors stated "For fair comparison, all pruning experiments follow the setting of SViTE." (Line 351). 6. [Experiments] It would help the readers to better compare the results if the authors can include the computational (training) time of different methods. 7. [Minor] There are unnecessary margins under Tab. 2 and Fig. 3. Please consider to remove them and make the layout of the figure better. 8. [Minor] Line "is illustrated in" -> "**are** illustrated in"

Questions

Below is a summary of my comments in the section "Weaknesses". 1. What is the relationship between the rank of the pruned weight matrix? Will the unstructured pruning with higher rank impairs the further acceleration on hardwares? 2. How are the SVD-reconstructed terms dealt with during back-propagation? 3. What are the formulation of $\mathcal{L}(\text{task})$? 4. How are the weights with multi-layers processed to compute Eq. (2.6)? 5. Why are the baselines not consistent within one table/across different tables on the same tasks? 6. What how does the training efficiency of different methods look like?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I do not have additional comments on the limitation of this work. Please refer to the "Weaknesses" and "Questions" sections.