Dynamic Sparsity Is Channel-Level Sparsity Learner

Summary

This paper investigates practical deployment and speedup of Dynamic Sparse Training methods on general hardwares. The author's motivation stems from observations that many channels naturally exhibit high sparsity rates. Based on this finding, the authors propose a sparse-inducing method to gradually remove entire channels during sparse training. This ultimately produces a network with certain structured sparse characteristics, enabling favorable acceleration. The authors validate the effectiveness of their proposed method through experiments on extensive datasets.

Strengths

1. The motivation is highly reasonable, and the proposed method has great application prospects, as it overcomes the bottleneck of unstructured sparsity being difficult to deploy on off-the-shelf hardware. 2. The observation on the sparsity distribution across channels of DST is novel and provides inspiration for the community. 3. The proposed method is easy to implement and even surpasses traditional non-structured pruning methods while achieving practical GPU acceleration.

Weaknesses

1. Given the author's focus on practical acceleration in the pruning field, it would be interesting to compare this method with structured pruning methods [1,2] even after obtaining accelerated and deployable networks. [1] Pruning filters for efficient ConvNets. In ICLR, 2017. [2] Channel Pruning via Automatic Structure Search. In IJCAI, 2020.

Questions

Regarding weight regrowing, I understand that the author selects weights with the highest magnitude globally for restoration. How about restoring weights with the highest UMM in each channel? Would this allow the network to gradually learn to become completely structured, i.e., some channels are automatically removed, while important channels are all preserved?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

n/a

Summary

This paper proposes a new approach to dynamic sparsity called Channel-aware dynamic sparse (Chase), which gradually translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end sparse training process.

Strengths

- The technical novelty of Chase mainly lies in two aspects: on the structured sparsity level, it adopts the gradual sparsification schedule to gradually remove Amenable Channels during training with smallest UMM scores, and on the unstructured sparsity level, it globally redistributes parameters based on their magnitude and gradient, which significantly strengthens the sparse training performance. I am a fan of this main idea. - Chase differs from other approaches to dynamic sparsity in that it seamlessly translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end training pass, without any ad-hoc operations. Therefore, it is designed to be a more efficient and effective approach to dynamic sparsity that can be accelerated by commodity hardware. This is in contrast to other approaches demonstrated on unstructured sparsity which receive limited support in common hardware. - One of the main new observations is that sparse amenable channels, which are channels that rapidly become sparser than others at the very early training stage, cause marginal damages to the model performance than their counterparts when pruned. The paper proposes a new metric called Unmasked Mean Magnitude (UMM) that can be used to precisely discover sparse amenable channels during training by monitoring the quantity and quality of weight sparsity. This finding reveals that a more nuanced approach to channel pruning can be more effective in achieving sparsity in neural networks. - Chase achieves up to 1.7× inference throughput speedup on common GPU devices without compromising accuracy with ResNet-50 on ImageNet. Even without pruning skip connections, the model is able to provide 31% throughput speedups, while outperforming the SOTA RigL by 0.39%. Additionally, Chase establishes a new SOTA performance bar for unstructured sparse training, outperforming all the presented DST methods by good margins.

Weaknesses

- Chase appears to be a mixture between “unstructured sparse training” and “structured sparse inference”. So, what is the benefit of Chase over early-stage structured pruning methods, such as Early Bird lottery ticket? How about its comparison to N:M dynamic sparse training, which can be done now from end to end? And how about instead just training a slimmer dense network from scratch? Those important questions are not discussed. - Experiment choices seem flawed to me. Both VGG19 and ResNet50 are known to be (ridiculously) over-parameterized for CIFAR-10/100. Comparison results on those testbeds cannot be reliable for sparsity algorithms: for CIFAR perhaps ResNet-32 or Res-20 would have made more sense. The only persuasive setting in this paper is ResNet 50 + ImageNet. - Important detail is missing on how “GPU-supported FLOPs” is computed? On what GPU? By channel level or N:M level? Also, how is the “Throughput” computed in Table 5? - Chase requires additional memory to store the mask for each channel. This can be a limiting factor for models with a large number of channels, as the memory requirements can become prohibitively large. - Additionally, the paper does not provide a detailed analysis of the end-to-end computational cost of Chase compared to other dynamic sparse training methods, which could be an important consideration for practical applications.

Questions

Please see above weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors shall discuss more limitations of their method

Summary

This paper proposes Channel-aware dynamic sparse, which translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end training process without using any particularly sparsity-aware hardware accelerators. The authors also observe the sparse amendable channels and proposed a new metric - Unmasked Mean Magnitude. Experiments demonstrate that the proposed method achieves 1.2x - 1.7x inference throughput speedups on common GPU devices with less to no accuracy degradation.

Strengths

1. This paper is well-written and gives sufficient experiments. 2. This paper observes the channel-wise sparsity during dynamic sparse training and proposes a framework Chase to translate unstructured dynamic sparsity to channel-level sparsity with less to no accuracy loss.

Weaknesses

1. The proposed method does not seem to have much performance benefit for training compared to unstructured methods. The computation is still unstructured in most iterations. 2. In Figure 3-b, the accuracy of RigL+GACP and RigL+GACP+SM is lower than RigL for ResNet-50 with sparsity 90% on CIFAR-100. While I understand GACP improves efficiency, it is not a good way to show the advantage of GACP as accuracy drops. It will be more convincing if you can show the runtime at the same accuracy or the accuracy with the same runtime.

Questions

The experiments use 50% channel sparsity. Why 50%?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I don't find a discussion on the limitations.

Summary

The paper introduces a novel approach called Channel-aware dynamic sparse (Chase) for training deep neural networks with high sparsity. Unlike previous techniques that focus on unstructured sparsity with irregular patterns, Chase seamlessly translates the benefits of dynamic sparsity to channel-level sparsity, which is more hardware-friendly. This is achieved through an end-to-end training process without requiring additional ad-hoc operations or specialized hardware accelerators. By identifying and removing sparser channels during training, Chase enables small sparse networks that can be efficiently accelerated on commodity GPUs. Experimental results show that Chase achieves a significant 1.7× inference throughput speedup without sacrificing accuracy when applied to ResNet-50 on ImageNet. The authors plan to release their code to facilitate further research and adoption of their approach.

Strengths

The paper exhibits exceptional clarity and a well-structured format. I particularly appreciate the bullet-point summary in the introduction, which effectively outlines the paper's roadmap. However, it must be noted that the paper can be somewhat dense and requires some additional time to fully comprehend. The proposed method is technically sound and yields commendable empirical results. The authors offer ample implementation details that would greatly facilitate other researchers and practitioners in reproducing the key findings outlined in the paper. It is worth mentioning that the authors have also committed to releasing the code upon the acceptance of the paper, further enhancing its accessibility and potential for future advancements.

Weaknesses

The paper includes experimental results that demonstrate the improvement in inference speed. However, it would be valuable to clarify whether the proposed method also enhances the training efficiency of the model. From my understanding, the primary advantage of employing dynamic sparse training lies in improving training efficiency. Otherwise, one could simply train a dense model initially and subsequently prune it to achieve sparsity. By addressing this aspect, the authors can provide a more comprehensive understanding of the benefits of their proposed method. This paper primarily conducts its main experiments and ablation studies using CIFAR-10, which is widely regarded as a small-scale dataset. Consequently, the results obtained from this dataset may not be entirely representative. It would significantly enhance the robustness of the findings if the authors could replicate these experiments using the larger-scale ImageNet dataset. Additionally, it would be highly advantageous if the authors could extend their validation to encompass other models and tasks, such as object detection and semantic segmentation, in order to establish the general effectiveness of their proposed method across diverse domains.

Questions

Please carefully address the weaknesses listed above in the rebuttal.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper lacks an adequate discussion on the limitations or potential negative societal impact.

Thanks

Thanks for the rebuttal. My concerns are well addressed and I think the paper is ready for publishing. I increase my score to Accept.

Reply

Thanks for the rebuttal. Overall, I am satisfied with the quality of this paper and believe it is worth publishing due to its contribution on deploying unstructured sparsity methods on off-the-shelf hardware.