S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning

Summary

The paper titled "S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning" introduces a novel framework designed to address the challenges associated with the discretization gap in neural network pruning techniques. The authors propose a method that bridges the gap between the representation capacities of a continuously relaxed network (soft network) and its pruned counterpart (hard network), aiming to improve the pruning performance without requiring post-pruning fine-tuning.

Strengths

1. The S2HPruner framework is innovative in that it directly addresses the issue of the discretization gap, which is often overlooked. By incorporating soft-to-hard distillation, the method ensures that the hard network maintains a similar level of performance to the soft network, which is a significant improvement over traditional pruning methods. 2. The overall writing is good and the work is easy to follow.

Weaknesses

1. The figures can be further improved in this work. For instance, figure 2 is not good engouth to illustrated the overall pipeline of the work. It is recommanded to explictly show soft net and hard net instead of thier outputs only. 2. The formats of the tables are not consistent.

Questions

See weaknesses above

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

It is recommanded to discuss the training time and GPU RAM cost during training and inference. The comparisons between the proposed one and existing SOTAs are necessary.

Summary

In this article, the author proposes using a 0-1 mask (hard network) and a differentiable mask (soft network) with an accuracy gap as a starting point for network distillation, where the distillation function selects Kullback Leibler divergence as the gap measure（S2HPruner）. This method was tested on datasets including CIFAR-100, Tiny ImageNet, and ImageNet for Resnet-50, ViT Swin, and other models.

Strengths

* This paper is easy to follow. * The experiments and investigation into gaps and gradients are well-developed.

Weaknesses

**Q1:** The abstract section mentions the concept of 'bidirectional', but it's incorrect. My understanding is that S2HPruner only uses sparsity in forward flows and does not involve sparsity in backward flows. Please explain the meaning of this "bidirectional" in detail. **Q2:** Is there a reason to utilize Kullback Leibler divergence when selecting a gap measure? Is it possible to add corresponding experiments or proof explanations to the article that does not provide an explanation? **Q3:** How does S2HPruner control network's sparsity? The method section of the article lacked any description. From the description in Figure 2, it appears that the mean is calculated from a differentiable soft mask as a threshold, and if this is the case, it seems that the purpose of controlling the sparsity rate will not be achieved.

Questions

Please see the weakness part

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

Discretization in pruning poses a huge threat to network performance. To alleviate this issue, the paper proposes S2HPruner, a pruning method that leverages distillation. In details, the pruning process involves two networks that share the same architecture. The difference is that the teacher network has a weight covered with soft, differentiable masks, while the pruned student network weight is covered with non-differentiable binary masks. The optimization target involves a distillation loss that narrows the gap between the differentiable and non-differentiable network. The proposed method achieves good performance on various datasets.

Strengths

The paper highlights the harm of discretization in pruning, which I think is the core problem. The paper distill knowledge from a soft-mask network to a hard-mask one, which I think is novel.

Weaknesses

1. Line 9 in Abstract: SH2->S2H 2. Notation problem: the authors should unify the gradient notations in the pseudocode and the equations. 3. For Table 4: as different methods use different training settings, the number of the training epochs should be indicated. 4. The baselines in Table 4 are a bit out-of-date. The proposed S2HPruner should be compared against latest channel pruning methods, like SCOP [1] and CHEX [2]. [1] github.com/yehuitang/Pruning/tree/master/SCOP_NeurIPS2020 [2] github.com/zejiangh/Filter-GaP

Questions

1. Baselines reflect the actual training capability of a certain codebase. Why do you use 76.15 instead of 76.8 as the ResNet-50 baseline? Please justify. 2. From a high-level perspective, the method is a combination of distillation and pruning. I wonder if the method could outperform the simple combination of pruning and knowledge distillation (at equal training costs)?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are addressed.

Response to Rebuttal

I am satisfied with the results presented by the author, and I will raise my score. I also hope that the author could include these results in the paper upon acceptance.