Summary
The paper focuses on DNN pruning and proposes a novel approach using a soft mask during training. The soft mask is designed to encourage weights around the pruning threshold to actively switch their states, aiding in the recovery of pruned weights. This method is simple, efficient, and introduces no additional parameters. Experimental evaluations of the proposed approach have been conducted on ResNets, MobileNets, and BERT. It also explores different pruning settings such as N:M pruning, random pruning and channel pruning.
Strengths
* The paper proposes a soft mask during training, which smooths the participation of weights around the pruning threshold in the neural network. This can give more chances for weights around pruning threshold to change their states, especially help pruned weights actively recover themselves.
* The compared methods are enough in experiments.
* The paper discusses relevant works and provides a comprehensive analysis.
Weaknesses
* Method:
* Lack of differentiable analysis: The title and content of the paper stress the differentiable pruning, however, method part (3.2) does not give differentiable analysis.
* Insufficient explanation of the design of "t" and calculation of "m(w)": Why is “t” in PDP training flow designed in this way? The paper lacks clear explanations about the design of “t” and calculation of “m(w)”.
* Lack of analysis and ablation study on the efficacy of the proposed soft mask: I find that the paper adopts progressive pruning as illustrated in the algorithm chart in supplementary material. As progressive pruning is an effective technique for final accuracy, I am doubtful about the real effect of the proposed method. To better understand the effectiveness of the proposed soft mask, we need more analysis and ablation studies.
* Experiments:
* More experiments about Transformer architectures are preferred. The paper provides BERT models experiments on MNLI dataset. Experiments with ViT models or BERT models on the whole GLUE benchmark will be preferred.
* In the main body, there are not enough sparsity ratio selections. Table 4 provides only one sparsity ratio for each model. More sparsity ratios, ranging from low to high are necessary to understand the effect of the proposed method.
* Writing:
* Redundancy between Figure 5 and Table 4: It seems that Figure 5 and Table 4 convey the same information. What is the difference between Figure 5 and Table 4 or why illustrate the same information twice? I think each table or figure should contain specific information, especially in the main body.
* Improper organization of the paper: The paper first illustrates the benefits of PDP over existing pruning approaches, then introduces the PDP algorithm, which can be confusing. Before readers know the method clearly, it would be difficult to understand its benefits. The better solution would be to change their order or to reconsider the effects of section 3.1, such as serving it as a motivation section.
Questions
Please see the method and experiment points in the weakness part. If my concerns are addressed, I would like to increase my score.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
There is no potential negative societal impact of their work.