Pruning vs Quantization: Which is Better?

Summary

This submission conducts a series empiricial experiments and anlysis between neural network pruning and quantization. It first used some statistics method to compare pruning/quantization. Then it measure the per-layer error based on a post-training compression framework. Finally, it conducted some experiments using LSQ for quantization and iterative magnitude-based iterative pruning.

Strengths

Statistics method for compression comparison as a non-parameteric and non-trainable method is interesting.

Weaknesses

1. The main drawback of the submission is that it barely proposed new algorithm or insight. The submission is a collection of empiricial experiments and analysis. 2. Comprehensive empiricial experiments can be regarded as contribution. However, the compression methods used in the submission is limited and the experimental setting is not solid. The submission basically focused on magnitude-based pruning and uniform quantization. Author may refer to [1] for some examples. [1] https://arxiv.org/pdf/1810.05270.pdf

Questions

N.A.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N.A.

Summary

The authors compare the performance of post-training quantization and pruning methods with the same compression ratio using a signal-to-noise metric, a kurtosis metric, and, ultimately, model accuracy. They study the expected performance analytically and in simple toy problems, i.e., Gaussian- and Student's t-distributed weights. They also run experiments on real pre-trained models from the PyTorch Model Zoo. They conclude that post-training quantization alone generally outperforms post-training pruning alone based on their per-layer weight fidelity metrics and full-model accuracy.

Strengths

- The paper represents a complete analysis of post-training pruning alone vs. post-training quantization alone when targeting the same compression ratio. - The authors analyze the problem at multiple levels (more theoretical/analytical and more empirical/experiment driven) and find the two viewpoints complement and support each other. - The toy model analysis, though straightforward, illuminates the general issues at play for pruning vs. quantization.

Weaknesses

- The SNR appears to be well-motivated, but it would be ideal to demonstrate explicitly that it maps to model accuracy, e.g. by adding a subfigure with accuracy to Fig. 3 and/or adding the SNR metric to Table 1. - The authors note that the quantized models naturally exhibit a larger compression ratio than the pruned models because they have a larger fraction of real 0s, which is a minor caveat. - Some additional references could be added to the related work, e.g., - combining pruning and quantization: https://arxiv.org/abs/2102.11289, - quantization-aware training: https://arxiv.org/abs/2006.10159, https://arxiv.org/abs/1905.03696 - In the Impacts section, there is no discussion of the potential decrease in model robustness and generalization when pruning or quantization.

Questions

- Why does the SNR metric use weights and not activations? - How does the layer-wise SNR metric you evaluate look for the real models in Table 1? Does it match the conclusions based on accuracy? - Can you indicate how computationally costly the pruning/quantization methods you employ are?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors discuss the limitations that they do not consider combinations of pruning and quantization, hardware considerations, and other quantization or pruning schemes.

Summary

This paper sets out to answer the question whether quantization or pruning is better. It first provides an analytical analysis of the two methods in terms of signal-to-noise ratio (SNR) and establishes an early relationship between kurtosis and SNR. It then provides a mathematical breakdown of the compression error in both methods. Then, it empirically compares the two methods on 46 models from the PyTorch model zoo to validate the early analytical results. The authors then compress and fine-tune a set of vision models to compare the two methods after fine-tuning and show that quantization still outperforms in this region. Finally it explores the hardware considerations since dense pruned models can be difficult to accelerate on most hardware.

Strengths

The authors use the reasonable FP16 baseline instead of the more common FP32. This paper makes an interesting connection between kurtosis and the relative performance of quantization over pruning. Observation that pruning tends to recover the original representation but quantization builds new ones is intriguing. The empirical evaluation spans across image classification and detection tasks.

Weaknesses

Especially when considering the hardware, it would be useful to consider block sparsity like 2:4, which is supported in some modern GPUs. However, element-wise sparsity presumably has strictly better accuracy / memory performance, so block sparsity would likely be worse. Given the recent success of transformers, it may be helpful to include more (in addition to ViT) in the empirical evaluation. That said, I do not expect this will change the conclusion. You correctly point out that weight distributions are typically quantized symmetrically but they also typically use channel-wise quantization, which is especially helpful with low bitwidths. I expect channel-wise quantization could further shrink the region where pruning is preferred. I am slightly concerned about the novelty of this comparison, but I will do a better literature search later in the review process. One other potential critique may be that this conclusion may not be too surprising given the exponential decay in the importance of bits. This decay likely does not hold for individual weights. Minor: L87 is a little confusing for the definition of T, which is more the error of the magnitude pruning, as opposed to the magnitude pruning itself.

Questions

Did you consider clipping to make the distribution less sensitive to outliers? This seems standard with weight quantization too since weights are easy to profile and optimize the quantization settings. Figure 1 (right) is computed with the analytical equations with the normal distribution substituted? Are there any regions when the combination of the two techniques outperforms the techniques individually? I'm confused about Appendix H, where the conclusion is that quantization tends to relearn different representations but pruning recovers the previous ones. The figure shows that the distance between the original and new activations is larger for pruning, which should imply that it doesn't relearn the original distributions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Summary

In this paper, the authors try to answer whether pruning or quantization is better for network compression. The paper start by analyze the qunatization error and pruning error under standard normal distribution and heavy-tail distributions. Full-model comparison are done between quantization and pruning with a set of 8 models. In most cases, the quantization method performs better than pruning. Quantization is much better than pruning under extreme compression targets.

Strengths

1. The paper is well-written. The whole writing is to the point. The content is well organized, which makes the paper easy to understand. 2. Thorough full-model comparison is done for a set of 8 models.

Weaknesses

1. The standing point of this paper might be biased. Quantization and pruning are not two competitive methods. Each of them have a specific application scenario. They could be used sololy and jointly. According to the experiments, quantization is almost always better than pruning. Yet, this does not mean that pruning is not useful. 2. The comparison is done such that the model size after pruning and quantization are the same. Not sure whether this comparison is proper. 3. Even without the investigation in this paper, quantization is still used more often than pruning.

Questions

1. Quantization and pruning are two different network compression techniques. But this paper tries to compare them under the setting of same model size. Nevertheless, model size is not the most important factor for efficient applications.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The novelty of this paper is quite limited. It is more like an empirical study.

Response to Authors

I agree that "a paper with novel insights is also a valuable contribution". And my point is that the submission is lacking in insights, due to the following reasons: 1. Only maganitude-based pruning and uniform quantization is compared. 2. The submission can be summary as: (a speific form of) pruning and quantization is compared under same budget and we find that quantization is better in performance. However, for example, pruning (with a same method) can lead to very difference performance under different settings (initialization, training strategy), as shown in [1] (provided by me). Not even to mention quantization. Novel insights do not come from listing numbers but thoughtful comparison and anaysis of the difference. Besides, I don't think reporting a SOTA metric is important and contributes novelty to empricial comparison. Pruning and quantization are two parallel (not mutually exclusive) methods, their performance under same budget can be compared (in complete settings) and indeed important for decision, but their applicability is also very important. Author may focus more on: 1. Thoughtful comparsion (including but not limited to initialization, training, applicability) between pruning and quantization, if the submission's ultimate is still finding which (pruning / quantization) is better in most cases. 2. Add more analysis on statistics methods, and their relation to final performance. Overall, I keep my scores.