Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of repetitive train-prune-retrain routine of iterative magnitude pruning (IMP) which worsens with increasing model size. This paper comprehensively studies induced sparse patterns across multiple large pre-trained vision and language transformers. We propose the existence of -- essential sparsity defined with a sharp dropping point beyond which the performance declines much faster w.r.t the rise of sparsity level, when we directly remove weights with the smallest magnitudes in one-shot without re-training. We also find essential sparsity to hold valid for N:M sparsity patterns as well as on modern-scale large language models (Vicuna-7B). We also present an intriguing emerging phenomenon of abrupt sparsification during the pre-training of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate a counter-intuitive finding that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). Our codes are available at \url{https://github.com/VITA-Group/essential_sparsity}.
Paper
Similar papers
Peer review
Summary
This paper presents a comprehensive study on the induced sparse patterns across multiple large pre-trained vision and language transformers. The authors propose the existence of "essential sparsity" and present intriguing findings on abrupt sparsification during the pre-training of transformers and the effect of pre-training data on knowledge condensation.
Strengths
The main strength of this paper is that it validates the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language, irrespective of the training strategy used for pre-training them. The authors used various datasets and models to validate the existence of "essential sparsity" and presented intriguing findings on abrupt sparsification during the pre-training of transformers and the effect of pre-training data on knowledge condensation. The authors also compared the performance of models with and without sparsification and found that one-shot sparsification without re-training does not significantly affect the performance of downstream tasks. In Section 6, the authors further carefully analyzed the connection between LTH and essential sparsity, and showed the former to become potentially less necessary or relevant in larger models. Those are valuable insights of broad interest to the sparsity research community. Based on the methodology and results presented in the paper, the experiment presented is sound enough to support the idea of the existence of essential sparsity in large pre-trained models (bert-base, OPT-125m, OPT-350m, OPT-1.3B). In particular, the authors present several quite surprising findings related to the existence of essential sparsity in large pre-trained transformer models. - Firstly, the authors found that BERT suddenly becomes heavily sparse after a certain number of training iterations, which is unobserved before and not well-understood. This finding suggests that there may be underlying “phase transition”-like mechanisms in the pre-training dynamics of transformers that are responsible for inducing sparsity, and further research is needed to understand it. - Secondly, the authors also found that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in relatively fewer parameters. This finding is counter-intuitive because one would expect that increasing the amount of pre-training data would lead to an increase in the number of parameters required to capture the additional information. - Thirdly, the authors found that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). This finding is also intriguing because one would expect that supervised learning, which provides more explicit information to the model, would lead to better knowledge condensation.
Weaknesses
- One main downside of this paper is that the paper did not report any result on hardware-friendly sparsity such as N:M, nor it discussed any GPU run time benefit from the induced essential sparsity (if any). While one can understand the study is mainly conceptual, just like the original LTH, I believe the real hardware support is especially important for LLM pruning/sparsity research due to their exploding costs - More baselines are desired – currently only LTH is reported. Would essential sparsity meaningfully outperform the simplest baseline of random pruning? How it compares with the pruning result of SparseGPT [18]? - In both SparseGPT [18] and Sparsity-May-Cry [75], it was found that larger LLMs are harder to prune. This paper seems to pinpoint the opposite conclusion. Could the authors elaborate why their conclusions seem to contradict [18,75], or not? - Sparsity has more benefits beyond efficiency, such as few-shot transfer, robustness to noisy label or other distribution shifts. Many of those were previously demonstrated under LTH or dynamic sparse training settings. I will be curious to see if essential sparsity will own the same merits too.
Questions
Please see the Weaknesses
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
4 excellent
Contribution
4 excellent
Limitations
Not discussing more hardware-friendly sparse patterns
Summary
This research paper focuses on the following notion for large pre-trained models: "essential sparsity", the idea that a sharp drop in fine-tuning performance occurs after one-shot pruning relative to the level of sparsity. The authors propose that large and overparameterized models can be pruned without additional computational expense, showing this to be true across a range of tasks in both computer vision and natural language processing. Additionally, the authors found an interesting occurrence of abrupt sparsification during pre-training, indicating that models trained with larger datasets tend to achieve knowledge abstraction with fewer parameters. Their findings also revealed that self-supervised learning objectives tend to trigger stronger emergent sparsification properties than supervised learning. They argue that identifying and understanding these inherent high-quality sparse patterns could make fine-tuning large models more practical and environmentally friendly.
Strengths
The paper makes a number of interesting observations about the notion of essential sparsity, its relation to "winning ticket" networks (see LTH), the idea that better knowledge abstraction may be feasible with fewer parameters (as long as we train with more data), and that self-supervised learning tends to have better emergent sparsification than supervised learning.
Weaknesses
Unfortunately, these interesting observation mentioned in the Strengths section, are also too "empirical" or anecdotally shown in the paper. It is hard to be convinced that these are general results -- and they are not substantiated with any form of of theory or at least some explanations/justifications based on results from other papers. In general, I would describe this paper as "it is moving in a good direction but it is not ready for publication yet" -- at least not at the top-tier conference of deep learning.
Questions
There is a completely different approach to get the benefits of essential sparsity -- namely to start with and train a sparse network. That would be even less computationally intensive than training a dense network and then doing OMP on that. Are the authors familiar with methods such as SynFlow, PHEW, Synflow++, etc that do pruning before training? If not, I suggest that they also consider those and compare the performance they get from OMP versus those networks. The two more interesting observations of the paper (the idea that better knowledge abstraction may be feasible with fewer parameters (as long as we train with more data), and that self-supervised learning tends to have better emergent sparsification than supervised learning) are presented very briefly, without a sufficiently deep analysis in my opinion.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
One limitation of course is that the claims of the paper are not substantiated in the case of really large foundation models used today -- but of course it is hard for academic researchers to experiment with the training of such models. Another issue is that the paper does not explicitly state the limitations of the proposed method. For example, in the end of section 6 there are some rather "hidden statements" about the benefits of LTH and IMP, which is a very important point.
Summary
This paper defines essential sparsity and conducts various experiments to analyze the sparsity property of pre-trained models.
Strengths
1. This paper investigates the potential of directly sparsifying the pre-trained models 2. CV and NLP models are both explored.
Weaknesses
1. Evidence on large models is lacking. 2. Some conclusions are not new, e.g., sharp dropping point.
Questions
What about the conclusions on large models like LLaMA 65B or Bloom 175B?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The insights brought from this paper is limited.
Summary
The paper postulates existence of "Essential Sparsity" in large pre-trained transformer models. The "Essential Sparsity" is defined by the paper as a sparsity threshold beyond which further pruning of weights leads to a large performance drop. It considers (1) one-shot pruning and (2) lottery ticket pruning of networks with fine-tuning. Experiments are conducted on both large language and vision transformer models.
Strengths
While the sparsity has been studied a lot in the neural network literature, this work studies the sparsity for pre-trained language and vision models. Given the wide adoption of these networks, the study is important. The paper covers a number of experiments for large vision as well as language models. The comparison of self-supervised vs fully supervised models is interesting.
Weaknesses
1. Existence of sparsity in transformers in both language [a,b] and vision [c] has been discussed in the prior literature. The drop of performance in networks beyond certain threshold has also been covered in the literature. I do believe the current work is the first one to formally define it, however it will be good if the work can put the contributions in proper context. 2. Many of the "surprising" findings of the work appear to be over-claimed. For instance, i. The abrupt drop in the performance of networks beyond certain sparsity thresholds is shown by the original LTH paper and the current work extends it to pre-trained network. ii. The similarity of sparse masks for various downstream tasks is also not very surprising given that the networks was pre-trained on a large massive dataset and the downstream tasks comprise of very small similar sets of data (often even the subset of pre-training data). iii. L50 (and L66) says that "a significant proportion of the weights in them can be removed for free" while it is hardly free given the network has been heavily pre-trained. 3. The mathematical definition of Essential Sparsity is somewhat unclear (what exactly does $1-\frac{||m||_0}{|m|}$ represent?). Also calling it essential for the network is a bit misleading. I agree dropping it would hurt the performance of the network but it is not "essential" for the network to perform well. A fully dense network usually performs the best. 4. Minor fix - Labels for x-axis is missing in most of the figures. References: a. Chen, Tianlong, et al. "The lottery ticket hypothesis for pre-trained bert networks." NeurIPS 2020 b. Dettmers, Tim, et al. "Llm. int8 (): 8-bit matrix multiplication for transformers at scale." NeurIPS 2022 c. Girish, Sharath, et al. "The lottery ticket hypothesis for object recognition." CVPR. 2021.
Questions
While some of the analyses of the paper is interesting, I would appreciate authors comments on above weaknesses.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The paper doesn't cover discuss the limitations of the study in details. For example, while the sparsity and efficiency of models both during training and inference is an important problem, it should be kept in mind that unstructured sparsity alone is not good enough to obtain speed gains. There is no discussion of potential speed gains with the unstructured sparsity (this doesn't necessarily mean FLOPs but speed improvement during inference/training), or training time of different networks, etc.
Summary
This paper delves into the natural sparsity of large-scale pretrained transformers that could be found without additionally training the models with sparsity optimization targets. The authors find that when simply low-magnitude weights are zeroed out, 1) up to a certain sparsity level, the pruned transformers do not exhibit performance drop on downstream tasks after fine-tuning. 2) the sparsity of weights abruptly increases at certain training iterations 3) simple magnitude-based pruning shows similar performance as the Lottery Ticket Hypothesis method up to a certain sparsity level. The phenomenon is observed across many different LLM and vision transformer architectures. Also interestingly, self-supervised weights are more sparse than supervised weights.
Strengths
This paper presents interesting findings on the nature of pretrained weights of large-scale models based on substantial amount of experimental grounds. The authors emphasize that the use of essential sparsity to prune weights can be done without repetitive train-prune-retrain routine. Considering the massive scale of recent large-scale transformers, it is a promising and practically efficient approach to find and utilize the sparsity of transformers. In addition, this paper opens up several research topics to be studied. I believe they will be impactful. - Sparsity pattern from self-supervised learning and supervised learning - Sparsity pattern change by training on larger-scale datasets - Acceleration of large-scale transformers by pruning weights
Weaknesses
- Practical gains after finding the sparsity pattern In contrast to learning-based pruning methods (such as LTH), the essential sparsity has the raw pattern of weight magnitude. Although it is interesting to find the sparsity is there by nature, as mentioned in L90-91, such patterns can’t be directly used to bring actual acceleration of neural networks. Training-based methods can set optimization goal on sparsity patterns to alter the sparsity pattern to make the neural network accelerable, however, such approach seems to be difficult for essential sparsity. While it is easier to find the sparsity pattern with essential sparsity, what would be the practical gains of the proposed method after finding the sparsity? - Why does sparsity occur? The authors present interesting comparison of impact of sparsity on model accuracy on various conditions - model size & architecture, supervised vs self-supervised learning. It is interesting to know of such results, but why does such differences happen? How can we understand the phenomenon? - typos L36 pre0trained -> pre-trained
Questions
Please refer to the weaknesses.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
No negative societal impact is expected.
Thank you for generating some new results to address one of my comments. I appreciate that and I will increase my score from Borderline to Weak Accept (only considering the comments posted by the other reviews).
Author Response to Reviewer AZn4
Dear Reviewer AZn4, We thank you for your time to review our work and your constructive comments to improve it, and we really hope to have a further discussion with you to see if our response solves your concerns. We have replied to the important points raised by you such as novelty concerns, limited gain from unstructured sparsity, over-claimed observations, clarification regarding the definition of essential sparsity etc., in our rebuttal response. To augment our rebuttal response, we would like to further add more clarification related to novelty concerns of abrupt drop in our work. We certainly agree that *almost all pruning methods will observe their performance drop after a certain sparsity level - there is not any surprise nor our main finding*. We would like to highlight the key message as follows: * The primary goal of this paper is to show the easiest pruning option, *one-shot, magnitude-based, training-free, and data-free pruning* echoing exactly the same behavior as any sophisticated method like LTH within a sparsity range. **The most important finding, as overlooked as prior work, is summarized** as: *within the induced sparsity range induced by “essential sparsity”, the simplest possible pruning technique as aforementioned performs the same good as any fancy technique like LTH. and even their identified sparse masks are extremely similar*. **This is NEVER revealed by any other work, and it seems other reviewers have appreciated this main merit.** * For first time we reveal how **“easy”** large model pruning is. At least within a certain sparsity range, one need not look beyond the *simplest one shot, magnitude-based, training-free, and data-free pruning* - that both presents a strong baseline for future pruning and reveals a strong “in-situ” pruning option (e.g. pruning is as simple as mag sorting). Hence our finding comes with profound practical value too, e.g., for cheap “on-the-fly” LLM pruning at test time adaptive to varying resource availability. * We provide some surprising and counter-intuitive findings related to **emerging abrupt sparsification of BERT during pre-training, sparsity dynamics in supervised vs self-supervised settings, first-time controlled pre-training data experiments** which illustrate more data make models more sparser, etc. We also included ViT experiments. All other reviewers have appreciated the significance of our interesting findings and the solidness of our thorough experiments. Since the author-reviewer discussion period has started for a few days, we will appreciate it if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends. If our response resolves your concerns, we kindly ask you to consider raising the rating of our work. We again thank you for your time and efforts. Best, Authors
2nd Reminder on feedback
Dear Reviewer AZn4: This is our 2nd reminder for you to please read our rebuttal and hopefully update your opinion. As the deadline for the discussion period is approaching, we would appreciate it if you could kindly let us know whether any further questions remain. We’re very confident that our rebuttal responses will address your concerns. We would again like to highlight that we have addressed your concerns related to novelty, limited gain from unstructured sparsity, over-claimed observations, clarification regarding the definition of essential sparsity etc., and additionally provided more experimental results related to structured N:M sparsity patterns, which significantly increase the value of our work. Authors,
Thank you for your response (and apologies for late reply). I've gone through the rest of the reviews and author responses. - Regarding mine (and other reviewers') concern on efficacy of unstructured sparsity - authors have provided some additional experiments on 2:4 sparsity patterns (although it appears that performance of model drops a bit on the downstream tasks and it is still unclear how much is the improvement in inference speed with the achieved 2:4 sparsity) - "We are not competing with LTH or any other pruning work, but we are interested in whether it is necessary to use LTH or other expensive methods" - IMHO given the repeated claims that the one-shot pruning works as well (or nearly as well) as LTH, the work is competing with pruning works such as LTH - Regarding my concerns on overclaims - the authors continue to call it a data-free, training-free pruning method which I believe is misleading for the readers. Overall my main concern is **not** the experiments on big models but presenting the paper in a fair manner and providing an essential analysis or trade-offs between one-shot pruning vs other pruning methods (say IMP). There are obvious advantages in terms of less training time and such. But this proper trade-off between training time / test-time accuracy / speed improvements is not properly revealed in the paper. I am currently keeping my score in a hope that authors would significantly tone down their claims and provide a fair assessment and limitations of the work.
We would happily adjust the tone - but hope for more clarifications
Dear Reviewer AZn4, Despite last-minute, we are delighted to hear further from you and are thankful for your time. **TL;DR; we are happy to adjust our tone and to "provide a fair assessment and limitations of the work" and we proposed a few actions below, although we would appreciate if you could also clarify some concerns a bit more**. From what we understand, all concerns are caused by confusion or miscommunication, and shall be easily fixable in current writing. We would be very thankful if you could take another moment to review our (as timely as possible) reply, and we will keep standing by from now on, if you want to discuss more before the rebuttal closes in ~8 hours. (1) 2:4 sparsity is the standard convention of "structured sparsity" that is evaluated by current few LLM pruning methods such as sparseGPT and Wanda (available later than NeurIPS deadline); and is a commonly accepted practice in GPU model acceleration. We have reported other N:M forms too for completeness. We will happily add real GPU hardware inference time in the final version; meanwhile our reporting has followed the standard convention in prior/peer works (e.g., neither sparseGPT nor Wanda reported hardware time of their N:M models in latest arxiv versions) (2) "The work is competing with pruning works such as LTH" - sorry for the confusion. **We never have intention to hide or avoid comparing with LTH, and we re-state our comparison rationale in bullets below**: - Essential sparisty has two main characterization points: (i) within this range, the one-shot pruned model performs as well as or better than dense baseline; (ii) beyond this range, the sudden accuracy drop is observed - LTH at best preserves the dense model accuracy (or very slightly better), while our method achieves the same behavior within essential sparsity range. Therefore, at least within essential sparsity range, ours is comparable to LTH in accuracy (through the *indirect, yet sound* comparison using dense model as "anchor") - Meanwhile, ours has "obvious advantages in terms of less training time and such", which is our main point in this paper: one can achieve good/best pruning without paying the expensive cost, within the essential sparsity range - Accurately performing LTH/IMP on billion-level LLM is hardly practical for most (if not all) research teams. Despite so, we tried my best to report *direct* comparison with LTH and ours on bert-base and ViT-base in Section 6. While that section is currently focused on reporting mask similarity not accuracy (due to the first bullet), we do have all "accuracy" results of LTH for Section 6 experiments and we can add them all if the reviewer feels helpful We sincerely hope the above has further clarified and solidified our "repeated claims that the one-shot pruning works as well (or nearly as well) as LTH". (3) "data-free, training-free pruning method" - we suggest you perhaps **unintentionally misunderstand our terms in this context**. This claim is made w.r.t the context of existing LLM pruning methods. We hereby clarify more below, and we can integrate the clarifications in final paper to avoid any future reader confusion: - **Data free**: this is commented w.r.t. sparseGPT or Wanda (available later than NeurIPS deadline), both of which will use a small set of "calibration data" to run inference and compute activations (because they need activation values as part of their pruning criteria, thus dependent on input). Our method relies on nothing but pre-trained weight value, hence no such need. We believe our "no activation/inference" criteria is very helpful when pruning needs to be performed on a real-time, or resource-adaptive basis. To help clarify, we are willing to revise "**Data free**" into "**Inference free**": please let us know if that sounds clearer to you - **Training free**: this is commented w.r.t. sparseGPT, which relies on solving layer-wise Hessian-inspired optimization over pre-trained weights. Ours belongs to the same camp as Wanda, with no need of post-optimization, which incurs negligible time overhead compared with SparseGPT. (ours is even way cheaper, due to no inference) We apparently never talk about "free of pre-training" overhead or so. To help clarify, we are willing to revise "**Training free**" into "**Post-optimization free**": please let us know if that sounds clearer to you - Your mention of "this proper trade-off between training time ..." is confusing us, since indeed our pruning has no overlead related to post-training (except the time used to sort pre-trained weight magnitudes). We fail to comprehend why a post-(pre)training pruning method must be responsible for that "the network has been heavily pre-trained" - this deviates from common evaluation practice for this type of methods We will always do our part as serious & responsible authors :-) We hope our above responses settle more of your concerns and please don't hesitate to let us know if we could address more!
Thank you for your response and it certainly provides more clarity. My current belief is that the paper will greatly benefit from a major writing revision.
Thank you again for your prompt response and we are glad that our detailed explanation finally provided clarity and it was able to resolve your doubts and concerns. Please note that all of the **above-suggested changes are actually easy to incorporate in our submission draft** and we promise to make these modifications in camera-ready. Specifically, none of our main technical conclusions nor their associated analysis will be affected - and it seems now we agree that there also exists no "over-claim" after we clarified above. In our revision, we will clarify a few terms, mainly: - explaining "for free" (note we didn't use training-free/data-free in the original paper but rather used them to explain "for free" in the previous rebuttal) explicitly as "free of inference or post-optimization" - explaining our comparison with LTH explicitly as "same as the unpruned baseline performance as well as LTH performance within the essential sparsity range" - other appropriate tone adjustments, if further suggested by the reviewer explicitly etc. Yet other than those additional clarifications (which are very helpful, and we are grateful for you bringing them up), we honestly did not see a "major writing revision" needed. We would humbly appreciate it if you consider raising your score based on the technical merits of our work and our rebuttal; which has been key in resolving your doubts.
Author response to Vvp4
Dear Reviewer Vvp4, We thank you for your time to review our work and your constructive comments to improve it, and we really hope to have a further discussion with you to see if our response solves your concerns. We have replied to the important points raised by you such as novelty concerns, limited experiments for large-scale models etc., in our rebuttal response. Since the author-reviewer discussion period has started for a few days, we will appreciate it if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends. If our response resolves your concerns, we kindly ask you to consider raising the rating of our work. We again thank you for your time and efforts. Best, Authors
2nd Reminder on feedback
Dear Reviewer Vvp4: This is our 2nd reminder for you to please read our rebuttal and hopefully update your opinion. As the deadline for the discussion period is approaching, we would appreciate it if you could kindly let us know whether any further questions remain. We’re very confident that our rebuttal responses will address your concerns. We would again like to highlight that we have addressed your concerns related to large models and additionally provided more experimental results related to structured N:M sparsity patterns, which significantly increase the value of our work. Authors,
1. Thank the authors for adding Vicuna-7B experiments. However, we think it can not be defined as very large pre-trained models. 2. As for the shart drop point, we can refer to the paper such as https://arxiv.org/pdf/2301.00774.pdf. In figure 1, it has revealed an obvious drop from certain sparsity rates.
Response to Comment by Reviewer Vvp4
We would like to thank you for your time to read our rebuttal and responding back. **1. Vicuna-7B is not considered as very large pre-trained model:** We value your concern BUT we would again like to emphasize that 65B or 175B experiments are seriously impractical without the availability of big industry-scale hardware support. Considering your response came very close to the deadline of the discussion period (~1 day before it ends), we would not be able to secure the required hardware support and complete the experiments within the deadline. However, we promise to scale our findings to 65B in the final version of our paper. In addition, we are running experiments for Vicuna-13B and it is expected to complete within the next 12 hours and we will update our results as soon as we get it. We sincerely hope, you can understand the hardware constraints and several concurrent works (eg. LLM pruner https://arxiv.org/abs/2305.11627) also restrict their findings to 7-10B scale models. **2. Sharp-drop point behavior:** Thank you for raising this concern, and we certainly agree that *almost all pruning methods will observe their performance drop after a certain sparsity level - there is not any surprise nor our main finding*. Unfortunately, it seems **you have missed the key message of our work**. * The primary goal of this paper is to show the easiest pruning option, *one shot, magtitude-based, training-free, and data-free pruning* (different from SparseGPT who also requires calibration data and the more costly Hessian estimation) echoing exactly the same behavior as any sophisticated method like LTH within a sparsity range. **The most important finding, as overlooked as prior work, is summarized** as: *within the induced sparsity range induced by “essential sparsity”, the simplest possible pruning technique as aforementioned performs the same good as any fancy technique like LTH. and even their identified sparse masks are extremely similar*. **This is NEVER revealed by any other work, and it seems other reviewers have appreciated this main merit.** * Orthogonal to SparseGPT or any other pruning method: it for first time we reveals how **“easy”** large model pruning is. At least within certain sparsity range, one need not look beyond the *simplest one shot, magtitude-based, training-free, and data-free pruning* - that both presents a strong baseline for future pruning and reveals a strong “in-situ” pruning option (e.g. pruning is as simple as mag sorting). Hence our finding comes with profound practical value too, e.g., for cheap “on-the-fly” LLM pruning at test time adaptive to varying resource availability. * We provide some surprising and counter-intuitive findings related to **emerging abrupt sparsification of BERT during pre-training, sparsity dynamics in supervised vs self-supervised settings, first-time controlled pre-training data experiments** which illustrate more data make models more sparser, etc. We also included ViT experiments. All other reviewers have appreciated the significance of our interesting findings and the solidness of our thorough experiments. We reiterate to include Vicuna-13B experiments within next 12 hours, and sincerely hope that you will give a look at it before making your final decision. We hope for the best and still open to clarify any doubts if you have to convince you for a rating upgrade before the clock ticks out.
Response to Comment by Reviewer Vvp4
As per our promise, we additionally provide the performance of Vicuna-13B on the MMLU benchmark within the essential sparsity range (we found it to be ~35\% for Vicuna-13B). Note that MMLU is a comparatively challenging task and we have also provided a comparison with SparseGPT in the following table. |Sparsity Ratio | Dense | 5% | 10% | 15% | 20% | 25% | 30% | 35% | 40%| | ------------- |:-------------:|:-------------:|:-------------:|:-------------:|-------------:|-------------:|-------------:|-------------:|-------------:| |One-shot MAG| 39.2 | 39.4| 39.1| 39.30| 39.11| 38.86| 38.7| 37.5| 34.80| |SparseGPT| 39.2| 39.5| 39.30| 39.2| 38.75| 38.54| 37.98| 37.6| 36.2| It can be clearly observed that within the essential sparsity range, both one-shot and SparseGPT have comparable performance and SparseGPT doesn't bring any additional benefit even being computationally much expensive and dependent on calibration data. Removal of ~35% from Vicuna-13B without any cost again aligns with the key message of our work. We sincerely hope that these new results will further help you in making the final decision. We will continue our experiments for LLama-75 and BLOOM-175B and update results in the final version. But we want to **emphasize again that our original submission + all the additional results are already strong and solid for the reasons we mentioned, and we believe more evaluations are only going to help support our work.** Best, Authors
Thank you so much for the reply! I have read other reviews too. It appears that the authors have addressed all concerns properly. I think the paper has presented more-than-sufficient insight and back-up results to warrant its acceptance. N:M sparsity and Vicuna-7B results are even nicer additions. In particular, beyond LLM, this paper reports ViT results too, as well as pre-training dynamics from scratch (which were not observed in peer LLM pruning works). I also don't feel it necessary nor reasonable, to ask for some colossal model (LLaMA 65B, Bloom 175B) done within rebuttal time window. I am raising my score to 8 to champion this solid work.
Author Response to dYgU
We are extremely glad that you find our work solid. We deeply appreciate and thank you for your strong support for the work and identifying its merits. We’re particularly grateful for your agreement on the sufficiency of our current/newly added experiments, and the impracticality of running ‘colossal models’ within a few days.
After reading the rebuttal, I still think the essential sparsity to be far from being practical. In rebuttal Figure 2, the authors present performance drops from several different N:M sparsity patterns. However, I don't think any of the other N:M patterns other than 2:4 are valid. Albeit general N:M sparsity is theoretically possible, they require hardware support (such as tensor cores) to have actual acceleration. 2:4 sparsity has been supported by GPU hardware for a few years, however, no other patterns have been supported. Some patterns where n or m are powers of 2 are theoretically explored, however, the patterns in Rebuttal Figure 2 (1:10, 1:6, 2:10) are not practical. (Is it 1:10 or 9:10?) * A. Zhou et al., LEARNING N:M FINE-GRAINED STRUCTURED SPARSE NEURAL NETWORKS FROM SCRATCH, ICLR 2021 As 2:4 sparsity is losing accuracy, I don't consider it to be safely accelerable without additional training. Given that the authors could provide more objective comments in the revision, I am ok with it. I still find it an interesting paper. I'm between Accept and Weak Accept.
Decision
Accept (poster)