Compressing LLMs: The Truth is Rarely Pure and Never Simple

Authors Response to X7p4

We first like to thank you for your time to review our work. We greatly appreciate that you have found our work to be well-written, comprehensive, and have interesting insights. We are glad that you think our work can have a big impact on how practitioners use these models. For the ease of use, we reiterate that we will be releasing codes, experimental details, and host the datasets on Dropbox, which will help LLM compression researchers use LLM-KICK with ease for evaluating their novel algorithms. Next, we would like to address all the weaknesses pointed by you point-by-point below: > **1. More LLMs. Right now, it's only vicuna model (7B and 13B), having other architectures and perhaps bigger sizes (e.g. llama 70b) would add more evidence for the insights.** Thank you for your suggestion and we strongly agree that extending our experiments beyond Vicuna will bolster our work. In that direction, we have included some additional experiments with LLaMa-2 as you suggested in Appendix A6 which obey our previous findings. For larger model size, we additionally provide some results for Vicuna-33B as follows which again aligns with our conclusion. We also have plans to run our experiments on llama-2 70B as pointed out by you, and include in our final draft (rebuttal duration doesn’t permit us to complete this experiment). |Factoid-QA|Dense|10%|20%|30%|40%|50%|60%| | ------------- |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:| |Magnitude|76.92|76.72|71.73|59.44|46.75|42.46|3.10| |SparseGPT|76.92|73.93|70.43|61.24|50.85|42.16|29.37| |Wanda|76.92|73.63|72.73|65.03|50.65|52.05|39.36| > **2. Add prompt designs to the main body of the paper. This will make it easy to understand how each task is distinct.** Thank you for this great suggestion. We agree that the addition of prompt design/example will improve the distinguishability of our task settings. We will surely work towards incorporating this valuable suggestion into our final draft. > **3. More quantization methods. AWQ/SmoothQuant are interesting to see.** Thank you for your suggestion. As also pointed out by b73d, we considered evaluating two more quantization methods AWQ and LLM.int8() across our different task settings and we summarize our results on Vicuna-7B as in the following table. We observe that LLM.int8() despite its simplicity and ease-of-use, achieves better results than AWQ (8-bit), and GPTQ (8-bit) across all listed tasks. Note that we couldn’t complete our experiments on summarization and instruction following due to dependency and time constraints (on OpenAI) but we will update them in the final draft. | | GPTQ | AWQ | LLM-int() | | ------------- |:-------------:|:-------------:|:-------------:| |**Factoid-QA**|60.14%|60.31%|61.02%| |**MCR-QA (MMLU)**|47.10%|47.18%|47.82%| |**Retrieval Augmented QA**|75.55%|75.89%|75.91%|

Authors Response to b73d

We would first like to thank you for the time to review our work. We greatly appreciate that you have found our work to be important, timely, and revealing novel widely unknown gaps in compressed LLMs in comparison to their uncompressed counterparts. We again assert that we will be releasing codes, experimental details, and host the datasets on Dropbox, which will help LLM compression researchers use LLM-KICK with ease for evaluating their novel algorithms. Next, we would like to address all the weaknesses pointed by you point-by-point below: > **1. Extending experiments beyond Vicuna Family?** Thank you for your suggestion and we strongly agree that extending our experiments beyond a single family of LLMs will bolster our work. In that direction, we have included some additional experiments with LLaMa-2 as you suggested in Appendix A6 (Factoid-QA, MCR-QA, ICR-QA) which obey our previous findings. For larger model size, we additionally provide some results for Vicuna-33B as follows which again aligns with our conclusion. We are running more experiments on larger LLMs which will be included in our final draft. |Factoid-QA|Dense|10%|20%|30%|40%|50%|60%| | ------------- |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:| |Magnitude|76.92|76.72|71.73|59.44|46.75|42.46|3.10| |SparseGPT|76.92|73.93|70.43|61.24|50.85|42.16|29.37| |Wanda|76.92|73.63|72.73|65.03|50.65|52.05|39.36| > **2. Comparison with LLM.int8() ?** Thank you for your suggestion. As also pointed out by X7p4, we considered evaluating AWQ and LLM.int8() across our different task settings and we summarize our results on Vicuna-7B as in the following table. We observe that LLM.int8() despite its simplicity and ease-of-use, achieves better results than AWQ (8-bit), and GPTQ (8-bit) across all listed tasks. Note that we couldn’t complete our experiments on summarization and instruction following due to dependency and time constraints but we will update them in the final draft. | | GPTQ | AWQ | LLM-int() | | ------------- |:-------------:|:-------------:|:-------------:| |**Factoid-QA**|60.14%|60.31%|61.02%| |**MCR-QA (MMLU)**|47.10%|47.18%|47.82%| |**Retrieval Augmented QA**|75.55%|75.89%|75.91%| > **3. Table summarizing all the tasks' performance?** Thank you again for your great suggestion and we agree that a table summarizing all task performance will be very helpful for the readers to quickly digest our experimental insights. We promise to include it in our final draft with an additional page. > **4. Quantization works better than pruning?** To answer this question, we would like to bring attention to Junk DNA Hypothesis (JDH) recently proposed by https://arxiv.org/abs/2310.02277 . According to JDH, small-magnitude weights may appear "useless" for simple tasks and suitable for pruning (in most SoTA pruning methods), but actually encode crucial knowledge necessary for solving more difficult downstream tasks. Removing these seemingly insignificant weights can lead to irreversible knowledge forgetting and performance damage in difficult tasks. Their finding reveals fresh insights into how LLMs encode knowledge in a task-sensitive manner and pruning them significantly hurt the performance. On the other hand, quantization doesn’t remove these small-magnitude weights completely, which can explain why quantization is more friendly to pruning for LLMs. Another recent work, https://arxiv.org/abs/2307.02973 provides a detailed thorough analysis but wrt small-scale models like ResNet, EfficientNet, etc. *We sincerely hope our responses answer your questions, and we again would like to take this opportunity to thank you for highly rating our work.*

Authors Response to PUep

We would first like to thank you for the time to review our work. We deeply appreciate that you have found that our work provides a more fine-grained understanding of compression methods. We are glad that you think our work provides interesting insights for SoTA LLM compression methods. Next, we would like to address all the weaknesses pointed by you point-by-point below: > **paper does not include any new model, data or algorithm, which I'd say makes this paper more suitable for a workshop, not a full conference paper** We politely disagree with your comment that without proposing any new model, data, or algorithm, a submitted work is not suitable for a conference paper. To provide more context, [1,2,3,4,5,6,7,8] are some among numerous examples of top full conference papers (some cited >150) which doesn’t propose any new model, data, or algorithm. We kindly request you to acknowledge that scientific research can encompass novelty in various forms. We believe that thoroughly exploring existing solutions, and providing novel insights holds equal significance to the pursuit of new methods or architectures. We also very politely request you to look at other reviewers who find our work very important, timely, and can have a big impact on the LLM compression community. [1] NeurIPS’19. A Meta-Analysis of Overfitting in Machine Learning, https://papers.nips.cc/paper_files/paper/2019/hash/ee39e503b6bedf0c98c388b7e8589aca-Abstract.html [2] NeurIPS’20. The Lottery Ticket Hypothesis for Pre-trained BERT Networks https://proceedings.neurips.cc/paper/2020/file/b6af2c9703f203a2794be03d443af2e3-Paper.pdf [3] CVPR’21. The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models https://openaccess.thecvf.com/content/CVPR2021/papers/Chen_The_Lottery_Tickets_Hypothesis_for_Supervised_and_Self-Supervised_Pre-Training_in_CVPR_2021_paper.pdf [4] CVPR’23. Reproducible scaling laws for contrastive language-image learning, https://openaccess.thecvf.com/content/CVPR2023/papers/Cherti_Reproducible_Scaling_Laws_for_Contrastive_Language-Image_Learning_CVPR_2023_paper.pdf [5] ICLR’23 - Top25%: Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! https://openreview.net/forum?id=J6F3lLg4Kdp [6] ICLR’22: The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training https://arxiv.org/pdf/2202.02643.pdf [7] NeurIPS’22: Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation https://proceedings.neurips.cc/paper_files/paper/2022/file/408cf1a1d9ff35d5fea7075565dbf434-Paper-Conference.pdf [8] ACL’23: Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors https://aclanthology.org/2023.acl-long.650/ > **Some arguments are rather subjective. Why choose the 5% threshold? If we change the threshold to 10% it seems 4-bit quantization is then in the range in most cases, and sparse models can still be "competitive" for around 50% sparsity.** Thank you for bringing this up and we are very happy to clarify your doubts. Firstly, **our threshold is simply an indicator to illustrate the tolerance level of performance drop** when we start compressing any LLM. Many prior works [1,2,3] consider matching thresholds to be the same as the dense subnetwork performance or within the margins of 1%. However, in our work, we carefully relaxed it to 5% performance drop as an acceptable tolerance (before calling the compressed model useless) keeping in mind that the performance of compressed LLM on any of our task categories/disciplines remains above the random guess. For example, to provide context, as you have mentioned, 10% threshold on MMLU can leave performance on some disciplines like Algebra, Econometrics drop to 19.7%, 21.1% which is below the random guess (25%) given MMLU has one correct out of four choices. Again, note that this threshold has nothing to do with any performance or insight reported in our draft. We will add additional clarification in our final draft. [1] CVPR’21: The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models [2] NeurIPS’23: The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter [3] NeurIPS’20: The Lottery Ticket Hypothesis for Pre-trained BERT Networks

Author Response to Reviewer QryF

We would first like to thank you for the time to review our work. We greatly appreciate that you have found our work to be important, timely, and significantly necessary considering the growing interest in LLM compression, and the lack of any standard evaluation protocol to elucidate the merits and plights of novel algorithms. We also significantly value your suggestion to include the dataset statistics, and we will update our Appendix A5 to provide more details. We also have plans to release codes, experimental details, and host the datasets on Dropbox, which will help LLM compression researchers use LLM-KICK with ease for evaluating their novel algorithms. We again take this opportunity to thank you for highly rating our work.