Additional Results and Important Clarifications.
Thank you for your review.
>This work discussed a traditional idea, i.e., the low-rank approximation using SVD, for language model compression.
**Clarification:** We want to emphasize that the focus of the paper is not to compress models but to demonstrate instead that selective rank-reduction interventions can offer a significant boost in terms of predictive performance.
We also emphasize that most past works (Lv et al., 2023; Hajimolahoseini et al., 2021; Yu et al., 2017) that do a low-rank approximation of neural networks or transformers perform low-rank approximation on every weight matrix and/or every layer of the model. In contrast, LASER is layer-selective, meaning we intervene in selective weight types (e.g., MLP input matrix ($U_{in}$)) and selective layer numbers. In fact, doing low-rank approximation unilaterally across layer numbers and layer types often leads to a significant reduction in performance. We believe this subtle difference with past work is an important finding.
To our knowledge, papers that use a low-rank approximation of matrices for model compression at best only obtain roughly accuracy-preserving models. We instead demonstrate that selective reduction of this sort can offer significant performance improvements.
>Current LLMs are often evaluated from multiple aspects including their reasoning abilities such as commonsense reasoning, world knowledge, reading comprehension, etc...such as MNLU, AGIEval, and BBH...this work uses none of them...
**Additional Experiments on 3 new datasets:** We evaluate LASER on 5 benchmarks and 3 LLMs, which gives 15 different setups in total. The sizable improvements given by LASER across these setups demonstrate a clear and robust result.
However, we agree that evaluating LASER on more setups will bolster our arguments. In light of this, we have added results on additional new datasets, including Epistemic Reasoning (from Big Bench Hard) and QA Wikidata (from Big Bench) alongside the Truthful QA benchmark. Of these, the big bench hard was specifically suggested in the review. We were unable to locate the MNLU dataset.
These new results are in the general response and in Table 4, Appendix C of the paper. All relevant edits are written in red. These results show similar notable improvements as in our main paper. For example, we notice a 14% point increase in test accuracy on QA Wikidata with GPTJ and an 18.6% point increase in test accuracy on Epistemic reasoning with Llama2. We also notice that, similar to Table 1 in our main paper, some setups have more modest gains of 0.5-2%. Understanding which datasets have more improvements and why is an interesting avenue for future work.
We will release our code and provide results on more datasets in the future, but we believe that **positive results across 8 datasets and multiple LLMs and tasks establish the general usefulness of LASER and that these results cannot be a coincidence.**
>The authors do not provide the final search results of rank reduction...It is very important to provide these results to show that the selected model is indeed in a reduced rank
**Final Search Results:** Thank you for pointing this out. We have added the final search results of the reduced ranks in Table 5 and Table 6 of the updated paper. As noted in the paper, the largest gains typically occur when intervening in higher layers of the LLM. For reference, Llama2 has 32 layers, GPTJ has 28 layers, and Roberta has 12 layers.
>I find that the dimension of GPTJ is 4096, which should be in your notation. So in Fig2, what is the rank of Reduction 99.5%/99.75% and others with .5%? (4096*0.0025=10.24, not an integer?)
**Rounding Clarification:** As mentioned in Section 4, last line of paragraph 1: “We replace it with a rank ⌊ρ · d⌋-approximation.” Here, “⌋” represents the floor of the number, i.e., the number is rounded down to the nearest integer value.
>The used CounterFact is very similar to the table-to-text generation task (but for a QA task), which is not a frequently used dataset to test even the QA and factual/world-knowledge reasoning performance of LLM. Any reason for choosing the dataset?
**Justification for using Counterfact:** The CounterFact dataset (introduced by Meng et al. 2022 ) is derived from the PARAREL dataset (Elzar 2021) and WikiData. This dataset is well-cited and commonly used for research on LLM understanding, and this was the main reason why we picked it. Including this dataset afforded the measurement of robustness to paraphrases with and without LASER (See Section 5.1 para 3). This study deepened our understanding of LASER’s robustness to paraphrases. Additionally, we provide results on 7 other datasets, including 3 new datasets that we added in this rebuttal. Specifically, we have also included the effect of LASER on WikiData QA, a similar benchmark from Big Bench.