Low Rank Adaptation (LoRA) has gained massive attention in the recent generative AI research. One of the main advantages of LoRA is its ability to be fused with pretrained models, adding no overhead during inference. However, from a mobile deployment standpoint, we can either avoid inference overhead in the fused mode but lose the ability to switch adapters rapidly, or suffer significant (up to 30% higher) inference latency while enabling rapid switching in the unfused mode. LoRA also exhibits concept-loss when multiple adapters are used concurrently. In this paper, we propose Sparse High Rank Adapters (SHiRA), a new paradigm which incurs no inference overhead, enables rapid switching, and significantly reduces concept-loss. Specifically, SHiRA can be trained by directly tuning only 1-2% of the base model weights while leaving others unchanged. This results in a highly sparse adapter which can be switched directly in the fused mode. We further provide theoretical and empirical insights on how high sparsity in SHiRA can aid multi-adapter fusion by reducing concept loss. Our extensive experiments on LVMs and LLMs demonstrate that finetuning only a small fraction of the parameters in the base model significantly outperforms LoRA while enabling both rapid switching and multi-adapter fusion. Finally, we provide a latency- and memory-efficient SHiRA implementation based on Parameter-Efficient Finetuning (PEFT) Library which trains at nearly the same speed as LoRA while consuming up to 16% lower peak GPU memory, thus making SHiRA easy to adopt for practical use cases. To demonstrate rapid switching benefits during inference, we show that loading SHiRA on a base model can be 5x-16x faster than LoRA fusion on a CPU.
Paper
Similar papers
Peer review
Summary
This work proposes a new PEFT method, SHiRA, which finetunes 1-2% of pretrained model weights. The authors demonstrate that the resulting sparse adapter weights can be combined in multi-adapter settings with less concept loss than LoRA as the sparse adapters are mostly orthogonal. Further, the authors empirically demonstrate that the scatter operation used in SHiRA has a lower latency than fusing LoRA weights at inference time for model embedding dimensions > 1024.
Strengths
* This is a timely work as adapters as the multi-adapter setting offers great promise for memory constrained devices such as mobile phones. * The proposed method outperforms LoRA across a diverse set of tasks and models. * The paper is presented well, with clear figures, tables, and discussion.
Weaknesses
* The primary weakness is the relatively modest novelty of this work in the context of Diff Pruning [1] and SFT [2]. Both of these prior works follow a very similar strategy as SHiRA, namely, fine tuning a very small, sparse set of the original pretrained parameters. In my view, the most unique aspect of SHiRA is the emphasis on composability in the multi-adapter setting and related analysis. A comparison between the performance differences between SFT-AG and SHiRA would help establish the potential benefits / drawbacks of using dynamic mask selection (SFT) vs. static (SHiRA). In any case, [2] is a very relevant work which should be discussed in the related work section. * The primary motivation for this work is low latency adapter switching, providing end-to-end profiling would be much more convincing than focusing only on the scatter-op as in Appendix B. Perhaps Appendix J was originally intended for this analysis? It is unclear how significant the overall effect of the latency reductions in Appendix B are in the context of online or batched inference on an edge device. Does SHiRA provide any latency benefit for smaller embedding dimensions such as those used in StableDiffusion? * One of the main benefits of LoRA is the reduced memory footprint for fine-tuning with adaptive optimizers such as Adam. Due to the momentum buffers typically requiring full float precision, LoRA greatly reduces the memory overhead required for fine-tuning. In contrast, it appears that based on Appendix C.3 SHiRA must materialize the full grad buffers for the pretrained weights which are selectively set to zero for frozen parameters with a post gradient accumulation hook. I believe SHiRA could be reparameterized as $W_{new} = W_{pretrained} + \theta_{shira}$ such that only the $\theta_{shira}$ parameter tracks gradients and therefore Adam would not allocate buffers for every parameter in $W_{pretrained}$. Acknowledging the difference in memory footprint between LoRA and SHiRA during training and suggesting how this overhead may be avoided would help extend SHiRA to low-memory PEFT settings. * The pruning criterion studied for mask initialization could be expanded to include some additional, more modern criteria designed specifically for transformer based models such as Movement Pruning [3] and Wanda [4]. Further discussion on the surprisingly high performance of the SHiRA-Rand mask would be beneficial as well. * An analysis on how to distribute the sparse fine-tuned parameters across layers was not discussed. It would be particularly interesting to explore how to allocate the fine-tuning parameter budget across various modules in the network. For instance, how does the uniform strategy showcased here compare with OWL [5]? [1] D. Guo, A. M. Rush, and Y. Kim, “Parameter-Efficient Transfer Learning with Diff Pruning.” arXiv, Jun. 09, 2021. doi: 10.48550/arXiv.2012.07463. [2] A. Ansell, I. Vulić, H. Sterz, A. Korhonen, and E. M. Ponti, “Scaling Sparse Fine-Tuning to Large Language Models.” arXiv, Jan. 29, 2024. doi: 10.48550/arXiv.2401.16405. [3] V. Sanh, T. Wolf, and A. Rush, “Movement Pruning: Adaptive Sparsity by Fine-Tuning,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020, pp. 20378–20389 [4] M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A Simple and Effective Pruning Approach for Large Language Models.” arXiv, Jun. 20, 2023. doi: 10.48550/arXiv.2306.11695. [5] L. Yin et al., “Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.” arXiv, Oct. 08, 2023. doi: 10.48550/arXiv.2310.05175.
Questions
## Questions * What are the key differences between SHiRA and SFT? How does SFT compare with SHiRA in terms of generalization performance? * What are the end-to-end latency profiles of LoRA vs. SHiRA when performing both online and batched inference? I would be interested in the profiles for both the single adapter case and the multi-adapter setting. How does the latency scale as the number of adapters is increased in the multi-adapter setting for both methods? Based on Figure 7, it does not appear that SHiRA would benefit StableDiffusion 1.5 since it has a much smaller embedding dimensions than Llama (320 for the UNet vs. 4096), does SHiRA provide a latency benefit at this small embedding dimension? * What is the memory overhead required for fine-tuning with SHiRA vs. LoRA as currently implemented? Can SHiRA be engineered to be more memory efficient than currently implemented, and if so, how? * What is the effect of the dataset order in the SHiRA-WM-Non-Overlap setting? Is there an appreciable difference in performance for the dataset that is fine-tuned on the top 1% of weights vs. those datasets trained with lower magnitude weights? * The performance of SHiRA-WM-Overlap was surprising, I speculate that this may be due to the three datasets being relatively similar (QA based reasoning). Does SHiRA-WM-Overlap maintain its high performance when fine-tuned on very different datasets? For example, combining adapters trained for multiple-choice based QA reasoning and in context retrieval augmented QA (TriviaQA, for example). Another way to perform this analysis could be to compare the L2 distance between the pretrained weights and the individually trained single adapters vs. the distance between the single adapters. Are the single adapters trained on different datasets more similar to each other than the pretrained weights? * How are the number of trainable parameters determined? While performance remains competitive at 1%, it would be worthwhile to examine the trade-off between latency and generalization performance of smaller or larger adapters. * Did the authors experiment with different layerwise distributions of the fine-tuning parameter budget? For instance, only fine tuning the MHA modules or only the MLPs? Another interesting approach could be to focus the fine-tuning budget on earlier blocks as recent works [6] have found that these early blocks have a disproportionate impact on the model output. * In Appendix B.3, the SNIP code example determines the mask by selecting the topk elements in a gradient_dict member variable of the SFTTrainer class. Does this dict contain the products of the gradient and weight magnitudes? ## Suggestions: * Adding a discussion of PERP [7] to related work. While PERP aims at restoring LLM performance post pruning, it is related to this work in that it also finds that a very small portion of the pretrained parameters (<1% in some cases) can be used to fine-tune the network. * Add citation for PEFT library on L70 * The typical LoRA fusion formulation is $W_{new} = W + BA$. I suggest revising to match the original paper unless the authors prefer to explicitly define the shapes of matrices B and A. * The SHiRA-Rand baseline in Table 10 appears to be very strong. I note that random selection has been established as a robust baseline in the dynamic sparse training literature. Adding a discussion of the performance of the random mask and potentially expanding the main paper’s results to include the random mask would be of interest to the reader. [6] S. He, G. Sun, Z. Shen, and A. Li, “What Matters in Transformers? Not All Attention is Needed.” arXiv, Jul. 07, 2024. doi: 10.48550/arXiv.2406.15786. [7] M. Zimmer, M. Andoni, C. Spiegel, and S. Pokutta, “PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs.” arXiv, Dec. 23, 2023. doi: 10.48550/arXiv.2312.15230.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Adequately addressed; however, adding additional commentary on memory required for fine-tuning would help reader understand if this method is applicable to memory-constrained fine-tuning setups.
Remaining minor concerns (due to lack of space)
Continued from Author Rebuttal… Q5 [L2 distance analysis for SHiRA overlap: Are adapters more similar to each other than to base model weights?] Table 4 (and section 5.3.2 in submitted paper) have results for unstructured SHiRA-WM masks. Fig. S2 (rebuttal PDF) shows AWOM and AWOR for unstructured SHiRA masks such as SHiRA-WM overlap and non-overlap masks. As evident, the $\mathcal{A}_1^T \mathcal{A}_2$ has very similar number of zeros for both overlap and non-overlap cases. This suggests that the relative orthogonality properties between SHiRA-overlap and non-overlap would be similar and that explains why SHiRA-overlap performs well. Hence, for unstructured masks, overlap and non-overlap adapters have similar orthogonality properties. Since overlap adapters are trained on top-1% weight magnitudes, they tend to achieve slightly higher single adapter accuracy. Table S5 (rebuttal PDF) shows the L2 analysis for the adapters trained in Table 4 (submitted paper). We compute the L2 distance between each adapter and the original pretrained weights (all adapters train top 1% weights in the overlap setting) as well as the L2 distance between each adapter. Clearly, each adapter is closer to the pretrained weights compared to the other adapters. This demonstrates that the tasks are sufficiently different. We hypothesize that the main reason for the good performance of SHiRA-WM-overlap is its orthogonality properties as shown in Fig. S2 (rebuttal PDF). Q6 [#trainable params] is purely a user defined metric and depends on what size adapters we want. For all experiments, adapter sizes are kept close to LoRA for a fair comparison. Q8 [gradient-dict?] Yes, that dictionary contains the products of the gradient and weight magnitudes. Thanks for the remaining suggestions. We will incorporate the rest of them in the next version of the paper. With these new results, the paper has improved significantly. We would greatly appreciate if the reviewer could increase the rating of our work.
I thank the authors for their detailed rebuttal, clarifications, and additional analysis. I agree that SFT can be considered contemporaneous work. My other concerns regarding memory overhead and latency were adequately addressed by the rebuttal figures and discussion. Based on the above, I've elected to increase my inital score.
Thank you!
Thank you so much for increasing the rating of our work and for the very detailed feedback. It genuinely improved the quality of our work.
Summary
This paper presents a PEFT method, which applies gradient masks for downstream adaptation. It claims three main contributions: rapid adapter switching, lower concept loss, and higher rank. Experiments are conducted in the area of LVMs and LLMs. The proposed method presents SOTA performance and adapter switching efficiency.
Strengths
1. Leaving the LoRA framework may indeed bring additional benefits, such as adapter switching and high rank, etc. The motivation of this article is intuitively reasonable. 2. This paper is easy to understand. 3. It is reasonable to use classic (initialization) pruning methods such as SNIP for adaptation.
Weaknesses
1. This article highlights too many contributions. I respectfully admit that there are lots of requirements to be met in the PEFT field, such as performance, efficiency, resource costs, adapter switching, etc. However, the premise for proposing a method that is good at adapter switching is that it must perform well under a single adapter set up and on tasks in a variety of fields. Compared with tasks such as image generation and LLaMA, more basic tasks may better reflect the strength of the PEFT method. Therefore, I suggest that the authors refer to methods like VeRA, LoRA to perform experiments. In short, GLUE and image classification tasks are suggested to be added. 2. For efficiency, which is also a very necessary property for PEFT, can the authors provide the *time per epoch* and *peak GPU memory usage* on the LLaMA-2-7B model or maybe larger models? This article repeatedly emphasizes that the proposed method has very few parameters, but this does not mean that it can be more efficient than LoRA. This is because the number of parameters has no absolute relationship with GPU cost and training time. To save trouble, you can only provide the efficiency comparison between proposed method and LoRA on the LLaMA-2-7B model under single adapter (when they achieve similar performance). 3. I think this is a strong claim for high rank. Many works show that high rank may be better, but it does not necessarily mean that the higher the rank, the better the effect. This phenomenon varies depending on the model and data. For example, [1] shows that if the rank is too high, LoRA's performance may deteriorate. This is intuitively reasonable (my personal view), because a large number of parameters does mean strong expressiveness, but it may lead to a more complex optimization process, making it difficult to converge to its peak capability. Therefore, high rank is an extremely strong conclusion, which needs to be verified by comprehensive experiments. I do not recommend that authors continue to claim this contribution unless there are more detailed experimental results for support. 4. Minor: Why not consider SynFlow as one of your strategies? Because it seems that you don't need to initialize the mask for all datasets with SynFlow. [1] Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning, ICLR 2024.
Questions
N.A.
Rating
5
Confidence
5
Soundness
3
Presentation
3
Contribution
2
Limitations
N.A.
Summary
The authors propose a new type of adapter, SHiRA, for parameter-efficient fine-tuning. SHiRA selects a part of parameters for update and thus triggers both rapid adapter switching and multi-adapter fusion, while traditional methods like LoRA can't have it both ways. Experiments based on Stable Diffusion and LLaMa show the effectiveness.
Strengths
1. The writing is good and clear. 2. SHiRA significantly reduces the fusion time of LoRA-type adapters.
Weaknesses
1. The technical novelty is limited. SHiRA could be seen as a type of partial fine-tuning works which are already well explored in the past years. This paper just applies similar approach on larger models than previous works, but there is no evidence that previous works like [35,27,3,32,8] can't be used for current large models. 2. Another weakness is the value of the solution. As the authors stated, this paper aims to solve the problem that traditional methods like LoRA can't rapidly fuse weights when we need to switch between multiple adapters frequently. However, the fusion time might be far less than inference time, making the objective less meaningful, In addition, there are not many application scenarios with such requirement.
Questions
The authors are encouraged to further demonstrate the novelty and value of their proposed method in the rebuttal.
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Yes
Summary
Low Rank Adaptation (LoRA) is a crucial technique for fine-tuning LLMs and LVMs. This paper addresses two limitations of LoRA: 1. inference overhead while enabling rapid adapter switching; 2. concept loss with multiple adapters. The paper proposes Sparse High Rank Adapter (SHiRA), which directly trains a small portion of the model’s weights while keeping the other weights frozen. SHiRA has the following advantages compared to LoRA: 1. no inference overhead, 2. rapid adapter switching, and 3. less concept loss. Experiments on LVMs and LLMs validate its performance.
Strengths
1. Efficient fine-tuning techniques for large models are a significant topic in the current field of deep learning. 2. Well structured and easy to follow. 3. The proposed technique is simple and sound. 4. Extensive experiments demonstrate the effectiveness of SHiRA.
Weaknesses
My main concern lies in the novelty of the approach. Directly fine-tuning a small subset of parameters is a straightforward idea and should have been widely used even before the introduction of LoRA. I don’t see the unique innovation in the proposed method, which makes its superior performance surprising to me.
Questions
See "Weaknesses".
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
2
Limitations
The authors provide a discussion on limitations.
Summary
The paper introduces Sparse High Rank Adapter (SHiRA), a novel method to address the limitations of Low Rank Adaptation (LoRA) in some settings. SHiRA aims to minimize inference overhead, facilitate rapid adapter switching, and reduce concept loss when using multiple adapters. By training only 1-2% of the base model weights, SHiRA maintains high sparsity, enabling efficient on-device deployment. The paper provides both theoretical insights and empirical evidence to support the effectiveness of SHiRA, showcasing its superiority over LoRA in various experiments on large vision and language models.
Strengths
1. SHiRA introduces a novel PEFT method that focuses on high sparsity and selective training of base model weights. 2. SHiRA significantly reduces the memory and latency overhead associated with adapter switching, making it highly suitable for mobile and edge device deployment. Additionally, it will not introduce any inference overhead. 3. By enabling multi-adapter fusion without significant interference, SHiRA addresses a critical limitation of LoRA, thereby preserving the integrity of concurrent adapters. 4. SHiRA effectively resolves a major drawback of LoRA by allowing multiple adapters to function together seamlessly. 5. The method is tested on various large models, including language and vision tasks. 6. The paper provides solid theoretical foundations for the high sparsity and high rank properties of SHiRA.
Weaknesses
1. Lack of baseline: From Table 2, we observe that SHiRA works better than LoRA but perform worse than DoRA. Therefore, I was wondering whether the authors can provide some comparison with DoRA in vision tasks. 2. Lack of end-to-end efficiency analysis: In the Appendix, the authors provide some evidence of the efficiency of sparse adapter. However, it do not provide an end-to-end task switching time for both LoRA, DoRA and SHiRA. I think it will be better to include this result in the main paper. 3. Limited applications: Although this method emphasizes its advantages in rapid adapter switching and multi-adapter fusion, its practical applications remain questionable. As noted in Appendix B, the method primarily accelerates scenarios with a hidden dimension of 8192, potentially reaching the I/O bottleneck. However, for other settings, the fuse operation may not experience significant slowdowns. For the multi-adapter fusion, SHiRA performs better than LoRA-based methods. However, as it still lead to about 4% performance drop, making it difficult to be applied in real-world settings.
Questions
See weaknesses.
Rating
5
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
See weaknesses.
Response to Rebuttal
Thank you for your rebuttal. I will maintain the score.
Thank you for the rebuttal. Considering all the comments and the author’s replies, I’ll keep the score.
Dear Reviewers, The author-reviewer discussion period will end within two days (Aug 13). Please respond to/acknowledge the author rebuttal and indicate whether it addressed your concerns. Your help is greatly appreciated. Best, Area Chair
update
Thanks to the author's rebuttal, I believe it solves most of my concerns. However, I still have one comment, which is a continuation of weakness 1. As a PEFT method applicable to multiple fields (even if it is only applicable to image generation), it should have relatively concentrated and easy-to-understand conclusions. For example, LoRA will tell users the approximate relationship between rank and performance. As for SHiRA, as a user, what mask strategy should I adopt? The author does not seem to give specific guidance or conclusions. In addition, a very personal view is why the author hides the random mask in most of the experiments? Judging from the results in Table 10, the random strategy is not bad. I never think that this strategy should not be shown and analyzed because it sounds simple. On the contrary, I think that if the author's core argument is centered around the following, I think the insight and contribution of this article will be more meaningful: Even the simplest random gradient mask is an extremely effective means of PEFT. For the ML community, simple and effective methods have always been respected and encouraged, especially for PEFT, a field that is highly relevant to engineering applications. Therefore, for my concern, the author does not need to add additional experiments. Can the author give a simple conclusion, for example, in what filed, what mask strategy is more effective?
Further clarification on conclusions
We are very grateful to the reviewer for their feedback on our rebuttal. We are happy to see that our rebuttal addressed most of their concerns. We completely agree that it is important to have clear conclusions for any study. To address this final concern, we propose to include the following separate “Discussion” section in the paper before the “Conclusion” section, where we will summarize our key findings. ================================== __Discussion__ To summarize our main contribution, we highlight that SHiRA – when used with even the most basic pruning metrics (such as weight- or gradient-magnitude, SNIP, structured masks, etc.) – significantly outperforms LoRA on a variety of large-scale tasks in both large vision and large language domains. For LVM style transfer applications, we found that SHiRA-Struct is the most effective masking technique due to its special orthogonality properties that aid multi-adapter fusion. However, SHiRA-SNIP and SHiRA-Grad are not too far behind and achieve competitive performance as SHiRA-Struct. On the LLM commonsense reasoning side, SHiRA-SNIP is the best strategy out of the masking techniques we have considered in this work. Specifically, as discussed in the main paper, SHiRA-Struct did not achieve good results on the more complex commonsense reasoning tasks since it is a combination of a rank-1 + a highly sparse diagonal adapter. SHiRA-Grad on LLMs is about 0.8% worse accuracy than SHiRA-SNIP (76.6% vs. 77.4% average accuracy on commonsense reasoning for LLaMA-1). Therefore, in conclusion, for the applications/fields and the masking techniques considered in this paper, SHiRA-SNIP works well across both language and vision domains. Hence, we recommend that SHiRA-SNIP is one the strongest candidates for sparse finetuning. ================================== As for the SHiRA-Random, we completely agree with the reviewer that the random mask is in fact a strong baseline (Reviewer AaSE also pointed this out). The main reason we could not include it in the main paper was the space limitation during the initial submission. We wanted to include it in the main paper Table 1 to present HPS scores for SHiRA-Rand. However, that would have also required us to put the generated images of the SHiRA-Rand baseline in the main paper. We really did not have the space to include it at the initial submission time. For the next version of the paper, we will move some of these results from the Appendices to the main paper. About the final but important comment by the reviewer: "_On the contrary, I think that if the author's core argument is centered around the following, I think the insight and contribution of this article will be more meaningful: Even the simplest random gradient mask is an extremely effective means of PEFT. For the ML community, simple and effective methods have always been respected and encouraged, especially for PEFT, a field that is highly relevant to engineering applications._" We completely agree with the reviewer on this statement (as we also mentioned in our rebuttal previously). We have also included it in the new “Discussion” section as specified above. Further, we will modify the abstract, introduction, and conclusion sections accordingly to make it clearer that this is one of the fundamental contributions of our work. We really appreciate the reviewer’s detailed feedback. It has helped improve our work significantly. Please let us know if you have additional concerns. We would be grateful if you could please raise the rating of our work. Thank you.
update
Most of my concerns are addressed, and it is obvious that authors make enough efforts for rebuttal. Thus I increase my score to 5. Overall, from my personal view, the biggest pros and cons with simple words are as follows: Pros: the proposed method has almost all properties as a good peft method. (better to consider more general tasks to verify effectiveness) Cons: there seems no clear insight proposed. From my view, deep insight and more discussion about random mask manner will be a significant contribution and a very attractive element for others to follow this work.
Thank you!
Thank you so much for all the feedback and for increasing our rating. As a result, the quality of our paper has significantly improved. And yes, we agree that the properties of the random mask would be a very interesting follow up work.
Thank you for providing the rebuttal. After reading other reviews and rebuttals, I tend to maintain my initial recommendation.
Decision
Accept (poster)