Expanding Sparse Tuning for Low Memory Usage

Summary

This paper proposes a method called SNELL for vision model fine-tuning. It extends matrix decomposition from a kernel perspective and designs a novel sparsification mechanism for end-to-end parameter selection. Experimental results show that the proposed SNELL achieves low memory requirements and high performance on downstream tasks simultaneously.

Strengths

1. The proposed method is novel and can achieve low memory requirements and high performance concurrently, which is valuable for practical applications. This is particularly noteworthy as many existing methods primarily focus on reducing tunable parameter volumes rather than memory usage. 2. The proposed method can perform the data-dependent parameter selection and tuning in low-resource scenarios. I think this approach has the potential to advance the study of other areas, for example, model editing. 3. This paper conducts extensive experiments, especially on large-scale vision models including ViT-L and ViT-H, demonstrating the high performance and memory efficiency of the proposed SNELL. 4. This paper extends LoRA with a kernel perspective for sparse tuning. The kernel perspective contributes to new insights into LoRA and promotes its further improvements. 5. This paper is well-written and easy to follow.

Weaknesses

1. It would be better to introduce the utilized kernel function in the method section for the first time rather than in the experiment section. 2. The authors should mention in the paper that the implementation details of Figure 4(a) are located in Appendix A.3, for readers to easily access the specific details of this figure. 3. For Figure 3(b), authors should explain why the memory savings of SNELL are more significant for larger models. 4. Although the paper includes comprehensive experiments, I still want to know the comparison between the proposed SNELL and other visual PEFT methods, such as MOSA [1] and VQT [2]. [1] Zhang et. al., “MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning”, Arxiv:2312.02923. [2] Tu et. al., “Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning”, CVPR 2023.

Questions

Please check detailed comments in the Weaknesses part.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

NA

Summary

This paper focuses on the sparse tuning task of the PEFT methods, by introducing the kernel function modeling applied into the LoRA two low-rank matrices or adapters, which costs low memory usage and leads to better performances. To achieve this target, a Kernelization for LoRA is proposed to map two low-rank matrices dot-production from a lower rank r to a implicit higher rank d, such that leading to stronger model capacity based on relatively marginal memory cost. Further, a soft mask adaptive strategy is used to implement the sparsification for fine-tuning so as to reduce the memory usage. Extensive experiments are presented to validate both the memory efficacy and performances improvements.

Strengths

1. Mapping the low-rank matrices dot-production into an implicit space using conventional kernel function sounds like a promising direction, since some typical machine learning methods help with the representation ability of the model while reducing marginally small computation cost. 2. The experimental results are extensive and the ablative studies showcase convincing result. 3. The performances look superior to most of previous works, including both the adapter methods and weights decomposition methods. 4. The presentation and writing of this paper are easy to follow.

Weaknesses

1. The Competition-based Sparsification Mechanism is kind of overclaim by this paper, because the primary idea of this soft-thresholding is widely utilized to re-activate or de-activate some neurons or bits so that leading to sparsification. But competition-based is not really demonstrated on this point, neither showing competition among each bits of the mask nor clear explanation of the ∆W computation. Can you make a better clarity? 2. For the memory usage comparisons among LoRA, Adapter and SNELL, it seems SNELL displays little advantages over these works. I suppose that the improvements comes from the kernel function modeling as this implementation enable the model with stronger capacity to adapt for downstream task, such as the non-linear modeling ability, etc. Can the author support more comprehensive explanation and experimental comparisons? 3. Some mistaken typos are presented in the paper writing. In Line6, 'accompanied by increases in' should be 'accompanied by increasing in'.

Questions

1. Among Line47-48, I am confused about fine-tuning implementation that storing only the tunable weight into the optimizer using such as Pytorch is still practical. Hence, we may not need to put all the pretrained and frozen weights into the specific optimizer during the training phase. Can the authors make a clear explanation about this part? The other questions please refer to the weaknesses above.

Rating

5

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

Overall, the presentation of this paper and the extensive experiments of this paper sounds promising. However, I still have the confusions above in terms of the motivation or the actual contribution of the Sparsification part. I will keep watching the responses from the authors to evaluate if my concerns are well addressed.

Summary

This paper proposes a method called SNELL for achieving sparse tuning of pre-trained models with low memory usage. The authors employ LoRA to reduce the number of learnable parameters in the optimizer and utilize kernel tricks to ensure the merged matrix maintains a high rank. Additionally, during the sparse training phase, the authors introduce a competition-based sparsification mechanism to avoid the storage overhead of weight indices. Experiments on two benchmarks demonstrate that this method achieves leading performance while reducing memory usage.

Strengths

1. This work introduces SNELL, a sparse tuning method based on LoRA, which combines the memory efficiency of LoRA with the performance of sparse tuning. 2. Experiments conducted on two benchmarks show that SNELL consistently outperforms LoRA and compares memory usage with the baseline. The authors also conduct comprehensive experiments on different pre-training methods, model architectures, and component ablations. 3. The paper is well-organized and well-written, making it easy to follow.

Weaknesses

1. The idea of this paper isn't that novel. The overall approach of this work is a combination of existing fine-tuning techniques, LoRA and sparse tuning, with improvements to LoRA relying on existing kernel tricks. The competition-based sparsification mechanism merely sets a soft threshold. 2. As a work on sparse tuning, it’s really a pity that this paper does not compare with the latest pure sparse tuning work, GPS[1], nor is it mentioned in the related work section. This severely limits the contribution of this work. In fact, comparing with the state-of-the-art sparse tuning work is essential. If the sole aim is to reduce memory usage at the expense of performance, it may not be very meaningful. 3. As a PEFT method, the paper does not report the number of trainable parameters, citing the difficulty of calculation due to the sparse tuning method. This is unreasonable to me, as other sparse tuning methods (e.g., GPS) have reported this metric. This metric is also an important evaluation standard for PEFT. 4. Generally, reparameterization-based methods refer to those where the additional modules can be reparameterized into the original model after training, such as LoRA and SSF[2] (which is also not compared with SNELL). In contrast, methods like Bitfit, Partial, and sparse tuning are not in this category and are referred to as specification-based methods in reviews like Delta-Tuning[3]. [1] Zhang Z, Zhang Q, Gao Z, et al. Gradient-based Parameter Selection for Efficient Fine-Tuning[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 28566-28577. [2] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123. [3] Ding N, Qin Y, Yang G, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models[J]. arXiv preprint arXiv:2203.06904, 2022.

Questions

1. During the experiments, was there a performance comparison with sparse tuning applied to the entire pre-trained model? 2. The paper mentions that the competition-based sparsification mechanism can save memory usage for weight indices. What proportion of the total learnable parameters does this memory usage represent? 3. In the sparse tuning part, have you tried fixing the sparse matrix during training to evaluate its effect?

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The paper mentions that to achieve sparsification, the re-computation of the LoRA merged adaptation matrix ΔW can lead to reduced computational efficiency. Although the authors explain that designing appropriate GPU operators can solve this problem, this may not be feasible in practice or may incur higher costs. If this approach can solve the problem, then the high memory usage issue of sparse tuning itself could also be addressed similarly.

Summary

The paper introduces SNELL (Sparse tuning with kerNELized LoRA), a method aimed at reducing memory usage during PEFT of large pre-trained models. SNELL achieves this by decomposing the tunable weight matrices into low-rank matrices and utilizing a competition-based sparsification mechanism, thereby avoiding the storage of large weight matrices and tunable weight indexes. This approach is demonstrated to achieve SOTA performance on multiple visual recognition tasks while maintaining low memory usage, making it viable for large-scale models.

Strengths

1. The introduction of kernelized LoRA for sparse tuning is a novel approach that effectively combines the advantages of LoRA's low memory usage with the performance benefits of sparse tuning. 2. SNELL significantly reduces memory usage compared to traditional fine-tuning and other PEFT methods, which is critical for scaling to larger models. 3. Extensive experiments demonstrate that SNELL achieves state-of-the-art performance on various benchmarks, including FGVC and VTAB-1k, while maintaining low memory usage.

Weaknesses

1. While the focus is on visual recognition tasks, it would be beneficial to explore the applicability of SNELL to other domains such as natural language processing. I will expect the authors can further show some results on the large language models. 2. While LoRA offers benefits such as reduced memory usage, quick task switching, and the ability to serve multiple adapters, SNELL's use of dynamic masking during fine-tuning negates these advantages. I'm curious if the authors could provide results using pre-defined fixed masks instead, which might help retain some of LoRA's beneficial properties.

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

See weaknesses.

Response to Rebuttal

Thank you for your response. Regarding the additional results on LLMs, I noticed that the accuracy is lower than that of Dora [1]. As a result, the performance of SNELL is not sufficiently impressive. Concerning W2, the issue is that loading and computing adapters stored in an unstructured sparse format can be slow, which, in my view, limits SNELL's applicability compared to LoRA. Given these considerations, I will maintain my current score. Reference: [1]: Liu, Shih-Yang, et al. "Dora: Weight-decomposed low-rank adaptation." arXiv preprint arXiv:2402.09353 (2024).

Concerns largely addressed

I appreciate the detailed explanations provided by the authors. Your responses have largely addressed my concerns. However, I believe it is still important to conduct a thorough literature review and compare your work with significant existing works. In the attached PDF, I noticed that the performance of SNELL is not particularly impressive, with minimal differences compared to other methods and its own ablations (such as removing kernelized LoRA and fixing the sparsification method). Additionally, the authors should emphasize the motivation and novelty of the paper more prominently. Considering the above points, I have increased my score to 5.

Rebuttal Response

Somehow the authors have addressed my concerns. Therefore, I maintain my initial score.