Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone

Summary

This paper provides a unified framework (called Res-Tuning) to combine different efficient tuning methods. It introduces an unbinding form that integrates existing methods and allows combination flexibility. A memory-efficient variant is also introduced for the sake of training memory efficiency. Experiments are performed on discriminative tasks (CIFAR-100 and VTAB-1K benchmark) and generative tasks (Text-to-image generation on COCO), demonstrating the proposed framework's flexibility and efficiency.

Strengths

1. This paper is well-organized, and the presentation is clear. 2. The formulation of the unified and unbinding form is sound and straightforward and is novel to this field of research. 3. The unbinding form allows combination flexibility, which is the main strength of the proposed methods. 4. Empirical evaluation and theoretical analysis verify the equivalence between the unbinding form and existing approaches. 5. Empirical evaluation on different standard benchmarks shows better performance among existing approaches.

Weaknesses

My main concerns about this paper are the comparisons with previous works. From my perspective, this paper partially draws inspiration from [13] (MAM-Adapter), which also struggled to unify parameter-efficient tuning methods such as Adapter, Prefix or Prompt tuning. All these methods are evaluated in NLP tasks rather than vision tasks that this paper used. On vision tasks, VPT [18] is particularly proposed to explore PETL methods in vision (rather than providing a unified view), Thus, the evaluation following VPT is not sufficient for this work. Besides, VPT can work well with Convolution-based foundation models and self-supervised pre-trained models (e.g., MAE, MoCo V3). I don't find such discussion in this work.

Questions

Why does this paper not evaluate the proposed method on NLP tasks that most parameter-efficient tuning methods evaluated on? This paper only includes the reproduced results of MAM-Adapter on CIFAR-100 without any discussion on NLP benchmarks. Since this paper aims to introduce a unified framework for efficient tuning methods, it is necessary to include empirical evaluations on commonly used benchmarks. I would increase my score if sufficient results or discussion were provided.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are not adequately discussed in this paper. It is unknown whether the proposed method can generalize well. Negative societal impacts are unknown.

Summary

This paper proposes a new tuning paradigm, dubbed as Res-Tuning. They first introduce the basic building blocks of foundation models and then unbind three popular tuners from foundation models. They provide theoretical and empirical evidence to support their structural disentanglement. By detaching from the foundation models, they further propose a memory-efficient variant of Res-tuning, dubbed as Res-Tuning-Bypass. They conduct extensive experiments on both discriminative and generative tasks to demonstrate the superiority of their method.

Strengths

- This paper is well-written and the illustration of this paper is concise and easy to understand. - The idea is simple yet effective. - They conduct extensive experiments and the results demonstrate superior efficacy and efficiency on both discriminative and generative tasks.

Weaknesses

- There are some detail errors in line 124 and line 130. The reference seems to be Fig.3c, not Fig.3b. And there are some punctuation errors in Eq.(8). There are many similar issues in the article, the author should check this paper again to correct them. - Code is unavailable now and open-source as soon as possible is beneficial for expanding the influence and credibility of this paper. - The author provides detailed derivation in supplementary materials, but the manuscript is hard to understand. It would be helpful if the deriving process have a more detailed explanation.

Questions

- Please see the weaknesses part. - There are no queries in FFN/Block, How did you apply ResPre/ResPro in FFN/Block? - The form of adapter tuning in Eq.(6) and Fig. 2c is inconsistent. Is there an FFN before the adapter of the parallel branch? - LoRa [1] is an effective method in PETL. Adding a comparison with Lora in discriminative tasks can make this paper more solid. - The full fine-tuning and linear probing results in Tab.3 are inconsistent with the results in SSF[2]. The results of the same task (CIFAR-100) and model (ViT-B/16) are relatively low. #### Reference [1] E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In Int. Conf. Learn. Represent., 2021. [2] D. Lian, D. Zhou, J. Feng, and X. Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Adv. Neural Inform. Process. Syst., 2022.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

2 fair

Contribution

3 good

Limitations

The transfer ability of this method depends to a large extent on the performance of the upstream model. This method shares the same vulnerability as existing PETL solutions.

Summary

This paper shows some existing parameter-efficient tuning methods can be decoupled from backbones, which can be formulated as a unified Res-Tuning model. Furthermore, the authors conduct empirical experiments to seek the optimal Res-Tuner. Additionally, a memory-efficient variant of Res-Tuning is introduced by combining outputs of backbone with previous Res-Tuner, which can only compute the gradients for Res-Tuning module. The experiments are conducted on several downstream tasks.

Strengths

+: Compared with parameter-efficient tuning, memory and speed efficient tuning methods show more practical in real-world applications. The proposed Res-Tuning-Bypass show clearly superior to memory-efficient counterparts, i.e., Side-Tuning, LST and linear probing. Meanwhile, Res-Tuning can show speed-efficient on multi-task learning. +: The proposed method is easy to implement, and is clearly described.

Weaknesses

-: The contribution of this work could be further clarified. (1) The statement on unbinding formulation seems a bit overclaimed. For tuning parameters of large models, parameter-efficient modules can be naturally designed in a cascaded or parallel manner. For parallel structures, outputs of frozen backbones and learnable branch can be fused by element-wise addition or multiplication, and such kinds of structures are often used for design network architectures. For Res-Tuning model, it can be regarded as a parallel structure with element-wise addition for fusion. Therefore, it is hardly regarded as a novel or special structure. (2) The authors show existing PETL methods have equivalent counterparts in the unbinding formulation, but I feel a bit confused about necessity of this conclusion. Besides, derivation of equivalent on adapter seems be less rigorous. I am not sure could such conclusion help to design Res-Tuner? Because the optimal Res-Tuner seems be decided by empirical experiments. (3) I feel a bit confused about the relationship between the unbinding formulation and memory-efficient variant of Res-Tuning. In my opinion, memory-efficient Res-Tuning can be regarded as an improved LST, where an adapter is used for outputs of frozen backbones before fusing with learnable branches. Besides, learnable modules of LST and Res-Tuning are different. The authors would better give more discussion about above issue. -: Some issues about experiments. (1) As shown in Table 3 (c), why Linear probing can be improved when only Bypass is used? The authors would better give more discussion. (2) In line 172, improvement of 0.94\% should be corrected to 0.92\%? -: The writing can be further improved. (1) Line 124, Fig. 3b -> Fig. 3c (2) FFN$_{adapter}$ in Eq. 6 is inconsistent with description of Fig. 2c.

Questions

Please see paper weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes an unbinding formulation of parameter-efficient methods and further leverage structural disentanglement to develop a memory-efficient variant. Sufficient experiments on both visual discriminative and text-to-image generative tasks are performed.

Strengths

1. novel implementation. 2. good motivation, which is reasonable for me. 3. good evaluation, promising results, and easy to follow.

Weaknesses

1. The proposed Res-Tuning is not that novel and the analysis of several PET approaches is similar to that in Towards a unified view of parameter-efficient transfer learning (ICLR2022). Parallel adapter design is already a consensus, minor changes seem trivial. Rather than this boring repetition, I find the memory-efficient variant i.e., Res-tuning-bypass is meaningful and novel. I suggest that the authors make this section a priority and restructure the article. 2. Following Lader Side Tuning, the proposed Res-tuning-bypass can free the backbone from backpropagation. In this way, the training time will be reduced, however, only the training time for generating tasks is reported. Thus I am curious about the training time for visual recognition tasks. 3. The authors compute inference time for multi-tasks, although it seems a bit trivial, I recognize the significance of this time reduction. However, I cannot accept the comparison with just non memory-efficient methods in Figure 4. The authors should report LST's results and discuss them carefully.

Questions

The authors employ the CLIP model for text-to-image generation, however, the generative model is more difficult to evaluate its performance. CLIP-based Parameter efficient approaches have also gained extensive attention recently, such as CoOp[1], PLOT[2], MaPLe[3], and CoPrompt[4]. I am curious whether the proposed Res-tuning-bypass can also reduce the memory cost for vision-language PET approaches with small performance degradation. [1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022. [2] Chen G, Yao W, Song X, et al. PLOT: Prompt Learning with Optimal Transport for Vision-Language Models[C]//The Eleventh International Conference on Learning Representations. 2022. [3] Khattak M U, Rasheed H, Maaz M, et al. Maple: Multi-modal prompt learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 19113-19122. [4] Roy S, Etemad A. Consistency-guided Prompt Learning for Vision-Language Models[J]. arXiv preprint arXiv:2306.01195, 2023.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have addressed the limitations.

Summary

This paper pays attention to Parameter Efficient Tuning and propose a unified framework namely Res-tuning. More importantly, based on the proposed unified framework, this paper constructs a memory optimization scheme similar to the LST in the language model. Several experiments are performed to validate the effectiveness , including over visual recognition tasks including VTAB-1k, Cifar-100 and text to image generation tasks on COCO and Oxford Flowers and Food-101.

Strengths

1. I am gald to see the success of Res-Tuning-Bypass, an efficient memory PET that is the counterpart of LST in visual tasks. 2. Sufficient experiments are conducted and Res-Tuning achieves impressive performance gain compared to state-of-art approaches. 3. The paper is well structured and easy to follow.

Weaknesses

1. The novelty of unbinded Res-tuning framework seems limited. As for me, Convpass [1] and AdaptFormer adopts a parallel module for PET, which is similar to Res-tuning. 2. The analysis of prompt tuning, prefix tuning, and adapter tuning is similar to that in [2], which is also cited in the paper. For me, the major contribution of this paper is the Res-tuning-bypass, which brings the success of memory-efficient side tuning to visual tasks. 3. The research of parameter efficient tuning has gained extensive attention recently. More recent related works should be added and described or even compared. In the work, only VPT, SSF, NoAH, and Adaptformer are used as baselines. More recent work such as Convpass[1], SNF[3] should be carefully examined. 4. Lacking some important experiments, such as few-shot experiments on FGVC and Domain Generalization experiments on four ImageNet related datasets (ImageNet-A, ImageNet-R, ImageNet-V2, ImageNet-Sketch). [1] Jie, Shibo, and Zhi-Hong Deng. "Convolutional bypasses are better vision transformer adapters." arXiv preprint arXiv:2207.07039 (2022). [2] He J, Zhou C, Ma X, et al. Towards a Unified View of Parameter-Efficient Transfer Learning[C]//International Conference on Learning Representations. 2021. [3] Wang Y, Shi B, Zhang X, et al. Adapting Shortcut With Normalizing Flow: An Efficient Tuning Framework for Visual Recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15965-15974.

Questions

1. This approach has only been evaluated on ViTs, and I'm curious if Res-tuning-bypass will work on CNNs. 2. The performance of Side Tuning is quite poor, and VPT proves that this memory-saving design is not suitable for visual tasks. This paper replicates the success of LST for the first time in a visual task, and I am very curious about the implementation process. However, the source code is not submitted in the supplementary material. If the authors can provide anonymous github project link in the rebuttal and I am glad to raise my score.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have discussed the limitaions and the potential societal impacts in the supplementary material.

Please respond to authors' rebuttal

Dear Reviewers, Thanks for your contributions! Please respond to the authors after carefully reading their rebuttals and other reviews. If your assessment of the paper changes, please update your score with a short justification for the new rating. Thank you, AC

Comment by Reviewer hNoJ

Thank the authors for their feedback. The authors provide sufficient evidence to resolve my concerns. I will raise my score.