Parameter-efficient Tuning of Large-scale Multimodal Foundation Model

Summary

This paper proposes a novel approach to address the challenge of high learning costs when migrating large models to specific downstream tasks. The proposed method aims to reduce task complexity and improve the consistency across the different modal outputs of multimodal models. The authors employ a LoRA-like technique that involves fine-tuning the transformer by appending adjustment matrices to the qkv matrices. This process is further optimized through CP Decomposition, an approach used to reduce the scale of the fine-tuning parameters. Then use Informative Context Enhancement mechanism to compute weights and the mixed graph-text features are adjusted according to these weights, facilitating the generation of superior fused features. To prevent the loss of textual information during the alignment of deep multimodal networks, the authors propose using a Gated Query Transformation. This approach calculates the blending ratio of text features for enhancement. The paper claims that these techniques allow the model to outperform the current best methods on several downstream tasks, using fewer fine-tuning parameters, and even surpasses fully fine-tuned methods.

Strengths

（1） Innovative and Effective Approach: Building upon the foundations of similar works like LoRA, this paper makes significant strides by achieving better fine-tuning performance with lower parameter usage. （2） Proposes Novel Techniques: The paper introduces innovative methods such as Informative Context Enhancement and Gated Query Transformation to enhance the fusion of modalities. These methods appear to be highly effective in improving model performance, and could potentially be applied in a range of different contexts. （3） High-Quality Visualizations: The paper includes aesthetically pleasing and informative figures and tables, which contribute to the clarity and overall quality of the work. The visualizations effectively aid in the understanding of complex concepts and methodologies.

Weaknesses

(1) Not Clearly Written: There appears to be some confusion in the terminology used, especially around 'soft prompts'. While the paper asserts that its fine-tuning approach can be seen as a 'soft prompt', the proposed method is more akin to parameter-efficient transfer learning. Therefore, the use of the term 'prompt' may not be accurate in this context. A clearer and more precise usage of technical terminology would be beneficial for readers and would also strengthen the overall quality of the paper. (Now author states that in the final version, they will replace the inappropriate term "prompt" with "multimodal parameter-efficient transfer learning based on mode approximations" to better reflect the essence of our method.) (2) Lack of Details on Gated Query Transformation: The paper does not provide a clear and thorough explanation of the implementation details for the Gated Query Transformation. For future improvements, it would be beneficial to include more technical details of this novel technique, which would improve the clarity of the paper and make it easier for others in the field to replicate and build upon this work. (Now author states the final version will be more detailed) (3) Manual Parameter Tuning: The paper suggests that the rank hyperparameter needs to be manually adjusted, which could pose an obstacle to scalability and efficiency. More research is required on how to optimally choose this hyperparameter's value. (Author said they use a detailed grid search experiments to get good rank)

Questions

(1) Justification for Terminology: Could you clarify why you choose to refer to your method as a 'multimodal prompt'? Given the technique's resemblance to low-rank fine-tuning, the terminology might be misleading. Understanding the rationale behind this terminology would greatly aid in interpreting your work. (2) Clarification on Gated Query Transformation: The paper could benefit from further details on the application of the Gated Query Transformation. Specifically, is the text information fused with the image features before entering the cross-attention, according to the gate values? This part of the methodology was not entirely clear in the paper, and further explanation would enhance the clarity of your methods and potentially support the broader understanding and application of your techniques.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

(1) Hyperparameter Selection: The need for manual selection of the rank hyperparameter is a key limitation of the proposed method. While it is acknowledged by the authors, there is no substantial discussion on how this limitation can be addressed. (2) Training Time: Another potential limitation that should be acknowledged is the impact on training time. Although the proposed method reduces the parameter count for downstream tasks, it doesn't seem to significantly decrease training time.

Summary

This paper aims to design a lightweight prompt tuning method (i.e. Aurora) for cross-modal transfer. The main idea follows the observation by LoRA [15] that most of the features are redundant and a low-rank ∆W can be learned to adapt the features. Different from LoRA, they adopt CP decomposition [47] to decompose the learnable parameters into a series rank-one tensors. They also propose Informative Context Enhancement and Gated Query Transformation for better modality alignment. However, the connection between the two parts is not clear. Experiments show that Aurora performs better than LoRA and is at least comparable to full finetuning methods.

Strengths

Strength 1. Figure 1 highlights the difference between the proposed method and the baselines. 2. The proposed method is lightweight and effective. 3. Aurora is applicable to both image-text and video-text retrieval.

Weaknesses

Major 1. The main idea of the paper is lightweight adaptation achieved by adopting CP decomposition [47] for the adapter weights. While it is shown to be effective and parameter-efficient, there is limited technical novelty introduced by this work. In terms of story-level novelty, it mainly follows the observation and approach in LoRA [15], and it’s not considered novel. 2. The improvement of Aurora over UniAdapter [32] is marginal in Table 1. 3. What are BLIP, BLIP+LoRA zero-shot performance in Table 4? 4. The new thing of this paper is Informative Context Enhancement and Gated Query Transformation. However, there is loose connection between the main idea and these two modules. These two modules are orthogonal to the low-rank approximation part. What will be the performance if LoRA is integrated with these two modules? 5. Which part of the method does Parameter Sharing in L248-L255 correspond to? Minor 6. What are x-axis and y-axis in Figure 7? 7. The main flow of the architecture is not clear. In section 3.3, Gated Query Transformation is presented after Context Enhancement. However, Gated Query Transformation manipulates f, but Context Enhancement depends on f. Is the former performed prior to the latter? 8. The insight for Gated Query Transformation is unclear. How about replacing t’ in L191 with t?

Questions

The author is suggested to address the concerns in the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation is mentioned at the end of the paper.

Summary

This paper proposes a parameter efficient adaptation technique Aurora for multi-modal models. Particularly, the proposed method motivates their design by suggesting that the original pre-trained weight matrices have redundancies due to their high dimensional nature and the downstream task often requires low-dimensional reparameterization only. Aurora supplements the original weight matrices with series-one rank tensors which are only learned during the fine-tuning process. In addition, to enhance the modality alignment between the vision and text representations, Aurora utilizes informative context enhancement module and gated query transformation which fuses and explicitly relates the textual representations with the multi-modal fusion representations in the cross-attention block of BLIP. Extensive experiments over various benchmarks shows the effectiveness of Aurora in comparison with fine-tuning and parameter efficient adaptation approaches.

Strengths

Strengths: (1) The idea of decomposing learnable pre-trained matrices into small rank tensors is encouraging, as it explicitly allows to adapt only necessary amount of parameters for efficient and effective adaptation. (2) The paper has performed extensive evaluations with proper ablation studies which justifies their design choices. (3) The proposed methods performs favorably well with very less number of learnable parameters.

Weaknesses

Weaknesses: (1) The overall paper presentation style is very confusing, specially the main methodology section. There is no preliminaries on the baseline architecture on which the proposed solution has been built. It is very difficult for the readers to grasp the contents without knowing the main model architecture. For example, in line 161-162, the authors have mentioned the cross-attention module, but unfortunately no prior information about that block is provided anywhere in the manuscript. Also the writing is not clear and I found it difficult to understand the manuscript. (2) How the proposed solution is considered as a prompt learning variant? If I understood correctly, the additional learnable parameters are utilized as part of the model and the input tensor has to be multiplied with it. It is not the case where the learnable parameters are part of the inputs, which is the core definition of prompt learning. (3) The proposed multi-modal alignment module seems to be heavily designed for the BLIP multi-modal model. It is not clear if these components could be utilized in other multi-modal models. It will be good to see the generalization of proposed approach to other recent VL models.

Questions

Please refer to the weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have discussed the limitations and societal impacts are highlighted.

Summary

The paper addresses the problems of (i) transfer learning and (ii) reducing the multimodality gap in multimodal models. To address (i) it presents a technique which can be viewed as a generalization of LoRA; instead of independently factoring representing matrices as a low rank representation, all the matrices of the transformer get stacked together and are represented as a low rank tensor. To address (ii) two techniques are presented: one which aims to improve the representations by allowing information exchange between different examples in the batch (Informative Context Enhancement) and another which aims to prevent the loss of text information for deep models (Gated Query Transformation). Ablation studies are performed to justify the different design choices.

Strengths

1. Mode approximation (Generalizing LoRA by stacking the matrices of all the transformer layers and using CP decomposition to represent that) is a nice idea and experimentally shows to be a parameter efficient way (beats LoRA) of adapting frozen models to new domains 2. Thorough ablation studies are performed to demonstrate the impact and justify existence of each presented component in the final model.

Weaknesses

1. Informative Context Enhancement seems to allow information exchange between different examples in the batch. That makes it dependent on the batch size, but the impact of changing it is not evaluated.

Questions

1. Why are some methods present in ‘Methods with frozen backbone’ in Table 2 omitted from the same section in Table 3? (e.g. LoRA) 2. Is |B| in the ‘Informative Context Enhancement’ section referring to the number of tokens in the entire batch?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

The paper proposes AURORA, a method which uses mode approximation to boost the knowledge transfer in Vision Language models and enhances alignment between the modalities in a lightweight parameter efficient manner. In addition to this, the paper further proposes Context Enhancement module and Gated Query Transformation module to boost the modality fusion in an adaptively controllable way. The proposed method is evaluated on six cross-modal tasks and two zero-shot tasks and is compared to existing PETL methods.

Strengths

- The paper seems to be the first which has used mode approximation in Vision-Language models to efficiently achieve modality fusion. - The proposed method is novel with respect to existing prompt tuning methods. - The proposed methods works by tuning a very small number of parameters which can save time and computational resources.

Weaknesses

- The motivation for using Mode approximation is not very clear in the paper. For instance there could be other methods which can solve the redundancies in the attention weights. There is no detailed explanation of the theoretical basis or formal analysis of how mode approximation aids in prompt learning. Adding a brief theoretical background could provide more insights into the method's underlying principles. - The authors haven't provided any solid proof behind the argument regarding loss of textual information in multi-modality fusion branches, which essentially forms the basis of introducing Gated Query Transformation. - The authors should have compared their method with previous prompt learning methods [1, 2] for zero-shot out-of-distribution classification tasks. [1] M. U. khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, “Maple: Multi-modal prompt learning”. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023 [2] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models”. International Journal of Computer Vision (IJCV), 2022

Questions

- Can the method be applied to a broader range of vision-language tasks, and how well does it adapt to different modalities or data distributions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Please have a look at Questions and Weaknesses

Response to reviewer Srms

Thank you once again for providing us with your valuable feedback on our paper. We are grateful to learn that our responses have successfully addressed your concerns. Your efforts in reviewing our work, as well as your insightful comments and support, are sincerely appreciated. Your suggestion has been invaluable in shaping the final version of our paper. We genuinely value your contributions and will ensure that your valuable suggestions are carefully incorporated.