Summary
The paper discusses the transfer of a visual prompt generator (VPG) across different vision-language language models (VL-LLMs) to reduce computational costs. VL-LLMs include a VPG module that bridges the gap between vision and language, encoding visual inputs into fixed-length soft prompts. To reduce the cost of building new VL-LLMs, the authors propose a two-stage transfer learning framework, VPGTrans, for VPG transfer across different LLM sizes and types. VPGTrans includes a projector warm-up (stage-1) and vanilla fine-tuning (stage-2). They conduct exploratory analyses to determine key factors for transfer efficiency and show that VPG transfer across frozen LLMs is feasible and can lead to substantially lower computational costs. The authors demonstrate the practical value of VPGTrans by customizing two novel VL-LLMs with recently released LLMs. The paper highlights the need for further research into VPG transfer across LLMs to further reduce the cost of building new VL-LLMs. Overall, the paper's main contributions include identifying the feasibility and effectiveness of VPG transfer across LLMs, smaller LLM sizes leading to more speed-up and better performance during transfer across LLM sizes, and VPG trained on smaller LLMs outperforming larger ones in most conditions, and then propose a transfer learning framework with empirical evidence supporting its efficiency.
Strengths
1. The multimodal large language model (MM-LLM) is considered the most promising path towards achieving Artificial General Intelligence (AGI). However, a major challenge is the significant computational cost associated with training an MM-LLM. This obstacle hinders the widespread research and exploration of this topic. The presented paper offers an excellent pilot study that demonstrates the feasibility of VPG transfer. By showcasing the viability of VPG transfer, the paper provides valuable insights into mitigating the computational cost of training MM-LLMs. This finding has significant implications for future research in this field. The pilot study serves as a promising starting point, paving the way for further investigations and advancements in the development of MM-LLMs with higher efficiency.
2. I quite enjoy the didactic/exploratory nature of this paper. I like that the paper reads like an investigation, and begins with a preliminary exp, as well as diagnosing reasons for the case, before then presenting the VPGTrans method. And then they delve into the transfer of VPGs under two settings. This, from the shallower to the deeper, feels more organic and the lessons learned along the way are insightful. Also the introduction with clear bullet point allows easy reading.
3. The proposed method, VPGTrans, can be fairly straightforward but effective, which is a merit.
4. The experimental work in this study is extensive and solid. It includes validation across a wide range of LLM sizes and types, providing thorough coverage.
5. This work yields numerous interesting and meaningful findings, which will serve as important empirical evidence for future explorations in LLM/MM-LLM efficiency. The conclusions drawn from this research have the potential to guide and shape subsequent investigations in this field.
6. Additionally, this paper makes good contributions by introducing two new state-of-the-art MM-LLMs in the community: VL-LLaMA and VL-Vicuna. Overall, I like this paper, and I believe this work will show important impact to the community.
Weaknesses
1. The major possible limitation of this method is its reliance on the existence of a well-functioned VPG model, and without this assumption, this work is of less meaningfulness. How does the quality of the existing VPG influence the transfer efficacy and efficiency?
2. This work evaluates on merely the captioning and VQA tasks. It would be interesting to see the performance on other different VL tasks and more benchmark datasets, to see the broad-coverage trends.
3. On the other hand, the proposed VPGTrans, though straightforwardly simple, relies too much on the empirical tuning, e.g., when to inherit the raw VPG and when to tuning or fixing some modules. Are there any intuitions and theoretical supports to build up the method?
4. When transferring the VPG, what has been changed of the visual soft prompts? Are the visual prompts of the raw VPG changed, to adapt to the new LLM backbone?
5. The claim of ‘our VPGTrans helps achieve a BLIP-2 ViT-G OPT2.7B→6.7B transfer with less than 10% of the GPU hours and 10.7% training data required for the original model training’ can be a little misleading, as the comparison between VPGTrans and the training of the raw VL-LLM is not strict fair and counterpart; should compare the VPGTrans with the hard VPG transfer method and then draw the conclusion.
6. Lack of exploration of potential limitations or drawbacks of the proposed method, which may limit its generalizability or practical usefulness.
7. How did you choose the specific VL-LLMs for your experiments?
8. In Section 4.2, you mention that the VPG trained on smaller LLMs may outperform larger ones in most conditions. Can you provide more explanation and insights for this observation of why it brought better task performance?
9. Can you explain in more detail the warm-up training of the linear projector and how it can prevent performance drops?
10. In Section 6, you mention that you customized two novel VL-LLMs with your VPGTransfer framework. How generalizable do you think this framework is to other novel VL-LLMs, and what steps would need to be taken to adapt it to different models?
Rating
9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
I do not foresee any potential for negative societal impact from this work.