What matters when building vision-language models?

Summary

This work conducts comprehensive experiments for re-examing common choices in the VLM area, such as unimodal backbone, connector, number of visual tokens, data, etc. Based on their experiments, they observe several findings important to the VLM research community. Finally, they rely on their key findings to collect corresponding training datasets and train a powerful VLM.

Strengths

The writing of this work is clear and easy to follow, and the experiments are thoroughly conducted. I have to say, I appreciate the thorough experiments and clear reasoning presented in this work. The findings obtained during the experiments can also contribute to the advancement of the VLM community.

Weaknesses

Since the authors ultimately chose a fully autoregressive training approach, why did they still use cross-attention training for the ablation study in Section 3.1 to support finding 1? Is it because finding 1 does not hold under the autoregressive training approach?

Questions

See weaknesses above.

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

N/A

Summary

This paper empirically investigates several essential design choices of vision-language models, such as language backbones, visual encoder backbones, and different architectures. Through extensive experiments, they evaluate different model architectures, training methods, and data utilization strategies. Based on their findings, the authors trained a foundational VLM with 8B parameters that shows state-of-the-art performance.

Strengths

One of the key strengths of this paper is its focus on an important and timely topic. By addressing the critical decisions in VLM design and highlighting areas where improvements can be made, this research contributes to a deeper understanding of how to build more effective and efficient models. The analysis and conclusions presented in the paper are particularly insightful and have the potential to significantly benefit the AI research community. By identifying key factors that influence VLM performance, such as the choice of model architecture and training methods, the authors provide clear and actionable recommendations for building better models. Besides, the availability of the trained models and public code is a notable strength of this work. Additionally, the paper is well-written and well-structured, making it easy to follow. Overall, this is a good paper, and I recommend accepting it.

Weaknesses

The baseline models compared in the study are primarily open-source models. It would be beneficial to include and discuss state-of-the-art (SOTA) commercial models to provide a more comprehensive evaluation of the model's performance. Additionally, incorporating performance reports of more open-source models, as listed on platforms like [MMBench](https://mmbench.opencompass.org.cn/leaderboard), in Tables 8 and 9 could better position the model's ability within the current research community. It is noticed that several open-source models of similar size achieve better performance. For instance, models like [InternLM-XComposer2-VL](https://github.com/InternLM/InternLM-XComposer) demonstrate superior performance despite having a comparable size, indicating potential areas for improvement in the proposed model. The evaluation of the model's in-context learning ability is relatively limited. It would be useful to investigate whether the model's performance improves with an increased number of demonstrations, which could provide further insights into its capabilities and limitations in real-world applications.

Questions

**Availability and Openness** - Will the training code and datasets also be public for the community? Ensuring the availability of both the training code and datasets would greatly enhance the reproducibility of the study and allow the research community to build upon this work. **Real-World Applicability** - How well does the model handle real-world scenarios involving noisy or low-quality data, which are common outside controlled benchmark environments? It is crucial to understand the model's robustness and performance in less ideal, real-world conditions to evaluate its practical applicability. **Comparison with State-of-the-Art Models** - How does the model's performance compare with state-of-the-art commercial models such as GPT-4v, GPT-4o, and Gemini? Understanding the discrepancies and similarities in performance between this model and leading commercial models would provide a clearer picture of its competitive standing. **Architectural and Training Insights** - What is the performance impact of using cross-attention while unfreezing all parameters? - Can a column for the number of optimized parameters be added in Table 3 to highlight the differences? Including this information would help to better illustrate the comparative efficiency and complexity of the models. - Does the final model use both LoRA and DoRA? As discussed in the exploration, a full autoregressive structure is better when equipped with LoRA finetuning on the unimodal backbone. Also, the paper mentioned that DoRA is used for instruction finetuning. So, does the final model use both LoRA and DoRA for different purposes during training? **Model Design and Inference Capacity** - How many images can the model support during inference? Minor Issues: Missing citation detail in line 22

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Generally, this is a good paper with several noticeable limitations: - The training resources are quite large and may not be accessible to many researchers, limiting the reproducibility of the results. - The evaluation benchmark is limited. More results on commercial SOTA VLMs like GPT-4v, and on more tasks such as reasoning, hallucination, and in-context learning, would be useful to fully assess the model's performance. - The red-teaming procedure could be more thorough to prevent malicious use of the proposed model.

Summary

The compares different methods and strategies involved in training VLMs - impact on inference efficiency by model architecture (fusion module: cross-attention versus autoregressive) & on training stability by multimodal training procedure - compare different design choices in a controlled environment and extract experimental findings - Build a new 8B model XXXXXX and a new instruction training dataset YYYYYY that surpasses or closely-matches the performance of many large-scale models

Strengths

- The paper has an interesting scientific-report like approach to find a good recipe for training VLMs which would be beneficial to the community - Using the derived findings, the proposed model XXXXXX performs higher scores than larger models - The authors also curate a new vision-language finetuning dataset YYYYYY. - The authors consider many important, yet overlooked design choices, which would help expedite research and development of VLMs

Weaknesses

- While I fully appreciate the importance of the work in standardizing a recipe for training VLMs, I believe it lacks vigour to make such bold claims. For example, Finding 4 does contradict results in Table 9, TextVQA. - Some results seem a bit too overarching. For example, while the authors make a generalised finding “the quality of the language model backbone has a higher impact on the performance of the final VLM than the quality of the vision backbone” (Finding 1), they do mention that this might be because the vision encoders might not be well trained (L108), which seems conflicting. - It would be nice to have most/all the findings validated for XXXXXX. - Can the authors elaborate on how YYYYYY is different from existing VLM instruction tuning datasets? Is there any scope of test dataset leaks? - How do the authors ensure that the superior performance of YYYYYY is not driven by the dataset alone and only because of their findings?

Questions

see weaknesses

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Discussed in section A.4

Summary

This work studies a question: what matters when building vision-language models? To this end, this work provides analysis from the following aspects, 1) Are all pre-trained backbones equivalent for VLMs? 2) How does the fully autoregressive architecture compare to the cross-attention architecture? 3) Where are the efficiency gains? (Number of visual tokens and Preserving the original aspect ratio and image resolution) 4) How can one trade compute for performance? Based on these observations, this work further proposes a vision-language foundation model.

Strengths

- First of all, I like the idea of analyzing the impact of each module on the VLMs (e.g., LLaVA). This work considers several aspects, such as language model, vision encoder, and input resolution. - Second, introducing a VLM model and giving the specific training recipe. I think the three main stages are straightforward, including multi-stage pre-training, Instruction fine-tuning, and optimizing for chat scenarios. The last one is not used in other VLMs. I think it is useful. Please comment on does this strategy can help other existing methods such as LLaVA?

Weaknesses

- Some of the findings are already studies in previous works. For example, the impact of a vision encoder has been studied in [a,b,c]. The effect of the language model is studied in [a,d]. The impact of input resolution [a,b,d]. However, I do not see a clear comparison and discussion with these works. Namely, although I think the analysis is interesting, it is not new and does not provide any new insights. [a] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models [b] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs [c] BRAVE : Broadening the visual encoding of vision-language models [d] VILA: On Pre-training for Visual Language Models - Regarding the improvement or training of a new VLM model. While the training recipe is provided, it is crucial to compare it with other methods that also aim to improve VLMs. For example, [b,c,d] all improve the VLMs. I would suggest the authors give a direct comparison or a clear discussion on how the proposed method contributes and why it is important.

Questions

- I understand YYYYYY and XXXXX are set for anonymization. But they indeed affect the reading. [Just a feedback] - Please discuss the observations/findings in the context of existing works. Moreover, another work which is released after NeurIPS would be useful to look at [d] [d] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs - Please discuss or compare with methods that also aim to improve VLMs.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

This work provides a discussion of the potential limitations in Section A. I agree with the common issue of hallucinations. Could you provide some suggestions on how to avoid them? Second, I do not quite get "lack of openly available". Are the datasets used in this work not publicly available? [I did not force the authors to release the data but a clarification would be helpful]. Lastly, it would be better if the authors could provide some advice on handling "jailbreak" cases.

Answer to the authors

Thanks for your response. I choose to keep my score unchanged.

Thank You

Dear Authors, Thank you for your detailed response, which has effectively addressed the initial concerns. Please include a paragraph in the revision that highlights the key observations and insights from the existing works. Best, Reviewer Ww8w

Answer to reviewer

Thank you for your review. We appreciate your feedback and have addressed your points below: > Analyze Data Quality: Provide details on how data quality was ensured. We conducted over 15 ablation studies to determine the optimal data mixture presented in Table 14. While it wasn't feasible to assess the individual impact of each dataset due to their sheer number, we rigorously evaluated the influence of each dataset group, categorized by task (e.g., document understanding, image captioning), both quantitatively and qualitatively. To manage computational resources effectively, we halted runs once we obtained sufficiently clear conclusions. > Discuss Diversity: Explain the demographic diversity of the datasets and how it affects model fairness. > Mitigate Biases: Outline steps taken to detect and reduce potential biases. We have combined our responses to these two points: The dataset YYYYYY, which we propose for fine-tuning, is a curated compilation of academic datasets from the literature. It is important to note that the vast majority of these datasets do not include images of people. Instead, they primarily consist of charts, figures, documents, and synthetic images. When real-world images are included, they mostly depict objects or signs with text, minimizing the risk of introducing significant social biases. For the pre-training dataset, specifically OBELICS and LAION-COCO, we took several steps to mitigate potential biases. We first excluded any content that the creators opted out of for AI training. We then applied NSFW filters to substantially reduce the presence of nudity in these datasets and manually verified the effectiveness of this filtering by reviewing thousands of examples. Finally, we conducted a red-teaming analysis, as detailed in section A.4.