Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight

Summary

This paper introduces Chain-of-Sight (CoS), a novel vision-language bridge module designed to accelerate the pre-training of Multimodal Large Language Models (MLLMs). The key innovation is a sequence of visual resamplers that capture visual details at various spatial scales, allowing for a significant reduction in visual tokens during pre-training while maintaining or improving performance. The authors conduct sufficient experiments, including the evaluation on diverse benchmarks and three scaling experiments, to justify their assumptions and model designs.

Strengths

1. The Chain-of-Sight approach is a novel combination of multi-scale visual processing and token scaling, addressing a critical efficiency bottleneck in MLLM pre-training. 2. Extensive experiments support the claims, demonstrating significant pre-training acceleration (up to 73% reduction in wall-clock time) without sacrificing performance. 3. The paper is well-organized and clearly written. 4. The work addresses a crucial challenge in MLLM development: the computational cost of pre-training.

Weaknesses

One minor weakness is that the authors have not discussed the potential impact of the scale of training data. Multiple compared MLLMs (e.g. Qwen, VILA) in Tables 5 and 6 are probably pre-trained on unequal sizes of data from diverse sources. It will help to clarify how the proposed Chain-of-Sight improves the performance on various tasks if the authors rule out the impact of the differences in the pre-training data.

Questions

Q1: See the Weakness above. Q2: Will there be differences in the convergence speed of different compound scaling strategies? Which combination of window scale, resolution scale, and compound scale exhibits the fastest convergence? Q3: Do the authors interpolate the positional embeddings when increasing the input resolution mentioned in Line 175? Q4: Do the authors mean to refer to Table 3 instead of Table 6 in Line 222?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

None.

Summary

This paper proposed a post-pretrain token scaling strategy, Chain-of-Sight, to accelerate the pre-training of Multimodal Large Language Models (MLLMs). Through the proposed method, the authors were able to achieve a significant improvement in pre-training speed. The authors confirmed through an ablation study that the proposed method exhibits performance similar to existing approaches.

Strengths

The paper is well-written, clearly and coherently presenting ideas. The proposed method demonstrates performance similar to or better than existing methods, despite having shorter training times. An ablation study was conducted across various tasks and settings.

Weaknesses

The paper only includes experimental results using CLIP-ViT-L/14 as the visual encoder and Vicuna. Experimental results based on other models are needed. The explanation of how the number of tokens during pre-training and fine-tuning (e.g., PT:80 FT: 1296) was determined is insufficient.

Questions

see weakness.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed both the limitations and potential negative social impacts in Section 5 (Conclusion) and Section A (Broader impact).

Summary

1. This work proposes Chain-of-Sight, a training method of MLLM that leverages global and local visual contexts effectively. 2. To boost efficiency in the pretraining stage, the authors propose a post-pretrain token scaling strategy. During the pretraining state, it requires significantly fewer visual tokens and cuts down the wall-clock training time by 73% in the pretraining stage. To effectively increase tokens during fine-tuning stage to enhance performance, they propose a compound strategy by manipulating input resolution and window size at the same time. 3. When both use the same number of tokens during fine-tuning, the results achieved by the Chain-of-Sight model pre-trained with 32 tokens match or surpass those obtained using 336 visual tokens throughout the training process.

Strengths

1. The writing is mostly clear and easy to follow, except for the compound scaling part where a clearer illustration is suggested. 2. Comprehensive ablation study. Variations of pretraining token number / pretraining strategy(CoS, Resample) / FT are compared on various tasks including image captioning, visual question answering, text recognition, referring expression comprehension, etc. 3. According to Table 5, CoS-7B achieves good results on VQA benchmarks compared to 7B-level baselines.

Weaknesses

1. The method of hierarchical (multi-scale) visual input has already been explored in many former works, which is contradictory to the expression in L91-92 "Despite this, the potential for harnessing multi-scale visual hierarchies remains under-explored in the context of MLLMs". For example, LLaVA-NeXT [1] and InternLM-XComposer2-4KHD [2] both adopted multi-scale strategies and proved their effectiveness. 2. There are mistakes in the table of experiment results. For example, in Table 3, RefCOCO+, test-A, the second-best one should be 90.57. [1] Liu, Haotian, et al. "LLaVA-NeXT: Improved reasoning, OCR, and world knowledge." https://llava-vl.github.io/blog/2024-01-30-llava-next/ [2] Dong, Xiaoyi, et al. "Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd." *arXiv preprint arXiv:2404.06512* (2024).

Questions

1. In Table 2 and Table 3, why are many of the results of "CoS, pre-train 80, FT 336" better than those of "CoS, pre-train 336, FT 336"? Analysis or explanations are suggested. 2. The settings of ablation experiments are not explained. Is that the same as sec 3.1 training settings? If so, why do the results in Table 5 not agree with those in Table 2&3&4?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The experiment is only conducted on 7B-level model with PEFT. Larger scale model with full-parameter training is suggested to prove its effectiveness and extensibility in modern pretraining.

Response to the authors' rebuttal

Thank you for providing the detailed answers. My concerns have been resolved and the rating of 7 is maintained. Good luck!