Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model

Summary

This paper introduces a new method to make a strong connector between a language model and a vision encoder, in order to build a vision-language model. The method is derived from the Perceiver Resampler, but uses a clever initialization for the queries. The authors validate their approach with a series of ablations, comparing the other connectors to theirs.

Strengths

- This work meets a demand in the field of vision-language models, by creating a stronger connector between a language model and a vision encoder. - The authors obtained better scores with their connector than with the most used and best ones researchers currently use, while being efficient too.

Weaknesses

- The Anchor Selector is to me the meat of the paper. The AcFormer is a Perceiver Resampler with queries benefiting from a better initialization that depends on the input image, instead of the same queries for all images. However, I find that the description of the Anchor Selector algorithm and the justification for why it could work could be better explained in the main paper (the full algorithm is in the appendix). Providing a pseudocode instead of the code could also help.

Questions

- What is the effect of N, the depth of the AcFormer, on the performance on the benchmarks? Which depth do you recommend taking in practice? - In the paper, you mention that the strategy of the Anchor Selector is to take the attention map already computed to avoid having to re-compute something, and use it after modification for the initialization to the queries in your AcFormer. Have you tried other approaches where you are not focusing on efficiency and you are allowed to do additional operations, to see if it performs better? - In Figure 2, we see the effect of the layer on the attention map. Which layer are you considering for the final attention map that you are using for your algorithm? Have you tried different layers to see if it performs better?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Yes

Summary

This paper propose AcFormer, a novel vision-language connector in MLLMs. AcFormer is driven by visual anchors observed in vision transformer's PCA of the feature map and attention map of [CLS] token, where high value is observed in both of the maps. The author use the attention value of [CLS] token to select the visual anchor in a progressive way, the selected visual anchors are then used as information aggregation tokens through cross attention. Through extensive experiments the author prove the effectiveness and efficiency of AcFormer comparing with other multimodal connectors.

Strengths

* As is shown in the main experiment result, AcFormer achieves about 2.3/1.7 time faster pretraining speed while retaining the overall performance. * Extensive ablation result is provided to further prove the effectiveness of AcFormer. * The writing and experimental setup is clear.

Weaknesses

* It seems to the reviewer that the main advantage of AcFormer comparing with original LLaVA-1.5 is its efficiency. However, only the training time is reported in the main table. Since all of them is less than 20 hours as is shown in table 6, apparently training time is not the main bottleneck of developing LLaVA-1.5 model. To further justify the efficiency of AcFormer, it's enouraged to report the inference time per token of the resulting models. * The motivation is not clear enough. The visual anchors discovered by the authors is a common phenomenon recently in vision tranformers, for example, [1] finds the outliers in ViT and provs that the outliers emerge because the ViT need some additional tokens to preserve golbal information. In this paper, more experiments should be done to unveil the reason behind the emergence of visual anchors, otherwise it not convincing enough to directly use the anchors to aggregate information, especially given that the performance improvement is limited comparing with the origial model. * AcFormer selects the visual tokens without considering the actual question, which means the token reduction could be harmful when the question is not about the major subject of the image. [1] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2023). Vision transformers need registers. arXiv preprint arXiv:2309.16588.

Questions

* In line 186-195, what is the difference between AcFormer on $i-1$ th layer and selecting some features from $i$ th layer? * In line 203-214, how are the 6 layers selected and how are the features fed before the projection layer? This part is not clear enough.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

yes

Summary

This paper introduces a novel vision-language connector, Anchor Former (AcFormer), designed to enhance the efficiency and accuracy of multimodal models. By identifying visual anchors within Vision Transformers and utilizing a cost-effective progressive search algorithm, AcFormer leverages these anchors to aggregate information more effectively. Extensive experiments demonstrate that AcFormer significantly reduces computational costs while maintaining or improving performance across various vision-language tasks compared to existing methods like Q-Former and Perceiver Resampler.

Strengths

1. The motivation of this paper is strong, attempting to improve a fundamental building block in recent multimodal models. 2. The paper is well written and easy to follow. Tables and figures are helpful for readers to quickly understand. 3. Experiments are extensive and solid. They cover various types of connectors (linear projection (LLaVA), Q-Former (BLIP-2), and Perceiver Resampler) and datasets (POPE, MME, MMB, MM-Vet, TextVQA, GQA, VQAv2, VisWiz and SQAimg). 4. This paper is insightful and can be useful to the community.

Weaknesses

1. The proposed method is not very effective. The key result comparison for this paper is AcFormer vs LLaVA-1.5 (linear), but in Table 1 and 2 the accuracy results are mixed (slightly in favor of AcFormer). The proposed method is not significantly better than the linear baseline. 2. Linear connector results are missing in Table 3. 3. Minor: The term Multimodal Large Language Model (MLLM) is self-conflicting. I suggest authors use Large Multimodal Model (LMM).

Questions

1. Besides slight accuracy gain and training time speedup, are there other benefits of the proposed method over the baseline connectors? 3. Only pretraining time results are provided. Is the proposed method faster at inference time as well? 2. It seems that the proposed method is more efficient in terms of number of visual tokens. Will this make a big accuracy difference on tasks with high resolution images (documents for example)? I will raise my rating if the authors can convince me the significance of the key technical contribution of this paper.

Rating

4

Confidence

5

Soundness

4

Presentation

3

Contribution

2

Limitations

The authors discussed them in the limitation section.

Summary

This paper proposes a way to select visual tokens, e.g. token pruning, by using attention map scores. This reduces the number of tokens needed in the network which saves compute costs. The method is evaluated on multiple datasets and shows reduced compute while maintaining performance.

Strengths

The paper is fairly well written and the experiments are good and show the benefit from the approach.

Weaknesses

The approach isn't especially novel, many works have explored token pruning, token learning, token reduction before. While this approach is different from the previous ones, the differences are small.

Questions

The compute cost overall seems to go down, but I'm curious about the implementation of the anchor selection. How long does that part take? How optimized is that algorithm?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Answer to authors

Thank you for providing additional work and answering the questions.

Request for Review of Rebuttal and Clarifications

Dear Ac and Reviewers, Thank you for taking the time to review our paper, We are delighted to see that four reviewers recognized the strengths of our work, Specifically, Reviewer i5LS and Reviewer Ld45 found our method to be effective and insightful. Reviewer r5Ke acknowledged the extensive and meaningful ablation experiments we conducted. However, the reviewers also raised some concerns. For instance, Reviewer i5LS and Reviewer r5Ke wanted to know the comparasion of the inference time, while Reviewer U61n was interested in the implementation of the anchor selection. We have provided a detailed rebuttal addressing these concerns and kindly ask the reviewers to review our responses when convenient. Thank you once again for your valuable feedback. Sincerely, The Authors of Submission 4497

Reply to Reviewer i5LS

Dear reviewer i5LS, Thanks for your reply! If there are any other concerns, feel free to tell us. We are grateful for your response. Sincerely, The Authors of Submission 4497