MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Summary

The paper proposes a MoVA to extract and fuse task-specific knowledge from different visual encoder. Besides, the paper designs a coarse-to-fine mechanism, which dynamically selects the most suitable vision experts, then extracts and fuses task-specific knowledge from various experts. The experimental results are encouraging and validate the performance of the proposed method.

Strengths

1. This paper proposes a new perspective, which is to improve the multi-task reasoning ability of MLLMs based on the different perception ability of the different encoders. 2. The paper is fluent and easy to understand. 3. The extensive experiments evaluate the effectiveness of the proposed approach.

Weaknesses

1. The number of components in line 102 does not match the serial numbers. 2. During the inference phase, the process involves two rounds of LLM inference and two Base Encoder, which results in significant inference time and GPU memory usage. This poses a challenge for the practical application of MLLMs. 3. This is a well-completed article, but I am concerned about its novelty. The method seems similar to the adapter+agent approach described by Wang et al. [1]. Could you explain the differences and advantages of your method compared to theirs? [1] Wang X, Zhuang B, Wu Q. Modaverse: Efficiently transforming modalities with LLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 26606-26616.

Questions

Why is the last row of MoVA results in Table 1 different from those in Table 3? In other words, The caption of Table 1 shows that the same data was used to train each model. Does this mean that all models used LLaVA-1.5 data? If so, I think this table will inspire the community and I would like to ask if you can release the checkpoints of these models.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

1. As mentioned in the conclusion, the performance may be affected by failure cases of the context-relevant vision experts, leading to potential degradation. Additionally, it is influenced by the choice of encoder results in the first stage of inference. 2. The model's complex inference structure increases the inference burden.

Summary

This paper proposes MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. MoVA first leverages the tool-use capabilities of LLM, routing the most appropriate experts from expert candidates and then uses the MoV-Adapter module to enhance the visual representation.

Strengths

1. The writing is easy to follow. 2. The idea of using LLM to route the most appropriate experts is interesting.

Weaknesses

1. Is there an adapter structure in MoVA? Why do we call it MoV-Adapter? 2. The experiments use SFT data from different MLLMs should be aligned (the same). The comparison is not fair with different SFT data. 3. More vision encoder enhancement methods should be compared using the same SFT data, including [1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models. [2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs) [3] MouSi: Poly-Visual-Expert Vision-Language Models [4] BRAVE: Broadening the visual encoding of vision-language models 4. The inference speed of LLM is very slow. I think MoVA using LLM as a router selection will decrease the speed of the whole process. How to solve this problem?

Questions

In line 102, I just found four components, but the paper claims five.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations.

Summary

This paper considers a critical issue in vision language models, i.e., the vision encoder design. The authors proposes MoVA, a coarse to fine routing strategy, adaptively routing and fusing task specific vision experts which best suit the task. Experiments conducted on a wide range of MLLM benchmark demonstrate the advantages of the proposed MoV adapter.

Strengths

1. The motivation is clear, and the method is easy to follow 2. The proposed coarse to fine routing strategy is reasonable

Weaknesses

1. As for OCR and related tasks, it has been validated that high resolution is much more import than model size, and image split like LLaVA Next, Monkey, Ferret etc. have been widely used. I wonder the importance of including task specific models. It looks redundancy, and importantly, we cannot include all models that cover different tasks exhaustively 2. As shown in Table 1, the main improvement results from the OCR related Doc and Chart task, which may resolved by current MLLMs baseline such as LLaVa Next, I wonder the necessity of the using MoE experts, as other module, adding resolution is more reasonable. In Table 3, since the PT, SFT and pretrained models are different, it is hard to validate the gains are from the MoV experts. 3. Does the method introduce error propagation, say, the first stage LLM routing is not correct, which may harm the performance of the whole framework. On the other hand, if the coarse routing is simple, could we say that this stage we do not need to carefully design, and a tiny LLM or even rule based according to the prompt is enough? Since current framework seems complex using two LLMs, the comparison is not fair, and I wonder if there is a better routing strategy.

Questions

The main concern is the rationality of using MoV on vision side, while two step routing is interesting, it adds the complexity (using two LLMs), and it is not clear the advantages of MoV over the latest MLLM benchmark.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

None

Summary

In this paper, the authors propose MoVA, a powerful MLLM composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter. Based on multimodal context and model expertise, MoVA fully leverages representation from multiple context-relevant vision encoder experts flexibly and effectively while avoiding biased information brought by irrelevant experts. MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of benchmarks.

Strengths

1. The authors comprehensively investigate various vision encoders and analyze the performance of individual vision encoders versus the plain fusion of multiple encoders across various tasks. 2. The authors propose MoVA, a powerful MLLM composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter. 3. The proposed MoVA can achieve significant performance gains over state-of-the-art methods in a wide range of challenging benchmarks.

Weaknesses

1. Despite the proposed MoVA in this paper achieving state-of-the-art performance across multiple tasks, leveraging multiple visual encoders in MLLM design does not introduce novelty. Additionally, the design of MoV-Adapter also lacks innovation. 2. This paper lacks a comprehensive evaluation regarding the routing selection of multiple visual encoders using LLMs, necessitating a more targeted and comprehensive assessment from the authors.

Questions

Please refer to Weakness Section for more details.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have clearly illustrated the weakness of this work by pointing out the hallucination and the model performance may be affected by failure cases of the context-relevant vision experts.

Looking forward to your post-rebuttal feedback

Dear Reviewer dPiw: We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work. If your concerns have been well addressed, please consider raising your rating, thanks. Best, Authors