MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

Summary

The paper proposes to mitigate task interference during multimodal instruction tuning with a mixture of experts, in both the language and the encoder side. The paper is well written, contains insightful analysis and shows improvements over baselines.

Strengths

- Mixture of experts is an important topic that is heavily studied in LLMs but little in multimodal models. - The problem of task interference is an important problem that is more present in a multimodal setting. - The paper is well written and easy to follow. - The proposed approach is well motivated with insightful analysis and show improvements over baselines

Weaknesses

1. While the paper shows improvements over different baselines in Tab.3, the scores are still lagging behind, compared to methods without MoE, such as LLaVA (which the paper builds on top of). Knowing that the proposed approach have significantly more parameters, pretrained for more steps (stage 1) and use different visual encoders. 2. The paper claims that “This means the language experts in our MoLE module gradually specialize in distinct task domains during training.” However the visualization does not support that. For instance, in Fig.5 E4 is used in most tasks types, while E1 is relatively less used. But it is not clear if we can map different experts to different task groups. 3. The paper is not evaluated on common and recent multimodal benchmarks: SEED, MME, MM-Vet, POPE, VQAv2 … that are considered in other methods like LLaVA. 4. The design of MoE are very different for the LLM and visual encoders. Each visual encoder is considered as an expet. Did the authors experiment with typical MoE (replicating FFN) in a single visual encoder? 5. The papers used deformable cross-attention, but I did not find any experiment to support this design choice, compared to using simple cross-attention. Did the authors conduct this experiment? 6. The paper states that load balancing did not help. Any insights on why this is the case? knowing that these losses are typically used in most MoE papers. Also, did the authors encounter any instabilities during training? It is important to discuss these in the paper, as instability is a major problem in MoEs training. 7. The paper main novelty seems in applying MoE also in the vision encoders. This question is also explored in previous works with slightly different context such as [1] and [2]. I found the novelty limited in this regard. [1] Mustafa, Basil, et al. "Multimodal contrastive learning with limoe: the language-image mixture of experts." Advances in Neural Information Processing Systems 35 (2022): 9564-9576. [2] Shen, Sheng, et al. "Scaling Vision-Language Models with Sparse Mixture of Experts." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.

Questions

Please the weaknesses section (e.g. 4-5-6)

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Some limitations and societal impacts are discussed in the paper or the appendix.

Summary

The paper proposed a MoE design for MLLM, it utilize MoE in both visual encoding procedure and LLM decoding procedure. The paper utilized a dynamic routing module to mix visual features from different experts, and adopted a multi-adapter structure to combine the knowledge of differnent language experts. Widely conducted experiments showcase the effectiveness of the proposed method on various downstream tasks.

Strengths

The paper widely conducted quantitative results on several multimodal downstream tasks to prove the effectness of the proposed MoE design. The paper also adds qualitative results to intuitively showcase the performance.

Weaknesses

The design proposed in this paper seems to be not innovative enough. Its key design philosophy has already been brought out in many previous works. As for MoE in vision, the CVPR 2024 paper “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs” combines CLIP and DINOv2 features to improve visual representations, the arxiv 2023 paper “SILC: Improving Vision Language Pretraining with Self-Distillation” combines the learning objectives of CLIP and DINOv2 for better pretraining outcome. Moreover, the references [12, 5, 8, 54] cited in this paper has already investigated MoE in LLMs, as stated in the introduction section. It is already a consensus that MoE in MLLM, especially combining multimodal pretrained feature (such as CLIP) and pure visual pretrained feature (such as DINOv2), can improve model performance. Combining a third feature to improve performance (in this paper, the feature from Pix2Struct) besides CLIP and DINOv2 is a pure engineering design, which could not add up to the technical contribution of this paper. Moreover, the ‘Dynamic Router’ proposed in this paper is also a simple MLP network which is widely used for combining different feature. To sum up, it is hard to claim all these designs as a contribution of this paper.

Questions

1. What is the differnece between this work and the previous works discussing MoE in vision (the several papers I listed in the weaknesses section), how can this paper brough new knowledge or designs that is perviously unknown or not sufficiently expored by the community? 2. How does the MoE structure affects the inference speed, are there any quantitative results?

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

The authors already adequaately addressed the potential negative societal impact.

Summary

In this paper, the authors introduce a mixture of multimodal experts (MoME) to reduce task interference and develop a generalist MLLM. MoME consists of two main components: a mixture of vision experts (MoVE) and a mixture of language experts (MoLE). MoVE can adaptively adjust features transformed from different vision encoders and boasts strong compatibility with various transformation architectures. MoLE integrates sparsely gated experts into LLMs, achieving seamless improvements while keeping inference costs nearly unchanged.

Strengths

1. The analysis of task interference and mixture of vision experts in this paper is clear, highlighting the necessity of a vision mixture of experts 2. The experiments are well-conducted and quite comprehensive 3. The study demonstrates strong performance on most datasets compared with other generalist and MoE MLLMs

Weaknesses

1. From Table 1, we can see that both ADT and Router have achieved notable improvements. Could you explain the internal mechanisms behind this? I am quite curious as to why the improvements are so significant. 2. In Table 3, many values are missing. Could you add some results from more general multimodal benchmarks to make the experiments more comprehensive, such as MME, MMbench, MM-Vet, LLaVA$^W$, Science$^{QA}$, etc.

Questions

One advantage of MoE is its fast inference speed. Could you conduct an experiment to verify the model's inference speed?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

yes, the author explains the limitations of their study and potential negative societal impact.