CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

Summary

This paper is motivated by the fact that traditional scaling approaches are computationally expensive and overlook the significance of efficiently improving model capabilities from the vision side. The authors introduce CuMo, a method to enhance MLLMs by incorporating sparsely-gated Mixture-of-Experts (MoE) blocks into the vision encoder and MLP connector. CuMo employs a three-stage training process and has achieved impressive results across multiple benchmarks.

Strengths

1. This paper is well-written, and the proposed method is simple and easy to follow. 2. This paper has addressed an important issue: traditional MoE-based MLLMs are computationally expensive and overlook the vision encoder. 3. CuMo achieves good performance across various benchmarks.

Weaknesses

1. The author employ the MoE method in the vision encoder and connector, but don't explain why using this approach enhances the model's capabilities. 2. The use of the MoE method inherently expands the trainable parameters to some extent. The author should consider whether merely increasing the parameters of the vision encoder alone will also enhance the model's capabilities. This will provide readers with valuable insights. 3. CuMo's LLM is mainly limited to Mistral-8$times$7B. Since this paper primarily explores the expansion of the vision encoder and connector, I think the author should include comparisons with other LLM backbones, such as Qwen[1] and LLaMA[2], to demonstrate that the model's improvements are not due to differences in the inherent capabilities of the LLM itself. 4. Since CuMo is three-stage training, the increased amount of training data may lead to an unfair comparison with other baselines, such as LLaVA1.5. [1] Qwen Technical Report. [2] LLaMA: Open and Efficient Foundation Language Models.

Questions

See weakness.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors mention that the main limitation of this work is the hallucination problem and indicate that it will be improved in future work.

Summary

The paper introduces CuMo, a novel approach to enhancing multimodal large language models (LLMs) by integrating sparsely-gated Mixture-of-Experts (MoE) blocks into both the MLP connector and vision encoder. CuMo addresses the challenge of scaling multimodal LLMs effectively by leveraging MoE's efficiency in parameter usage during training and inference. Key contributions include a detailed exploration of MoE integration strategies across different components of LLMs, a three-stage training methodology to stabilize model training, and the introduction of auxiliary losses for expert load balancing. Experimental results demonstrate that CuMo outperforms existing state-of-the-art multimodal LLMs on various benchmarks, showcasing its efficacy in enhancing model performance while managing computational efficiency

Strengths

Innovative Integration of MoE: Integrating sparsely-gated Mixture-of-Experts (MoE) blocks into both the MLP connector and vision encoder of multimodal large language models (LLMs) represents a novel approach. This approach is not only innovative in terms of architecture but also in its application to enhance multimodal understanding. Experimental Rigor: The authors provide a thorough experimental evaluation, comparing CuMo against state-of-the-art models on multiple benchmarks. This comprehensive evaluation includes ablation studies, which validate the effectiveness of their proposed approach.

Weaknesses

Issue: While the paper touches upon the scalability benefits of MoE, there's limited discussion on the computational efficiency and training time required for integrating MoE blocks into multimodal LLMs. This is crucial as MoE designs can potentially introduce additional computational costs during training. While the paper focuses on performance metrics, there's a lack of discussion on the interpretability of MoE decisions within CuMo. Understanding how MoE blocks make decisions and whether they exhibit consistent behavior across different inputs is crucial for deploying models in real-world applications. The paper shows that Mixtral-8×7B's CuMo does not consistently outperform Mini-Gemini on several datasets, particularly on TextVQA and MME. This discrepancy could be due to architectural differences, especially Mini-Gemini's specialization in high-resolution data. The author should conduct a comparative study with non-MoE architectures like Vicuna or Llama3, focusing solely on CuMo's projection MoE and Vision Encoder MoE. This would provide insights into how CuMo's MoE-based approach performs against architectures not leveraging MoE. Table 1 highlights CuMo's results in datasets like VQAv2, SeedImg, and MMbench, even though it does not consistently achieve the best performance. This can mislead readers about CuMo's comparative performance against other models. The experiments primarily focus on Mistral or Mixtral-8×7B architectures with MoE integration. There's a lack of exploration into how CuMo's MoE-based enhancements compare against non-MoE architectures like Vicuna or Llama3.

Questions

see weaknesses

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

see weaknesses

Summary

The paper presents upcycling for large multimodal models (LMMs). It specifically looks at how to enable upcycling for the different components of an auto-regressive based multimodal model (e.g., LLaVA). It shows how the MLP and vision-encoder (in this case, CLIP) are the two modules that should be upcycled and instead of relying on upcycled LLMs, it is better to use pretrained MoE models (for their specific example of upcycled-Mistral vs Mixtral). The paper outlines the training recipe, which is based on 3 stages, with the first focusing on enabling a stable multi-modal model, and then incorporating their CuMo recipe in two stages. To enable stability and balance across the introduced experts, the authors enabled both load balancing loss for the experts and $z_{loss}$ (as suggested in the ST-MoE paper) as auxiliary losses. They apply these auxiliary losses to the two different upcycled MoEs separately. This is followed by a detailed training recipe and evaluation (both qualitative and quantitative) and ablations that explain their design choices.

Strengths

1. The paper outlines a clear recipe for upcycling in the context of multimodal models, which has not been explored in literature before. 1. The authors support their design choices through well-designed ablations - particularly for the MoE blocks (which form the core of their method) - for the MLP connector and CLIP model. They show the benefits of using pretrained MoE LLM models over upcycling a dense LLM. 2. They present the benefits of using different auxiliary losses (balancing + $z_{loss}$, which they term as bzloss) to train the upcycled model. 2. The paper relies on fully open datasets, and it presents all training settings and hyper-parameters, enabling the reproduction of its results. 1. The authors show quantitative results, which show for models of similar # active parameters (during inference), the models are competitive with other popular LMMs of similar sizes and outperform some benchmarks. They show this with two different LLMs (Mistral + Mixtral), showing their method is composable with different LLMs. For results that use the GPT4 API, they present the statistical average of 3 API calls to calibrate their results. They give a full breakdown of how the # active parameters are computed for their models based on the upcycled components. 2. They follow this up with some qualitative analysis for the balance in experts (relying on the bzloss), seeing an approximately equal distribution of tokens through the experts. They also show some dialogue examples based on sample images for 3 different LMMs, highlighting the benefits of using their method. 3. For some of their training recipes, they show studies of using high-fidelity data and how relying on the latest training methods, such as multi-resolution features, helps boost their overall model quality.

Weaknesses

1. One main criticism of the upcycling setup is the limit on the gains from upcycling. The original setting [1] compares the upcycled model to the original MoE model in a 100% dense compute setting and shows that it takes ~20% additional capacity to catch up to the upcycled model (on the LLM setting). While it does take more computing on the ViT models, do the authors have any intuition on when training MoE-based ViTs (such as V-MoE [2]) and MoE connectors from scratch will be much better / potentially start outperforming the dense-only case? 2. Another criticism of the setup is the diminishing gains as the base model size increases (see Figures 2 & 3 in [1]). Do the authors have an intuition of how this will apply to the CuMo setup? If we scale the CLIP-ViT model or use a much stronger model like SigLIP-ViT, will the gains from CuMo still hold? Please note that I'm not expecting new experiments or results here — just a sense of the authors' intuition. [1] Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints, https://arxiv.org/abs/2212.05055

Questions

__Suggestions__: 1. In Table 1, please highlight all the best numbers, not just where the CuMo model is the best (both for the 7B dense & MoE sections) 2. In Table 2, TextVQA is highlighted as the best with the CuMo model, but the QwenVL model has a higher score. How is the highlighting done in this case - if it is still the "best" result (despite # data samples used), then it will be good to highlight the appropriate model. 3. For both table captions, it would be good to mention "best results are highlighted in bold" so that it is clear to the user. 4. For Figure 6, please mention which model generates the responses - Cumo with Mistral or Mixtral. 5. In the Appendix intro paragraph, the authors mention the "M3" model in the last line, but no other reference has been made to this model before. It would be good to clarify this. __Questions__: 1. The authors mention a non-society license in the impacts section - is there a specific license being applied that the authors cannot speak about as it will break double-blind? Or is it a general enough license so it is well understood how the release is regulated? 2. To clarify, the authors mention only the datasets in the checklist (question 5), but in the impacts section, they mention releasing both code and model. It would be good to be consistent on this front. 3. For the usage guidelines (question 11), there is nothing directly mentioned in the paper. Is the idea that the license, when released, will have clear guidelines for this?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. The authors list hallucination as the main limitation of the model, which they propose can be corrected using RLHF and help improve reliability. While RLHF can help to some extent with the hallucination problem in the context of multi-modal LLMs, it firstly aligns models with human preferences on tasks of interest (e.g., improving captioning capabilities). Additional systems such as RAG also play a role in this case. I'd recommend potentially mentioning this. 2. In a similar vein to RLHF, since the model has not been aligned, I'd also recommend that the authors mention the potential for biased / non-helpful outputs from their model.