Summary
This paper introduces the Mini-Gemini, which is a simple and effective approach to enhance the VLMs. More specifically, the improvement comes from three aspects, which are higher resolution images, higher quality datasets, and VLM-guided generation. The Mini-Gemini can achieve performant results. In addition, this framework can support a series of dense or MoE LLMs from 2B to 34B.
Strengths
1. The authors propose a new approach to use high-resolution images to elevate the model performance. Although this approach uses one more visual encoder, the cost is still relatively acceptable as shown in Table 2.
2. The authors use a series of experiments to showcase the performant results on different zero-shot benchmarks. Although there is still a gap between the proposed method and current SoTA private models, the Mini-Gemini has already outperformed many open-sourced models.
3. The authors employ a series of ablation studies to confirm the effectiveness of high-quality datasets and high-resolution images.
4. The paper is well-written and easy to follow. Not only the quantitative results, this paper also provides ample qualitative results.
Weaknesses
### 1. Efficiency and Novelty
From the experimental approach perspective, increasing the image resolution and dataset quality will elevate the performance of VLM[1].
In order to enhance the visual representation, the paper introduces one more visual encoder.
Also, the authors collect a higher quality dataset to improve the performance.
From an overall construction perspective, the authors leveraged additional resources to enhance the performance. However, from a research novelty standpoint, I feel this paper does not provide substantial new insights. Although the proposed approach is somewhat instructive, I believe this work primarily focuses on stacking resources to achieve better results rather than offering sufficient genuine innovation.
### 2. Error Analysis
All qualitative results showcase the capability of Mini-Gemini, but I think the error analysis also plays an important role in seeing why the current model fails and which direction we can improve in the future.
### 3. Some More Experiments
As far as I see, this work focuses on the zero-shot settings. However, nowadays, with the increase in training data, achieving a purely or strictly zero-shot scenario is quite difficult, as good zero-shot performance may often result from the downstream tasks containing elements similar to the training set.
Therefore, rather than focusing on zero-shot scenarios, I am more interested in exploring few-shot capabilities. Few-shot not only allows us to observe the final performance but also reveals the model’s instruction-following ability. If the authors could conduct more few-shot experiments, I think it would make the study even more interesting.
[1] MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Questions
1. I can see many experimental results in Table 1, but why not put more results in the radar chart (bottom left corner) in Figure 1?
2. In Table 4, when you utilize a stronger visual encoder ConvX-XXL, why is the performance of MME and MM-Vet in this setting lower than ConvX-L? Do you have more analysis about it?