Dense Connector for MLLMs

Summary

The document presents a novel approach called the Dense Connector (DC), which is a simple and effective plug-and-play vision-language connector that enhances existing Multimodal Large Language Models (MLLMs) by leveraging multi-layer visual features from the frozen visual encoder.The authors propose three intuitive instantiations of the Dense Connector.The authors demonstrate the versatility and scalability of their approach across various visual encoders, image resolutions, training dataset scales, LLM sizes, and MLLM architectures, achieving state-of-the-art performance on 19 image and video benchmarks.

Strengths

1. the dense connector is easily plugged into mainstream MLLM architecture, and could enhance visual representation of existing MLLM with little additional computation. 2. authors demonstrate the compatibility and scalability of proposed approach across visual encoders, image resolutions, scales of training datasets, sizes of LLMs and even for video understanding tasks.

Weaknesses

1. authors do experiments on several MLLM models, however don't summarize the compatible scope of models architectures.

Questions

I am curious whether the DC is compatible with BLIP2-like architectures which have cross-attention visual resamplers? Could authors give generalization of the compatible types of MLLM architecture?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

authors adequately addressed the limitations

Summary

This paper endeavors to delve into the visual representations in MLLMs, introducing a module, plug-and-play component named as Dense Connector (DC). This DC is designed to enhance visual representation. To this end, three instantiations are presented, including Sparse Token Integration(STI), Sparse Channel Integration(SCI), and Dense Channel Integration(DCI). Thorough experimental evaluations are meticulously carried out to validate the compatibility and scalability, thereby highlighting their potential in enhancing MLLM performance.

Strengths

1. A plug-and-play module can be applyed into various MLLMs. 2. The experiments, including different training datasets, diverse MLLMs, are conduted to demonstrate the performance of DC. The video benchmarks are also adopted as an extension.

Weaknesses

1. The insight of enhancing visual representations from multi-level features of vision encoder have been explored in previous works, like [1]. The work [1] has provided the simliar evidences from multi-level features, and more comprehensive analysis based on a amout of experimetal results. Would authors provides any possible fair comparsions with the work[1] (Only CLIP vision encoder)? [1] Jiang et al., From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models, Arxiv 2023. 2. While this work outlines three architectural configurations for the Dense Connector (DC), it primarily employs a single variant – the Dense Channel Integration (DCI). It is important to note that the DC module serves more as an empirical trick rather than a theoretical methodology. 3. I think the performance gain is limited, especially on high-resolution MGM baseline in Table2. The scalability of DC may be limited. To comprehensively assess its effectiveness, I encourage the authors to extend their comparisons utilizing a Vicuna-13B model based on the MGM framework, incorporating additional benchmark datasets such as VQAv2, MME, and POPE. It is speculated that the incremental performance gain might become even less pronounced when employing larger language models (LLMs),

Questions

It would be beneficial to include a line chart that incorporates a broader range of combinations, facilitating a clearer visual comparison and enabling the selection of groups more evidently.

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

Suggestions: The presented DC can be furthre to combine multiple vision encoder, like DINOv2.

Summary

This paper introduces the Dense Connector, a simple idea that aligns visual and language modalities by utilizing multi-layer visual features. The authors explore three instantiation methods and demonstrates the effectiveness of the Dense Connector across various settings, including different backbones, modalities, datasets and architectures.

Strengths

1. This paper focuses on further exploring the utilization of visual signals in MLLMs. Compared to the attention given to language models, this perspective indeed lacks sufficient attention. It provides insights into utilizing MLPs for integrating multi-layer visual features at the channel level in MLLMs, reducing visual tokens and enhancing computational efficiency. 2. Dense Connector demonstrates effectiveness across a range of model architectures, including LLaVA and Mini-Gemini, as well as LLMs scaling from 2.7B to 70B parameters. 3. This paper is well-written and includes thorough experimental validation of the Dense Connector. 4. I also tried running the code provided by the author, and it is indeed simple and effective. Personally, I feel dense connector has the potential to become a fundamental module in future MLLMs.

Weaknesses

1. High-resolution approaches are widely utilized in vision-language models. This paper explores the application of the Dense Connector with dual visual encoders, specifically Mini-Gemini, in high-resolution settings. However, it lacks an exploration of dynamic high-resolution scenarios, such as those implemented by LLaVA-NEXT. I am curious about whether the Dense Connector could further enhance MLLM performance in this scenario. 2. My another concern is that although the Dense Connector performs well on the Vicuna 7B and 13B models, its effectiveness seems to diminish when scaling to larger sizes. I suspect this might be because training LoRA is not as effective as updating the LLM. If possible, the authors should provide results for updating the LLM to give readers a better reference.

Questions

Beyond the section on weaknesses, I have the following additional questions: 1. The decision to fine-tune ViT during MLLM training is still debated. Research findings are mixed; some [1] indicate that fine-tuning ViT may impair performance, while others [2] show it can lead to performance improvements. I wonder whether the Dense Connector, which utilizes multi-layer visual features, could benefit from fine-tuning ViT. 2. The authors could specify the additional training and testing time required by the Dense Connector. [1] Karamcheti, Siddharth, et al. "Prismatic vlms: Investigating the design space of visually-conditioned language models." arXiv preprint arXiv:2402.07865 (2024). [2] Chen, Lin, et al. "Sharegpt4v: Improving large multi-modal models with better captions." arXiv preprint arXiv:2311.12793 (2023).

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have addressed the limitations.