CogVLM: Visual Expert for Pretrained Language Models

Summary

This paper introduce CogVLM, a powerful open-source visual language foundation model. CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. The contributions are summarized: 1. introduce the CogVLM model, which deeply integrates visual and linguistic features while retaining the full capabilities of a pretrained large language model. CogVLM-17B, trained from Vicuna-7B, achieves state-of-the-art across 17 classic cross-modal benchmarks. 2. validated the effectiveness of our proposed visual expert module and the importance of deep fusion. 3. the weights of CogVLM and the dataset used in the SFT phase available to the public.

Strengths

1. The paper's results have achieved state-of-the-art (SOTA) performance, demonstrating excellent capabilities across multiple benchmarks. 2. The paper is well-organized and the expression is clear. 3. The starting point of the paper: addressing the issue of performance degradation in LLM due to parameter updates during multimodal training, is very meaningful and valuable.

Weaknesses

1. A new set of QKV and FFN would nearly double the number of model parameters. Why not consider using LoRA (Low-Rank Adaptation), where text tokens use QKV/FFN and image tokens use QKV+LoRA/FFN+LoRA? Additional experiments are needed to support this. 2. There is a lack of ablation studies to compare the impact of different RoPE (Relative Positional Encoding) strategies on visual tokens on the results.

Questions

1. Why train a separate grounding model? Would it degrade the chat model if the chat model also had grounding capabilities? 2. Why not try tuning the visual encoder?

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The paper introduces CogVLM, an new open-source visual language foundation model. Contrary to the popular method of adapting LLMs by fine-tuning their original weights, CogVLM introduces new weights specifically for processing the visual tokens. Concretely, CogVLM copies all weights of the LLM to form visual expert modules at every attention and FFN layer. During training only the visual expert weights are updated such that the LLM retains its language modeling capabilities when no image input is given. CogVLM is evaluated extensively on image captioning, VQA, LVLM and Referring Expression Comprehension benchmarks.

Strengths

1. The design choice to add additional parameters for the vision tokens is reasonable and implemented well such that it defaults to the original model without image input. 2. The experimental evaluation is extensive. CogVLM achieves convincing results across the board. 3. Ablations shed light on important questions such as the importance of the visual expert parameters and the causal attention mask. 4. As an open-source model, CogVLM can have a high impact on the research community.

Weaknesses

1. The prompting differences between the tasks is not clear. For instance, how can a user make sure that CogVLM returns bounding box information or prevent CogVLM to output bounding boxes? 2. While the results of CogVLM are impressive, it is not a parameter efficient method. Trying to scale CogVLM to LLMs of better language modeling capacity will double the parameters which is not sustainable for big LLMs. 3. As VLMs become more powerful over time, there will be higher expectations of their capabilities. One aspect that has not been mentioned at all is whether CogVLM can handle multi-image inputs or multi-turn conversations with interleaved image-text context. Based on the examples given and the training data, the paper currently suggests this is a model limitation. 4. Limitations are not discussed at all. The checklist states that the paper has no limitations without explanation, which is a superficial statement and hardly ever true. The previous points are likely limitations. 5. Similarly, the authors do not provide a discussion on the broader impact of the paper. Since CogVLM is an open model it has the potential to have a high impact, both positive and negative. Since CogVLM puts a particular effort into retaining the original LLMs performance, it also means that it carries over all biases and limitation from the base LLM. It is important that papers of generative models reflect on the societal impact.

Questions

I would be great if the authors could clarify my concerns from the weaknesses section. Apart from that, here is one more minor suggestion. - Figure 1 shows interesting qualitative example. I suggest to reference the figure in the main text and extend the caption to explain the displayed capabilities.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations and broader impact are not discussed although they certainly exist. The authors should add a discussion on both (see weaknesses).

Summary

This paper aims to add multimodal capability while maintaining the language capability of LLM. The authors thus propose a feature fusion strategy, without sacrificing any performance on NLP tasks. The experimental results on multiple datasets are impressive, which demonstrate the validity of the proposed method to some extent.

Strengths

1. The motivation of this paper sounds reasonable. Most of current VLMs do not take into account the degradation of LLM when training multimodal data, which is a good research point. 2. The method looks interesting. A simple feature fusion strategy seems to tackle the above problem. 3. The writing of this paper is good, and the structure is easy to follow.

Weaknesses

The experiment still has room for improvement: (1) The results on the representative VLM benchmark MME are not given, which is widely used by current methods, such as LLaVA, InternVL, and Qwen-VL. It is suggested to add the results of MME in Table 2. (2) In order to maintain the capability of LLM, we can also add text data while using multi-modal data, is there a corresponding comparison? Some common text data might also serve this purpose. (3) Alternatively, we could freeze the parameters of the LLM but train the visual encoder and the connector, which might also do the trick. (4) It would be good to see CogVLM's ablility on the representative multimodal video benchmark Video-MME [1]. The authors could input multiple video frames, i.e., generalize to multiple images. GPT-4V, InternVL-Chat, Qwen-VL-Max, and Qwen-VL-Chat all have good generalization on Video-MME. It is suggested to add the results of CogVLM in this paper. [1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. Given the limited time available for rebuttal, the authors could incorporate the above experimental results and discussions into the final accepted version or the next version.

Questions

None

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

None

Summary

This paper proposed a large vision language model, CogVLM, which shows capabilities in various benchmarks. In contrast to popular solutions that fuse the vision and language token in LLM with shared trainable parameters, CogVLM bridges the gap between the frozen pre-trained language models and image encoders by trainable visual expert modules in the attention and FFN layers. In this way, CogVLM excels in maintaining the language abilities of the original LLM and achieving good performance on various vision-language benchmarks, including captioning, VQA, and visual grounding.

Strengths

1. This paper is well-written and easy to follow. 2. The idea of Visual Expert is simple but effective. It is impressive that newly introduced Visual Expert Modules with large parameters can be well-optimized and perform well. 3. The proposed CogVLM is extensively evaluated using various benchmarks. It can be widely adopted for captioning, VQA, and Visual Grounding.

Weaknesses

1. The visual expert will introduce a large number of parameters. As a result, the CogVLM with a 7B LLM backend will finally have 17B parameters. Although this won't lead to high computational costs in inference, the significantly increased parameters will require much more GPU memory. It would be helpful if the author could discuss whether there are alternative solutions to save memories. 2. CogVLM adopts Visual Expert Modules to retain LLM abilities via freezing it. However, how about involving pure language data in the SFT stage, as in [1]? Whether the LLM ability can be retained this way needs to be discussed. 3. CogVLM allows all visual tokens to share a single position ID in the RoPE. An ablation study should be reported. [1] InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

Questions

See weakness

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are not discussed in this paper.

Correction and Apology

We sincerely apologize for an error in our initial rebuttal. In our discussion of the LLaMA model used for the MME benchmark results, we incorrectly stated that we used LLaMA-3 32B. This was a mistake. The correct model used was LLaMA-3 8B. Thank you for your understanding, and we apologize again for this oversight.