VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Summary

The paper proposes a model that supports a wide range of multimodal tasks, beyond text generation. The approach leverages an LLM, modality specific encoders and task specific decoders to effectively handle different tasks. The LLM communicates with task decoders via “super link”, which consists of special tokens and soft prompts for each task.

Strengths

- The work is extensively evaluated on many benchmarks - The model competes with other more specialized approaches - The paper is well written and easy to follow - The paper addresses an important problem, building efficiently a generalist models is still an open-research question

Weaknesses

1. The paper title is a bit misleading. It gives the impression that the paper proposes a single model that can handle many tasks, while it is basically an agglomeration of many powerful pretrained models (CLIP, LLM, Stable Diffusion, UniPose…). 2. The contribution is limited. The super link is basically soft prompts (learnable tokens) and special tokens per task. Soft prompt is widely used in other approaches (shared learnable query as in InstructBLIP) or (learnable query per task as in eP-ALM/MAPL). For the special tokens, FROMAGe uses special tokens to handle multimodal outputs. The main contribution of the paper is in scaling this approach to many multimodal tasks. It is also important for the paper to discuss these similar approaches. 3. The paper lacks details about the model training. In particular which training objectives are used on top of the task-specific decoders? And how the different losses are weighted? 4. Did the authors experiment with one-stage training instead of the 3-stages training? I didn’t find any experiment to support this design choice 5. The paper claims “Multimodal In-Context Ability. our model … exhibits superiority over the previous in-context models …” However there is no quantitative comparison to support the superiority of the proposed model in a a few-shot ICL setting. InstructBLIP: Dai, Wenliang, et al. "Instructblip: Towards general-purpose vision-language models with instruction tuning." Advances in Neural Information Processing Systems 36 (2024). eP-ALM: Shukor, Mustafa, Corentin Dancette, and Matthieu Cord. "ep-alm: Efficient perceptual augmentation of language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. MAPL: Mañas, Oscar, et al. "Mapl: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting." arXiv preprint arXiv:2210.07179 (2022). FROMAGe: Koh, Jing Yu, Ruslan Salakhutdinov, and Daniel Fried. "Grounding language models to images for multimodal inputs and outputs." International Conference on Machine Learning. PMLR, 2023.

Questions

Check weaknesses section (e.g. 3-4)

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discuss the limitations and the broader impact in the paper.

Summary

This paper introduces an advanced multimodal large model (MLLM) that integrates visual perception, understanding, and generation in a unified framework. Unlike traditional models limited to text outputs, it expands its capabilities to tasks like object localization, pose estimation, and image generation and editing via a "super link" mechanism to connect the MLLM with task-specific decoders, facilitating flexible information and gradient feedback transmission while resolving multi-tasking training conflicts. The model is trained on data from hundreds of vision and vision-language tasks, allowing it to generalize across these tasks with shared parameters and achieve performance comparable to task-specific models. VisionLLM v2 aims to enhance the generalization of MLLMs.

Strengths

1. The starting point of the paper is very good: enabling LLMs to use a variety of tools so that the model can handle various tasks. 2. Model's performance is excellent, reaching the state-of-the-art for general models across various tasks. 3. The experiments are extensive, involving a significant amount of engineering work, and integrating various tasks and datasets.

Weaknesses

1. The training process is relatively complex, consisting of three stages, making it difficult to promote and use in the industry. The method described in the paper seems like it could also be done in a single end-to-end training stage, similar to LLMs. [1] is the one-stage training general vision model. Discussing the general models [1][2] trained in a one-stage manner with a pure language interface would be better. :) 2. In Table 3, the detection performance is basically on par with the single-task decoder performance later on. Does this imply that the performance mainly depends on the task-specific decoder? Is the method proposed in the paper primarily aimed at facilitating the integration and scheduling of various decoders? Will the computation of MLLM enhance the performance of the subsequent decoder? 3. The advantage of LLMs is not only their compatibility with various tasks but also their ability to mutually enhance different tasks, which is the so-called multi-task capability. In the method described in the paper, different tasks share the same MLLM and encoder. So, can joint training of different tasks, like GiT[1], lead to mutual enhancement? Points 2-3 are just areas where I think there might be room for improvement, but they won't significantly affect my decision on whether to accept this paper. It's just for discussion. :) [1] GiT: Towards Generalist Vision Transformer through Universal Language Interface. (ECCV2024) [2] Fuyu-8B (blog)

Questions

This question is just to hear the authors' perspectives: Should a general model use a bridging approach (LLaVa or VisionLLM) or a simple multi-layer transformer (Fuyu8B, GiT, Gemini)? The former might be easier to implement but has complex training and scaling issues, while the latter has greater advantages in scaling and simpler training, similar to GPT. (Authors can answer freely without considering that the paper uses a bridging framework. This will not affect the final score.) :)

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

This is purely academic research and does not involve any potential negative societal impact.

Summary

This paper proposes an end-to-end generalist Multimodal Large Language Model (MLLM) for a variety of vision-language tasks, including captioning, detection, segmentation, and image generation. It introduces the concept of "Super Link" for triggering different tasks and attaches corresponding task decoders to the MLLM through query embeddings for end-to-end gradient updates. The model demonstrates strong results on the problems of interest and opens a new avenue for future multimodal generalist studies.

Strengths

The paper is well-written, and the idea is well-motivated. The evaluation results are generally sufficient.

Weaknesses

The paper lacks a comprehensive review of related work. Relevant works such as AnyGPT [r1], Chameleon [r2], the CM3 series, and other earlier publications that unify image generation in MLLM with image quantization and decoding should be reviewed and acknowledged. Clarification is needed on how super-link queries are generated. For [DET] and [SEG], these can be boilerplate templates, but what about text-to-image generation? Are the super-link queries generated on-the-fly based on the context? Examples for [GEN], similar to the one at the bottom of Page 5, would be appreciated. Regarding the joint multi-task training stage elaborated in Sec. 3.3, ablation studies on how each task benefits the others should be conducted. Does performing fine-grained vision tasks enhance the model's image generation capability, or do multimodal understanding and generation remain conflicting as shown in the literature? Do detection/segmentation tasks lead to better controllability or spatial reasoning for image generation? These questions need to be studied. [r1] Zhan et al., AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. [r2] Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models.

Questions

see previous sections

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

appear to be sufficient