4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities

Summary

The paper presents an advanced vision model capable of handling a wide range of tasks and modalities, demonstrating the potential to train a single model on tens of diverse modalities without a loss in performance compared to specialized models. Specifically, the model is trained on a multitude of modalities, including RGB images, depth, semantic segmentation, CLIP features, surface normals, and more, enabling it to perform various tasks such as image generation, retrieval, and understanding.

Strengths

+ The presentation is clear and detailed. + The performance of the proposed model is good. + The workload of this paper is impressive.

Weaknesses

1. Do the authors try to measure the quality (e.g. FID ) of generated images (based on caption input) and can the proposed method surpass some existing diffusion-based models? 2. The authors should illustrate the detailed configurations of B, X, and XL models, for example, the number of layers and the width of channel dimensions.

Questions

see weakness

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

see weakness

Summary

The paper addresses the limitations of current multimodal and multitask foundation models, such as 4M and UnifiedIO, which are constrained by the limited number of modalities and tasks they can handle. The authors present a model trained on a wide variety of modalities and tasks using large-scale multimodal datasets and text corpora. This includes training on images, text, semantic and geometric modalities, feature maps from state-of-the-art models, and new modalities like image metadata and color palettes. A key technique used is discrete tokenization of various data types. The new model can handle at least three times more tasks and modalities than existing models without losing performance. This approach also enhances fine-grained and controllable multimodal generation and explores the unification of diverse models into a single one.

Strengths

1. The paper has a good starting point, noting that the network structures across various AI fields are converging (mostly to transformers). 2. This paper involves a significant amount of engineering work. Organizing large amounts of data and conducting large-scale training is not easy. 3. The experiments and performance in the paper are quite good.

Weaknesses

1. The writing of the paper has room for improvement. Not all reviewers have read the 4M paper. The paper does not detail the training framework, such as how to control different tasks and how to conduct multimodal masked training. These aspects are not simply described and are only mentioned in section 2 as following the 4M paper. Including a simple diagram would significantly improve clarity. 2. Some claims are somewhat inaccurate. There are works that use very simple methods to unify modeling for multiple tasks, maybe not as many as 10. It is necessary to discuss the difference and challenges between completing 5 tasks and 10+ tasks, as some tasks can actually be categorized as a single task.

Questions

1. The paper uses an encoder-decoder paradigm, whereas we know that the commonly used LLMs nowadays typically employ a decoder-only structure or simply a multi-layer transformer structure. Why does the paper adopt the encoder-decoder paradigm? The decoder-only approach[1][2] seems simpler and better suited for unifying with the LLM structure. The authors are advised to include some discussion on encoder-decoder and decoder-only approaches, referencing papers such as the following. [1] GiT: Towards Generalist Vision Transformer through Universal Language Interface. (ECCV 2024) [2] Fuyu-8B: A Multimodal Architecture for AI Agents. (Blog) The two articles mentioned above also tokenize everything within a simple unified framework. 2. Do different feature maps count as different modalities? I thought different modalities referred to the basic levels such as image, language, and speech, etc. The formulation in the paper is somewhat strange, and I hope the authors can explain this a bit. After all, these features are generated by cost-intensive multi-modal encoders, which differs from the lightweight multimodal tokenizer proposed in the paper. I also hope the authors can discuss this in comparison with the approaches of GiT[1] and Fuyu-8B[2], which use light-weight yet simple tokenizers on original image and language. 3. Regarding the challenge of negative transfer, you can also refer to the GiT paper. It demonstrated that joint training on five tasks has better performance than single-task training. (Considering that this paper is also recent, a direct comparison may not be necessary, but it is still recommended to discuss it.) :) Overall, I am inclined to accept this paper, but I still hope the authors can address my concerns. If these issues are addressed well, I will accordingly raise my score. :)

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

There are no comments regarding the limitations.

Summary

The authors present a new vision model that can generate tokens in all directions that represents multiple modalities like RGB images, depth map, segmentation maps, color palette, DINOv2 features etc. given a conditioning on a subspace of the modalities. They develop a new way to tokenize certain modalities and scale both model’s parameters and dataset’s size.

Strengths

- Good performances on a lot of downstream tasks. The authors evaluate the model on a wide variety of tasks and show promising results that are either on-par or above very strong baselines. - Showing scaling to bigger models isn’t easy - Multimodal retrieval is interesting to see and could be leveraged for a lot of new interesting tasks.

Weaknesses

- The added value over 4M is mostly due to engineering. - No multimodal retrieval nor conditional generation metrics. The actors emphasize a lot the generative/multimodal part of the model but the only quantitative experiments are shown in Table 3. Qualitative figures are interesting but should be supporting quantitative results. The conditional generation evaluation needs to be compared to other models like ControlNet [1] and you should at least provide precision/recall@k on some modalities for the multimodal retrieval (like comparing DINOv2/ImageBind with one modality against your model with one modality and your model with multiple ones). - The tokenization might reduce the burden of finding good losses’ weight, but tokenizers trained on a specific domain (like CC12M) might not generalize well. Moreover, tokenizers might not capture high frequency details that are very important like text for OCR, good quality faces, or smaller details. In [2], the authors speak about the fact that “quality limitations of the VQ image reconstruction method inherently transfer to quality limitations on images”. They also focus on the different domains that are not well captured by the tokenizers. The current paper only includes one ablation on the tokenizer, and I’m skeptical of the tokenizers’ performances on out-of-distribution images. [1] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang, Rao and Agrawala [2] Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors, Gafni et al.

Questions

- What are the performances of the trained tokenizers? Given that your model will be somewhat bounded by the tokenizers’ performances, could you provide a table including the performances of all tokenizers versus the original model and yours? - How did you chose each vocabulary size? - The metadata picked by the authors seem arbitrary, could you explain your decision process? l.223 I don’t understand why you treat RGB tokens and pixels differently: ‘we leverage RGB patch embeddings learned during the pre-training, as RGB pixel inputs are used alongside the tokenized modalities’. Could you expand on this please? The paper is interesting nonetheless but could be way, way stronger with quantitative evaluations on the new claimed capabilities.

Rating

5

Confidence

4

Soundness

2

Presentation

4

Contribution

2

Limitations

The authors talk about all the limitationsof their work.