4M: Massively Multimodal Masked Modeling

Summary

The paper addresses the importance of a versatile model that is not limited to single modality and task and proposes a multi-modal pre-training scheme called 4M. 4M is a single encoder-decoder architecture trained on a large set of image and sequence-like modalities. The modalities including text, images, geometric and semantic are brought into a joint representation as tokens through modality-specific tokenizers. The training procedure relies on multi-modal masked training, with only a small set of tokens used as inputs and targets. Extensive experimentations showed that 4M can solve many common vision tasks out of the box, can be fine-tuned to unseen tasks as well as perform multi-modal controllable generation.

Strengths

This publication has several strengths including: 1) The writing is very clear and easy to understand. 2) The proposed approach is scalable across three key aspects - data (more training samples increase performance), architecture (improve performance with model size as well as remain stable) and training objective (handle growing number of modalities without incurring excessive computational costs). 3) Good experimental methodology with carefully designed ablations that justifies architectural design decisions especially impacts of input modalities and target tasks, multi-modal masking strategy as model and data scaling. 4) Very exhaustive in-depth experimentation showcasing the key capabilities - zero-shot generalization to diverse set of vision tasks, fine-tune to unseen tasks, multi-modal controllable generation.

Weaknesses

1) The paper seemingly lacks any quantitative evaluation of its generation capabilities or comparison with existing state-of-the-art methods. I'd appreciate it if the authors can elaborate on this. 2) Another concern is that the paper lacks any discussion on the robustness of the proposed approach to the quality of the datasets, since low-quality data is usually readily available compared to high-quality data. This will be critical for data scaling as well as will align with model scaling to 4M-XL and beyond. 3) Minor comment: I am curious if the authors have performed any out-of-distribution analysis.

Questions

The paper in its current form lacks discussion on the robustness of 4M to dataset quality as well as evaluation of its generation capabilities. Please refer to the “Weakness” section for details.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Yes, the authors discuss the limitations in the paper.

Summary

The paper proposes a multi-modal masked modeling pre-training scheme (4M) that unifies a several modalities – including text, images, geometric, and semantic modalities, neural network feature maps. The tokenization and masked modeling enable the efficient pretraining of 4M. The pretrained model can 1) achieve reasonable finetuned performance on vision tasks, 2) achieve conditional generation under different modalities.

Strengths

This work is a good practice on multi-modal masked modeling pre-training and achieve reasonable performance on both finetuned downstream tasks and generative tasks.

Weaknesses

1. This work is a combinational work of existing methods and lacks technical novelty. - The multi-modal masked modeling has been utilized by various existing works. E.g. MultiMAE has proven the feasibility of multi-modal masked modeling. Vision-NLP multi-modal is also explored by exsting works,e.g. MAGVLT. - The tokenization is claimed to be a key part for the efficient pretraining of 4M, while the tokenization has been widely used existing works, and the tokenizers used in this work are mostly borrow from other works. - The major contribution seems to be the the multi-modal pretrained data based on the CC12M dataset. But this is no novelty on the pseudo-labeling of the data. 2. The performance is not impressive given the large-scale data used in this work. - The paper claims 4M can perform a diverse set of vision tasks out of the box, but no experimental results are showed in the manuscript. - In Tab.1, the finetuned performance is relatively weak compared other pretrained methods. In addition, many stronger pretrained methods are not included in this paper. - The conditional generation results are interesting. While there is no quantitative performance comparison with other methods such as contorlnet. Also, one drawback of 4M based conditional generation is that the model accepts fixed modal after pertraining, while controlnet can be easily extended to new modal. - The paper claims the pretraining efficiency is an advantage, yet no experimental result is given to prove this point.

Questions

1. Clarification of the novelty of this paper. 2. The performance comparison and experimental results to support the claims in this paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

The limitation of this work is mainly on the lacking novelty and the experimental results cannot well support the claims.

Summary

The paper presents a foundation model for a variety of vision tasks. The authors show it can perform many key vision tasks out of the box and can also be fine-tuned to achieve highly competitive performance on unseen downstream tasks and input modalities. To handle the variety of modalities, the inputs/outputs are encoded into sequences of discrete tokens, and the model is trained on all the tasks simultaneously via a multi-modal masked modeling objective.

Strengths

The paper presents strong results and a scalable method to perform a variety of vision tasks. The ablation study covers almost all the aspects of the model. The results indeed prove the superiority of multimodal training over the baselines, without any need for augmentations. The paper is well-written and paves the way for a variety of research questions about the interactions between different modalities.

Weaknesses

While to tokenization method allows the model to train on a variety of tasks with a single architecture and cross-entropy loss, it also introduces quantization of the space of inputs/outputs. While this quantization does not harm the results for text, it might decrease the quality of the results for other domains (the segmentation boundaries might not be fully aligned with the objects for example). This idea induces an upper bound on the performance of such an algorithm and should be discussed. One way to evaluate this upper bound is by encoding and decoding back the ground-truth results of different domains (e.g. - quantizing the ground-truth segmentation masks and decoding them back), to verify the reconstruction quality and the downstream task-specific performance.

Questions

One suggestion is to mention in the related work other papers that deal differently with multimodal inputs/outputs for solving various vision tasks altogether. This line of work includes: - Wang et al., "Images Speak in Images: A Generalist Painter for In-Context Visual Learning", CVPR'23 - Bar et al. "Visual prompting via image inpainting", NeurIPS'22 One question that I had (and I am not sure how to evaluate) - what is more helpful for downstream performance - inputting during training tokens that correspond to the same image position but from different domains, or using more tokens from the same domain but from different locations in the image?

Rating

9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The limitations of the paper are discussed and addressed in the last section.

Summary

The paper presents a unified transformer model by using an effective multi-modal pre-training scheme. The authors propose to perform masked modeling across different modalities. This is made possible by unifying the representation space of the considered modalities by mapping them into discrete tokens and then performing multi-modal masked modeling on a small subset of tokens. Experimental results demonstrate several promising results.

Strengths

- This paper is technically valid and interesting. By conditioning on arbitrary modalities, the model can have great potential for a variety of multimodal intelligence capabilities. - The authors present comprehensive experiments and ablations, providing insightful discussions. The paper can be a good reference for future researchers. - The paper is well-written and easy to follow.

Weaknesses

- The multi-modal masking strategy is highly similar to prior works, like MultiVAE. The mask-modeling part of this paper is somewhat less interesting and less innovative. The innovation is more in the developed system framework. - I don't find other significant concerns in the proposed method.

Questions

N/A

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors discussed some of the limitations, but this is more like descriptions of future work. No method is developed in address the limitations.

Summary

This paper proposes a multimodal pre-training framework named 4M, which employs the masked data modeling style to train a transformer encoder-decoder archtecture that is capable of performing different downstream tasks. Experiments show that 4M delivers competitive transfer ability on these tasks compared with MAE / DEiT III / BEiT v2.

Strengths

1. The baseline settings in the experiments are fair and sound, especially the self-baselines to control other variables. 2. The ablation studies provide insightful discussions on the design choices of the pre-training strategy.

Weaknesses

1. The multi-modal and multi-task training of 4M needs datasets with all required modalities and labels. However, this kind of well-annotated dataset is hard to obtain and not scalable. This research employs pseudo labeling to extend existing image-text datasets such as CC12M. Therefore, the performance of the off-the-shelf labelers are important. The authors should provide more detailed and careful discussions and ablations on that. 2. The downstream tasks are limited, especially considering the target of this paper, i.e. ``massively pre-training''. Quantative results on more diverse tasks / datasets should be examined, especially on the transfer ability to novel tasks.

Questions

1. In the Table.2 of ablation studies, why use loss instead of corresponding task metrics as the mesure? 2. Could the authors provide some quantative results on the generative capabilities of 4M?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The model size and the data size could be further scaled up. The authors have discussed some limitations in their paper.

Thanks for your response.

- After seeing the response, my major concern about the novelty remains. - MAE cannot be regarded as a strong baseline now since many pretraining methods have been proposed with much stronger performance. - On the good side, this work is a good practice to combine exsiting methods to achieve the multi-modal pretraining. - Considering the multi-modal pretraining is a promising, I would like to rise my rating.

Thanks for the rebuttal

Thank the authors for their response. I maintain my score as "weak accept".