Language Is Not All You Need: Aligning Perception with Language Models

Summary

This paper propose a pretrained multimodal large language model, which can achieve impressive zero-shot performance on many downstream tasks. The experimental results verify the transfer knowledge from language to multimodal or from multimodal to language. However, compared to the very similar model FROMAGe and BLIP-2, this model structure seems to be of little difference. The most advantage of this model is that it uses more multimodal data. It seems that the performance on visual question answering is less than BLIP-2.

Strengths

The motivation is clear and the writing can be easy to be followed. The multimodal COT reasoning ability is analyzed which is interesting. The pretrained multimodal large language model achieve impressive zero-shot performance on many downstream tasks.

Weaknesses

This paper presents a lot of experiments to demonstrate how good their model is, but it does not compare GPT-4 or BLIP-2 to show how much room for improvement their model has. Directly adopting a good image captioning model to convert images into diverse text and feeding them into a large language model (gpt-3.5 or llama) will also achieve good results (please see the multimodal instruction-following data construction way shown in the paper ``visual instruction tuning’’ ). The technique novelty advantage of this model compared to the common approach should be elaborated. This paper claims that they proposed a multi-modal large language model that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot), but the experimental baseline always lacks some key models that are relatively strong in the field, such as the experiment results (Table 6) shown in Section 3.8: language tasks. They only compared the language model trained on their own corpus. In fact, the multimodal version of KOSMOS-1 sees more text data and image description, leading to the comparison a little unfair. On the one hand, the comparison is not sufficient because there is no comparison with other large language models. What are the overall advantages and core contribution of KOSMOS-1 compared to LLMs (such as LLaMA, BLOOM, OPT) and previous multimodal models (BLIP-2, Flamingo)? In addition, Sec 3.6 Multimodal Chain-of-Thought Prompting analyzed the multimodal COT reasoning ability, but it is only evaluated on the classification task SST-2. There is no demonstration of corresponding examples and no evaluation on complex reasoning tasks (e.g., ScienceQA), so it is difficult to judge whether this KOSMOS-1 really has the multimodal COT reasoning ability. Because prior work has shown that the ability of the chain of thought may emerge when the number of parameters of large language model is greater than 10B. The conclusion drawn by this paper is hard to convince. Finally, Sec 2.4:Training Objective does not specify in detail how the visual representation model and language model are trained. If you train separately, how do you perform multimodal alignment and fusion? From the description, what is the difference between this whole training process and Oscar's training method? If following your description to train KOSMOS-1, will the model have the multi-modal COT reasoning ability? In sec.2.1, should the input textual token embedding and visual token embedding be pre-aligned and then feed to the decoder? Or is it entirely dependent on the learning ability of the following decoder.

Questions

See the weaknesses part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

None

Summary

This work introduces KOSMOS-1, a multimodal large language model trained on large-scale text corpora, image-caption pairs, and interleaved image-text data. KOSMOS-1 can perform classic captioning/VQA tasks in a zero-shot or few-shot in-context prompting fashion. It can also perform OCR from visual document, and answer questions based on a webpage. It consists of a pre-trained CLIP-L/14 model for image representation and MAGNETO for language decoding. KOSMOS achieves competitive results on a wide suite of vision-language tasks with 1.6B parameters compared against Flamingo-9B. Furthermore, it introduces a Raven’s IQ benchmark to evaluate nonverbal intelligence.

Strengths

The paper presents comprehensive technical details about the proposed system, including pre-trained models, loss design, and datasets. It also shows impressive qualitative sample usage of the proposed system on a wide range of tasks in supplemental, including multi-turn multimodal dialog, image classification with description, few-shot multimodal in-context learning, multimodal chain-of-thought prompting, and reasoning with webpage screenshots.

Weaknesses

I do not find the proposed system particularly novel, because its architecture design, training losses, and pre-training datasets are widely adopted in prior art such as Flamingo [1] and BLIP-2 [2]. Furthermore, BLIP-2 [2] achieves stronger zero-shot performance on VQAv2 (Table 2b) 65% compared to KOSMOS-1's 51.0%, even though it is trained on much fewer data (LAION114M) with much fewer tunable parameters. The small-scale Raven IQ test with 50 samples is undoubtedly challenging and interesting, however, it is hard to say KOSMOS-1 is better than random chance because it is only marginally better by 5.3%. Could authors discuss what are some promising directions to improve KOSMOS-1 on this benchmark? Language-only instruction-following paradigm shows worse performance on COCO but leads to better results on Flickr30K, VQAv2, and VizWiz (Table 7). Could the author explain why language-only instruction-tuning might lead to better or worse results on different vision-language tasks? To demonstrate that the KOSMOS-1 has better commonsense reasoning capabilities, the authors show language-only zero-shot evaluation results on RelativeSize, MemoryColor, and ColorTerms even though KOSMOS-1 is a multimodal model. Is there a reason not to report the multimodal zero-shot performance? The authors do not promise a model release. [1] Flamingo: a Visual Language Model for Few-Shot Learning. 2022. [2] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. 2023.

Questions

- How can one improve KOSMOS-1 for better performance on Raven-IQ test? - Why is the zero-shot VQAv2 performance of KOSMOS-1 worse than BLIP-2? - Could authors report KOSMOS-1 performance on commonsense reasoning tasks while using multimodal inputs? - What is the intuition behind language-only instruction-tuning? Why could this benefit vision-language tasks? - Will the code and model be released to the public? The writing can be further improved. For example: L24: “it is still struggling to natively use LLMs for multimodal data, such as image, and audio.” -> “it still struggles to natively use LLMs for multimodal data, such as image and audio.”

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes, the authors discuss limitation in appendix.

Summary

This paper presents a vision/language model trained on text and interleaved image/text data. It uses a ViT to encode the image into tokens and a transformer predict output tokens from the previous ones. It is trained on web-scale data and then fine-tuned on NLP instruction tuning data.

Strengths

- Despite a lot of interest in instruction following for V/L models, training them from scratch is a somewhat less well explored area so I think contributions to this direction are useful. - Many evaluations that help understand how effective the model is and what it has learned in different settings - Showing NLP instruction tuning benefits multi-modal tasks is very interesting, although I would have liked see the ablation consider tasks with more complex instruction then just captioning/VQA kind of tasks. I also found the the chain-of-thought and visual common sense results interesting.

Weaknesses

- The models appears to be limited in its ability to take advantage of few-shot data, which I think should be one of the main theoretical advantage of supporting interleaved V/L data and pre-training on such data. Captioning seems to be the only task where we really see a clear benefit. - The instruction following shown in the paper is a bit simple, just for tasks like captioning, QA or classification which are tasks that are pretty close to the pre-training tasks. While it is interesting that this data helps the model. it has not really been shown the model can follow instruction for very different tasks in the same way LLM can. - Related work is poorly discussed. There is no related work section, and as far I can see there is not much discussion about similar works such as other V/L foundation models or models like BLIP that adapt a pre-trained LLM in other parts of the paper or appendix either. Overall I feel like the model and many experiments have scientific value despite the relatively smaller scale compared to other recent models, but I also feel like the lack related work is a non-trivial issue and I am not really sure how to balance those two points. I have recommend accepting the paper for now since I still think the paper would a benefit for the conference.

Questions

Will the model or the interleaved data corpus be released? Is there any evidence the model "forgets" any of its multi-modal knowledge during instruction fine-tuning, which only has NLP data? Is this a concern? Did the authors consider initializing the model somehow with a pre-trained LLM? With so much work following that approach recently it would be interesting to hear why everything was trained from scratch.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

This paper proposed KOSMOS-1, a Multimodal Large Language Model (MLLM) that can take image embeddings as additional input to auto-regressive LLM. Trained on both text-only and image-text web-crawled data, KOSMOS-1 can get strong performance across a variety of tasks including language understanding, perception-language tasks, and vision tasks, demonstrating the utility of cross-modal knowledge transfer.

Strengths

1. Although there are no specific novelties in terms of loss and module design, the introduced model is simple, unified, versatile, and effective. 2. The experiments cover both language-focused, vision-language, and vision-focused evaluations and demonstrate the strong performance of the proposed models. 3. The study about whether one modality can benefit the other (cross-modal transfer) is interesting and showcase some insightful observations.

Weaknesses

1. Comparison with recent MLLM works either in related work or experiments was missing. For example, MiniGPT4, LLaVA, etc. 2. Although in ablation about cross-modal transfer, authors show that Language-Only Instruction Tuning can help several visual-language tasks. However, that experiment is only about language instruction tuning data. I wonder if the general Text Corpora (not just language instruction tuning data) in pre-training help general vision-language downstream tasks? 3. Can Visual Instruction Tuning data (LLAVA's or MiniGPT4's) help NLP tasks?

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Discussed in Supp.

The authors have addressed my concerns

I will retain my positive rating as the authors promised to release the model and dataset and have addressed all of my concerns.

Official comments from BRko

I have read the authors' rebuttal as well as the other reviewers' comments. First the authors have addressed my comments as well other reviewers' comments. Second, the proposed method does have merits, which can use different data ( text corpora, image-text pairs and interleaved image-text data), and good model performances. Therefore, I am increasing my rating from Borderline Accept to Weak Accept.