Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.
Paper
Similar papers
Peer review
Summary
This paper introduces the first attempt to extend instruction-tuning paradigm to multimodal domain. This work has several major contributions: (a) the curation of the first vision-language instruction-following dataset by converting public image-text pairs into appropriate format using ChatGPT, resulting in over 100K+ multimodal instruction-following samples, (b) results indicating that a multimodal model (consisting of a CLIP visual encoder, a linear projection layer to convert visual tokens into language prompts, and a LLaMA language decoder) trained on this dataset can achieve robust multimodal chatting abilities. All assets used in their research including datasets and models are open-source.
Strengths
There are two major novel contributions of this work: (1) it introduces one of the first large-scale instruction-following multimodal datasets by leveraging public image-text pairs, and (2) it releases all training code, pre-trained models, and evaluation benchmarks to the wider public. These assets (outlined in supplemental L64) are undeniably valuable to the multimodal research community.
Weaknesses
I have several major concerns about (a) the evaluation benchmarks and metrics, (b) the lack of simple baselines such as captioning-based approaches, (c) missing implementation details such as the sampling procedure. A: Issues about quantitative analysis for multi-modal chatting. - The paper uses rather small evaluation sets (L217-235) to construct the LLaVA-Bench, including 30 randomly selected COCO images and 24 in-the-wild images. Why is this subset is much smaller than the pre-training dataset with 100K+ multimodal instruction-following samples? And how do you select the 24 in-the-wild images? I couldn't find evidence in the current draft to suggest that these 24 images are not cherry-picked. - The evaluation is text-only and the authors use GPT4 to explicitly assign a score. While prior works such as Vicuna [1] also uses GPT4 to score their responses in a text-only fashion, it is unclear how robust is GPT4 for multimodal reasoning while doing text-only evaluation. For more robust quantitative analysis, I would encourage the authors to split the instruction-following datasets into train/val splits and also include the results of classic text-only scoring metrics. Small-scale human evaluation will also be beneficial. B: Language prior of ScienceQA benchmark. - I am shocked that a text-only (vision-blind) GPT4 can achieve as high as 82% accuracy on ScienceQA, suggesting that this particular VQA benchmark has severe language prior [2]. Even though prior works also adopt this benchmark in their evaluation, this makes it hard to interpret the progress achieve by LLaVA towards a truly “multimodal” instruction-following agent as this benchmark can be largely addressed by language prior information. - Is it possible to report zero-shot LLaVA performance on ScienceQA? C: Simple baselines such as dense captioning: - Even though the model architecture of LLaVA looks elegant as it only uses a linear projection to connect CLIP’s visual tokens to soft language prompts, I believe an even simpler baseline is to train a dense captioning model (using the existing rich descriptions generated by prompting ChatGPT with caption+bounding box information). During inference time, the dense captioner can turn an image into a rich textual description, which can be sent to an instruction-following text-only LLM (Vicuna/GPT4). D: Missing implementation details such as sampling. - The sampling procedure (e.g., top-k/nucleus sampling/beam search) can have profound impact on the quality of generated texts. However, the current draft does not discuss how to perform sampling for LLaVA. Also, when using GPT4 for text-only evaluation, the exact hyperparameters used such as temperature should also be reported. E: Generalization or bias? - Fig. 5 in appendix suggests that LLaVA is able to generalize to unseen domains, i.e., correctly identifying that the person holding a doge coin is Elon Musk, while Elon Musk does not appear in LLaVA’s training dataset. However, it is unclear whether this is a result of generalization or language bias of LLMs. Perhaps your model tends to answer “Elon Musk” when asking about the name of the person, or perhaps it tends to answer “Elon Musk” when there is a doge coin in the image. One minor typo: L99: “and curate such a questions list” -> “to curate such a list of questions” Finally, I have an doubt about "multimodal instruction-following" (this is not a weakness but open to discussion): - Studies in NLP such as [3,9] have suggested that instruction-following is effective mostly because LLMs such as LLaMA are already capable foundation models, and therefore instruction-following can effectively align the model output with human interest. However, it is unclear whether multimodal foundation models such as CLIP (as used in LLaVA) is powerful enough. For example, a wide range of recent works and benchmarks [4,5,6,7,8] suggest that CLIP behaves like bag-of-words and do not have strong vision-language reasoning capabilities. As we do not yet have strong enough vision-language foundation models, it is unclear if the multimodal research community is ready to embrace the instruction-following paradigm. [1] Vicuna. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [2] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. Goyal et al. 2016. [3] LIMA: Less Is More for Alignment. Zhou et al. 2023. [4] When and why vision-language models behave like bags-of-words, and what to do about it? Yuksekgonul et al. 2022. [5] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. Thrush et al. 2022. [6] CREPE: Can Vision-Language Foundation Models Reason Compositionally? Ma et al. 2022. [7] Equivariant Similarity for Vision-Language Foundation Models. Wang et al. 2023. [8] Visio-Linguistic Reasoning with Multimodal Generative Pre-Training Scores. Lin et al. 2023. [9] The False Promise of Imitating Proprietary LLMs. Gudibande et al. 2023.
Questions
I summarized my most concerned questions about this work: - Why did you not sample train/val splits for evaluating LLaVA's multimodal chatting abilities? Are there specific concerns? - Is it possible to extend LLaVA's evaluation to other VQA benchmarks (as reported by GPT4) such as VQA2.0 which has balanced language prior? - Why is the architecture design of LLaVA more superior than a dense captioning model + instruction-following LLM, if both are trained on the same dataset? - What is the sampling procedure of LLaVA? Given that this paper presents a significant dataset contribution, I would be happy to revise my rating if the authors can address my above-mentioned weaknesses and questions. Updated in Aug 17th: I have increased my rating based on the author's promise to revise the paper by including more discussion on more scientific evaluation metrics and benchmarks for MLLMs.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
4 excellent
Contribution
4 excellent
Limitations
Yes, the authors discuss about limitations in supplemental material.
Summary
This presents a multi-modal instruction following model and evaluates it. The model is trained by using a frozen vision encoder whose features are used an input to a LLM, which is fine-tuned. It is trained first on simple captioning tasks using a large amount of data, and then on a multi-modal instruction following dataset built by having a LLM generate tasks using the captions and bounding boxes of images.
Strengths
- Presents an impressive model that seems to have advanced instruction following abilities for images and text. - Present a detailed set of comparison and evaluation using LLaVA-Bench, and the ScienceQA experiments. I also appreciated the re-use of a qualitative example from the GPT-4 paper. - Some interesting insights, such as using the second to last layer and getting a sense of the benefit of each kind of instruction-tuning task.
Weaknesses
- I think it would be valuable to see if this same pipeline could work with an open source LLM, using OpenAI's closed LLMs are not ideal for scientific understanding and reproducibility. I in particular wonder if other LLMs can use the bounding boxes annotation of the images effectively, which I imagine in much harder than the using the captions. - Seeing an evaluation on standard vision/language benchmarks would have been interesting, such as zero-shot VQA.
Questions
Would the data generation pipeline benefit from even more detail image descriptions? There are other annotation like region captions or visual narratives that could have also been used. I am also curious if the model can handle very low-level detailed questions, like "What color is the rope the man is using?" for Table 3. 278: year -> layer
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
The authors have a detailed section in the appendix.
Summary
This paper studies instruction tuning in the multimodal domain. Instruction tuning has recently drawn a lot of attractions in the large language model (LLM) field, and hence it is interesting and important to study similar capabilities in multimodal models. This paper is a pioneer work in this direction. It constructs the first instruction tuning dataset LLaVA and conducts comprehensive analysis. This is an important step towards general-purpose multimodal language model (MMLM) that can interact with humans using natural languages.
Strengths
1. To the best of my knowledge, this is the first work of multimodal instruction tuning. This is an important direction and hence the impact of this paper is huge. 2. The dataset constructed in this paper is both useful and inspiring. It constructed an impactful first step for future studies in this field. 3. The paper is well written and contains a lot of detailed studies.
Weaknesses
I don't see any major weakness of the paper. There are some minor ones that can be improved but I understand they may go beyond the scope of this paper. For example, it will be more comprehensive to conduct ablations on the models used in this paper other than Vicuna etc.
Questions
What did you use 2e-3 lr for pretraining? This seems to be a very large value given the 128 batch size.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
NA
Summary
This paper introduced LLaVA, an effective visual instruction tuning method to turn Large Language Models (LLMs) into multi-modal LLMs. LLaVA is first pre-trained on image-text pairs to connect a visual encoder (CLIP) and a LLM (Vicuna). Then the authors utilize GPT4 to generate ~150K visual instruction data for training visual instruction-following models. LLaVA is evaluated on two diverse and challenging benchmarks, as well as a science question answering dataset. Overall, LLaVA is a very early attempt for enhancing LLMs with multi-modal capacity. I believe it will inspire a lot to the research community.
Strengths
- Expand instruction tuning to the vision domain. Visual instruction tuning is a new research problem for vision-language models. It endows vision-language models with powerful comprehension and reasoning capabilities. - A new pipeline for visual instruction data generation. LLaVA takes image captioning or object detection results as GPT4’s input for visual instruction generation. This is an effective way to quickly generate a large amount of visual instruction data. - A strong visual instruction model LLaVA with available pretrained models and demos. - Multi-modal instruction-following benchmark.
Weaknesses
- Data quality. The proposed visual instruction data is automatically generated by GPT4. But there seems to be a lack of validation of the data quality. For example, MiniGPT4 [1] will manually verify the correctness of each image description. High quality instruction data is also proved to be important for pure LLMs in LIMA [2]. - Is 595K image-text data enough for vision-language alignment? Blip2 uses >100M image-text pairs for vision-language alignment, while LLaVA only uses 595K data from CC3M. Have the authors tried to use more pre-training data, and will the model be further improved? Besides, I noticed that LLaVA inputs 256 CLIP visual tokens into the LLM, which is much large than Blip2 and MiniGPT4 (~30 tokens). Such a design will make the training much slower. So, do we really need 256 tokens? - The LLM in LLaVA is fully fine-tuned in the second stage. Will this lead to degradation of LLM's ability? Are there verification results on traditional LM tasks? [1] Enhancing Vision-Language Understanding with Advanced Large Language Models [2] LIMA: Less Is More for Alignment.
Questions
See Weakness
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
Yes. The authors have addressed the limitations and potential negative societal impact.
Follow-up questions
Thanks for the detailed response. Some of my questions in the original review were not answered and I would still appreciate a discussion on these topics: **Q1. "While prior works such as Vicuna [1] also uses GPT4 to score their responses in a text-only fashion, it is unclear how robust GPT4 is for multimodal reasoning while doing text-only evaluation."** After reading the response, I am even more concerned about GPT-4 text-only evaluation: - It costs money to perform evaluation, which limits the custom test-set in this paper to 30 COCO and 24 in-the-wild images. - GPT-4 performance is not stable across time. - As mentioned by authors in response to Q5, the textual annotation of an image may not capture all the details in an image. - While prior works such as Vicuna also use GPT4 to output numerical scores, it is hard to measure how accurate the numerical score is. **Q2. "Are the 24 in-the-wild images cherry-picked?"** By cherry-picking I actually meant **selecting test samples based on the performance of LLaVA**. While I would like to believe that the authors did not do this, this evaluation set just seems too small for robust and scientific benchmarking. Although it covers a diverse range of skills, I would like the authors to provide more discussion in the revised paper about **what is the scientific way to benchmark such multimodal instruction-following models.** For example, should one be using larger and standardized benchmarks such as VQA2.0 and GQA (or some more recent multimodal benchmarks)? Are there better and more reproducible evaluation metrics than GPT-4's raw numerical scores? Because LLaVA is one of the first works on this trending topic, I believe it is important to answer these questions such that the community can move in the right direction. After reading the response to Q4, I would like the authors to clarify a few things about ScienceQA experiments: **Q3. What is the "random-guess" mode, and why does it help with text-only GPT4? I couldn't find the term "random guess" in the original ScienceQA paper or their github repo, so I would like a more detailed explanation here.** **Q4. How does the text-only GPT4 and zero-shot/fine-tuned Vicuna baseline utilize the image context? Do they use the image caption from the ScienceQA dataset?"**
Discussion: the scientific way to benchmark multimodal instruction-following models
We thank the reviewer for the insightful comments, and we are happy to discuss the evaluation aspect for multimodal models. For existing benchmarks, they usually focus on a single aspect. However, what is unique to the recent large multimodal models (LMM), like multimodal GPT-4, is that they can perform visual tasks in the wild that require integrated capabilities. For example, in Supplementary Table I, to correctly complete the user’s request, it requires the model to have: (1) OCR capability to understand the captions; (2) visual recognition capability to understand that it is a pan of nuggets that looks like a world map; (3) reasoning capability to combine the information together and answer why this can be interesting. **Having a model that excels in each single aspect, does not necessarily guarantee a capability to combine and reason about them in a single answer**. For example, BLIP-2 is one of the best models that ranks top across the board of academic benchmarks, while lacking the complex reasoning capabilities. Further evidence can be found in Table 5, where BLIP-2 is capable at answering short-form “conversational” questions and fails to tackle more complex tasks. Some recent evaluation benchmarks on LMMs also revealed similar results. At the time of submission, such a benchmark was lacking. This motivates us to construct such a benchmark that requires the model to leverage different capabilities to correctly complete a user’s task. We tried to utilize the resources we have at the time of submission, to construct such a benchmark, with an aim to create scientific and controlled settings. **We do not cherry-pick test samples based on the performance of LLaVA. We do not tune any model/data design choices, based on any of the results we obtain on LLaVA-Bench-In-the-Wild, and we use that solely for benchmark purposes**. However, we fully agree with the reviewer that to have a more comprehensive and complete understanding of the model’s capability, we would need a benchmark at a larger scale, which we were unable to achieve due to the cost limit. Some recent works show a clever way to address this. Instead of directly evaluating with GPT-4, one can use ChatGPT to extract the answers, or key aspects that are required to solve the problem. This can allow the model to still answer with natural sentences, while enabling evaluation at scale. We are happy to see such progress in this field. Of course, despite being cheaper, ChatGPT still incurs a cost. During the rebuttal, we find the largest open-source model, LLaMA-2-Chat-70B, elicits impressive capabilities in following complex instructions like ones we used to query GPT-4 to create LLaVA-Instruct-158K (see response to Q1 of MLCz). It can also be a cost-free alternative. Meanwhile, we believe that it is also important to leverage academic benchmarks, like VQAv2, GQA, etc. The current LLaVA is only trained on natural instruction and responses, making it challenging to evaluate on those standard benchmarks that have ground truth with a single or a few words. Given the large size of these datasets, it is an open research problem to design efficient and effective metrics. We find the recent VisualGPTScore [1] can be considered as an inspiring way to construct a cost-efficient metric. For example, one can evaluate $P(text|image)$ after the model outputs the long-form answer. We thank the reviewer for bringing up this important topic for discussion, and we are happy to discuss and clarify any further questions or doubts. We will include these discussions in our revision, and we believe our draft will be stronger by including the insightful suggestions from the reviewers. [1] Visio-Linguistic Reasoning with Multimodal Generative Pre-Training Scores. Lin et al. 2023.
Response to follow-up questions
**Q1. "While prior works such as Vicuna [1] also uses GPT4 to score their responses in a text-only fashion, it is unclear how robust GPT4 is for multimodal reasoning while doing text-only evaluation."** Below, we further clarify some aspects of GPT-4 evaluation, and the measures that we have taken to make sure that the experiments are conducted under a controlled setting. > It costs money to perform evaluation, which limits the custom test-set in this paper to 30 COCO and 24 in-the-wild images. We agree. We have discussed some alternatives above. > GPT-4 performance is not stable across time We find the comment relevant to the release of an iteration of GPT-4 on June 13. We do want to note that the gpt-4-0613 is released after the NeurIPS submission deadline, and we have taken two measures to make sure the experiments are conducted in a controlled setting. We fixed the GPT-4 endpoint version to gpt-4-0314, after the release of gpt-4-0613. The same endpoint outputs consistent evaluations as verified in Table 5. This ensures that the evaluation is consistent before the retirement of the gpt-4-0314 endpoint (no earlier than July 5, 2024). We found that the relative rankings of different methods remain consistent as long as the same API is used (across gpt-3.5-turbo-0301, gpt-4-0314, gpt-4-0613, respectively). We suggest that future researchers should fix the same API and evaluation pipeline throughout their experiments for all the methods, and report API endpoints in the paper. > As mentioned by authors in response to Q5, the textual annotation of an image may not capture all the details in an image. We would like to clarify that when we construct the instructions in the LLaVA-Bench, we make sure that the instruction is answerable by referring to the provided image context. In the context of Q5, we want to emphasize that, when the model is generating captions, without the access to the user instruction, it may not capture all details in the image that the user cares about. We believe the instruction-aware property can be of great value to future research. > While prior works such as Vicuna also use GPT4 to output numerical scores, it is hard to measure how accurate the numerical score is. MTBench [1] evaluates the agreement between GPT-4 scores and the human evaluation, and shows that there is a strong agreement between them. A recent study to evaluate LMM shows that the agreement between GPT-4 score and human evaluation is quite consistent (4.2% absolute discrepancy). **Q2.** Answered with the discussion above. **Q3. Clarification of "random-guess" mode.** We apologize for the confusion. In the official implementation (`models/run_gpt3.py#L70`), when the model does not output a valid answer, it randomly picks an answer. This improves the GPT-4’s accuracy when it refuses to answer. For fine-tuned LLaVA models, it follows the predefined format, so adding this does not further improve the model’s performance. When we evaluate zero-shot LLaVA and Vicuna, such cases also happen. We also treat them as incorrect and do not perform random choice. **Q4. How does the text-only GPT4 and zero-shot/fine-tuned Vicuna baseline utilize the image context? Do they use the image caption from the ScienceQA dataset?"** We do not add image captions from the ScienceQA dataset. We made this choice as we find that adding the image caption harms the Vicuna’s zero-shot performance (IMG 56.27 -> 52.95). We hypothesize that it can be due to the fact that the image caption does not always provide accurate context (similar to observations in Fig. 17 and relevant discussions in ScienceQA paper). This can confuse the model and thus make incorrect predictions, after such irrelevant context is incorporated. [1] Zheng, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena."
Thank you for the detailed discussion
I genuinely thank the authors for their discussion on more scientific and cost-efficient evaluation metrics for MLLMs. I believe such a revision of the paper will make the submission much stronger, and thus I have increased my rating. I also appreciate authors' detailed response on all my other questions. The above response seems to be missing a few references. For example: 1 - *"Some recent works show a clever way to address this. Instead of directly evaluating with GPT-4, one can use ChatGPT to extract the answers, or key aspects that are required to solve the problem."* -- What are the recent works? 2 - *"A recent study to evaluate LMM shows that the agreement between GPT-4 score and human evaluation is quite consistent (4.2% absolute discrepancy)."* -- What is the recent study?
We sincerely appreciate the reviewer's insightful comments and the increased rating. Regarding the missing citations, we would like to clarify that the paper is cited by those publications. We have consulted with the ACs, and they advised us not to include citations of these papers to maintain the anonymity of the review process. We will provide the exact references to the ACs and PCs. Once the reviewing process is concluded, we will post the citations here via an "Official Comment" for your reference and for the benefit of the community.
Thank you for answering my questions, I think discussion about the weaknesses of these systems is also very valuable. I continue to be very positive about this paper.
Decision
Accept (oral)