Why are Visually-Grounded Language Models Bad at Image Classification?

Summary

This article notes that current top-performing VLMs, like GPT-4V and LLaVA, are unable to perform image classification tasks like the CLIP model. Despite having a larger number of parameters and incorporating a vision encoder from a pre-trained CLIP model. This author thinks the main reason for this underperformance is data-related. Critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. The author proposes to incorporate classification-specific datasets (such as ImageNet) into VLM training; its performance in classification and complex visual tasks can be significantly improved. For example, after fine-tuning ImageNet, VLM's performance on the ImageWikiQA dataset improved by 11.8%.

Strengths

The article clearly points out the performance issues of the current Visual Language Model (VLM) in image classification tasks. It systematically analyzes the possible reasons, filling the research gap in this field. This author thinks the main reason for this underperformance is data-related. By incorporating classification-specific datasets (such as ImageNet) into VLM training, the performance of classification and complex visual tasks can be significantly improved. For example, after fine-tuning ImageNet, VLM's performance on the ImageWikiQA dataset improved by 11.8%.

Weaknesses

The novelty is limited. This article assesses how well current VLMs perform on classification datasets and fine-tunes them using the same datasets. These contributions are insufficient to warrant publication of this paper.

Questions

I would like to know how well the fine-tuned VLMs perform on the original benchmarks such as MME, TextVQA, and others. Can we achieve similar results if a classification dataset is included in the second stage of training with LLaVA?

Rating

3

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the author has addressed relevant limitations and discussed the broader impacts adequately.

Summary

This paper analyzes the issue of large vision language models (VLMs) that perform poorly in common image classification datasets such as ImageNet. The authors analyze different perspectives on the problem, including trying different inference and training methods. For inference, the authors tried using different prompt variations, shrinking the number of classes, and computing conditional probabilities of class names. All these methods still leads to performance gaps between VLMs and CLIP. Furthermore, the authors confirm that sufficient class information is encoded in the output features of the visual encoder in VLMs, via linear probing. The authors also discover VLMs can be trained to generate class labels with a comparable accuracy as their visual encoders. Finally, they analyze the frequency of different classes in the original datasets that were used to train the VLMs, and find that the frequency of samples for a class is positively correlated to the accuracy of the VLMs on that class. This reveals that the poor performance of VLMs in image classification is due to the lack of class labels in their instruction-tuning data. They also construct a new question-answering dataset, ImageWikiQA, to test the model ability of answering questions related to the finegrained image class. The authors then finetune VLMs using a dataset that consists of both the image classification data and the original nstruction-tuning data. The resulting VLMs can perform well on the ImageWikiQA dataset.

Strengths

- This paper provides a relatively extensive analysis of why SOTA public VLMs perform poorly in image classification. The results generally look correct but there are some issues (as detailed in Weaknesses and Questions) that make them less sound. - The authors create a new dataset, ImageWikiQA, to evaluate the question-answering capacity of VLMs related to the fine-grained classes in ImageNet. - The authors show that incorporating the ImageNet classification data into VLM's instruction-tuning data can improve the model performance on ImageWikiQA.

Weaknesses

- It seems that the authors did not consider there may be multiple textual labels for many ImageNet classes. For example, for class with ID n01496331, the class name can be electric ray, crampfish, numbfish, or torpedo. When calculating VLM classification accuracy, the authors can match the model-generated text with any of these class names. - It may harm the VLM's instruction-following capacity to incorporate the ImageNet 1.28M classification data into the original LLaVA instruction-tuning data to train the VLM, because the classification data has a single fixed template "What type of object is in this photo? ASSISTANT: <Class Name>." Did the authors perform any evaluation of the VLM's instruction-following capacity? - It is expected that the information necessary for classification is largely preserved in the visual encoders in the VLMs, as these visual encoders are typically pre-trained on ImageNet and are kept frozen during the integration into VLMs. If the instruction-tuning data for VLMs do not contain class-specific information, it is unlikely that the VLMs can automatically align the class information in the visual encoder's output to the text generation. This explains why inference methods do not work in the paper. - Section 4.2 is a bit unclear. For example, it mentions that the authors randomly sampled at most 3 ImageNet images for each question, but then it mentions there are 2000 multiple-choice questions, each with an image. So how many images are there per question?

Questions

- Can the authors confirm the results in Table 3 (the right sub-table) are the accuracy on the validation/test sets? The accuracy on ImageNet looks very high to me. I have the same question for the accuracy mentioned in line 279. - In Table 4, LLaVA1.5-7B with GT Class provide still has a relative low accuracy. Why is that? - Can the authors confirm there are no training data in ImageWikiQA (i.e., it is only a testing dataset)?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Please see the Weaknesses.

Summary

The paper presents an interesting observation of VLMs lagging in image classification performance as compared to the visual encoders lie CLIP used within them. Several hypothesis are explored to explain this observation including train-time (information loss, training objective used), inference-time (prompt variations, label set size) and data related reasons. Their analysis shows the primary reason for the observed gap to be data prevalences, showing a correlation between class prevalence during VLM training and performance in those classes for image classification. The paper then simply proposes inclusion of these datasets into VLM training as a way to fix this issue, showing image classification improvements.

Strengths

1. The paper is nicely structured. An interesting performance trend is observed, hypothesis are explored to explain the phenomena, and conclusions drawn from the experiments are further tested. The evaluation and experimental details, the different hypothesis spanning both training and inference, and the well crafted control experiments make it a good study. 2. The paper is also well motivated and studies a relevant failure mode of VLMs. The results show that even though these modes might encode object or image level concepts, they still suffer from an inability to classify images which is supposed to be a more simpler, and more importantly a more fundamental visual task. This makes it a valuable research problem. 3. The specific experiments to explore each hypothesis are interesting in their own right. For example studying the effects of prompt ordering or CoT to image classification or different variations of inference or label set size, type of objective used, and linear probing results, all provide useful cues towards the hypothesis, but also showcase interesting VLM behavior. 4. The paper proposes a very simple and effective way to bridge the VLM performance gap by fine-tuning on the classification data. It shows how doing so leads to not just regaining high image classification performance, but also improves VLM-specific tasks on the classification dataset, for which they curate a new dataset.

Weaknesses

1. I'm not completely convinced with the paper's final conclusion of data being the reason why CLIP models are superior to VLMs. The fact that linear probes can extract good classification performance from VLMs shows that its a decoding problem since even these models were trained with the same data having skewed prevalences for certain badly performing classes, but the linear probe tuning manages to bring that out to the same degree as CLIP. This implies that either what's missing is a task-aligned fine-tuning (which is what a linear probe does, and which is what the paper does when they fine-tune) or perhaps even a modification of how these are inferred on for image classification which can lead to a more a task-aligned inference (CLIP is trained for text/concept alignment and is evaluated using knn whereas VLMs are trained for vision-text conditioned text generation, but evaluated in a very specific way by attaching tokens of the different options for image classification). In either case, the data argument for explaining the gap might not be the only, or even the strongest reason. 2. The paper proposes inclusion of the image classification datasets as a solution for this problem. If this VLM limitation stems for other reasons such as multimodal confusion and text feature interference or hallucination problem specific to the generated text space, adding more data for different tasks as a solution might not generalize to similar VML issues. In this case, since the new dataset involves tasks which necessitate good image classification, performance after fine-tuning goes up on both sets of tasks. This might not hold true for other equally fundamental visual tasks.

Questions

1. In Figure 2, the label set size does seem to have some effect on the performance gap b/w CLIPs and VLMs. Since this relates to the way VLMs are evaluated for image classification (attaching class options as tokens in the prompt which grow with the label set), do you think this experiment shows the evaluation difference between the two methods (and also how they are originally trained), can explain some of the gap? Also, why do you think the Caltech and ImageNet datasets behave a little differently than Cars and Flowers in Figure 2? 2. Why do you think CLIP does not suffer from the class prevalence bias but LLava does in Figure 3? What happens if we plot this for linear probed LLava models? What happens if we plot this for linear probed Llava models which are fine-tuned on similarly biased data having similarly skewed frequencies? If those models do not show this strong correlation, it might be more about the fine-tuning and less about the data distribution.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper has discussed limitations.

Summary

In this paper, the authors explored why Vision-Language Models (VLMs) significantly underperform as image classifiers. They compared several publicly available with proprietary VLMs on several classification benchmark datasets, including ImageNet, Flowers102, StanfordCars, Caltech101, and their newly collected ImageWikiQA dataset. The paper presented multiple hypotheses regarding the underperformance of VLMs, addressing questions related to inference, training objectives, and training/finetuning data. The authors conducted empirical studies and concluded that the performance of VLMs in classification is determined by the data used. They fine-tuned LLaVa1.5-7B on ImageNet and the LLaVa instruction-tuning dataset to enhance classification accuracy.

Strengths

1. The paper is well-organized and easy to follow. 2. The motivation is clear, and the empirical study is thorough. 3. The authors analyzed the training data of LLaVA1.5 models to show the strong correlation between class frequency over accuracy.

Weaknesses

1. The authors did not provide much analysis on the proprietary VLMs. Can the authors address the hypotheses related to inference, training objectives, and data for these VLMs? 2. The authors introduced “open-world setting” and “closed-wold setting” as evaluation protocols in section 2.3. However, the protocol seems to be only applied to Table 1. It is not clear which settings are being used for sections 3 and 4. 3. The authors need to provide more explanation for Table 2. What is the difference between the “Closed-World Setting” performance of Table 1 and the “Base Prompt w/ Label (Fixed Order)” in Table 2? Why are their accuracies different? 4. The LLaVA-7B and BLIP2-7B models are not large enough to demonstrate intrinsic capabilities such as chain-of-thought reasoning [1]. The authors could include the prompt variation results of Table 2 for one of the proprietary VLMs to show the impact on performance. 5. In Lines 178-179, the authors observed that the information necessary for classification is preserved in VLMs' latent space; however, it cannot be effectively decoded. If the VLMs have the essential information, then why are they not able to decode it? The authors should explain the possible reasons for this. 6. In Table 4, the finetuned VLM performance is worse than the zero-shot performance of GPT4 on ImageWikiQA. What are the possible reasons for this? Why is the “Finetuned on ImageNet” model performance (30.6) worse than the non-finetuned model performance (37.8)? Does finetuning always enhance the general capabilities of the VLMs? [1] Zhang, Zhuosheng, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. "Multimodal chain-of-thought reasoning in language models." arXiv preprint arXiv:2302.00923 (2023).

Questions

Please respond to the points of weakness I mentioned above.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes.