LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

Summary

The paper proposes a novel method for improving zero-shot classification of VL models, in an unsupervised manner. This is done without additional visual labeled examples, yet with additional unlabeled examples. They rely on LLM to generate a dataset of text describing the desired classes. Then, they train a text classifier on top of the VL textual encoder using the generated dataset. As a second stage, they use a set of relevant unsupervised images as well as a set of designed augmentations, to finetune the classifier and visual prompts, adjusting it to the VL models image representations. In an extensive evaluation, the authors show a significant performance enhancement, and ablate their method methodically.

Strengths

- The paper is clearly written and easy to follow. - The performance enhancement shown by LaFTer is very significant and consistent across most test cases. - Although inspired by previous works, the idea of tunning the visual encoder of VL models using text supervision is highly interesting. Overall, I believe the idea of the paper is very interesting, and the results are a significant improvement over CLIP.

Weaknesses

My main concern is the performance of CLIPPR baseline. In most cases, it has been found to be less successful than the original CLIP, which is not intuitive with the method. Could the authors explain this gap please? One other minor concern is the statement in the abstract about LaFTer being the first to reduce the gap from the supervised baseline. Table 1 shows LaFTer is not the first. Table 2 shows that it does not close the gap for about 50% of the cases, using only few-shot supervision. I believe this is misleading statement.

Questions

- Could the authors please verify if the experiments denoted by LaFTer* are just the second stage of LaFTer using CLIP as the initial text classifier? If so, I think these results are more suitable to appear in the ablations section (Table 4).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

- The method requires access to a set of unlabeled images from the same distribution. - The method is limited by the quality of the visual representation of the VL model.

Summary

This paper proposes a finetuning approach for Vision-Language models that does not require any labels. It begins by demonstrating the transfer of information between different modalities and training a classifier using natural language inputs. This classifier achieves the successful classification of visual data. Furthermore, it utilizes this pre-trained classifier that relies solely on text inputs in our pseudo-labeling pipeline. This enables it to effectively and efficiently fine-tune the Vision-Language models with fewer parameters.

Strengths

1- The proposal introduces a label-free approach to zero-shot learning, capitalizing on the benefits of pretrained LLM models. Additionally, it enhances the model's performance by finetuning it in an unsupervised manner. This novel approach presents a new direction for zero-shot learning research, eliminating the need for labeled datasets even for seen classes. 2- The proposed method extends its experiments to include a few-shot framework and demonstrates a noteworthy improvement compared to the baseline approaches. 3- The experiments are conducted on twelve (12) diverse datasets, encompassing coarse-grained, fine-grained, and natural scene datasets. A comprehensive ablation study is conducted to demonstrate the effectiveness of the proposed model and the contribution of its individual components.

Weaknesses

1- Upon analyzing Table-1, it is evident that the proposed model exhibits lower performance for the Flower and SUN datasets, both commonly utilized in zero-shot learning and are fine-grained datasets. This observation raises surprise as the proposed model struggles specifically with fine-grained datasets. 2- The visualization of embeddings obtained from both the text encoder and image encoder would provide a clearer understanding of the effectiveness of the pretrained CLIP (LLM) model. Additionally, generating images based on textual descriptions of classes would offer valuable insights into how the proposed model operates within the zero-shot framework. 3- The distinction between the weakly-augmented view and the strongly-augmented view is unclear when it comes to unsupervised fine-tuning. What are the benefits of utilizing the weakly-augmented view for pseudo-labels instead of the strongly-augmented view?

Questions

Please answer all my concerns raised in the weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

In this paper, the authors acknowledge a limitation of their proposed model. They state that in their current work, they employed a straightforward neural network-based classifier and consider exploring more complex neural networks as part of future work. However, it is worth noting that from my observation of Table-1, the proposed model also faces challenges when applied to fine-grained datasets.

Summary

This paper proposed a new approach to improve zero-shot vision recognition capability which leveraged the common embedding space for image and text. Specifically, with a pre-trained VLM (vision-language model), the authors leveraged LLM to automatically generate multiple language prompts for training a text-based classifier, which can be adapted for vision classification thanks to the shared embedding space. The classifier is further applied to augmented images to provide pseudo labels for finetuning the visual encoder in a label-free manner (pseudo labels generated from the text classifier). The authors perform extensive experiments to validate the effectiveness of the proposed training paradigm and on multiple benchmarks the model achieved significant performance gain.

Strengths

1. The proposed method is intuitive and theoretically sound. The shared image-text embedding space naturally serves as a bridge for applying text-trained classifiers on image domains. The simple idea turns out to be very effective and brings significant performance boost across multiple datasets. 2. The conclusion is supported by extensive experiments with various ablation studies, showing many interesting patterns. 3. The paper is well-written and in good shape, it's easy to read.

Weaknesses

1. Section 3.2 is named as "unsupervised finetuning" but technically since the model is trained with smoothed cross entropy loss, it's not truly "unsupervised", just that the labels are from text-based classifier (thus "pseudo labels"). The name is somewhat misleading as audience may get confused with unsupervised approaches. 2. CLIP is pre-trained with massive data thus demonstrating good zero-shot capability. Although this paper experimented on multiple benchmark datasets, the images of these datasets are all natural thus similar in style. The proposed approach achieves significant performance gain without real labels but the assumption is that the pseudo labels are relatively accurate due to image consistency. Therefore it is better to emphasize the point at least as a potential limitations. It might be better to be more careful with some claims e.g. L317 regarding the cross-modality transfer capability.

Questions

1. Some other recent work used similar ideas for finetuning VLMs so maybe it's worth mentioning and provide a comparison. For example, [1] used a similar idea to generate multiple text and image prompts for finetuning VLMs domain adaptation. [1] https://arxiv.org/abs/2306.16658

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The paper has a dedicated section discussing about the limitations with the single linear layer of text classifier. Some more important limitations may need emphasizing as I suggested in the weaknesses section that the general assumption of the effectiveness of the proposed approach is the high quality of the underlying VLM model. No obvious potential negative societal impact.

Summary

This paper proposes a new method to improve the zero-shot classification accuracy for a pre-trained vision-language (VL) model. By leveraging the shared embedding space between the vision and language modalities, and relatively accessible text data generated by large language models (LLMs), the proposed method trains a classifier on the embedding of generated texts for the zero-shot classes, which can also work on the image features. The proposed method also includes a pseudo-label strategy using the trained classifier to augment unlabeled image data. The paper shows compelling performance improvement over the original VL model and other non-label fine-tuning methods.

Strengths

1. Overall I think it is an interesting idea. By using the shared embedding from the two modalities, it generates relatively "cheap" data by using an LLM to train a classifier directly for the target classes. This classifier further gives a better "starting point" for the pseudo-labeling in the next stage. 2. The evaluations on the benchmark datasets show considerable improvements over other label-free methods. It also outperforms 1-shot fine-tuning methods on certain datasets.

Weaknesses

1. What bothers me the most is that the performance of only using Llama descriptions is quite underwhelming (Table 3). It has a big gap to only using class names. Even the pure GPT description is only 1% better than class names on average. I think the failure of Llama needs more explanations. Also, these results might imply: 1) the generated texts are not that effective; 2) you need a really good LLM (GPT3 or better) to generate decent text descriptions to make this method work. IMO, both points will limit the contribution and impact of the proposed method. 2. I feel the diversity and quality of the generated texts will be very important in the proposed approach, as it will affect the generalization ability of the trained classifier. It is better to have some ablations on these aspects.

Questions

LLM can just generate random text descriptions of the class. For example, it might generate the history of an object which is not very useful in visual classification. Did you consider imposing any constraints to make the LLMs generate only visually meaningful texts?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

To me, the biggest limitations are 1. The method might rely on a super good LLM which might not be accessible in many real-world scenarios. 2. The generated text for visual classification might not be visually meaningful. No negative societal impact was observed.

Thanks for the response!

I've read the response from authors and are mostly satisfied with the answers. I don't have other questions at this moment and would like to keep my rating for acceptance.