UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models

Summary

This paper investigates the task of data pre-selection by learning a better representation from the joint feature space of both vision and text in an unsupervised manner. The paper focuses on training text prompts to extract joint features with enhanced representation, specifically with the BLIP-2 parameters kept fixed. The aim is to achieve a diverse cluster structure that encompasses the entire dataset.

Strengths

**[New task]** This paper tackles data pre-selection for labelling without accessing the information of downstream tasks, which is quite new to the community. **[Well-illustrated figures]** The figures shown in this paper are clear enough for better understanding. **[Good presentation]** The paper is well-written and easy to follow.

Weaknesses

**[Unconvincing illustration]** In Figure 1, BLIP-2 is pre-trained with prompt and image together. This explains why using only image features yields poor performance. Consequently, the evidence presented does not convincingly demonstrate the superiority of multimodal features. **[Need in-depth analysis]** (i) It is unclear why the self-trained model can be used for sample selection. (ii) The motivation of medoid selection is not given. It would be nice to see the rationale. **[Missed ablation studies]** The ablation study for two hyperparameters are not given. **[Disorganized reference format]** Please reformat the references as per some published papers.

Questions

Please refer to the weaknesses part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please refer to the weaknesses part.

Summary

The paper addresses data pre-selection (akin to active learning) problem using the highly successful vision-language models (VLMs) of late. In relation to existing approaches, the proposed approach has a few advantages, e.g., no need to have a small initial set of labeled data, no need to have multiple rounds of selection, labeling and retraining, once selected the data can be used for multiple future, unknown downstream tasks etc. The authors start with a BLIP-2 model and an unlabeled set of data $D$. The BLIP-2 model is extended to have learnable context/prompt and a few MLPs, namely instance-level and cluster-level heads. The instance-level head is employed to produce a contrastive training between two views of each unlabeled instance. Cluster-level head, on the other hand, first assigns cluster memberships to the instances and then helps to train the model with a cluster-level contrastive loss. After training for a few epochs using these two losses, cluster-level MLP along with the learned contexts/prompts are used to get the cluster assignment of unlabeled data (which can come from different downstream dataset). Finally, the medoids from each cluster form the representative selections for active learning. Experiments performed on linear probing and domain generalization show the efficacy of the proposed approach over the state-of-the-arts on benchmark datasets.

Strengths

1. The use of VLMs for unsupervised active learning is appreciable. VLMs. Now-a-days, are known for good zero-shot transfer. The already well-learned representations can and did help the active learning cause. 2. The use of learnable contexts/prompts has been shown to be useful for few-shot transfer. The use of these for active learning is interesting. 3. The use of cluster-level contrastive loss cleverly avoids the use of any initial labeled set that is required in traditional cluster level losses in getting the initial clustering (akin to group contrastive loss in semisupervised literature e.g., [a]). 4. Experimental analysis and ablations show the efficacy of the proposed approach compared to sota approaches and the importance of different components of the approach as well. [a] Singh et al., Semi-supervised action recognition with temporal contrastive learning, CVPR 2021.

Weaknesses

1. One important ablation that could be useful is running the approach without the learnable prompts/contexts. What I mean is updating only $g_I$ and $g_C$ but not employing $V$ in Algorithm 1. This will help gauge the importance of the contexts/prompts vis-à-vis the instance and cluster level MLPs. Does ‘Initi_Prompt’ row in Table 2 do this? 2. Line 260+: This is more of a clarification query. When the datasets are described, I don’t see any mention of which dataset is used for pre selection. I am assuming these 7 datasets are downstream task datasets. The question is coming from Table 1. While in Table 2, it seems that the first column tells what is the dataset on which the prompts are learnt, in table 1, it is not clear. Is it that the learning is done on the same datasets on which the linear probing performances are shown for Table 1? 3. Line274+: I am not getting what is meant by 'with the learned prompts' in the baseline using USL. Does it mean everything else here is same as BLIP-2, but in addition a few prompts are learned also?

Questions

Questions are already asked above (in the ‘Weaknesses’ section). Those are mostly queries for further clarification. Here in addition, let me list a few presentation related issues (typos mainly). - Line 32, 78, Figure 1 caption: Will these be ‘data efficient learning’ instead of ‘data efficiency learning’? - Line 231: ‘map’ -> ‘maps’ - Line 244: ‘optimizing’ -> ‘optimization’ - Line 246: ‘combine’ -> ‘combination’ - In Figure 2a, which feature extractor is used? Is it anything different from BLIP-2?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are described well in the paper. At the same time, the authors tried to address the limitation in the supplementary material.

Summary

This paper presents an unsupervised approach for data preselection, which aims to select instances for labeling from an unlabeled dataset in a single pass. The authors leverage the text features in multimodal models, specifically BLIP2, to enhance the representation for data preselection. They argue that a well-designed joint feature space of vision and text can yield improved results. To achieve this, they train text prompts to extract joint features with enhanced representation, ensuring a diverse cluster structure that covers the entire dataset. The authors employ two loss functions, namely instance-level contrastive and cluster-level contrastive, to train the learnable text prompts. These loss functions encourage the adapted multimodal model's joint representation space to be more diverse and well-separated, suitable for clustering. Experimental results on seven different datasets, along with a comparison against three baselines, demonstrate the effectiveness of the proposed approach.

Strengths

I believe the strength of the paper is as follows: - The integration of vision and language models for data pre-selection holds promise due to the added benefits of leveraging text modality. - This paper introduces a novel approach that promotes diverse and clustered representations, addressing limitations in the current state-of-the-art BLIP2 model through the lens of prompt learning. - The proposed method is lightweight and more efficient in terms of training costs. The paper makes a significant contribution, evident in its clear presentation and compelling results.

Weaknesses

While the paper's results are strong, there are two notable aspects that could be addressed: 1. Missing CLIP baseline: Comparing the proposed approach with a CLIP baseline would provide a fundamental point of reference. Since the authors employ contrastive loss functions and prompt learning, which can be applied to any vision and language model, including CLIP as a baseline would enhance the comparative analysis. 2. Lack of integrability: A limitation of the prompt tuning method is the lack of interpretability. While prompt tuning improves model performance, it does not provide a clear explanation of why and how the model works in the combined language model embedding space. Addressing this limitation would enhance the understanding and justification of the proposed approach. 3. Performance/training-time trade-off: The paper utilizes the best-performing model of BLIPV2, which has over 7 billion parameters. However, it does not explore the performance and training-time trade-off with different BLIPV2 models. Investigating the performance of the proposed approach on BLIPV2 models with fewer parameters would provide insights into its scalability and suitability for models of varying sizes.

Questions

Following questions address the weaknesses and limitations mentioned earlier. Asking the authors these questions could help improve the paper and provide a more comprehensive understanding of the proposed approach: 1. Could you compare your proposed method when built on top of CLIP as well? Including a comparison with CLIP as a baseline would provide valuable insights into the effectiveness of your approach and its advantages over a widely used vision and language model. 2. Could you provide some visualizations, such as attention weight visualizations, to explain why the trained prompts lead to better data preselection and generalization? Visualizations would enhance the interpretability of your method and shed light on the mechanisms behind its improved performance. 3. Have you replicated your proposed method on other variants of the BLIPv2 model? It would be insightful to see how your approach's performance changes when applied to BLIPv2 models with varying levels of expressiveness, particularly when the model size decreases. This analysis would help assess the scalability and adaptability of your method to different model configurations.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

There are not any limitations.

Summary

The paper studies the problem of data pre-selection, which aims to select instances for labeling from an unlabeled dataset to enhance performance for downstream tasks with a limited annotation budget. The authors suggest that combining visual and textual features in a joint space can result in a better representation for data pre-selection. They introduce UP-DP, an unsupervised prompt learning approach that adapts vision-language models(specifically BLIP-2) for data pre-selection. The proposed approach outperforms the state-of-the-art on benchmark datasets and exhibits generalizability.

Strengths

1. The paper introduces a novel approach for data pre-selection that incorporates unsupervised prompt learning in the vision-language model. This approach effectively exploits the multimodal features and enhances the discrimination among classes. 2. The authors provide a clear motivation for the task of data pre-selection and highlight the unique challenges it poses compared to semi-supervised learning and active learning. 3. The paper compares with the state-of-the-art on multiple benchmark datasets, demonstrating its effectiveness and superior performance. Also, it highlights the generalizability of the learned prompts across different datasets.

Weaknesses

1. The authors claim that the purpose of the data pre-selection is to optimize performance for undefined diverse downstream tasks; however, they only conduct experiments on the image classification task. This limited scope is insufficient to demonstrate the effectiveness of UP-DP for various downstream tasks such as detection or segmentation. The paper would benefit from conducting more experiments on other tasks. And on top of this, it would be interesting to investigate the generalizability of the learned prompts across different tasks. 2. In Table 1, the "Zero-Shot BLIP-2" setting is unreasonable. It lacks a justifiable rationale to use the prompt for the CLIP model to evaluate the zero-shot performance of the BLIP-2 model. If the authors intend to use this baseline, they should train the learnable prompt for BLIP-2 from scratch. 3. In Table 1, the author does not include a baseline that utilizes only image features extracted from BLIP-2. By comparing with this baseline, the authors can demonstrate the efficiency of the proposed method. 4. In Table 1, the authors compare "Random", "USL-I/M", but it is suggested to compare with more approaches in the field. Including additional comparisons would strengthen the paper's evaluation and provide a better context for understanding the performance of the proposed approach. 5. The ablation studies of the proposed method are limited. It would be beneficial to conduct more comprehensive experiments to analyze the performance of UP-DP under different settings. For instance, the authors should provide a detailed analysis of the impact of the instance-level and cluster-level contrastive loss. 6. All the experiments are carried out using the BLIP-2 model during the data pre-selection stage. However, it remains unclear whether the proposed method is exclusively effective for this particular model. It is important to consider other visual-language models, such as CLIP, to establish the efficacy of the approach. 7. The paper contains several grammar issues. For example, on page 5, line 191, it states "presents an efficient pre-training," and on page 6, line 237, it states "Thus we can from a positive pair." These sentences require revision for improved clarity and grammatical correctness.

Questions

1. Is it fair to use the prompt for CLIP model to assess the zero-shot performance of the BLIP-2 model? It's supposed to train the prompt for BLIP-2 from scratch. 2. Please demonstrate that UP-DP is valid and effective for other tasks like detection and segmentation. 3. Table 1 requires revision and additional information. It should incorporate a baseline that utilizes only image features extracted from BLIP-2. Also, it is beneficial to compare with more approaches in the field. 4. Please include more ablation studies. At least, demonstrate the impact of the instance-level and cluster-level contrastive loss. 5. During the data pre-selection stage, all the features are extracted from BLIP-2 model. However, it remains unclear whether the method is exclusively effective for this specific model. It is recommended that the authors demonstrate the effectiveness of using other vision and language models as well.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

1 poor

Presentation

3 good

Contribution

2 fair

Limitations

The authors need to analysis the limitations of their work.

Summary

This paper presents a novel method to perform data preselection for the task of image classification. Data preselection refers to the task of finding the images for annotating labels and then used for training. The paper builds around the powerful visual-language model BLIP, and proposes learnable prompts as inputs to BLIP to help perform unsupervised clustering for data preselection. The paper then presents results on seven image classification benchmarks, showing the superb performance of the proposed method.

Strengths

+ The presented method is a novel application of visual language model to data preselection. + I find the proposed learnable prompting in conjunction with unsupervised clustering novel, and as suggested by experiment effective. + The presented method is effective in data preselection, as demonstrated by the comparison against baseline methods.

Weaknesses

- The decision to annotate 200 images per benchmark (LINE 265 - 273) seems arbitrary. Why this number? It would be great if the number can be varied and then plot the model performance accordingly to understand the effect of annotated data set size on model performance. - USL-M, which shares the same multimodal features from BLIP-2 as the proposed UP-DP method, isn't really outperforming the baseline USL-L on EuroSAT (Table 1). Such a result contradicts the claimed effectiveness. What is the explanation? - No results on linear probe on BLIP-2 using Random Sampling. This is needed in order to showcase the effectiveness of the proposed method.

Questions

See above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Post rebuttal comments

Thanks, authors, for the detailed response. I had a careful read of the responses to my concerns as well as to my fellow reviewers’. My clarification queries as well as the request for the ablation were addressed very good. The responses to my fellow reviewers’ queries are also apt e.g., the new experiments on additional tasks, additional backbones or linear probing results etc. I was already positive about the work and am seeing no particular reason to change.

Post rebuttal comments

Thanks, authors for their detailed clarifications and for providing the experiment I requested. I have also reviewed the comments made by other reviewers and the authors' responses to them. The authors' rebuttal effectively addressed my concerns, so I will not be lowering my score.