Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a "second generation" of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at https://github.com/BatsResearch/menghini-neurips23-code.
Paper
Similar papers
Peer review
Summary
This paper is about the exploration of using pseudolabels for VLMs on different downstream tasks. Specifically, the authors experimented CoOp, VPT and UPT with pseudolabels generated from CLIP on semi-supervised learning, transductive zero-shot learning and unsupervised learning. They proposed three training strategies, namely FPL, IFPL and GRIP, where the difference is mainly at whether the pseudolabels are static or dynamic. Experiments on 6 datasets show significant improvement.
Strengths
- Writing is good. The work is presented in a logical way and is easy to follow. - Performance improvement is impressing. - Experiments are thorough.
Weaknesses
- Novelty is limited. The use of CLIP's pseudolabels are not new and the proposed training strategies are also widely used in self-training methods. - I agree with the authors' statement that "If CLIP performs poorly on a task, we may struggle to obtain a reliable set of pseudolabels to begin with, potentially diminishing CLIP’s performance." Therefore, it is unintuitive why GRIP achieves such a big improvement even on datasets like MNIST where CLIP fails. One explanation I can think of is that the quality of the pseudolabels are drastically increasing as the training proceeds. However, results from Fig 3 seems to show the opposite. I am mainly concerned about this and believe further analysis is needed to explain what is the exact reason for the success of the proposed method. - I'm also concerned with the computational cost since a total of 10 iterations are used. Authors should include this when comparing with other methods.
Questions
- The authors stated that the proposed IFPL and GRIP are "novel and proposed here for the first time for prompt tuning". However, aren't these two methods similar in spirit to self-training? What are the differences? - According to Fig 3, the accuracy of pseudolabels are only marginally increasing for IFPL, and decreasing for GRIP as the training proceeds. Is it necessary then to train over 10 iterations? What are the detailed improvements for each iteration? - Why is it that CLIP's pseudolabel accuracy decreases in Fig 3? Shouldn't zero-shot CLIP remain an approximately constant performance irrespective of the number of iteration and the number of unlabeled data? - When using a larger image encoder, why does the performance decrease? If a larger backbone is used, wouldn't the quality of pseudolabels be higher? This further deepens my concerns on the proposed method.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Yes, the authors have addressed the limitations.
Summary
The authors provide compelling evidence that underlines the power of a repetitive prompt-training approach, which leverages CLIP-based pseudo labels. Regardless of the learning model (SSL, TZSL, UL) or the type of prompt (text, visual), this strategy significantly enhances the image classification capabilities of CLIP across multiple settings. Moreover, by using the Top-K pseudo-labeling approach, they ensure a balanced distribution of pseudo-labeled training samples for each class, thereby effectively addressing the inherent bias of CLIP pseudo labels.
Strengths
- Various learning paradigms, including semi-supervised, transductive zero-shot, and unsupervised learning, can all be viewed as unique instances of a single objective function when pseudolabels are used as a form of guidance. - As evidenced by Table 1and Table 2, the experimental results demonstrate a significant boost in performance, indicating that the proposed pseudo-labeling strategy holds considerable merit.
Weaknesses
The overall structure or layout of the manuscript needs to be further refined. - Table 5 ("There is a trade-off between quality and quantity of pseudolabels") is mentioned in L290, but Table 5 does not exist in the main manuscript. - Table 2 was not mentioned at all in the body text. - Since Fig 3 and Fig 5 are too small to be seen properly, efforts are needed to improve readability. - The formulas of L157-L158 should be specified as an equation in the form of (1), but it was not. Methodologically, the proposed approach is too naive. - Setting the trade-off parameters in unified objective function (L157-L158), e.g., gamma, and lambda, is too heuristic, and there is lack of the analysis. - Selecting the Top-K most confident samples by class is too naive. It would be nice to select a confident sample by improving class diversity through the application of techniques such as information maximization loss [1]. And it would be good to present comparison results with other attempts for class diversity. [1] Liang, Jian, Dapeng Hu, and Jiashi Feng. "Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation." ICML 2020
Questions
- In Table 1 and Table 2, there's a noticeable drop in some experimental results under the UL setting. I'm curious as to the cause of this decrease. - Within Table 2, the outcomes for the TRZSL learning setting in RESICS45 and DTD, under the "Multimodal prompts" category, show a decline. I'm interested in understanding why this is so. - In the experimental results presented in Table 3, where "GRIP benefits adaptation even for larger image encoders," there seems to be a reduced rate of improvement when using a larger image encoder. I wonder the reason or analysis behind this observation.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
I already pointed out in the weakness part.
Summary
In this paper authors extend empirical study of self-training for the case when pseudo-labels are generated by models (CLIP) in zero-shot regime (models are trained on unlabeled data with respect to a downstream task, but can be used for zero-shot prediction for the downstream task). Authors investigate self-training in case of visual-language models (CLIP) with tuning visual/text/visual-text prompts with pseudo-labels generated in the zero-shot regime by CLIP model. They formulate also different training regimes (unsupervised, semi-supervised, transductive zero-shot) in terms of simpler supervised training where part of data (or all) just has pseudo-labels. Also it is investigated how we could bootstrap further performance in self-training doing rounds of training while updating the pseudo-labels. In the end authors show that self-training is effective and improve across different settings and datasets. Interesting results are obtained in terms of debiasing original CLIP model when self-training is done afterwards.
Strengths
- extensive empirical study of self-training for the CLIP-based models and various configurations and settings - demonstrating that CLIP-style models can be used as seed model for self-training in variety of settings bringing significant boost for prompt finetuning - empirical demonstration how CLIP de-biasing can be done via self-training
Weaknesses
- absence of any ablations on balancing between labeled / unlabeled data - absence of ablation on the number of samples selected for every class in self-training (why K=16? is it important? why balanced across classes?), also it is not clear what is the effect of balanced selection here especially on the de-biasing the CLIP (if de-biasing happens only because of this balancing then it is very obvious/expected result in my opinion) - GRIP exact explanation / how it works should be extended in the text (many ambiguities in the current description) - Table 1 and other Tables: I don't get why for CLIP TRZSL is similar to SSL and UL while for self-training in any combination TRZSL becomes way better than SSL and UL? - Missed discussion on the results that text prompt alone gives superior results compared to text+visual prompt or visual-alone prompt **Summary** There is no novelty or any impactful design of the self-training algorithm itself, though extensive empirical study is done for zero-shot learners (CLIP) used to pseudo-label data. The particular choice of hyper-parameters (like balancing pseudo-labels per class) demonstrate significant improvement for the prompt fine-tuning task as well as de-biasing original model (latter I found particularly important and interesting).
Questions
- lines 50-53: I would not agree that VLM are trained on unlabeled data. We do mining of audio-visual pairs, so in that sense it is really unlabeled. On the other hand, yes, downstream task is different and we can use zero-shot inference to generate pseudo-label. However one could consider that task of pairing text and audio is more general than pairing class name and image. I would be here more concrete in phrasing. - There are a lot of repetitions of the same sentences / phrases along the text, e.g. lines 60-69 (other - lines 134-144) appear couple of times. I would reorganize the text to refer to the previous parts rather than repeat almost same ways in introduction / results / sections. - typo line 96: "such as such as" - lines 179 and 187 - selection of $gamma$ and $\alpha$ seems to be equivalent providing 1:1 balancing between labeled / unlabeled data. I don't get why then expression are made different if in the end mathematically optimization is the same. - IFPL: what happens if for some class we don't have K samples? say no any examples are predicted as class 0 for MNIST, what happens then? - How about IFPL variant but when we increase K over time, still taking top only pseudo-labels? This is I assume is different from the GRIP. - It is not clear if in GRIP classes for pseudo-labels are still balanced, and also how do we select which part of unlabeled set to take? Do we take every iteration top pseudo-labels increasing the number of them or we just increase randomly the data size and take whatever pseudo-labels will be there? Do we still re-initialize prompt every iteration here? Do we reuse or regenerate pseudo-labels for the part of data used in previous iteration here? - Why we don't train continuously prompt when we re-generate pseudo-labels? Why do we not fine-tune the whole model also but only prompting (if for efficiency purpose - ok, but interesting to see if we need to finetune the whole model)? - lines 308-311 - this is actually known fact. E.g. for fixmatch/remixmatch we do different augmentations, and they are the key to make self-training to work, so particular type of noise in data/labels make self-training to work. - Figure 3 (and similar in Appendix) is not clear. I got what is x-axis only after reading it several times. Either better notation or better caption is needed. Maybe "refers to the top x-axis (number of iterations) while .. to the bottom a-axis (amount of unlabeled data)". - I still don't get why GRIP over iterations can become worse than IFPL. Also maybe it is worth to have ablation where we do IFPL but for every iteration K is increased so that we increase amount of unlabeled data involved though we do this over the most confident every time still. - Robin Hood effect: is it because we do balancing of classes in self-training? I guess here this plays a huge role, assuming that some classes are unrepresented in CLIP pretraining + we know that in self-training different classes have different pace of learning, so that balancing can resolve issues on under representative classes or hard ones. - line 345 - what is "conventional pseudo-labeling"? - Table 5: typo - UPL -> UPT - lines 664-665 - is it because of the CLIP bias itself and the way classes are balanced in fine-tuning? - lines 692-693 - maybe not surprising as we balanced examples per class?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Limitations are discussed after conclusion sections. Formulation sounds reasonable to me. **Update: after rebuttal and discussion with authors the score is update from 6-weak accept to 7-accept and contribution from 2 to 3.**
Summary
The paper explores the use of CLIP for pseudo-labeling for various tasks such as SSL and on various datasets. Overall - while not being very surprising - the results can clearly outperform prior work (that is not using CLIP it has to be said) - and thus showing the potential of CLIP for such tasks.
Strengths
Clearly, the authors show what they set out to do: CLIP pseudo-labeling can improve via pseudo-labeling on a variety of tasks and datasets. That is indeed nice to see. The positive experiments are good - doing this for a variety of tasks and on a variety of datasets.
Weaknesses
While the improvements are good - they are not surprising either. CLIP is trained on a wide variety of data and thus it is clear that pseudo-labeling using CLIP should help in many cases In my view the main weakness of the paper is that it is not showing the limits of CLIP pseudo-labeling - while most of us do not know which exact data is used to train CLIP - the datasets used seem still reasonably close.
Questions
Overall the paper has a clear point and it does a good job making that point reasonably clear. As said above, after reading the paper I am not surprised that this can work, but I am left without an answer what the limits are of CLIP pseudo-labeling,
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
limitation discussion ok for me - except the point of the limits of CLIP pseudolabeliing as mentioned above
Summary
This paper explores the concept of prompt tuning in the context of limited labeled data. The authors propose a unified objective function that encompasses three different learning paradigms, and investigate three distinct training strategies for leveraging pseudolabels. The experimental results on six datasets demonstrate the effectiveness of the proposed GRIP strategy. The findings of this study highlight the value of utilizing pseudolabels generated by CLIP itself to enhance its performance in diverse learning scenarios.
Strengths
* this paper is well-written with a clear structure, making it easy to understand * this paper extensively explores a board design space, including the prompt modality, the learning paradigm and the training strategy * the authors conduct extensive expriments to verify the effectiveness of this method
Weaknesses
* The writing of the article is not clear in some details, such as the typo error in line 290 and inconsistencies in abbreviations. * please see questions listed below
Questions
* I am little confused that does `line 217` means reinitializing the prompts at the begining of each iteration? Or does it mean only reinitializing the set of pseudolabels while the prompts are kept? * Is the process of pseudolabeling done online or offline? If it is done offline, is it time-consuming when there is a large amount of unlabeled data? * Are the abbreviations TZSL mentioned in `line 182` and TRZSL mentioned later referring to the same concept? This might be a bit confusing. * The experimental results in Table 1 show significant improvements compared to CLIP or other baselines. However, there is a slight decrease in some cases. Perhaps the authors could try to explain the reasons behind this? For example, it could be related to the dataset or other factors. * I think there might be a typo error in `line 290`. It should be Table 2 instead of Table 5. * Have you validate the effectiveness on datasets with more categories (e.g, ImageNet-1K), because the quality of pseudo labels might decrease when the category space get larger.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
I think the authors have adequately discussed the limitations of their research.
Reviewer's response to rebuttal
Dear authors, Thanks a lot for detailed clarifications. I have carefully read all reviews and your responses. I strongly suggest you to incorporate all main points of the discussion into the final revision (e.g. the choice of balancing and top-k of 16). I still have some questions and comments to clarify: > Absence of any ablations on balancing between labeled / unlabeled data Ok, I buy your arguments here. But I think you should include this into text to provide for the future as reference of observations. > Absence of ablation on the number of samples selected for every class in self-training (why K=16? is it important? why balanced across classes?) As you have reference to prior work - nice to see it still works confirming robustness of prior work choice. Add this to the text as it reasons on the choice you made. > Table 1 and other Tables: I don't get why for CLIP TRZSL is similar to SSL and UL while for self-training in any combination TRZSL becomes way better than SSL and UL? I think this clarification should be included in the final revision to make readability clear. > (2) select for each class the K datapoints with the highest similarity score to the class. Could then we end up having some samples presented in data with two different assigned pseudo-labeled classes? > Based on our understanding, it appears that the variant of IFPL proposed by the reviewer is the same as GRIP. In IFPL, top-K pseudolabels are recomputed at each iteration while keeping K fixed. In contrast, in GRIP, K increases with each iteration and we assign pseudolabels still following the top-K schema. > GRIP maintains class balance by selecting the top-K samples at each iteration, with K increasing progressively. Similar to IFPL, both prompts and pseudolabels are reinitialized with every iteration, in order to avoid accumulating errors from earlier iterations. In other words, learning progresses from pseudolabels to new prompts to new pseudolabels, and so on. From description in line 223 I do not agree with your statement. "we use i/I -th of the unlabeled data" - there is nothing about top-K where K is growing. You need to be more precise in the description of GRIP in the text for the revision. But thanks for the explanation, now it makes sense to me :) (And general response answers entirely my concern here). Any comment on > Robin Hood effect: is it because we do balancing of classes in self-training? I guess here this plays a huge role, assuming that some classes are unrepresented in CLIP pretraining + we know that in self-training different classes have different pace of learning, so that balancing can resolve issues on under representative classes or hard ones. ? > As reported in Table 2, GRIP either outperforms or performs comparatively with IFPL. The suggestion of the reviewer is correct and it corresponds to GRIP. Here I look e.g. at Fig 6 in Appendix where accuracy drops over iterations for GRIP but not for IFPL. I found this opposite to the expected behaviour where GRIP should improve things over iterations as we become more confident (maybe this shows overfiting). Do you report last iteration performance for GRIP or the best in Table 2? Thanks, Reviewer kGE6.
Thank you again for all the suggestions. They are all valuable and clarifications will certainly be included in the final version of the paper. Below, we answer further questions and post the answers we wrote and left out because of space constraints. > (2) select for each class the K datapoints with the highest similarity score to the class. [..] Could then we end up having some samples presented in data with two different assigned pseudo-labeled classes? Yes, this can happen. However, we checked the pseudolabels at each iteration and it was rarely the case. This is a characteristic of the pseudolabeling strategy proposed in [15]. We believe this can be an object of study for future work motivated by the effectiveness of self-training. > Robin Hood effect: is it because we do balancing of classes in self-training? I guess here this plays a huge role, assuming that some classes are unrepresented in CLIP pretraining + we know that in self-training different classes have different pace of learning, so that balancing can resolve issues on under representative classes or hard ones. In our paper, we conducted a comprehensive investigation into the causes of the Robin Hood effect, which results in a more balanced distribution of class accuracies (Section 4.2). Our study revealed that the combination of the top-K pseudolabeling strategy and prompt tuning plays a crucial role in achieving this effect. We hypothesize that the parameter-efficient nature of prompt tuning also helps avoid overfitting to the easier classes. Surprisingly, this straightforward and intuitive approach, aimed at balancing classes, has not been explored in previous works that address imbalanced accuracy distributions, as evidenced in references [49,8]. While these studies emphasized the importance of examining class accuracies when employing pseudolabels in semi-supervised learning, we extend this analysis to the application of prompt tuning and CLIP-pseudolabeling. By doing so, we present a novel perspective on this phenomenon and its practical implications. > Here I look e.g. at Fig 6 in Appendix where accuracy drops over iterations for GRIP but not for IFPL. I found this opposite to the expected behaviour where GRIP should improve things over iterations as we become more confident (maybe this shows overfiting). Do you report last iteration performance for GRIP or the best in Table 2? We report the performance of the last iteration. The behavior of GRIP’s pseudolabels accuracy surprised us too. That is why we emphasize it by discussing the trade-off between the quantity and quality of pseudolabels (Sect 4.1, page 7). Although the accuracy of GRIP decreases we can still observe that it is higher than the accuracy of CLIP on the same amount of data. Moreover, the accuracy of pseudolabels at the last iteration is close to the overall accuracy of GRIP in Table 1 and 2. We speculate that one of the causes of the deterioration of psuedolabels accuracy could be overfitting, but we do not have evidence to entirely support this claim and leave the question open for further explorations. > lines 308-311 - this is actually known fact. E.g. for fixmatch/remixmatch we do different augmentations, and they are the key to make self-training to work, so particular type of noise in data/labels make self-training to work. In lines 308-311: we say “This suggests that numerous, slightly noisier pseudolabels can yield better results, highlighting a trade-off and offering insights for future approaches.” We clarify that the word choice of “noisier” is misleading and led the reviewer to think we are referring to augmentations. Instead, with “noisier” we refer to incorrectly assigned pseudolabels. Specifically, in that sentence we explain the trade-off between quality and quantity of pseudolabels. To avoid confusion we will replace the word “noisier” with “incorrect”. > lines 692-693 - maybe not surprising as we balanced examples per class? The class balance is imposed on both methods we compare. However, the class accuracy distribution of CLIP via linear probing shows a 30 times larger reduction in the accuracy of rich classes. Lines 692-693 comment on the class accuracy distribution obtained running GRIP with linear probing. While the class balance plays a central role to determine the Robin Hood effect, in this context we were commenting that with linear probing the reduction of accuracy of rich classes is on average 30 times larger than the reduction observed using prompts. > lines 664-665 - is it because of the CLIP bias itself and the way classes are balanced in fine-tuning? In the lines highlighted, we observe that the accuracy of GRIP on seen classes is worse than CoOp. During training, for large lambda, the loss component of unlabeled data (unseen classes) is the first to decrease, while the loss on the seen classes reduces later. Thus, we hyphotize that extra training steps might be needed to complete the learning on the labeled data.
Final comments and raising score
Dear authors, Sorry for the delay in response. Thanks for providing more answers to my questions. Just one last comment from my side: > We report the performance of the last iteration. The behavior of GRIP’s pseudolabels accuracy surprised us too. That is why we emphasize it by discussing the trade-off between the quantity and quality of pseudolabels (Sect 4.1, page 7). Although the accuracy of GRIP decreases we can still observe that it is higher than the accuracy of CLIP on the same amount of data. Moreover, the accuracy of pseudolabels at the last iteration is close to the overall accuracy of GRIP in Table 1 and 2. We speculate that one of the causes of the deterioration of psuedolabels accuracy could be overfitting, but we do not have evidence to entirely support this claim and leave the question open for further explorations. Yep, I could look into overfitting as first thing by checking how many times the same sample participated in training, as then it could be that we just overfit to the top-1 sample during the whole training if they appear in every GRIP iteration. Summary: after discussion most of my concerns were resolved, and the ones which are still there (e.g. some extra ablations) I don't think are so important regarding paper acceptance. I recommend the paper for acceptance, agreeing with the position of Reviewer 1u3c, that it is a necessary milestone in the research community for under-explored CLIP pseudo-labeling which can stimulate other works and deeper understanding of pseudo-labeling algorithm aspects. About the computational budget: this argument as the weakness is not strong for me as there are a bunch of methods developed in many areas, like vision, MT and speech to have computationally efficient pseudo-labeling (when we train one model with regenerating pseudo-labels during training) -- so here I would not be surprised that they are applicable with small modifications and could resolve the computational budget issue. I would like to raise my score from 6 to 7 to support the paper acceptance, though discussion should be incorporated in the final revision. Thanks again for productive discussion.
Thank you so much for all your valuable input and for supporting our paper! Thank you as well for the productive discussion. We will be sure to incorporate the great feedback into the final version.
After reading all reviews and the authors' rebuttal I personally still lean towards acceptance of the paper. While I agree that one could argue that methodologically there is not much novelty, I still consider it worth reporting given the reported improvements and the fact tat CLIP pseudolabeling is still rather under-explored. I also fully agree with on of the other reviewers that the authors should included the discussions and clarifications mentioned in the reviews and rebuttal to strengthen the paper overall.
We are glad the reviewer appreciated our work and emphasized the paper's contribution as extending beyond methodological novelty. Thank you again for your review. We will make sure to add valuable discussions and clarifications in the final version of the manuscript.
Thanks for the author's response, and I have already read all of the comments. My main concerns have already been solved, but I still believe that a scalable approach should have lower additional computational costs and be able to scale to datasets with a larger number of categories.
Thank you for your response! We are glad that your main concerns have been solved. In light of that, we hope that you will consider increasing your support for our paper. If the scalability remains a concern, Reviewer kGE6 addressed this in their last comment: > About the computational budget: this argument as the weakness is not strong for me as there are a bunch of methods developed in many areas, like vision, MT and speech to have computationally efficient pseudo-labeling (when we train one model with regenerating pseudo-labels during training) -- so here I would not be surprised that they are applicable with small modifications and could resolve the computational budget issue. We agree that speeding up pseudo-labeling should not be a major challenge. CLIP even offers new tricks. For example, if only one prompt modality (text or image) is used, the embeddings of the other encoder can be cached. This will easily lead to an almost 2x speed up over a naive approach. Thank you so much for your valuable feedback.
Response to rebuttal
Dear authors, Thank you for the feedback! Most of my concerns were addressed (e.g., performance on MNIST and with a stronger backbone). I've also carefully read other reviewers' comments and the authors' responses. However, since pseudolabeling of CLIP is already widely explored in the unsupervised setting, I still feel that extending such methodology to SSL and TZSL is a marginal step. The experiments are good, but I would prefer seeing something new in the design. Therefore I would remain my score.
Thank you for your response, we are glad your main concerns have been solved. For the sake of the final discussion among reviewers, we provide some clarifications about the goals of our work. > “since pseudolabeling of CLIP is already widely explored in the unsupervised setting” In our design space, for unsupervised learning we define 6 paths to explore, among which only 1 was previously investigated in the literature [15]. If there are other papers that have considered these choices in the context of vision-language models like CLIP, we would be grateful for pointers. > “I still feel that extending such methodology to SSL and TZSL is a marginal step” We respectfully disagree with the reviewer. We do not claim to propose or extend methodologies to work in the SSL and TRZSL settings, rather we extensively explore how pseudolabeling adapts to these settings in the context of prompt tuning with CLIP. The exploration of our underexplored design space is based on the observations that (1) VLMs' zero-shot capabilities extend the usability of pseudolabels to any limited-label data scenario, and (2) paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function, by using zero-shot pseudolabels as a source of supervision. We consider these two observations, and the experimental findings that generalize across these settings, as important takeaways from this paper. > “The experiments are good, but I would prefer seeing something new in the design” The goal of this paper is to study how to use pseudolabels with CLIP in a variety of unexplored learning settings. We do not propose new methodologies to use pseudolabels with CLIP. We explored 27 combinations of prompt modalities, learning settings, and training strategies, among which only 1 was already explored. We consider filling this gap in literature a significant contribution that shows useful takeaways to the community. In this context, our experiments not only demonstrate the effectiveness of using pseudolabels iteratively for prompt tuning CLIP in limited labeled scenarios, but also show that prompt tuning with pseudolabels can mitigate the biases of the original model. Reviewers generally appreciated our contributions: * Reviewer sr7c says: “ The authors provide compelling evidence that underlines the power of a repetitive prompt-training approach, which leverages CLIP-based pseudo labels. Regardless of the learning model (SSL, TZSL, UL) or the type of prompt (text, visual), this strategy significantly enhances the image classification capabilities of CLIP across multiple settings. [..] effectively addressing the inherent bias of CLIP pseudo labels.” * Reviewer kGE6 says: “ agreeing with the position of Reviewer 1u3c, that it is a necessary milestone in the research community for under-explored CLIP pseudo-labeling which can stimulate other works and deeper understanding of pseudo-labeling algorithm aspects.” * Reviewer qyTk: “ Performance improvement is impressing. Experiments are thorough”
Response to Rebuttal
Thanks for addressing my feedback; I'm inclined to increase my score from 4 to 6 in favor of the paper's acceptance.
Thank you for your feedback and the support! We will include all the necessary clarification in the final version of the paper!
Decision
Accept (poster)