Summary
The paper presents S-CLIP, a novel semi-supervised learning method for training contrastive language-image pre-training (CLIP) models in specialized domains where limited image-text pairs are available. Vision-language models like CLIP have achieved impressive results in natural image domains, but they struggle when applied to specialized domains like remote sensing due to the scarcity of paired data. To address this, S-CLIP leverages additional unpaired images and introduces two pseudo-labeling strategies: caption-level and keyword-level pseudo-labeling.
Caption-level pseudo-labeling assigns pseudo-labels to unlabeled images based on the captions of paired images. This is achieved by solving an optimal transport problem between the unlabeled and labeled images, ensuring that the pseudo-labels represent a probability distribution over the labeled images. On the other hand, keyword-level pseudo-labeling assigns pseudo-labels based on the keywords in the captions of visually similar paired images. It formulates the training as a partial label learning problem, considering a candidate set of target keywords instead of a single exact one.
The experiments conducted in various specialist domains, including remote sensing, fashion, scientific figures, and comics, demonstrate the effectiveness of S-CLIP. It outperforms CLIP fine-tuning and other semi-supervised learning competitors in zero-shot classification and image-text retrieval tasks. For example, in the remote sensing domain, S-CLIP improves zero-shot accuracy by 10.4% and image-to-text retrieval R@5 by 4.4% compared to CLIP fine-tuning. Moreover, S-CLIP remains robust even when the unlabeled images are from a different dataset. Overall, S-CLIP shows promise in enhancing CLIP training using only a few image-text pairs.
Strengths
- Effective Pseudo-Labeling Strategies: The paper proposes two pseudo-labeling strategies, caption-level and keyword-level pseudo-labeling, which effectively guide the training process. The caption-level pseudo-labeling leverages optimal transport to assign pseudo-labels based on the captions of paired images, ensuring robust training even with distribution shifts. The keyword-level pseudo-labeling assigns pseudo-labels based on keywords in visually similar paired images, capturing local components of unlabeled images. The combination of these strategies results in improved zero-shot classification and image-text retrieval tasks.
- Experimental Demonstrations: The paper provides extensive experimental evaluations in various specialist domains, including remote sensing, fashion, scientific figures, and comics. The results consistently show that S-CLIP outperforms CLIP fine-tuning and other semi-supervised learning competitors, achieving significant improvements in zero-shot classification and image-text retrieval tasks. The robustness of S-CLIP, even when unlabeled images are from a different dataset, further strengthens its effectiveness.
- Relevance and Applicability: The paper addresses an important and practical problem in the field of vision-language models. The proposed S-CLIP method has the potential to enhance the training of CLIP models in specialized domains with limited paired data. The findings are relevant not only for researchers working on vision-language models but also for practitioners in various domains where image-text understanding is crucial.
- Clarity and Presentation: The paper is well-written, organized, and presents the proposed method and experimental results clearly. The figures and illustrations aid in understanding the concepts and methodologies. The strengths and weaknesses of the proposed method are discussed in a concise and informative manner.
Weaknesses
- Baselines with other semi supervised approaches are severely limited with the added baselines being severely outdated. Several much newer few shot learning approaches are present and should be compared against. A summary could be found in the paper: A Survey on Deep Semi-supervised Learning by Yang et al. for example: SEMI-MAE: MASKED AUTOENCODERS FOR SEMI-SUPERVISED
VISION TRANSFORMERS by Yu et al or Semi-supervised Multimodal Representation Learning through a Global Workspace by Devillers et al
- A lot of the core content like limitations, experimental setup etc has been moved to the appendix which I feel is key to the understanding of the paper and unfairly grants the author extra writing space
- The approach combines keyword and caption level pseudo labelling, both of which have individually been well studied in literature. The combination of both is certainly a novel effort, but the contribution of the same doesnt seem like a strong enough novelty for a venue liek Neurips
- The choice of using OT as the distance measure between 2 images is interesting, it would be nice to see how other image distance measures compare to it
Questions
- Could you clarify why the soft-PL, hard-PL were the only semi supervised baselines chosen for comparison despite the presence of a vast body of literature for semi supervised learning?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
No ethical limitations as such