Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP

Summary

Authors provide a series of detailed theories to support the conclusion that the inter-modal distance of the CLIP model is large and the gap can be reduced by introducing proxy features in visual feature space. They also propose the intra-modal proxy learning method to improve the performance of CLIP model with only unlabeled test data. Experiments on several datasets and different network architectures validate their proposed method's effectiveness.

Strengths

1. Motivation is clear and theoretical analysis is very sufficient. 2. The mehtod derived from theoretical analysis is convincing. 3. The paper is well-organized.

Weaknesses

1. Temperature in CLIP paper is a learnable parameter which is initialized as 0.07 and becomes 100 after training. Is Tau_T in submission the same parameter as the temperature t in CLIP? And why set to 0.01. 2. According to Table 2, it seems alpha contributions little to the proposed method. If we ignore the step, the acc will drop from 63.74 to 63.49, not too much for a single experiment. 3. Can authors provide the results on UCF101 and eurosat, which are common datasets used in CLIP tuning methods. 4. Can authors provide the results of default prompt instead of 7. 5. About TPT results, authors copy the results from their paper, resutls of CLIP baseline is obvious higher than their bsaeline, why? If bsaeline uses ensemble prompt, please refer to Weakness(4). Using high baseline results and copied TPT results seems unfair.

Questions

Please refer to Weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

No potential negative societal impact.

Summary

This work theoretically proves that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks resides only in the vision space. Two strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate intra-modal proxy learning (InMaP). Experiments on ImageNet and extensive downstream tasks confirm the effectiveness and efficiency of the proposal. However, I have some concerns. Please refer to the comments below.

Strengths

- This paper is easy to understand. - To mitigate the problem of the gap between the text and vision space, the authors theorize that the optimal proxy for visual tasks is solely from visual space. This is reasonable and convincing. - Extensive experiments are performed on different datasets.

Weaknesses

- For NeurIPS, the innovation or contribution of this work is limited, especially the technical solution. Proxy learning is not a new technique. Furthermore, pseudo label refinement is a combination of existing methods. - Related work introduces few-shot transfer rand zero-shot transfer. It seems that this paper does not report the performance of few-shot transfer (this paper uses CLIP as the Baseline, and CLIP performs a few-shot transfer experiment). - The layout of this paper is not very reasonable.

Questions

- There seem to be some syntax issues, please check carefully. For example, Line 44 and Line 48. - Please distinguish between ablation studies and hyperparameter analysis. Does this paper include hyperparameter analysis in "Alation on ImageNet"? This confuses me. - For some comparison methods, this paper shows by lighter color, why is this? It doesn't seem to be clearly stated.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

- This paper does not analyze the reasons for achieving zero-shot performance gains or provide deeper insights into the field. - See Weaknesses and Questions.

Summary

Based on the phenomenon of modality gap, this paper analyzes that the proxy of visual space is better than that of text space, and explains this theoretically. This paper designs a test-time adaptation method for unlabeled test data

Strengths

1. The influence of the modality gap on image classification is analyzed from the perspective of the proxy, and the theoretical results are amazing, which is of great significance for CLIP community research. 2. The results of the experiment are very significant, and I think this article touches on the essence of CLIP.

Weaknesses

Some analysis of validity for the textual model is lacking.

Questions

I once thought about the role of textual models in downstream image classification. CoOp and other works try to optimize the textual model and get success, but from the perspective of this paper, textual models seem to play no key role in downstream tasks just as an initialization. What is the author's opinion?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The author has not provided any limitations of this article, so I hope you can add some analysis.

Summary

The paper aims to improve the zero-shot performance of CLIP, i.e.: the setting in which given a test image, we compute its visual embedding, computes its similarity to a pre-computer set of text query embeddings where each text query corresponds to a different class and finally output the classification results as class corresponding the most similar text query (in CLIP's joint embedding space). The paper argues that zero-shot misclassifications are due to the visual-textual modality gap, i.e. the inherent gap in CLIP's embeddings space between embeddings of image and embeddings of their corresponding text queries. The paper argues (and theoretically proves) that this gap cannot be bridged by optimising on the text query and that the optimal proxy lies within the vision space. The paper proposes a novel algorithm that recovers vision proxies for text queries from unlabelled image test data. The paper demonstrates the effectiveness of the method on 12 downstream vision tasks.

Strengths

* The paper addresses a key limitation of CLIP (and other vision-language models trained using contrastive loss) - the modality gap. The modality gap is an inherent fundamental "built-in" limitation of CLIP which hinders its performance. Tackling such a key problem in a wide spread and popular model (or model family) is of great important). * The paper provides both an in-depth theoretical discussion as well as a thorough experimental evaluation demonstrating the effectiveness of the proposed method.

Weaknesses

Nothing in particular that I can point to.

Questions

Nothing in particular.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors did not addressed the limitations of their work.

Response to Authors After Checking the Rebuttal

Thanks to authors' rebuttal for further clarification. I still have other concerns: i) As CLIP used large number of data for learning, which inevitably results in data discloure. As such, taking CLIP for zero-shot visual categorization is not very acceptable. I want to know how did author gurantee the real zero-shot generalization in the experiments. ii) As we all known, there are lots of STANDARD zero-shot leanring (ZSL) methods [r1-r6, etc.] applied into image classification/obeject recognition. Different to CLIP-based zero-shot generalization mehtods, STANDARD ZSL methods perform real zero-shot generalization. As such, I strongly encourage authors take more discussions between the STANDARD ZSL and CLIP-based generalization methods. [r1] Zero-shot Learning with Semantic Output Codes. In NeurIPS, 2009. [r2] Label-Embedding for Image Classification. TPAMI, 2016. [r3] Zero-shot learning on semantic class prototype graph. TPAMI, 2017. [r4] Zero-Shot Learning: A Comprehensive Evaluation of the Good, the Bad and the Ugly. TPAMI, 2019. [r5] HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning. In NeurIPS, 2021. [r6] TransZero++: Cross Attribute-guided Transformer for Zero-Shot Learning. TPAMI, 2022.