Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting

Summary

This paper proposes a method named Frolic to improve the zero-shot performance of vision-language models like CLIP. The method focuses on two key challenges: enhancing prompt distribution learning and correcting inherent label bias in pre-trained models without relying on labeled data. Experimental results across 16 datasets demonstrate performance improvements of the method.

Strengths

+ The paper is well-written and the method is easy to follow. + The method effectively addresses label bias inherent in pre-trained models, improving the robustness and accuracy of zero-shot predictions. + Ablation experiments are comprehensive.

Weaknesses

- Does the paper utilize the validation dataset in experiments, if not, this should be clarified. - How does the proposed method compare with the methods[1][2] in the few-shot settings? - Although Frolic is described as training-free, there may still be hyperparameters involved in the method, and how to choose these parameters? - How does the prompt distribution learning contribute to the performance? Does it outperform the original CLIP? - Can the method be performed with a few labeled samples? [a] Black box few-shot adaptation for vision-language models. ICCV 2023 [b] Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification. ECCV, 2022

Questions

The discussion of the limitations of the methods is unclear. The authors should discuss it in depth.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Please refer to the weakness and Questions parts.

Summary

This work aims to enhance the zero-shot performance of pre-trained vision-language models. Specifically, three strategies, including label-free prompt distribution learning, adaptive calibration and correction pre-training label bias, are proposed and work together to improve the performance on different downstream tasks. Experiments on diverse tasks confirm the effectiveness of the proposed method.

Strengths

1. Enhancing the zero-shot performance has attracted much attention recently. This work discusses multiple challenges in this process and proposes corresponding algorithms with theoretical analysis. 2. The proposed method is training-free without hyper-parameters, which is applicable to real applications. 3. Diverse downstream tasks are included for evaluating the performance of the proposed method. 4. The manuscript is well-written and easy to follow.

Weaknesses

1. The work works with a uniform prior that classes are balanced distributed. It is better to discuss the scenario with imbalanced prior. 2. Besides CLIP, there are many other vision-language pre-trained models, e.g., BLIP, etc. It is better to include other pre-trained models to evaluate the performance of the proposed method. 3. Why does $e_i$ follow a Gaussian distribution in L446?

Questions

I do not have critical questions about this work.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

In this paper, the authors introduce a promising method for enhancing zero-shot vision models through the utilization of prompt distribution learning and bias correction. The method is particularly notable for being training-free and label-free, which greatly simplifies the implementation process. The authors provide comprehensive experimental results, which effectively demonstrate the efficacy of the proposed method.

Strengths

1. This paper is easy to understand. 2. The paper provides an in-depth theoretical discussion and a thorough experimental evaluation demonstrating the effectiveness of the proposed method.

Weaknesses

1. The computation of second-order moments and the covariance matrix from the marginal distribution as discussed in Eq.(3) and (5) might be computationally intensive. 2. The method relies on the pseudo-labels estimated by CLIP. How does this process influence the final results? 3. The setting of results on the ImageNet variants dataset is unclear. 4. The results lack comparison with other prompt distribution methods, such as CoOp and CoCoOp.

Questions

The author stated that their method does not require hyperparameter tuning. How does the proposed method compare to those requiring hyperparameter tuning?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

Refer to the weakness and Questions

Summary

This paper presents Frolic, a label-free prompt learning methods aiming to improve zero-shot visual recognition of vision-language models like CLIP. The method is built upon estimating distributions over prompt prototypes to capture diverse visual representations and further bias correction. Experiment results demonstrate consistent improvement on various benchmarks over prior-art methods.

Strengths

1. The approach does not require access to large-scale datasets for estimation, which is required by many prior work. 2. The approach advances previous distribution estimation methods by removing the need of label information, which made this line of work possible for zero-shot recognition tasks.

Weaknesses

While this approach does not need access to large-scale pretraining data, it seems to make some weak assumptions on downstream tasks. If my understanding is correct, it assumes: 1. the downstream task is balanced (line 134), which is not always hold true in reality. The long-tail nature of real world does not guarantee that the testing class distribution will be balanced even though the benchmarks do. 2. A decent scale of the testing data on downstream tasks is available (e.g, testing data does not come in online fashion like one sample at a time) for estimating the beta term in bias correction. Would the approach for bias estimation still holds true in online testing scenario? I would assume this is more aligned with reality too.

Questions

1. The methods did not seem to evaluate on all benchmark datasets like previous works do. For example, CIFAR10/100 and RESISC. While some of these datasets have saturated performance, it might be still helpful to include results there. 2. It's not clear to me why [24] is not listed in tables for comparison as I would assume 24 to be the most direct baseline. It shares the same motivation of this paper with the requirement of accessing pretraining data. Arguably, while LAION has been taken back, [24] already does the job for us. I would not think an argument like "[24] requires large-scale pretraining data for estimating bias thus we do not compare with it" as a valid argument because this zero-shot visual recognition task itself is solely working with CLIP-ish models and estimating bias from the datasets like LAION sounds like a fair practice to me.

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Authors have included such discussion

Thanks for the rebuttal

Thanks for the authors for providing the rebuttal and additional results. Most of my concerns are resolved while a few remains: 1. In [A1], you mentioned "if the downstream datasets are imbalanced, we can derive the distribution vector ". It's not clear to me how can this be achieved when we are working with zero-shot recognition. Under zero-shot scenario, you wouldn't have class distribution information of downstream tasks beforehand and would have to either make further assumptions or use some sort of estimation. Can you elaborate more on this? 2. I see most results are provided for ViT-B models, what about stronger ones like ViT-L and ViT-L@336?

Response to Reviewer VFZ9

**We are grateful for your feedback and we agree with your suggestions to enhance the quality of our work.** **A1：** We make an assumption that the mean of the class distribution can be represented by the text features $\mathbf{z}_j$ via prompting. Given unlabeled samples, we can compute their sample mean $\mathbf{m}$ and denote the unknown label priors as $\pi_j$. The sample mean can be represented as a linear combination of $\mathbf{z}_j$: $\mathbf{m} = \int_{\bf x}\sum_{j=1}^K\pi_j\mathcal{N}({\bf x};{\bf z}\_j,\Sigma){\bf x}{\text{d}}{\bf x}=\sum_{j=1}^K \pi_j \mathbf{z}_j$. Let $Z = [\mathbf{z}_1, \ldots, \mathbf{z}_K]$, then the unknown label priors can be solved by $\mathbf{\pi} = Z^{-1}\mathbf{m}$. *** **A2：** We have conducted experiments using the stronger models mentioned, and the results are presented below. We find that: - Our Frolic outperforms the original CLIP across various datasets with both ViT-L/14 and ViT-L/14@336, - Our online version of Frolic maintains comparable performance to the original Frolic. - Our Frolic can outperform REAL with ViT-L/14 backbone. | ViT-L/14 | CIFAR10 | CIFAR100 | RESISC | | :---: | :---: | :---: | :---: | | CLIP | 95.8 | 78.6 | 65.7 | | Frolic | 96.5 | 79.8 | 66.8 | | ViT-L/14@336 | CIFAR10 | CIFAR100 | RESISC | | :---: | :---: | :---: | :---: | | CLIP | 91.3 | 79.2 | 66.7 | | Frolic | 92.6 | 81.1 | 68.1 | | ViT-L/14 | Caltech | Pets | Cars | Flowers | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 | Average | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Frolic | 97.3 | 95.4 | 83.5 | 81.8 | 92.4 | 42.1 | 77.3 | 66.9 | 71.0 | 82.2 | 79.0 | | Frolic（Online） | 96.7 | 95.6 | 83.3 | 82.1 | 91.7 | 41.9 | 77.7 | 66.7 | 70.4 | 81.5 | 78.8 | | ViT-L/14@336 | Caltech | Pets | Cars | Flowers | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 | Average | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | --- | | Frolic | 97.7 | 96.2 | 85.1 | 83.2 | 92.8 | 44.3 | 78.5 | 67.6 | 72.9 | 84.3 | 80.3 | | Frolic（Online） | 96.9 | 96.1 | 85.3 | 82.9 | 92.3 | 44.5 | 78.1 | 67.8 | 72.3 | 83.1 | 80.0 | | ViT-L/14 | ImageNet | Flowers | Cars | Aircraft | Pets | Food | DTD | Average | | :---:| :---: | :---: | :---: | :---:| --- | :---: | :---: | :---: | | REAL[24] | 73.7 | 82.4 | 89.6 | 28.2 | 92.8 | 89.4 | 65.7 | 74.5 | | Frolic | 74.2 | 82.9 | 90.5 | 29.1 | 93.3 | 90.1 | 66.6 | 75.2 |

Response to Reviewer jbMQ

We do appreciate the reviewer's positive support, and we are pleased to take the reviewer's advice to improve our work.