LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning

Summary

The paper proposed locally regularized Context Optimization for OOD detection inspired by CoOP. Their primary claim is that the CLIP feature contains many ID-irrelevant nuances, such as backgrounds. Hence pushing the ID and these embedding from each other will lead to a better separation between ID and OOD samples. It is a few shot OOD detection methods that utilize learnable prompts. The author conducted experiments on zero/one/few shot methods, which shows their proposed method outperforms existing approaches.

Strengths

The strengths of the paper are : 1. The paper is well-written and easy to follow. 2. It is a simple approach combined with CoOP, MCM, and GL-MCM scores for test time prediction, giving 3. SOTA performance in zero-shot settings. 4. Pushing apart the ID-irrelevant features from CLIP, similar to MaskCLIP, would lead to a better OOD classification is a novel idea and has much potential. 5. The results are also good; it outperforms most existing models using both ResNet and ViT.

Weaknesses

There are a small few weaknesses of the paper which the author needs to address : 1. The prompt learning used here is the same as CoOP, then why is it novel? 2. Table 1. Zero-shot uses CLIP or CoOP, it is not clear. 3. The performance using CoOP with MCM/GL-MCM is already quite good, and LoCoOP improves an additional 1-2% gain in AUROC only in the one-shot or few-shot setting. Hence I doubt its efficacy in large-scale OOD detection or in near OOD detection.

Questions

Are there any results on near OOD or LargeScale datasets?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

No negative social impact and limitations are addressed adequately

Summary

This paper focused on the problem of vision-language prompt learning for few-shot OOD detection, i.e., using CLIP model to detect OOD images from unseen classes using only a few labeled in-distribution (ID) images. Previous zero-shot methods may encounter a domain gap with ID downstream data, while fully fine-tuned methods require enormous training costs. This work focused on the few-shot learning setting and extends CoOp, a few-shot vision-language prompt learning, to OOD detection. The main difference is to extract ID-irrelevant regions based on region-class similarity for OOD regularization. Experiments are conducted on iNaturalist, SUN, Places, and TEXTURE as OOD datasets and take ImageNet-1K as ID data. Experimental results verified the effectiveness of the proposed LoCoOp, even performing better than zero-shot and fully fine-tuned methods with 1-shot training data.

Strengths

+ The problem of vision-language prompt learning for few-shot OOD detection is novel and interesting. Previous works focused on zero-shot or fully fine-tuning settings, while few-shot OOD detection is not well explored. Also, it makes sense to adapt few-shot prompt tuning method CoOp for the few-shot OOD detection. + The proposed method is simple but effective and intuitive. It directly extract pseudo OOD samples from few-shot ID samples. The motivation and solution are clear and inspiring. + The experimental results verified the effectiveness of the proposed method. Especially, it can perform better than zero-shot or fine-tuned methods even with one-shot training data. Compared to complex method CoCoOp, it can also achieve faster inference speed. + The writing quality is satisfying. It is easy to follow the idea and implementation.

Weaknesses

- Although Table 3 compares the ID accuracies of CoOp and LoCoOp, it is interesting to know how other zero-shot and fine-tuned methods perform on ID accuracy, as CoOp may not achieve good ID performance due to the limitation of sample numbers. - As the motivation is to extract OOD regions from ID images, it would be interesting to visualize the OOD regions to illustrate how well CLIP can output OOD regions. ------------------ After rebuttal: Thank the authors for providing further comparisons and discussions. The comparisons and discussions are comprehensive. I will keep my positive rating.

Questions

Please see weaknesses for my quesions.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations

Summary

In this paper, the authors propose a CLIP-based few-shot OOD detector named Local regularized Context Optimization (LoCoOp). The method LoCoOp uses learnable prompts and local tokens' class scores to optimize the OOD detection performance. During the few-shot training phase, the authors extract the ID-irrelevant regions by top-k ranking w.r.t. the ground truth class and minimize their class entropy. In this way, the affinities between local tokens and ID classes are suppressed, thus enhancing the distinction between ID and OOD samples. The experimental results show the proposed method outperforms baseline methods.

Strengths

1. The paper is well-organized and clearly presented. The proposed method is easy to follow. 2. LoCoOp is a novel method that uses OOD region regularization to suppress the ID similarity of the background components. The idea of ID-irrelevant local context suppression is interesting. The proposed method raises a valuable point that suppressing extraneous background helps distinguish ID from OOD data.

Weaknesses

1. The method needs more analysis. - The authors may provide some visualizations for ID and OOD samples to show which regions are suppressed. This kind of case study helps demonstrate that the proposed method works as the authors envision. - The foreground object F and its background context B may tend to co-occurrence, so there may be a spurious correlation issue [1]. Hence, the GT label may also rank higher in the background context. For example, for the background, like the beach, categories associated with the beach (e.g., beach chairs, surfboards) may rank higher than categories that have no relations. Based on this, for some categories with spurious correlation, will this training strategy fail? Because the ground-truth label may occur in top-k candidates for almost all background regions. - The OOD regularization loss (Eqn. 5) needs to be further analyzed. The authors may investigate the training dynamics of the regularization loss. How does this loss affect prompt learning convergence? Will it cause instability and affect ID accuracy in a longer non-few-shot training schedule? How does it affect hard samples? - The proposed method may need some theoretical analysis and support. 2. More experiments and validation are necessary. - I suggest that the authors extensively verify the effectiveness of the proposed method on various OOD benchmarks. In particular, it is necessary to pay attention to whether the proposed OOD regularization loss will behave as expected when the network is trained with fewer categories (such as CIFAR10)? - The authors only train the network with up to 16 samples per class. I'm curious if one can achieve better results by using more samples to finetune the network? Where is the bottleneck of OOD detection based on prompt learning? Can we unfreeze more layers to obtain stronger OOD regularization capabilities? - In Table 1, why does one-shot prompt learning of CoOp+MCM result in lower performances compared to zero-shot MCM? [1] Ming, Yifei, Hang Yin, and Yixuan Li. "On the impact of spurious correlation for out-of-distribution detection." *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 36. No. 9. 2022.

Questions

Some minor issues. - The concept *OOD image features* in Line 146 is prone to ambiguity. Readers may think that it is a feature taken from the OOD image. - There may be a typo in Figure 2 x-axis label *# of labels*. Perhaps *# of samples* is the correct one?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

The task of this article is to use the CLIP model for image classification, and the target scene is few-shot data and out-of-distribution (OoD). The author confirms the final OoD region by querying the ID-independent region in the image, and further optimizes the model through this region. Remarkably, the authors validated their method on the large-scale dataset Image-Net 1K. As far as I know the task is a new one, and its application makes sense.

Strengths

- This paper proposed a new and meaningful task. - The performance of the method is fine. - The author completed method verification on a very challenging dataset (ImageNet-1K full dataset). - It's an interesting idea to remove regions not related to IDs. - The author provides the source code to ensure the reproducibility of the article.

Weaknesses

The idea of this article is very interesting and theoretically feasible. However, I have the following concerns: - My biggest concern is the accuracy of locating regions not related to IDs. Because the author did not provide an experimental indicator of the accuracy of the ID-independent region positioning, it is only shown from the results of training and detecting OoD. My concerns come from another article I've read that points to the limitations of CLIP's current interpretation of location regions [1]. It is a pity not to see related visualization results in the article and supplementary material. If the author can provide some indicators to prove that his method can well locate regions that have nothing to do with ID, I think it will be more convincing. - In addition to OoD detection, the model should have Zero-Shot capability, should the author also compare the zero-shot performance with CoOp or CoCoOp methods? [1] Li, Yi, et al. "CLIP surgery for better explainability with enhancement in open-vocabulary tasks." *arXiv preprint arXiv:2304.05653* (2023).

Questions

My main questions are in the weaknesses section, and I would like to improve my score if the author can address my concerns convincingly. In addition, I still want to know how long (how many hours or how many days) does this method take to train the ImageNet dataset on an A100GPU?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have listed the limitation in the article.

Summary

This paper proposes a vision-language prompt learning method named local regularized context optimization (LoCoOp) for few-shot out-of-distribution detection. The proposed LoCoOp method performs OOD regularization that uses the portions of CLIP local features as OOD features during training. The experimental results show the effectiveness of the proposed LoCoOp method over zero-shot and fully supervised methods.

Strengths

- The addressed problem of few-shot out-of-distribution detection is interesting and practical. - The proposed CLIP based few-shot OOD detection method with local regularized context optimization is well motivated and achieves good results.

Weaknesses

- The proposed local regularized context optimization method is not specific to the few-shot setting, and can actually also be applied to fully supervised OOD detection setting, where the ID-irrelevant nuisances could also be learned. Then will the proposed method also improve the results in fully supervised setting? And what are the results? - Similar to the CLIP based segmentation task in [55], the ID-irrelevant regions are predicted based on CLIP prediction results. However, the segmentation results may not be good, will that be a key obstacle to the proposed method? - The training process of the proposed method is a little bit confusing. Does the L_coop loss apply to the whole image or only the ID-relevant regions? - In Table 1, it shows that the performance of one-shot LoCoOp method is even worse than the zero-shot CL-MCM method (and only comparable for 16-shot LoCoOp) on iNaturalist dataset. What is the reason?

Questions

- Will the proposed method also improve the results in fully supervised setting? And what are the results? - Will the region segmentation results be a key obstacle to the proposed method? - What is the reason that the performance of one-shot LoCoOp method is even worse than the zero-shot CL-MCM method (and only comparable for 16-shot LoCoOp) on iNaturalist dataset?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations.

Reply to the authors' rebuttal

Thanks to the author for answering my doubts in detail. However, since the author only provided very few visualizations, I still have concerns about the accuracy of the regional positioning, so I decided to maintain the current score.

ID and OOD accuracies in the same table

Thank the authors for the feedback. For the ID accuracy, could the authors show a new table combining both ID and OOD accuracies for a more clear comparison? It would be more straightforward to see how LoCoOp decreases ID accuracy and increases OOD accuracy, and whether it achieves a great trade-off or simply increases one by decreasing the other to the same extend.

Post rebuttal

The authors addressed most my concerns. So I keep my original rating and lean towards acceptance.