On the Powerfulness of Textual Outlier Exposure for Visual OoD Detection

Summary

The paper notices that while outlier exposure has shown promising potential in improving OoD detection performance, all previous studies on outlier exposure have been limited to utilizing visual outliers. The paper uncovers the benefits of using textual outliers by replacing real or virtual outliers in the image domain with textual equivalents. Then, it proposes various ways of generating preferable textual outliers. The extensive experiments demonstrate that generated textual outliers achieve competitive performance on large-scale OoD and hard OoD benchmarks.

Strengths

1. The paper is written well and is easy to understand. 2. The studied problem is very important. section 3 is quite interesting. 3. The results seem to outperform state-of-the-art.

Weaknesses

1. I am curious about section 4.1.1 and section 4.1.3, why do the authors choose to use the images from the validation set of the ID dataset for generation? 2. It is not clear that if the class labels of the outlier dataset is missing, how can we generation description-level supervision in section 4.1.2? 3. I am curious about the performance of the method without the large-scale pretrained models (CLIP) as the classification backbone. The current approach seems to be highly model-specific, which hinders its general usage. 4. The comparison with a reasonable baseline, i.e., NPOS [1] is missing. [1] Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023.

Questions

see above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

yes

Summary

The paper proposes a new method that takes the textual outliers to help the model better detect out-of-distribution samples. Specifically, they build the pipeline on top of CLIP models with a classifier and use different representations of the text sample to synthesize outliers in the CLIP space, then use them to train the model with ood awareness. The authors provide extensive experiments and analysis of the method.

Strengths

I think the paper is well-written, and the paper is in a great flow. Figures and texts are clear and easy to follow. The authors also give a good rationale and background of the problem. The authors approach the OOD problem with the textual outliers and the help of CLIP fundamental models, which is a novel and plausible method. The authors provide extensive experiments and detailed analysis of the problem and extensively worked on different types of textual outliers.

Weaknesses

The authors build the classifiers on top of CLIP, which is a large model with 151 Million parameters (~88 Million in the Image encoder) and implies a large hidden feature space. Comparing this with previous methods may lead to unfair comparisons. The use of CLIP may limit this to other fields like object detection/segmentations.

Questions

How can you prove the ood robustness is come from your method but not from CLIP itself? In the appendix, only the larger CLIP model is studied. However, in the open-sourced CLIP API, they provided smaller models. I wonder how much impact it would have on the metrics as the CLIP shrinks.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The method seems only experimented with classifiers. The appliance of the CLIP Image encoder makes it hard to extend to other problems like Object Detection, semantic segmentations, .etc.

Summary

This paper studies visual OOD detection by introducing the textual outlier under the outlier exposure paradigm. Different from previous research focused on utilizing visual outliers, this work explores the benefits of textual outliers in the image domain. Specifically, they propose different ways to generate textual outliers based on the powerful GPT model. Comprehensive experiments have been conducted to demonstrate the effectiveness of textual outliers in OOD detection.

Strengths

1. From the perspective of originality, to the best knowledge, this paper is the first work to explore the potential of textual exposure with multimodal neural networks for outlier exposure. 2. The overall presentation of this work is good, having intuitive illustrations and clear organization. The framework and the proposed textual outliers generation are easy to understand. 3. Instead of straightforwardly utilizing the LLM under the multimodal paradigm for visual OOD detection, this paper investigates the practical adaptation of different textual outlier types in the outlier exposure framework. 4. Experiments from different perspectives have been conducted to demonstrate the effectiveness of description-level textual outliers.

Weaknesses

1. Regarding the technical level, exploring the textual information (like description level or caption level) under the multimodal setting for the image domain has been studied in [1]. The general paradigm, e.g., using LLM to generate the corresponding description or caption and then utilizing them in the image task, is similar. 2. Compared with the conventional outlier exposure, the optimizing objective and test time OOD detection have limited advanced design beyond the previous method. 3. Compared with the generation part, the filtering part seems to be more important as it has a more close relationship with the data quality for outlier exposure. However, the current filtering process is a little bit heuristic. 4. Considering the performance, the improvement compared with other advanced methods of using pure image outliers is not very significant (like Table 5 compared with DOE). Could the authors provide more explanation or discussion? I generally appreciate the idea of introducing and excavating the potential of textual outliers. I hope the previous comments or the later questions can help the current contribution to be clearer and enhanced. [1] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023.

Questions

Regarding the current version of the draft, I have the following question: 1. Could the authors discuss more the unique contribution of OOD detection beyond the general paradigm of utilizing textual information? 2. Could the authors provide more examples or system comparisons of the three different types of textual outlier generation? 3. Could the authors discuss more why the performance of textual OE is not consistently performing better than visual OE?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This paper has discussed the limitation, e.g., the proposed textual outlier method involves a heuristic selection process to refine the outputs of the generative model and also provided the potential solution in future work.

Summary

This paper addresses the challenge of detecting Out-of-Distribution (OOD) data by introducing "textual outlier exposure" as an alternative to visual outliers. Instead of relying on visual examples, the authors explore the benefits of using textual equivalents in OOD detection. They propose various methods for generating textual outliers, which are validated through extensive experiments on large-scale OOD benchmarks. The findings demonstrate that the generated textual outliers outperform visual outliers and establish criteria for effective textual outliers, including their proximity to the data distribution, descriptiveness, and incorporation of visual semantics. The contributions of this work include investigating the potential of textual outlier exposure with multi-modal neural networks, utilizing large-language models for generating textual outliers at different levels of detail, and validating their effectiveness in various OOD detection scenarios. The paper presents a novel and promising approach to OOD detection, showcasing the advantages of textual outliers over visual counterparts and providing valuable insights for designing impactful textual outliers.

Strengths

- The paper presents an innovative and compelling idea, offering valuable insights for open-world learning, Out-of-Distribution (OOD) detection, and online learning. The findings of this study have significant implications for various research studies in these domains. - The clarity and coherence of the paper are commendable, making it easy to comprehend and follow the authors' methodological approach. - The inclusion of illustrative examples effectively enhances the understanding of key concepts. - The figures and plots presented in the paper are visually clear, aiding in the visualization and interpretation of the experimental results. - The paper offers comprehensive experimental studies, demonstrating the efficacy of the proposed method. The promising performance observed in these experiments further strengthens the validity and potential impact of the proposed approach.

Weaknesses

While the paper presents detailed experimental studies, there are two notable weaknesses that should be addressed: - The focus of the experimental studies is primarily limited to outlier exposure methods, neglecting the exploration of other types of Out-of-Distribution (OOD) detection methods. It would be valuable to consider and compare the proposed approach against alternative techniques in order to provide a more comprehensive evaluation. - The omission of an ablation study to assess the impact of network architecture on the proposed methodology is a notable oversight. Investigating the influence of different network architectures on the performance of the proposed approach would enhance the understanding of its strengths and limitations.

Questions

The experimental evaluation in your paper is commendable. However, I have two inquiries regarding the experiments: - It appears that the recent state-of-the-art methods, such as ASH [1] and GradNorm [2], have not been included in your experimental comparisons. Could you provide insights into the rationale behind this omission? It would be valuable to report and compare the performance of your proposed method with these state-of-the-art approaches to provide a comprehensive assessment. - Additionally, considering the impact of different network architectures on your method is crucial. Could you elaborate on how varying network architectures affect the performance and effectiveness of your proposed approach? Investigating this aspect would provide deeper insights into the robustness and generalizability of your method. [1] Djurisic, Andrija, et al. "Extremely simple activation shaping for out-of-distribution detection." arXiv preprint arXiv:2209.09858 (2022). [2] Huang, Rui, Andrew Geng, and Yixuan Li. "On the importance of gradients for detecting distributional shifts in the wild." Advances in Neural Information Processing Systems 34 (2021): 677-689.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No negative social impact.

Thanks for the response!

Thanks for the detailed response! Most of my concerns are well addressed by the further clarification and I appreciate the idea of exploring textual outliers in OOD detection.