Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models

Summary

This paper proposes a novel method utilizing pre-trained vision-language models (VLMs) to enhance out-of-distribution (OOD) detection. Their core idea lies in constructing a Conjugated Semantic Pool (CSP) to enrich the pool from which OOD labels are drawn. Unlike simply expanding lexicons with synonyms and uncommon words, the CSP leverages modified superclass names that serve as cluster centers for samples with similar properties across categories. This, the authors theorize, not only increases pool size but also amplifies the activation probability of OOD labels while maintaining low label dependence. Extensive experiments demonstrate significant performance improvements over existing approaches.

Strengths

Originality: The paper introduces a unique concept, conjugated semantic labels, for enhancing the semantic pool, offering a valuable contribution to OOD detection. Theoretical Underpinnings: The authors provide a robust theoretical framework, including mathematical models and justifications demonstrating how the CSP improves OOD detection. Empirical Validation: Extensive experiments showcase the effectiveness of the proposed method, achieving significant advancements over current state-of-the-art techniques. Generalizability: The method's applicability across diverse VLM architectures suggests broad utility and adaptability. Clear Problem Definition: The paper clearly identifies limitations in existing methods, particularly regarding simple lexicon expansion, and proposes a well-reasoned solution.

Weaknesses

1. Ablation Study & OOD Score Function: The ablation study in Table 11 showcases that competitive performance on iNaturalist and SUN datasets is achievable without the CSP. However, it remains unclear how the proposed OOD score function (designed by the authors) contributes to performance when CSP is not employed. Further clarification is needed on: (1)Performance Source: If strong results are achievable without CSP on these datasets, what factors besides the CSP are driving the overall performance improvement observed in the main experiments? (2) OOD Score Function Usage: How can the authors' OOD score function be effectively utilized in scenarios where CSP is not implemented? 2. Synonym Handling and CSP Overlap: The paper acknowledges the limitations of including synonyms in larger lexicons. However, it lacks a deeper exploration of how the CSP specifically addresses this issue. Additionally, the potential for semantic overlap within the CSP itself needs to be addressed. It would be beneficial to see a discussion on: (1) CSP vs. Synonyms: How does the design of the CSP inherently mitigate the problems introduced by synonyms in traditional lexicon expansion? (2) Mitigating Overlap: How do the authors handle potential semantic overlap between elements within the CSP? Are there strategies to ensure distinctiveness between labels? 3. Systematic Analysis on Place and Texture Datasets: A more systematic analysis of the CSP's impact on the Place and Texture datasets, particularly concerning the design principles of semantic pooling, would be valuable. The authors' own experiments suggest that designing effective semantic pooling requires considering multiple factors. Therefore, a more in-depth exploration of: (1) Place and Texture Performance: How does the CSP specifically improve performance on Place and Texture datasets? Can these improvements be linked to specific design choices within the CSP? (2) Semantic Pooling Design Factors: Building on the authors' findings, what factors are crucial for designing effective semantic pooling mechanisms, especially in the context of Place and Texture datasets? Overall, this paper proposes a promising approach to improve OOD detection by innovatively using conjugate semantic labels to expand the semantic pool, which is supported by theoretical analysis and empirical results. However, I still have some questions about the method that the authors need to address.

Questions

See Weakness

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

See Weakness

Summary

This paper presents a novel zero-shot out-of-distribution (OOD) detection pipeline that enhances performance by expanding the semantic pool with a conjugated semantic pool (CSP), which consists of modified superclass names that cluster similar properties from different categories. The approach aims to increase the activation probability of selected OOD labels by OOD samples while ensuring low mutual dependence among these labels. By moving beyond traditional lexicon-based expansion and using the CSP, the paper's method outperforms existing works by 7.89% in FPR95, demonstrating a significant improvement in OOD detection without the need for additional training data.

Strengths

1. The paper is well-written and easy to understand. 2. The supplement to the shortcomings of the NegLabel theory is very enlightening. 3. Extensive experiments demonstrate the superiority of the method.

Weaknesses

1. The primary contribution of this work is its enhancement of the semantic pool expansion issue within NegLabel, representing an incremental step within the NegLabel framework. 2. There is ambiguity regarding the distinction between citations and contributions. For instance, the content related to Lemma 1 in both the preliminary section and the appendix A.1, A.2 seems to originate directly from the original NegLabel paper, yet this source is not explicitly referenced in the main text and appendix. 3. In Section 3.3, the authors think that OOD images may not include their true label "white peacock" among the negative labels is a issue. However, this argument appears debatable. As evidenced in NegLabel's visualizations and Figures 6 and 7 of this paper, OOD images may not consistently exhibit the highest similarity to their ground truth (GT) label and could show significant similarity with multiple negative labels. Thus, even if the true GT label of an OOD image is absent from the negative labels, the NegLabel method can still effectively detect OOD data. 4. From an implementation standpoint, how does including terms like "smartphone" and "cellphone" as negative labels impact the overall OOD score? The claim in Section 3.2 about the impact of synonym disruption on Lemma 1's independence seems somewhat overstated. 5. I am curious whether, aside from the semantic pool, all other aspects of the technical implementation directly derive from NegLabel. Specifically, does the NegMining Algorithm, the OOD score calculation, and the Grouping Strategy align exactly with those in NegLabel? This aspect requires further clarification. 6. Additionally, is Figure 3 a representation of actual distribution distance data or a schematic created by the authors? Utilizing real experimental data to support the findings presented in Figure 3 would strengthen its validity. If it is a schematic, it may introduce subjective biases from the authors. 7. The paper extensively discusses the limitations of previous methods and the intended outcomes in Section 3, yet provides relatively minimal detail on the actual implementation of the proposed method. It would be beneficial for the authors to expand on the specifics of their proposed approach in Section 3.3. For instance, further elaboration on how superclass selection is conducted and integrated with adjectives would be valuable. 8. According to the description in the paper, the method used to select cluster centers appears crucial. Have the authors considered employing clustering techniques for superclass selection rather than relying solely on manual selection? 9. Another point of confusion arises regarding how variations in corpus size impact the acquisition of semantic pools. Does this necessitate selecting additional superclasses?

Questions

see Cons

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper explores how to set potential OOD labels to facilitate OOD detection with vision-language models. The paper first conducts a theoretical analysis, revealing that in addition to increasing the negative label space, it is also important to increase the probability of negative labels being activated and to reduce mutual dependence between negative labels. Therefore, the paper proposes a new strategy for constructing negative labels by introducing modified superclass names to construct a conjugated semantic pool (CSP). Experiments are conducted on standard benchmarks.

Strengths

1. The paper theoretically analyzes the criteria for selecting negative labels and proposes a new method for designing negative labels. 2. The motivation is clear, and the method design is straightforward and effective. 3. The proposed method achieves state-of-the-art results on standard benchmarks.

Weaknesses

While the overall method design is agreeable, there are some details that need clarification: 1. In Line 276, the authors claim that CSPs overlapping with ID classes will not be selected as negative labels. How is this implemented, and does it significantly impact the results? 2. In Line 321, the authors introduce the superclasses used and mention combining these superclasses with adjectives to form the CSP. Could you provide more details on the adjectives used and the selection process? Is the pairing with superclasses done randomly? 3. How many negative labels are ultimately used? Beyond the 10,000 classes in the negative label pool, how many new negative labels are introduced? Does the number of newly introduced negative labels significantly affect OOD detection performance? 4. In my experiments, on ImageNet, performance initially improves and then declines as the number of negative labels increases; however, in experiments with CIFAR as the ID dataset, performance continuously improves with more negative labels. Have the authors observed similar phenomena, and can they provide an explanation? 5. It is suggested to follow Neglabel and refer to the introduced classes as "negative labels" instead of "OOD labels" to avoid confusion with the labels of practical OOD datasets.

Questions

See weaknesses

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

NA

Summary

This paper presents improvements to zero-shot OOD detection methods based on pre-trained vision-language models. The study first models factors that influence the performance of existing pipelines and theoretically derive two necessary conditions for enhancing performance: expanding the OOD label candidate pool and maintaining low interdependence. Subsequently, the author analyzes why simple expansion methods do not meet these conditions and proposes constructing a conjugated semantic pool for expansion. This method meets the theoretical conditions and achieves SOTA performance on several public benchmarks.

Strengths

1. The article presents reliable mathematical modeling and theoretical derivations, and the proposed method is simple and efficient. 2. The proposed method achieves considerable performance improvements over the latest methods. 3. The article features a clear structure and logical coherence, demonstrating a high standard of presentation and writing quality.

Weaknesses

1. The description in the caption of Table 1 is unclear. In the experimental section, the author mentioned that the upper part of Table 1 represents traditional OOD detection methods, while the lower part pertains to detection methods based on pre-trained models like CLIP. However, this distinction is not explicitly marked in the caption, which might lead readers to misunderstand that all methods are based on the CLIP framework. 2. There is a possibility that in some cases, the chosen OOD labels from CSP may also have high similarity to ID images. The authors should provide more explanation on this point. 3. I do not quite understand the significance of the experiments in Table 5. I suggest the authors provide a more detailed discussion. 4. In Table 9, the meaning of "Size" of the corpus is not clear. Does it refer to the total number of words in the dictionary or the number of selected OOD labels?

Questions

NA

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Please see the weaknesses.

Summary

The paper proposes a method for zero-shot out-of-distribution (OOD) detection using an expanded semantic pool of modified superclass names. By leveraging a pre-trained vision-language model, the approach aims to improve OOD classification performance by ensuring low mutual dependence among selected OOD labels. This method outperforms existing techniques by 7.89% in FPR95, highlighting its effectiveness in handling OOD detection tasks.

Strengths

1. The theoretical analysis of expanding NegLabel using the u-function is very clear and intuitive. 2. The experiments are conducted very comprehensively.

Weaknesses

1. The method proposed in this paper lacks innovation; its main framework and basic performance are entirely derived from NegLabel. 2.The preliminary section lacks clear references to the original theory of NegLabel and the proofs in the appendix. 3.The description of the proposed method is very concise, which makes it unclear for readers to understand the details of the method design. It is recommended to provide pseudocode for the algorithm to clarify the method's design.

Questions

see weakness

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

YES

Official Comment by Reviewer QWh5

Thanks to the authors for the response. I am happy that most of my concerns have been addressed and I have decided to keep the score.

Summary of reviews and rebuttal

Dear Reviewers, ACs, and SACs, We would like to express our sincerest gratitude to the reviewers, ACs, and SACs for the time and effort invested in reviewing our work. As the review discussion period comes to a close, we would like to provide a brief summary of the main strengths, weaknesses, and our rebuttal. **Strengths**: 1. The theoretical analysis/motivation is clear and intuitive (pEkd, gvS3, KrTH), and the mathematical modeling is reliable/robust (QWh5, KrTH). 2. The analysis of the shortcomings/limitations of NegLabel/existing methods is enlightening (p3ao, KrTH), and the method design is simple/straightforward and effective (gvS3, QWh5). 3. The experiments are comprehensive (pEkd, p3ao, KrTH) , and the performance improvement is considerable/significant (gvS3, KrTH, QWh5). SOTA is achieved (gvS3,KrTH). 4. The paper is well-written/has high standard of presentation and writing quality (p3ao, QWh5). The above strengths were the ones appreciated by more than one reviewers, and we are deeply grateful for their recognition and positive feedback. **Weaknesses/Questions**: 1. This work is primarily based on the framework of the SOTA method NegLabel (p3ao, pEkd). **Rebuttal**: This work indeed builds on NegLabel. However, leveraging open vocabulary and pre-trained models is a natural and common paradigm for zero-shot tasks. Based on the existing framework, our work identifies key shortcomings and achieves much superior performance through our derived theoretical guidance and straightforward modifications to the semantic pool, showcasing unique innovation. 2. The description of the algorithm (pEkd) / table1 caption and table5 significance (QWh5) / technical details (gvS3, p3ao) are too concise. **Solution**: We have provided a more detailed introduction and the pseudo code in the rebuttal, which will also be added to the revised version. 3. Questions related to synonyms in the semantic pool (KrTH, p3ao). **Answer**: We have carefully explained the effect of including synonyms and how we handle them, which have addressed the concerns of reviewers. Please refer to the rebuttal for details. The above weaknesses and questions were the ones raised by more than one reviewers. Besides, all reviewers who responded agreed that our rebuttal addresses their concerns. Finally, thank you again for your valuable time and expertise throughout this process. Should you have any further suggestions or comments, we are more than willing to consider them and make the necessary adjustments. Warm regards, Authors of Submission 5027