Summary
This work proposed a refined framework for OOD detection that introduces clear definitions for Semantic Space and Covariate Space. By establishing these definitions, the paper addresses a critical limitation in existing OOD settings, where certain distributions are indistinguishable from in-distribution samples due to ambiguous shifts. It further introduces the concept of "Tractable OOD," ensuring OOD detection is feasible for post-hoc methods, accompanied by theoretical analysis and experimental validation.
Strengths
1. The manuscript provides a rigorous theoretical foundation for OOD detection by defining Semantic and Covariate Spaces, clarifying when OOD tasks become tractable or intractable.
2. The proposed "Tractable OOD" setting addresses a fundamental flaw in existing OOD frameworks, making it a valuable advancement for post-hoc OOD detection methods.
3. The manuscript demonstrates its theoretical findings through comprehensive experiments on both synthetic and real-world datasets, reinforcing the practical applicability and validity of the proposed definitions.
Weaknesses
1. The manuscript’s reliance on simplifying assumptions, such as linear separability and Gaussian distributions, may limit its applicability to complex, high-dimensional data in real-world Big Data settings. Additionally, it lacks a thorough discussion on the scalability and implementation challenges of the proposed "Tractable OOD" setting in deep learning models.
2. The experiments consider only one dataset, which is insufficient for drawing robust conclusions.
3. The code is not available; it would be beneficial for the authors to provide a link to their code repository.
Detailed Comments:
This paper identifies a key drawback in current OOD detection theory and introduces an intriguing framework to decompose the OOD detection problem for theoretical analysis. However, despite its innovative approach, the present version lacks practical impact on real-world applications. The quality of this paper could be greatly enhanced by incorporating the following improvements:
1. The manuscript’s theoretical analysis is built on simplifying assumptions, such as linear separability and Gaussian-like distributions, which, although useful for analysis, may limit its generalizability to real-world scenarios involving high-dimensional, non-linear data. This reliance on basic assumptions reduces the applicability of the findings to complex settings typical in Big Data applications, where more flexible models like mixture distributions or non-parametric methods often better capture the structure of ID distributions. Furthermore, the paper does not address the practical challenges or scalability concerns of implementing the proposed "Tractable OOD" setting in deep learning models for high-dimensional data, where Gaussian assumptions might be less informative. Expanding on these limitations and exploring potential extensions or architectural modifications to accommodate more complex data structures would enhance the relevance and practicality of the approach, particularly for diverse Big Data environments.
2. Although this paper primarily focuses on a new definition of semantic shift and its theoretical foundations, using only one real-world dataset limits the strength of its conclusions, given the diverse forms of covariate shift, such as domain shift (change in the data domain or context), feature distributional shift (distribution of individual features changes between training and testing), temporal shift (changes in data distributions over time), etc. To enhance the robustness of the findings, the authors might consider including additional real-world datasets in the experiments. For instance, the "Waterbirds" dataset [1] could be valuable for testing spurious correlations—a special case of feature distribution shift where irrelevant features influence model predictions.
3. The manuscript lacks explicit information on the availability of code or dataset access links, which are essential for reproducibility and community engagement. Providing a link to a code repository or outlining plans to make the code available upon publication would be beneficial.
[1] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization." arXiv preprint arXiv:1911.08731 (2019).
Questions
Q1: How could your proposed definitions and theorems support deploying state-of-the-art OOD detection methods in real-world scenarios? Please clarify the practical relevance of your theoretical framework and discuss how it could enhance or integrate with existing SOTA OOD techniques to address real-world challenges.
Q2: The experiments consider only one real-world dataset, "IMAGENET DOGS." The authors' results will be more convincing if they consider more real-world datasets, such as the Waterbirds [1] or other benchmark datasets appropriate for this context.
Please see my comments on the weaknesses as well.