Summary
The paper presents a universal method for integrating detectors and evaluating approaches for detecting out-of-distribution (OOD) data and addressing data distribution shifts. The authors propose a technique that normalizes detector scores into p-values using quantile normalization, transforming the problem into a multivariate hypothesis test. They combine these tests using meta-analysis tools, improving the effectiveness of the detector and consolidating decision boundaries. The authors also create an interpretable criterion by adjusting the final statistics of in-distribution scores. Through empirical investigation, they demonstrate that their approach enhances robustness and performance across various domains, shift types, and OOD detection scenarios. The paper contributes to the field of machine learning by providing a flexible framework for integrating detectors and addressing OOD detection challenges.
Strengths
The paper presents a universal method for integrating detectors, which can be applicable across different domains, including OOD detection and two sample test.
The use of quantile normalization to transform detector scores into p-values seems interesting. This transformation allows for treating the problem as a multivariate hypothesis test and enables effective combination for a set of predefined scoring strategies.
The paper proposes adjusting the final statistics of in-distribution scores to create a fully interpretable criterion. This feature is valuable as it provides insights and explanations for the detection decisions, enhancing the transparency and interpretability of the method.
Through empirical investigation, the paper demonstrates that the proposed method significantly enhances overall robustness and performance across various domains, shift types, and out-of-distribution detection scenarios. This finding highlights the practical effectiveness of the approach.
Weaknesses
This paper mainly uses the scoring strategies from OOD detection to build their method, which mainly considers the concept shift. However, in the discussion, they actually present another two different distribution shift, i.e., covariate shift and prior shift. Then, a natural question is why the basic method that handle concept shift can also be useful to tackling covariate shift and prior shift? Is it the attributed to the combination of different p values? More detailed discussion and empirical evaluation seem to be important.
Quantile normalization seems interesting, while there exist many other ways, typically more simple ways, in doing normalisation. For example, we can simply normalise the score to follow the N(0,1) Gaussian distribution, which can also normalise the data. Therefore, more discussion about the theoretical superiority in using quantile normalisation is interesting.
It seems that extra parameters are introduced when combine different scoring strategies in Section 4.2 and 4.3, while I cannot find how to tune the related parameters. Moreover, how to conduct hyper parameter tuning, epically for the choice of evaluation dataset, should be discussed. If the proposed method does not introduce additional parameters, seemingly different scoring strategies are treated equally in combination. I am not sure if it is a proper setup.
Different scoring strategies are effective for varying data, so another interesting question is that if the proposed method can be instance dependent in combination when facing different data points.
Questions
Please see the weaknesses above.
Rating
5: marginally below the acceptance threshold
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.