Summary
Out-of-distribution (OOD) detection is essential for deploying machine learning models in real-world scenarios, as it addresses the challenges posed by unseen data. This paper introduces a novel Neural Activation Prior (NAP) for OOD detection, based on the observation that fully trained neural networks exhibit a higher probability of strong neuron activation in response to in-distribution (ID) samples compared to OOD samples. The authors propose a scoring function that leverages this prior to enhance OOD detection without degrading ID classification performance or requiring additional training data. Their method uniquely utilizes intra-channel activation patterns, making it compatible with existing approaches and allowing for effective integration. Through comprehensive experiments, including an oracle validation, the authors demonstrate the effectiveness of their method, which significantly improves performance when combined with other OOD detection techniques.
Strengths
The paper is well-written and easy to understand. This paper designs a method called NAP, which utilizes visualization to demonstrate its motivation and validates the effectiveness of the approach through extensive experiments.
Weaknesses
At its core, this work is no different from Ash; both observe the most prominent activations in the feature maps. In this regard, the motivation can be considered overlapping with Ash, and the originality of the paper is not as high as claimed in the text. Furthermore, the observations regarding feature activations in this paper are consistent with the phenomena described in [1], which further diminishes the novelty of this observation.
Additionally, the score presented in this work is extremely unstable and essentially overfits to datasets with object-centric characteristics. In other words, it is designed with the assumption that the known ID datasets are of types like ImageNet and CIFAR. First, it imposes no constraints on the size of ID objects; the maximum value divided by the average value means that the magnitude of this result heavily depends on the pixel area of the ID semantics in the image. As shown in Figure 1(a), if the object were larger, resulting in high activations covering almost the entire image, the final score would actually be lower. This suggests that the effectiveness of the method when using ImageNet as the ID dataset may be coincidental. It would be worthwhile for the authors to test their method using Textures as the ID dataset to see how it performs.
Moreover, the method heavily relies on the semantic complexity within the images. Theoretically, the more semantic elements present in an image, the more likely it is to activate certain neurons' responses. Additionally, even within the same ID dataset, there can be significant differences in semantic complexity between different categories. The method tends to output lower scores for categories with sparse semantics and higher scores for those with complex semantics. It would be beneficial to discuss this issue further and to experiment with using datasets with sparse semantics as ID and more complex datasets as OOD to observe the effects.
Furthermore, there are concerns regarding the design of the Max operation. It is well-known that Max is particularly unstable. The purpose of this OOD detection algorithm is to address the issue of model overconfidence. If there is an OOD sample that closely resembles an ID sample (i.e., in a near-OOD setting), it may still conform to certain ID patterns in some channels, leading to a high score. This raises the question of whether the near-OOD experiments conducted are comprehensive enough. It would be beneficial to use larger datasets and to visualize and compare the activation heatmaps of both ID and OOD samples.
On another note, the issue is not limited to near-OOD samples; there is an even greater concern regarding spurious correlations. By using Max, the influence of spurious correlation features is amplified. It would be important for the authors to provide an explanation for this and to present experimental results on datasets with spurious correlations.
Lastly, the article appears to be based on an empirical modeling approach, lacking a reliable theoretical framework to explain its findings. As previously mentioned, there are numerous datasets and activation patterns that may not align with the assumptions made in this paper. The authors should provide a more detailed modeling and analysis to strengthen their claims.
Additionally, it is worth noting that the results reported in Table 3 for ReAct, ASH, and SCALE are significantly lower than those reported in their original papers. A similar issue arises in Table 13, where the results differ substantially from those reported in [2]. This raises concerns about the potential lowering of baseline performance in the current study.
[1] Pei, Sen, et al. "End-to-End Out-of-distribution Detection with Self-supervised Sampling." ICLR, 2024.
[2] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution
detection. In NeurIPS, 2021.