Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy

Summary

This paper contributes a new OOD detection algorithm based on energy score. It first equips the energy score with a trainable temperature. It then analyze the effect of forefront temperature and exponential divisor temperature in energy score by pointing out that the exponential divisor temperature is desired while forefront temperature is not. The trainable term is thus only for the exponential divisor temperature.

Strengths

1. The empirical performance of the work is good, especially in the segmentation task. 2. The discussion of the forefront temperature and exponential divisor temperature is reasonable. 3. It seems this method is able to perform well without a knowing the OOD samples for training/fine-tuning.

Weaknesses

The work in general seems to be incremental. 1. This work trains an objective similar to generalized ODIN and uses an energy score with trained temperature for OOD detection. A theoretical analysis between ODIN seems to be missing. For example previous energy score pointed out it was better than the maximum confidence score because of the contradiction of maximum logit and energy score. Even if it has, the contribution will still be limited. 2. Sec 5 only addresses the empirical understanding rather than theoretical.

Questions

1. (For weakness 1) Why using energy score with trainable temperature for energy-score is better than the OOD score in generalized ODIN? Maybe following this idea? What determines the performance in energy score is mainly the logsumexp of the logits, rather than temperature. Thus, its analysis was mainly in the denominator of the softmax function and temperature does not play a major role there. Given this fact/approximation, I feel like the function $\hat f_{y_i}(x_i; \theta)$ (just below equ(1)) could be analyzed similarly. Taking the log of $\hat f_{y_i}(x_i; \theta)$, we will have two terms, one of which will be the AbeT score. 2. Given the similarity between Generalized ODIN and energy score, it seems the really interesting result to me is it seems it does not need OOD samples during training. (2.1) Can you confirm this? (2.2) If so, can you explain why the trained temperature only could change the OOD performance theoretically? If this explanation is convincing, it may be more important than the empirical perspectives of the methods.

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Summary

This paper proposes the Ablated Learned Temperature Energy (or "AbeT" for short) for out-of-distribution detection. AbeT comprises two components: a learned temperature and an ablated energy score. Abet significantly boosts OOD detection performance, and shows efficacy in identifying OOD samples in object detection and semantic segmentation.

Strengths

1. The method is simple and the performance is promising. 2. The visualization on object detection and semantic segmentation is interesting.

Weaknesses

1. The idea of learned temperature has been fully studied in previous works like GODIN (Liu et al., 2020). This paper seems to be an incremental work. 2. In figure 1, as the Learned Temperature contradicts with energy score, how about using the negative temperature to boost the energy, rather than simply ablating the temperature? 3. The experimental results should be further discussed. For example, in Table 1, as vanilla AbeT achieves 40% of FPR95 in ImageNet-1k, why a simple combination with ASH leads to such a surprising 3.7% of FPR95?

Questions

Please check the weakness section above.

Rating

5: marginally below the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

This paper addresses the critical task of identifying Out-of-Distribution (OOD) inputs in deep neural networks. The authors introduce "AbeT" (Ablated Learned Temperature Energy), a method that combines existing techniques without complex training stages or additional hyper-parameters. Specifically, the proposed method combines the use of learned temperature parameters and energy scores for OOD detection. Crucially, it proposes a simple adjustment to the way energy score is computed with temperature parameters, leading to empirically demonstrated gains. AbeT significantly reduces false positives in classification, surpassing state-of-the-art methods. It also offers insights into how it learns to distinguish ID and OOD samples. Moreover, AbeT demonstrates improved performance in object detection and semantic segmentation tasks, making it a valuable tool for model robustness in critical domains.

Strengths

- Empirical Results: The paper supports its claims with empirical results, demonstrating the practical benefits and effectiveness of AbeT across multiple datasets and tasks. - Insightful Model Behavior Analysis: The paper provides valuable insights into how the model learns to distinguish between In-Distribution (ID) and OOD samples, contributing to a better understanding of model behavior. - General Applicability: The method's versatility is showcased by its successful application in object detection and semantic segmentation tasks, highlighting its potential for various computer vision applications. - Simplicity and Efficiency: Unlike some methods that require complex training stages, hyperparameters, or test-time backward passes, AbeT offers simplicity and efficiency in OOD detection, making it more practical for real-world applications.

Weaknesses

- Lack of novelty: While being able to achieve good performance empirically, the proposed method is a simple adaptation of two existing methods. - Lack of theoretical justifications: While empirically proven, the proposed decision on dropping Forefront Temperature Constant, a crucial part of the proposed method, lacks theoretical justifications. - Formatting: the paper is not in the ICLR conference formatting.

Questions

- Based on my understanding of table 1, no ablation study was done to compare the effect of dropping Forefront Temperature Constant. How does the performance of Energy + learned temperature compare to the proposed AbeT?

Rating

5: marginally below the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

This paper presents an ablated learned temperature energy for OOD detection. The paper is built on two existing works in OOD detection, which rely on a learned temperature and an energy score. The paper argues that the existing work does not consider any OOD examples while estimating the temperature; hence, it proposes to learn based on an input. The method is evaluated on multiple OOD datasets considering CIFAR-10, CIFAR-100, and Imagenet-1k as domain data sets. Experimental results show the method surpasses the existing methods.

Strengths

OOD detection is an important research problem and can be of interest to a large audience. The paper compares the performance with several existing methods on multiple benchmarks.

Weaknesses

Combining existing methods does not make the contribution less significant. However, I am not convinced that making the hyper-parameter learnable is a solid contribution. It is also unclear how the parameters of temperature are optimized and what the learning behaviour looks like. The paper proposes to make the temperature of the existing method learnable, which is a decent contribution; however, given the number of high-quality submissions that we receive in the ICLR, it can be challenging to find a spot in the main conference. The paper lacks insight into how the learnable temperature varies with different inputs. Sometimes, the paper is difficult to follow. The paper does not use the correct template of ICLR, although this does not have any role in my final rating.

Questions

Please see the weakness section.

Rating

3: reject, not good enough

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair