Long-Tailed Out-of-Distribution Detection via Normalized Outlier Distribution Adaptation

Summary

The paper proposed a novel approach, namely normalized outlier distribution adaptation (AdaptOD), to tackle this distribution shift problem, where ID classes are heavily imbalanced, i.e., the true OOD samples exhibit very different probability distribution to the head and tailed ID classes from the outliers. One of its key components is dynamic outlier distribution adaptation that effectively adapts a vanilla outlier distribution based on the outlier samples to the true OOD distribution by utilizing the OOD knowledge in the predicted OOD samples during inference. Further, to obtain a more reliable set of predicted OOD samples on long-tailed ID data, a novel dual-normalized energy loss is introduced in AdaptOD, which leverages class- and sample-wise normalized energy to enforce a more balanced prediction energy on imbalanced ID samples.

Strengths

1. The paper is written well and is easy to understand. 2. The studied problem is very important. 3. The results seem to outperform state-of-the-art.

Weaknesses

1. It would comprehensive if the accuracy is reported on ID data 2. I am curious why there is not a weight coefficient before DNE loss 3. It is better to ablate on the number of outlier data used during adaptation

Questions

see above

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

n/a

Summary

This paper addresses the issue of true OOD samples having distribution shifts in scenarios of Long-tailed Recognition (LTR) with heavy imbalance. To cope with this, the paper proposes normalized outlier distribution adaptation (AdaptOD) with two key components: Dynamic Outlier Distribution Adaptation (DODA) and dual-normalized energy (DNE) loss. DODA dynamically adapts the outlier distribution during inference to align better with the true OOD distribution, while DNE helps achieve balanced prediction energy across imbalanced ID classes. This method was tested on various benchmark datasets, including both low- and high-resolution datasets.

Strengths

- The proposed method, reducing distribution shift of OOD on training and inference stage, is novel. - The paper includes a detailed ablation study that demonstrates the generality of the proposed methodology. - The paper's class-wise and sample-wise normalization techniques improve the balance in prediction energy for imbalanced ID samples, enhancing overall performance compared to existing methods. - This framework shows good experimental results compare to several baselines.

Weaknesses

- The overview of Figure 2 is difficult to understand. Simplifying and highlighting only the necessary information would be beneficial. There are too many redundant details. - While recent methodologies improve long-tailed OOD detection performance using synthetic outliers without real outliers, this approach still relies on auxiliary outliers. This dependency can limit the practical applicability of the method. - The method's effectiveness is questionable when handling distribution shifts between training auxiliary outliers and actual real-world outliers, potentially compromising its performance if the shift is large.

Questions

N/A

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

see weakness

Summary

This paper addresses the challenge of OOD detection in LTR scenarios, where the distribution of classes is heavily imbalanced. The key issue highlighted is the absence of true OOD samples during training, which hampers the effectiveness of OOD detectors, especially the significant distribution shift between outlier samples and true OOD samples in LTR scenarios. The authors propose a novel approach called AdaptOD to tackle this problem by reducing the distribution gap between the outlier samples used for training and the true OOD samples encountered at inference time. AdaptOD introduces two main components: DODA performs test-time adaptation to dynamically adjust the outlier distribution to better align with the true OOD distribution using the OOD knowledge from predicted OOD samples during inference. And DNE is designed to balance the prediction energy for imbalanced ID samples during training, which helps in learning a more effective vanilla outlier distribution that is crucial for DODA's adaptation process. The authors demonstrate the effectiveness of AdaptOD through extensive experiments on CIFAR10/100-LT and ImageNet-LT. The results indicate its superiority in handling the long-tailed OOD detection problem.

Strengths

1. Exploring OOD detection in the context of long-tailed recognition has practical value, and the discovery of strong distribution shift between outlier samples and true OOD samples in LTR scenarios, as shown in the Fig. 1a, is meaningful. 2. The motivation is clear. AdaptOD utilizes the DNE to learning a better vanilla outlier distribution first, and then performs DODA to dynamically adapt the outlier distribution to the true OOD distribution. 3. The method is elegant. DODA only calibrates the outlier distribution, effectively eliminating the retraining or memory overheads, and well approximating the upper-bound performance. DNE effectively adapt the standard energy loss in LRT scenarios with batch energy normalization, and further eliminated the dependence of energy bar on imbalance factors and training datasets. 4. This paper is well-written, and the empirical evaluation is comprehensive.

Weaknesses

1. The goal of the CET component is to "enforce more balanced prediction energy for imbalanced ID samples" as line 88, which is similar to BERL[1]. The author should further clarify their differences. 2. AdaptOD performs better in near OOD dataset CIFAR, which cannot be achieved by previous methods. It would be better to further discuss the reason. [1] Choi, H., Jeong, H., Choi, J.Y.: Balanced energy regularization loss for out-of-distribution detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15691–15700 (2023)

Questions

1. DNE is designed for more effective DODA, what's the most important factor in applying TTA methods on OOD detection in LTR during training stage? 2. The updated form of the outlier distribution in Eq.4 is sample-wise, how does it compare to batch-wise updates?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations have been discussed in Appendix E.

Summary

This paper introduced the normalized outlier distribution adaptation (AdaptOD) which adapts the outlier distribution to the true OOD distribution from both the training and inference stages. Such AdaptOD includes two components: one is dynamic outlier distribution adaption (DODA), which adapts the outlier distribution to the true OOD distribution. The other is dual-normalized energy loss (DNE), which includes class-level and sample-level normalized energy to enforce more balanced prediction energy for imbalanced ID samples. The experiments are conducted on three OOD LTR benchmarks.

Strengths

1. The introduction of DODA and DNE is novel and addresses the key issue of distribution shift in OOD detection, especially in LTR scenarios. 2. The combination of DODA and DNE demonstrates significant improvements over existing methods, both individually and collectively, as evidenced by the ablation studies. 3. The paper is exemplarily structured and articulated. It offers a lucid and comprehensive elucidation of the AdaptOD methodology, its constituent elements, and the experimental framework, rendering it accessible to readers.

Weaknesses

The motivation and mechanism underlying dynamic outlier distribution adaption (DODA) are unclear and lack a theoretical foundation. (1) My primary concern is on the setting. The proposed AdaptOD with the introduction of test time adaptation (TTA) seems unfair. In previous work on OOD, the model's update process could not access the true OOD data and adopted the outlier as an alternative, while in this paper, the true OOD data is used to update the outlier distribution in the module DODA. As described in Tab. 5, without the DODA modules, the AUROC is 72.04 on ImageNet-LT, and such an increase is relatively small compared to SOTA. That indicates the main improvement depends on the knowledge leakage of true OOD samples during TTA. (2) I am confused about the motivation of DODA under the particular OOD setting, which adapts the outlier distribution to true OOD distribution. Suppose we could access the true OOD distribution in the test phase and update the outlier distribution with the predicted OOD sample. Why don't we use the true OOD distribution to calibrate the scores? The DODA provides access to the true OOD samples and provides the perfect solution for auxiliary OOD data that directly uses the predicted OOD sample as auxiliary OOD data. Moreover, as described in Fig. 1, the curves of adapted outliers almost coincided with the true OOD. The authors would better give more interpretation. (3) I am confused about the mechanism of DODA. Whether the DODA would adapt the outlier distribution to the true OOD distribution, as the initialized model detects the OOD sample at the beginning, the ID samples would be wrongly predicted as OOD, and that would lead to the optimization of the outlier distribution to the ID data distribution rather than the OOD distribution. The direction of the update of outlier distribution is uncertain how the DODA guarantees the direction to True OOD distribution. The authors would better give more theoretical analysis and support. (4) Is adapting the outlier distribution to true OOD distribution with specified OOD samples meaningful for OOD detection? As the OOD samples can be drawn from any unknown distribution, the benchmarks are just built to evaluate the models’ ability of unknown detection. The specified adaption of outlier distribution would improve the performance on corresponding benchmarks. However, the specified adaption can not generalized to other benchmarks and would be useless for the OOD sample from an unknown distribution.

Questions

1. The experimental comparisons for Tab. 1 seem unfair. The COCL and EnergyOE can not access the true OOD data, while the proposed method AdaptOD updates the global energy score with true OOD data in the test phrase. 2. For Tab. 2 and Tab. 3, the previous methods with DODA modules obeying the OOD setting should be considered.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of this paper, and there is no negative societal impact.