Learning to Augment Distributions for Out-of-Distribution Detection

Summary

This paper noticed that the discrepancy between the auxiliary and (unseen) real OOD data hindered the effectiveness of outlier exposure, which is a potential method for OOD detection. To tackle this issue, the authors proposed a learning framework that augments the auxiliary OOD dataset using a Wasserstein ball. By training the model on the worst-case OOD data in the ball, the performance of the model toward open-world OOD detection will be improved.

Strengths

- The paper is well-motivated: the authors formalized the challenges in existing outlier exposure methods for OOD detection and provide a solution to the challenges. - The proposed DAL framework is proposed with firm theoretical support with a convergence guarantee. - Experiments show that using ID and OOD auxiliary samples and using the proposed DAL can effectively improve the performance of OOD detection, even when ID and real OOD samples are very close. The results confirm the design of DAL.

Weaknesses

- Experiments are conducted on relatively simple datasets and models. It is worth to be shown that the proposed method still has improvements on more complicated samples.

Questions

- Given the Wasserstein ball is applied to the feature distribution, is it possible to train the classifier only and leave the encoder as pre-trained? - Will the false negative rate of ID samples be increased under DAL, i.e., ID samples being misclassified as OOD samples, especially in hard OOD detections? - Why choose 10 searches in experiments? Will increased searches improve performance? How long does it take for the DAL training?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors do not address the limitations of the method. Possible improvements can be: can the model be easily applied to (1) regression tasks? (2) models that are hard to be fully fine-tuned, i.e., transformers.

Summary

This paper proposes a theory and an algorithm to address the OOD detection problem better. The whole paper is well-structured, and the algorithm is designed via the developed theory. The basic idea is to simulation the OOD distributions as representative as possible, thus we can expect that the detector can work well in the most scenarios. Experiments verify the effectiveness of the proposed method. All theoretical results are proved in the appendix.

Strengths

1. This paper's structure is very clear, and readers enjoy reading this paper. It starts from a classical but powerful solution of OOD detection problem. Then, a theory follows to demonstrate the effectiveness of this kind of solution. Finally, am algorithm is designed via the proposed theory. 2. The proposed algorithm is verified in many benchmark datasets, including hard OOD situation, which is a strong evidence to support the claims made in this paper. 3. The studied problem is very important, and the focused research direction is practical. Although there are many non-outlier methods used to detect OOD data, we can always obtain some outliers to help our training, which is practical.

Weaknesses

1. There might be some gaps between the algorithm and theory. One might be addressed during the response phase: In Theorem 2, there are two assumptions regarding the loss function. How are the both assumptions satisfied in your algorithm? Some discussions should be added here. 2. In lines 218-219, the reason why we need to perturb data in the embedding space should be clarified. You indeed follow previous work, but more explanations are needed here. 3. In lines 228-235, more justifications should be added here. I did not see any support (from literature or experiments) for this kind of design. 4. Experiments regarding CIFAR-10 can be moved to appendix, as all methods have good performance. It is not meaningful to see that one method is better than the other by 0.2%. In this way, more experiments in appendix can be moved back to the main content. 5. It is very interesting to see the real validation set here, more details can be included in the main content.

Questions

See the weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper aims to address the outlier exposure in OOD detection from theoretical perspective. By developing a novel generalization bound based on the distributional augment principle, the paper proposes a novel outlier exposure method. Experiments indicate the effectiveness of the proposed method.

Strengths

1. A novel generalization bound to understand outlier exposure. It seems this theory is the first theory in outlier exposure field. 2. The proposed algorithm is theoretical guided and guarantee. From my personal view, this is important to reliable ML. 3. The features trained based on proposed DAL can be used with other post-hoc OOD detection method. This method can be combined with previous works well and naturally. 4. Extensive experiments have shown the effectiveness of the method. It seems the performance has achieved SOTA performance compared with latest works.

Weaknesses

1. I am confused that it seems the radius rho becomes larger, the estimation error in Theorem 3 will decrease. So, from the theoretical view, why not selected rho larger enough? 2. How to tune the parameters rho, gamma….? Validate set is allowed in outlier exposure? 3. It seems your algorithm and theory have a small gap. Could you clarity why you do that? It is better to give more clear explanations.

Questions

Please answer the questions in Weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper aims to tackle the problem of out-of-distribution (OOD) detection in supervised learning through a learning theoretic approach. The authors propose a method termed DAL, which intends to reduce the discrepancy between the detector's training distribution (auxiliary OOD distribution) and true OOD distribution by expanding the set of auxiliary distributions over which the detector is trained. The paper outlines a rigorous approach to open-world OOD detection. DAL is evaluated experimentally and shown to outperform competing techniques. Post-rebuttal update: increasing score to 7

Strengths

Strengths: 1. The authors recognize the shortcomings in prior work on OOD detection and motivate the problem towards their proposed solution 2. While I am not an expert in the area, using Wasserstein type metrics and controlling the distribution discrepency between true and auxiliary OOD distributions appeals to mean 3. Through Theorems 2 and 3, the paper shows that DAL has a theoretical grounding and gives an algorithm to achieve the DAL objective 4. While I have not checked the proofs in sufficient detail, I believe that the analysis is rigorous 5. The experiments show the improvements achieved by DAL over several recent competing approaches

Weaknesses

Weaknesses: 1. Theorems 2 and 3 can benefit from a detailed discussion in the main body 2. This could be a question as well - are common corruptions of CIFAR10 or Imagenet not used as standard datasets for OOD detection? Also, please comment on my questions below

Questions

Questions: 1. Line 88: Does this mean that the labels in ID and OOD are mutually exclusive? Could you please explain if this is a significant assumption 2. Line 124: Please provide references for the statement "the auxiliary and the real OOD data differ largely in practice". I would like to understand how significant is the typical difference. 3. As a follow-up, how large can the upper bound in (4) grow? Are there any standard assumptions made for theoretical analysis? 4. What does the assumption in Theorem 2 regarding l <= M imply- suppose l is cross-entropy or exponential loss, does the bound mean that f_w needs to be a reasonably good solution? Otherwise, the loss function could be unbounded, if I am not mistaken. How mild are these assumptions?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

It would be useful if the authors discussed a few most relevant open challenges, in light of their findings

Thank you for the response, increasing score

The authors have addressed my queries, thus I am glad to increase the score to 7. Thank you for a clear response.