AHA: Human-Assisted Out-of-Distribution Generalization and Detection

Summary

The paper presents a novel approach to the problem of OOD generalization and detection. The authors introduce the AHA (Adaptive Human-Assisted OOD learning) framework, which aims to enhance both out-of-distribution (OOD) generalization and detection by strategically leveraging human-assisted labeling within a maximum disambiguation region. The paper reports significant improvements over state-of-the-art methods with only a few hundred human annotations, demonstrating the efficacy of the proposed framework.

Strengths

- The AHA framework is a creative solution that addresses the challenges of OOD generalization and detection, which are critical for real-world applications of machine learning models. - The authors provide extensive experimental results that validate the effectiveness of their approach, showing robust performance across various datasets. - The incorporation of human feedback in a strategic manner is a strength, as it capitalizes on the limited labeling budget to maximize model performance. - The paper's contributions are articulated, with the novel labeling strategy and the integration of human assistance being the highlights. - The transformation of the problem into a noisy binary search is an intelligent methodological choice that allows for the efficient identification of the maximum ambiguity threshold.

Weaknesses

- While the paper demonstrates strong results, it is not clear how the AHA framework scales with larger and more complex datasets. - The reliance on human annotations could be a limitation in scenarios where such resources are not readily available or are cost-prohibitive. - The paper could benefit from a discussion on how the findings generalize beyond the tested datasets and scenarios. - The computational complexity of the AHA algorithm and its runtime performance on large datasets are not discussed. - The paper could address potential biases introduced by human labeling, especially in the context of OOD detection.

Questions

- How does the AHA framework perform as the size and complexity of the dataset increase? - What are the specific steps taken to mitigate potential biases in human labeling? - Can the authors provide more details on the computational efficiency of the AHA algorithm, especially for large-scale applications? - How does the framework handle a class imbalance in the context of OOD detection? - Are there any specific domains or applications where the AHA framework is expected to be more or less effective, and why? - Could the proposed method benefit the OOD detection with unreliable sources [R1] and inspire unsupervised OOD detection [R2]? ---- [R1] Out-of-distribution detection learning with unreliable out-of-distribution sources. NeurIPS 2023. [R2] Out-of-distribution detection with an adaptive likelihood ratio on informative hierarchical vae. NeurIPS 2022.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

NA.

Summary

This paper proposes to address both out-of-distribution detection and generalization within one joint framework under human-assistance. The proposed method first utilizes a noisy binary search algorithm to identify the most informative samples to be labeled. Then, it continues to annotate these samples with human feedback. The authors conduct experiments on CIFAR and PACS to evaluate the proposed method.

Strengths

- The proposed method can handle OOD detection and generalization at the same time, which is impactful to both these two individual research areas. - Covariat-shifts and semantic shifts are both inevitable in real-world applications. Thus the proposed method is straightforward and well-motivated. - Most parts of this paper are well presented with good visualization.

Weaknesses

- AHA may only work under a rather strict assumption. Compared to outlier exposure, AHA needs to access the real test data distribution $S_{wild}$ to selectively label some samples. While outlier exposure does not need such an assumption. Many previous OOD detection methods can only access training data distribution and an auxiliary OOD data distribution (noted that such auxiliary OOD has no overlapped samples with test-time OOD data in common settings). Thus I think AHA may only work under a more strict assumption (i.e., the test data distribution $S_{wild}$ is accessible) than previous outlier exposure. - I generally believe enhancing OOD detection and generalization with human feedback is laborious. I am aware the proposed method can get good performance with hundreds or thousands of labeled samples in many cases. However, such human feedback still seems laborious to me. For example, in CIFAR experiments, the images are only 32*32. Thus I think it would take a lot of time to label such samples. Not to mention the samples can be noised or corrupted (CIFAR-10-C). - The experiments are not adequate. It is widely acknowledged that OOD detection and generalization are more difficult on large-scale high-resolution datasets. For example, ImageNet benchmarks. The authors do not conduct evaluations on such datasets. - In some cases, there may be no clear boundary between covariate-shift data and semantic shift data. As mentioned in recent work[1], these two types of OOD samples can be very hard to separate (even for humans). Could the authors comment on this phenomenon? Given the above points, I tend to reject this paper because the overall quality do not meet the expectations of NIPS. However, I may adjust my score if there is a strong argument. [1] Yang, William, Byron Zhang, and Olga Russakovsky. "ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms."

Questions

Please refer to weaknesses and address my concerns.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper introduces a novel, integrated approach AHA (Adaptive Human-Assisted OOD learning) to simultaneously address both OOD generalization and detection through a human-assisted framework by labeling data in the wild. Extensive experiments validate the efficacy of AHA.

Strengths

1. this paper is well written and easy to follow 2. good visualization and extensive experiments 3. Maximum Disambiguation Region and reduction to noisy binary search are new to me

Weaknesses

1. there exsits a strong assumption that the weighted densities of semantic and covariance ood should equalize 2. what is difference between active learning and the proposed Human-Assisted Learning 3. what is the time used for noisy binary search 4. whether the lamda searched on the one dataset can be transfer to another dataset

Questions

see weakness

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

see weakness

The discussion has begun

Dear Reviewers, Thank you for taking the time to review the paper. The discussion has begun, and active participation is highly appreciated and recommended. Thanks for your continued efforts and contributions to NeurIPS 2024. Best regards, Your Area Chair

Official Comment by Reviewer Eoxk

Thank the authors for answering my questions. I would like to keep my rating "Borderline accept".

Response to Reviewer CyPp (Followup)

Thank you again for your feedback and comments. We address your comments below. > *The need for human assistance is both the motivation of this work and its inevitable weakness.* We would like to clarify that there are many scenarios where human annotation is particularly valuable, for example, in medical diagnostics. In these contexts, human assistance provides crucial expertise and contextual information, which should be considered a significant advantage rather than a weakness. We would greatly appreciate an increased overall rating if we have successfully answered your questions. Otherwise, we are happy to provide additional discussions to address any further concerns.