Neural Relation Graph: A Unified Framework for Identifying Label Noise and Outlier Data

Summary

This paper proposes a novel approach called the Neural Relation Graph framework for identifying label noise and outlier data in large-scale datasets with real-world distributions. The approach utilizes a relational structure of data in the feature-embedded space to detect label errors and outlier data, and introduces a visualization tool for interactive data diagnosis. The authors conduct extensive experiments on various tasks and demonstrate that their approach achieves state-of-the-art detection performance and is effective in debugging real-world datasets. The contributions of this paper include a unified approach for diagnosing and cleaning large-scale datasets, a data relation function and graph algorithms for detecting label errors and outlier data, and a visualization tool for interactive data diagnosis.

Strengths

Originality: The Neural Relation Graph framework proposed in this paper is a novel approach for identifying label noise and outlier data in large-scale datasets. The authors utilize a relational structure of data in the feature-embedded space to detect label errors and outlier data, which is a unique and innovative approach. The paper also introduces a visualization tool for interactive data diagnosis, which is a novel contribution to the field. Overall, the paper is highly original and presents a new perspective on diagnosing and cleaning large-scale datasets. Quality: The paper is of high quality, with a well-designed methodology and extensive experiments conducted on various tasks. The authors provide detailed descriptions of the proposed approach and the experiments conducted, which makes it easy to understand and replicate the results. The paper also includes a thorough evaluation of the proposed approach, comparing it to existing methods and demonstrating its effectiveness in detecting label errors and outlier data. The quality of the paper is further enhanced by the use of clear and concise language, making it easy to follow and understand. Clarity: The paper is well-written and easy to understand, with clear descriptions of the proposed approach and the experiments conducted. The authors provide detailed explanations of the technical terms used, making it accessible to a wide range of readers. The paper also includes visual aids, such as figures and tables, which help to illustrate the concepts presented. Overall, the clarity of the paper is excellent, making it easy to follow and understand. Significance: The paper is highly significant, as it presents a novel approach for diagnosing and cleaning large-scale datasets with real-world distributions. The proposed approach utilizes a relational structure of data in the feature-embedded space to detect label errors and outlier data, which is a unique and innovative approach. The paper also introduces a visualization tool for interactive data diagnosis, which is a valuable contribution to the field. The results of the experiments conducted demonstrate the effectiveness of the proposed approach, making it a significant contribution to the field of machine learning.

Weaknesses

One potential weakness of the paper is that the authors do not provide a detailed analysis of the limitations of their approach. While the proposed approach achieves state-of-the-art detection performance on various tasks, it is unclear how it would perform on datasets with different characteristics or in different domains. The authors could address this weakness by conducting experiments on a wider range of datasets and providing a more detailed analysis of the limitations of their approach. Another weakness of the paper is that the authors do not provide a detailed discussion of the computational complexity of their approach. While the paper mentions that the proposed algorithms are scalable, it is unclear how they would perform on very large datasets or in real-time applications. The authors could address this weakness by providing a more detailed analysis of the computational complexity of their approach and discussing potential strategies for improving its scalability. Finally, the paper could benefit from a more detailed discussion of the practical implications of the proposed approach. While the paper demonstrates the effectiveness of the approach in detecting label errors and outlier data, it is unclear how it could be applied in real-world scenarios. The authors could address this weakness by discussing potential use cases for the proposed approach and providing guidance on how it could be integrated into existing machine learning pipelines. Overall, the paper presents a novel and innovative approach for diagnosing and cleaning large-scale datasets, but could benefit from a more detailed analysis of its limitations, computational complexity, and practical implications.

Questions

Can you provide a more detailed analysis of the limitations of your approach? While the proposed approach achieves state-of-the-art detection performance on various tasks, it is unclear how it would perform on datasets with different characteristics or in different domains. Can you provide a more detailed discussion of the computational complexity of your approach? While the paper mentions that the proposed algorithms are scalable, it is unclear how they would perform on very large datasets or in real-time applications. Can you discuss potential use cases for the proposed approach and provide guidance on how it could be integrated into existing machine learning pipelines? While the paper demonstrates the effectiveness of the approach in detecting label errors and outlier data, it is unclear how it could be applied in real-world scenarios. Can you provide more details on the visualization tool introduced in the paper? While the tool is mentioned briefly, it would be helpful to have a more detailed description of its functionality and how it can be used to diagnose data. Can you provide more details on the datasets used in the experiments? While the paper mentions that experiments were conducted on various tasks, it would be helpful to have more information on the characteristics of the datasets and how they were selected. Can you provide more details on the hyperparameters used in the experiments? While the paper mentions that hyperparameters were tuned using cross-validation, it would be helpful to have more information on the specific values used and how they were selected. Can you provide more details on the implementation of the proposed algorithms? While the paper mentions that the algorithms were implemented using PyTorch, it would be helpful to have more information on the specific implementation details and any potential optimizations that were made. Can you discuss potential future directions for this research? While the paper presents a novel and innovative approach, it would be helpful to have a discussion on potential future directions for this research and how it could be extended or improved upon.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not explicitly address the potential negative societal impact of the proposed approach. While the focus of the paper is on diagnosing and cleaning large-scale datasets, it is possible that the approach could be used for other purposes, such as identifying individuals or groups based on their data. This could potentially lead to privacy concerns or other negative societal impacts. However, it should be noted that the paper does not provide any evidence that the proposed approach has been used for such purposes, and the authors do not make any claims about the potential negative societal impact of their work. Additionally, the paper does not explicitly address the limitations of the proposed approach, which could potentially lead to unintended consequences if the approach is used in real-world scenarios. Overall, while the paper does not explicitly address the potential negative societal impact of the proposed approach, it should be noted that the authors do not make any claims about the potential negative impact of their work, and the focus of the paper is on diagnosing and cleaning large-scale datasets.

Summary

The authors identified the issue of how existing label errors in the training data and the OOD in the test set can affect the model training and evaluation and further proposed a novel approach utilizing the learned feature embeddings and label information to compute the relations between data instances. The relations represent how similar the two data instances are in terms of their feature embeddings and also assigned labels. This is further used to construct a relational graph structure. Based on the graph structure, they introduce a min-cut algorithm based on the label noisiness score to identify a subset of label errors. They further visualized the derived graph structure for the purpose of interactive data error diagnosis. Extensive experiments are conducted on multiple data types (images, audio, texts), and all show superior performance over the baseline methods. Through comparison, authors showed their relational data structure provides complementary information not captured by the unary scoring methods. Generally, this paper is well-written, the ideas are pretty clear, the methods are novel, and results are solid.

Strengths

A substantive assessment of the paper's strengths, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See /faq. 1. This paper is well-written, and the authors explain the ideas clearly. The method is novel, and experiments and results are solid. 2. The example provided in Figure 1 explained well the limitations of using the unary scoring method when identifying the label error and the outlier data. 3. The design of the data relation function in Section 3.1 looks simple but still effective. 4. In section 3.2, the authors pointed out that simply aggregating all edges of a node can yield suboptimal results, which makes much sense, and they further proposed using the min-cut method as a walkaround. This looks interesting. 5. The experiments are conducted using various data types, including images, speeches, and texts. This also covers multi-class tasks and binary tasks. All experiments show the effectiveness of their methods, which is solid.

Weaknesses

1. In Figure 1, the difference between the Coil (label error) and the Envelope (outlier) is not very clear. More elaborations will be helpful. 2. Constructing the relational graph is a fully connected graph, the time complexity would be O(N^2), and the scalability to a large dataset is a concern. 3. In section 3.2, pp3 line 115, ‘a lower (label noisiness) score indicates a higher likelihood of label error’. A lower label noisiness indicates a higher error possibility, this is a bit confusing and not very intuitive.

Questions

1. In Figure 1, What makes the Coil example a label error, while the Envelope example an outlier? Some elaboration on this would be helpful. 2. In Section 3.2, pp3 line114, authors set $r(i,i)=0$, this is not very straightforward and the reason is not clear to me. More elaboration on this would be helpful. 3. In section 3.2, pp4 line 127, authors minimize the sum of the edges between two groups. One question comes naturally - what about the sums of edges within each group? 4. In terms of different datasets used in this work, several data types were involved (images, speeches, texts), it would be helpful to see what the embedding dimensions are for each dataset.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Minor edit suggestion: pp8 line309, ‘3.6%p’ should be ‘3.6%’.

Summary

This paper proposes a new method using graph structure for detecting label errors and outlier data. Briefly speaking, the algorithm utilizes data feature embeddings to generate relation graph and using the new defined data relation function, its algorithm can capture mislabeled and outlier data from a global prospective. A large number of experimental results show that the performance of this method is greatly improved compared with the existing detection methods.

Strengths

(1) This paper presents a novel relation graph-based approach to achieve better utility and considers error label and outlier detection in a global way. (2) Extensive experiments on datasets from different domains show the improvement of the new method is promising. (3) Comprehensive ablation study is performed, which helps to understand the proposed model better. (4) The overall expression of the article is clearer and easier to understand, and related works are adequately cited.

Weaknesses

(1) Certain statements in this paper lack justification. For example, the method is based on building graph on feature space, but there is a little discussion about feature spaces and how to generate it. (2) Lack of sufficient theoretical analysis (e.g. convergence but not only empirical analysis) , the proof of proposition and analysis of relation function are relatively simple. (3) Lack of explanation of choosing parameter in baseline method, like whether there is a finetune on the choice of K in KNN method.

Questions

(1) Line 159 in Complexity analysis, the paper demonstrates the complexity can be reduced to O(nk). But in the fourth last row in algo 1 each point in partition seems also need to calculate n relation score instead of k. Can you deeper explain the improvement in complexity？ And there is no empirical results of the acceleration. (2) The paper mentions it’s the first using data relation graph on the feature space. The “first” points to “relation graph” or “feature space”? Are there any other methods using graph to detect error data and need to be compared with?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

(1) Although extensive experiments on real-world datasets corroborate the effectiveness of the proposed method, there is lack of theoretical analysis of the effectiveness of the method. (2) Some content lacks detailed explanations.

Summary

The paper under review outlines a novel method utilizing graph structure to identify label errors and outlier data. It proposes an algorithm that makes use of data feature embeddings to produce relation graphs. By incorporating a newly defined data relation function, the algorithm can globally capture mislabeled and outlier data. The experiments conducted exhibit a marked improvement in performance compared to extant detection methods.

Strengths

+ Innovation: The paper introduces a novel method, which is an important contribution to the field. This new approach potentially provides a fresh perspective and further insights into the problem at hand. + Clarity and Comprehensiveness: The paper is well-structured and clearly written, making it easy for readers to understand the content. The authors have adequately cited related works, showing a thorough understanding of the existing literature and situating their work appropriately within that context. + Important Problem: The paper tackles an important problem, making its potential impact highly relevant and timely. This problem is pertinent to many real-world scenarios, amplifying the value of the proposed solution.

Weaknesses

+ Unclear Advantage: Despite the introduction of a new method, the paper lacks clear explanation of the motivation and advantages of the chosen approach compared to existing methods. The authors proposed to use the feature similarity ( 'semantic similarity between data points') to help learn a noise-robust classifier, which exploits $P(X)$ to help learn $P(Y|X)$. It shares the same underlying physiology with semi-supervised based methods (e.g., DividMix, ICLR20) or self-supervised based methods (e.g., UNICON, CVPR22). + Lack of Explicit Assumptions: The authors do not clarify the assumptions under which the proposed method is expected to perform well. This lack of clarity could impede understanding and application of the method in practical scenarios. + Unknown Motivation for Kernel Usage: The motivation for using a kernel in the proposed method is not explained. Without this, it's hard to understand the reason behind the choice of a kernel and how it contributes to the method's effectiveness. + Absence of Baseline Comparisons: The paper does not compare the proposed method with significant baselines in the field of learning with noisy labels (e.g., DividMix, ICLR20; ELR, NeurIPS20; CausalNL, NeurIPS21; C2D, WACV23; UNICON, CVPR22). Such comparisons are crucial for evaluating the performance of the new method and understanding its standing relative to existing techniques.

Questions

+ Could you elaborate on the unique advantages of your proposed method over existing semi-supervised and self-supervised techniques, especially considering the shared approach of using feature similarity to help learn a noise-robust classifier? What distinguishes your method from others that also exploit $P(X)$ to help learn $P(Y|X)$? + Could you specify the assumptions under which your method is expected to perform well? + Could you provide the reasoning behind the choice of using a kernel in your method? How does the kernel contribute to the effectiveness of the method and why was this specific kernel chosen over potential alternatives? + To convince others about the method's practical significance, could you include a comparison of your method with DividMix (ICLR20) and UNICON (CVPR22) in the field of learning with noisy labels?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I did not come across a sufficient discussion of the limitations of the proposed method. I would highly recommend the authors include a section on potential limitations in their revision.

Discussion period

Dear Reviewers, I would like to express my sincere gratitude for your thorough examination of this paper. Now that the authors have provided their rebuttal, I kindly ask you to evaluate whether their response sufficiently addresses the concerns you have raised. Should you require any additional information or have further questions, please feel free to request clarification directly from the authors. Your insights and contributions to this process are greatly appreciated! Best regards, AC

Awaiting Your Feedback on Authors' Rebuttal

Dear Reviewer Yk3p, Thank you for your hard work. The Author-Reviewer discussion ends on August 21. The authors and I are eager to learn whether their responses have adequately addressed your concerns. You are encouraged to directly reply to the authors' rebuttal. Please note that this is a public thread. If you prefer to reply to me individually, please use the internal discussion thread. Kind Regards, AC