Deep Insights into Noisy Pseudo Labeling on Graph Data

Summary

This paper aims to provide in-depth insights into pseudo labeling (PL) in the context of graph learning models. The authors first present an error analysis of the PL strategy, demonstrating that the error is bounded by the confidence threshold of PL and the consistency of multi-view predictions. Furthermore, they theoretically illustrate the impact of PL on convergence properties. Building upon this analysis, they propose a careful pseudo labeling methodology that involves assigning pseudo labels to samples with the highest confidence and multi-view consistency. Finally, extensive experiments demonstrate that the proposed strategy enhances the graph learning process and outperforms alternative PL strategies in link prediction and node classification tasks.

Strengths

1. This paper theoretically analyzes how the PL strategy affects convergence properties in GNN. 2. Based on the analysis, this paper proposes a cautious pseudo labeling methodology, and extensive experiments demonstrate the effectiveness of the proposed strategy.

Weaknesses

1.There might be some mistake in Theorem 2.5. since the covariance between the cross-entropy loss and PL strategy and $\beta$ is non-negative, the inequality $\beta Cov + L \leq L$ holds only if $\beta Cov = 0$. 2.It might be better to clearly illustrate $q(t)$ in Figure 2, which demonstrates the framework of the algorithm. 3.It is suggested to remove the first picture in Figure 1 and add two more pictures to show the performance on node classification. 4.The word ‘noisy’ in the title is redundant for pseudo labeling. Furthermore, the whole paper rarely mentions ‘noisy PL’.

Questions

How much time does the proposed method consume on the dataset like PubMed?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Some figures could be further polished up, and check the correctness of the theorem.

Summary

Pseudo labeling is significant for GNN. This paper first theoretically analyzed the effect of pseudo labeling by showing the error bound and the convergence property. Then, accordingly, the paper proposes a cautious pseudo labeling based on confidence and multi-view consistency. The experimental results demonstrate the effectiveness of the proposed cautious pseudo labeling strategy.

Strengths

The paper is well written. The analysis of Pseudo labeling for GNN is valuable.

Weaknesses

However, the analysis or the conclusion is general to all fields instead of highly related to the graph. Besides, the solutions, including high confidence and multi-view consistency, are not that novel. More experiments are required. The time complexity should be provided when comparing with previous methods. Performance comparison on link prediction should also contain the previous PL methods.

Questions

1) The paper is well written. The analysis of Pseudo labeling for GNN is valuable. However, the analysis or the conclusion is general to all fields instead of highly related to the graph. Besides, the solutions, including high confidence and multi-view consistency, are not that novel. 2) Some contents may contain errors. For example, the results of Figure 1 may contain some errors. When the samples of PL and CPL are equal to/similar to 0, the GAE/GAE+PL/GAE+CPL should have equal/similar performance. 3) More details need to be provided. What are the details of multi-view teachers? Does it refer to multi teachers or one teacher with multi-data-augmentation inputs? If the multi-view is obtained based on the data augmentation, what types of data augmentation are used? What is the data augmentation in Figure 4? 4) More experiments are required. The time complexity should be provided when comparing with previous methods. Performance comparison on link prediction should also contain the previous PL methods. 5) Other questions. In Algorithm 1, the student was fine-tuned based on the teacher model. I wonder whether there exists overfitting since the observed data set has been put into the model again and again. In 3.1, I wonder whether the data from the testing would be selected by CPL to be labeled for training. I wonder whether CPL may reduce the performance of nodes of low-frequently labeled classes.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There is no potential negative societal impact.

Summary

The paper provides an error bound for pseudo labeling on graphs. Moreover, the authors propose a cautious pseudo labeling method and validate it through experiments.

Strengths

1. The paper presents good experimental results, demonstrating the effectiveness of the proposed cautious pseudo labeling method. 2. The authors have conducted comprehensive experiments 3. The error bound proposed in the paper is deemed useful and practical

Weaknesses

The paper suffers from unclear notation and contains numerous typos. These issues hinder the reader's understanding and make it challenging to follow the presented ideas. Clarifying the notation and addressing the typos is necessary to improve the clarity of the paper.

Questions

1. Line 86: For "err(g)", does it mean the expectation is taken over the test points? Please clarify this point. 2. Line 91: Is the GPI property introduced in this paper, or has it been introduced previously? Please provide the necessary context for understanding this property. 3. Line 104: When stating "whose probability is higher than a threshold," does it mean this condition must hold for each "y ∈ U"? Please clarify this point. 4. Line 105: Since "g(G) ∈ ℝ^(N×M)", what do you mean by "ŷ ∈ g(G)"? Please explain this notation. 5. Line 102: Do you mean we can find "p_f(⋅)" or we can find "α, η"? Please clarify this statement. 6. Line 108: In "p_{αf}", there should be no "α" here. Please correct this notation. 7. Line 122: When stating "the teacher predictor satisfies additive expansion," the additive expansion is defined for a probability density, not a classifier. Please provide clarification and ensure consistency in terminology. 8. Line 124: Could you explain what is meant by "E_{Y_test}"? 9. Line 145: The first term should change "t" to "t+1". 10. Line 147: What is the definition of the covariance term in this context? Please provide clarification. 11. Lines 163-164: There seems to be a typo in the sentence, "we calculate multi-view prediction of the by." Please clarify this phrase. 12. Do you have any formal proof of Theorem 2.5? I am willing to increase my score if these concerns are resolved.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

1 poor

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations

Summary

The article discusses noisy pseudo labeling (PL) on graph data and proposes a new cautious PL methodology (CPL) to improve the graph learning process. The authors conduct experiments to evaluate CPL strategy for link prediction on various datasets and apply it on popular models in node classification task. The result shows that the proposed strategy outperforms other PL strategies. The paper also provides a theoretical analysis of the impact of noisy labels introduced by PL on the graph training procedure. 1. The error introduced by PL is bounded by the confidence of PL threshold and consistency of multi-view prediction. 2. PL can be designed to contribute to the convergence property.

Strengths

The paper makes a highly original and significant contribution to graph learning. It proposes a new cautious Pseudo Labeling (PL) methodology that addresses limitations in prior works by introducing a confidence threshold and a consistency criterion for selecting high-confidence PL samples. This methodology, combined with a new consistency-based PL (CPL) strategy, improves the convergence property of graph learning and outperforms other PL strategies in link prediction and node classification tasks. The research methodology is rigorous, and the paper is well-structured, clear, and provides practical solutions to the challenges of limited and noisy labeled data.

Weaknesses

The paper could be strengthened by addressing several weaknesses. Firstly, in Table 2, the authors can also compare with other PL methods in the link prediction task. Secondly, conducting experiments with larger sample sizes or exploring different configurations would provide a more comprehensive evaluation. Thirdly, clarifying the methodology by providing implementation details and explaining data preprocessing steps would enhance replicability and understanding. Finally, considering the applicability of the proposed approach to different domains with highly imbalanced class distributions or datasets with different types of noise would broaden its practical relevance.

Questions

1. Were there any specific assumptions made when applying the proposed approach to the benchmark datasets? It would be helpful to understand the compatibility of the approach with different dataset characteristics, such as class imbalance or noise types. Insights into these considerations would shed light on the generalizability of the approach. 2. Could you provide additional information on the hyperparameters used in the experiments? Specifically, how were the hyperparameters set for the proposed cautious Pseudo Labeling (PL) methodology and the baseline models? Sharing these details would aid in replicating and fine-tuning the approach in future research. 3. In the discussion of results, could you provide further insights into the potential limitations or failure cases of the proposed approach? Understanding the scenarios where the approach may not perform optimally would help in setting realistic expectations and identifying areas for further improvement.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not mention their limitations.

Rebuttals are visible now

Hi reviewers, Please take a look at the rebuttals when you have some time. Thanks. AC

About the rebuttal

1. The authors have addressed most of my concerns. 2. The authors mention that the covariance term, denoted as Cov, is never greater than 0 in the rebuttal. However, in Page 5, line 184-185, they say that "the second inequality holds because the covariance between the cross-entropy loss and PL strategy is non-negative". Does there exist a mistake?

Post-rebuttal comment

Thank you for the detailed explanations. I am willing to champion this paper.