Towards Label Position Bias in Graph Neural Networks

Graph Neural Networks (GNNs) have emerged as a powerful tool for semi-supervised node classification tasks. However, recent studies have revealed various biases in GNNs stemming from both node features and graph topology. In this work, we uncover a new bias - label position bias, which indicates that the node closer to the labeled nodes tends to perform better. We introduce a new metric, the Label Proximity Score, to quantify this bias, and find that it is closely related to performance disparities. To address the label position bias, we propose a novel optimization framework for learning a label position unbiased graph structure, which can be applied to existing GNNs. Extensive experiments demonstrate that our proposed method not only outperforms backbone methods but also significantly mitigates the issue of label position bias in GNNs.

Paper

Similar papers

Peer review

Reviewer tP9u5/10 · confidence 3/52023-06-25

Summary

This paper presents a study on Label Distance Bias in Graph Neural Networks (GNNs). The authors investigate the performance disparity across various GNN models on three datasets and analyze three metrics related to Label Distance Bias. The study reveals that Label Position Bias is a significant factor affecting the performance of GNNs. The proposed Label Proximity Score consistently exhibits a robust correlation with performance disparity. The findings highlight the importance of further exploring Label Position Bias and developing strategies to mitigate its impact on GNN performance. The paper provides valuable insights into the performance variation of GNNs and underscores the need for addressing Label Distance Bias to enhance the effectiveness of GNN models.

Strengths

This paper offers three key advantages in its study on Label Distance Bias in Graph Neural Networks (GNNs). Firstly, it identifies Label Position Bias as a widespread phenomenon across all GNN models and datasets, highlighting the importance of understanding and addressing this bias in GNN performance. Secondly, the paper introduces the Label Proximity Score as a metric that consistently shows a robust correlation with performance disparity. This score proves to be an effective measure of Label Position Bias, outperforming Degree and Shortest Path Distance metrics. Lastly, the study demonstrates that the severity of Label Position Bias is more prominent at lower labeling rates, but becomes less noticeable when 60% of nodes per class are labeled. This finding provides insights into the impact of labeling rates on bias and suggests strategies for mitigating its effects.

Weaknesses

Although your paper presents well and the proposed method seems to be very targeted towards relevant issues, I cannot determine the actual performance of your work as you have not provided any code.

Questions

1. Can the proposed method effectively mitigate label position bias in datasets with imbalanced class distributions? 2. What are the limitations of the selected baseline models (GCN, GAT, APPNP, MLP, Label Propagation) in addressing label position bias? 3. How does the proposed method compare to existing methods in terms of computational efficiency? 4. Can the proposed method be applied to larger and more complex datasets, and if so, how does its performance scale?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Reviewer fFnJ6/10 · confidence 5/52023-06-28

Summary

This paper demonstrates a new bias in GNN models called label position bias. A metric called the Label Proximity Score is introduced to quantify the bias. The paper proposes a framework to learn an unbiased graph structure, which can be applied to existing GNN models.

Strengths

Label position bias is an interesting phenomenon uncovered by the authors. This has important fairness implications in GNN model training.

Weaknesses

1. The label position bias is demonstrated empirically. It would be good if the authors can give an explanation for why this phenomenon occurs. Is this issue present in *all* GNN models inherently? 1. There are no statistical significance tests done to verify that the observed label position bias is indeed statistically significant. 1. More insights can be gained if the LPSL unbiased B can be tested using more diverse GNN models.

Questions

1. In the LPSL problem in (2), is there any constraint that the entries of B have to be non-negative? How do you interpret negative entries? 2. The constraint in (2) mimics (1) but the optimal solution B is not equal to P (unless the constraint is absent). How does B mitigate the LPS issue identified in Figure 3? There are no experimental results (taking the optimal B found as the adjacency matrix) showing this, which is the motivation for the paper. 3. The definition of the influence I_i(j) is incorrect. From [23], it should be the sum of the absolute values of the entries of the Jacobian. How does this affect the proof in the appendix? Furthermore, it is difficult for the reader to guess what "sum" notation means. 4. Same issue in (5) as Q1. 5. Is label position bias apparent in heterophily graph datasets like Chameleon and Squirrel?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N.A.

Reviewer Lj4a5/10 · confidence 5/52023-07-01

Summary

This paper pointed out that message-pass based GNNs training usually suffers from label position bias problem for semi-supervised node classification especially for quite few-shot settings. That is, the unlabeled nodes which far away from the labeled ones might suffer from performance degradation. The paper first proposed a new metric named “Label Proximity Score” based on PPR matrix to measure label position bias, and then put forward a label unbiased graph learning method by learning a dense graph structure to guide the graph representation learning, the dense matrix will mitigate the label position bias.

Strengths

1.Induced bias in GNNs for graph representation learning is one challenge problem which affects the fairness or robustness of down-streaming supervised learning tasks. The paper first found out the phenomenon that label position bias in GNNs (especially for one-hop neighbor message passing based GNNs) for node classification task. A quantified metrics is proposed to measure label position bias. 2.A position unbiased graph representation learning framework is proposed by learning a dense matrix to guide feature aggregation to mitigate the position bias. Meanwhile, the paper also gave analysis of the induced dense matrix to connect with existing PPR based APPNP.

Weaknesses

1.Lack of the analysis of the root to cause label position bias in GNNs over node classification. For example, the nodes being far away from the labeled ones requires deep GNNs to receive label information. The performance degradation might due to shallow GNNs with limited receipt field or deep GNNs suffering from over-smoothing or over-squashing. 2.More backbone GNNs should be evaluated to support the phenomena of Label Position Bias. GCN is a kind of one-hop neighboring based message passing which might suffer from over-smoothing to multi-hop neighbor information aggregation. Although APPNP can aggregate multi-hop neighbor information, it is sensitive to selection of iteration number K and re-start probability parameter alpha. It is better to select some GNNs (such as JKNet, Multi-hop Attention Neural Networks etc.) which is robust to over-smoothing or other dense matrix based GNNs (such as Graph Transformer)

Questions

1.Introducing l1 norm will sparse the induced matrix B and increase the computation efficiency, however, it seems that sparsity also diffs different nodes. Does sparsity make a tradeoff to relieve label position bias? Does this sparsity find better aggregation patterns? Does unbiased graph structure mean a dense induced matrix? 2. How to get the sensitive groups with respect to different bias metrics (Degree, Shortest Path Distance and Label Proximity scores)? Is the distribution of labeled node number balance among different groups? 3.The performance of GNNs over graph with few node labels heavily depends on the graph split, how to split the graphs to conduct experiments in this paper?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

1.Backbone GNNs are limited and lack of deep insight of the cause of label position bias. 2.Some important references are missing. For example, it is noted that GNNs might be not sensitive to position representation learning (Position-aware Graph Neural Networks). This is another kind of position bias, what is the connection of label position bias to such position embedding learning?

Reviewer VK135/10 · confidence 4/52023-07-06

Summary

This paper presents a new bias problem on graph neural networks: label position bias, which implies that nodes perform better closer to labeled nodes. A new metric - the Label Proximity Score(LPS) is proposed to quantify this bias, and a new optimization framework is proposed to learn a new unbiased graph structure. This framework motivates all nodes to have the similar LPSs and is experimentally shown to alleviate the label position bias problem and improve model performance.

Strengths

1) This paper mines a novel problem on GNNs: label position bias. This is a valuable problem, which can help us better understand the role of labeled nodes for test nodes. 2) The newly proposed metric label proximity score can measure the importance and influence of test nodes and labeled nodes, and the experiments in the paper show that certain trends can be identified by the label similarity score compared to the degree and shortest path distance. 3) The extensive experimental evaluation shows that LPSL not only outperforms standard methods but also significantly alleviates the label position bias in GNNs. This work could stimulate further research and development of methods aimed at enhancing label position fairness in GNNs.

Weaknesses

1) The proposed label position bias problem is similar to Topology-Imbalance Node Representation Learning, which has not been mentioned in related works, and I do not understand the difference between the two problems. Chen, Deli, et al. "Topology-imbalance learning for semi-supervised node classification." Advances in Neural Information Processing Systems 34 (2021): 29885-29897. 2) In the design of the framework only the new unbiased graph structure is calculated and its contribution may not be sufficient enough to even do ablation experiments to verify the role of the different parts. 3) The authors believe that the graph structure can solve the label position bias problem by ensuring that the LBS of each node is similar in the proposed framework, in order to fairness of all nodes. But there is confusion about this, is it possible that unfair LBS will get better results? Each node has different structural characteristics and has different relationships with labeled nodes. For example, for nodes with high degree and strong association with labeled nodes, or nodes with high degree and weak association with labeled nodes, is it necessary to distinguish between the former type of nodes, whose labels are more credible. Does the model get better performance if the degree or the shortest path distance is taken into account together with the label position bias problem? Typos: 1) Reuse of symbols. The lowercase letter c is defined as the number of classes in Section 2, which in turn is defined as the hyperparameter serving as the uniform Label Proximity Score in Equation 2. 2) In line 312 the reference of the figure should be 4 instead of 4.4.

Questions

Please see the above

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

In the paper, the authors mention that the existing methods may not scale to heterophily graphs.

Reviewer fFnJ2023-08-11

Thank you for your clarifications. To ensure correctness, I will require that the non-negativity constraint is added to the problem formulation (2) and the optimization steps (3)-(4) be updated to incorporate this constraint.

Authorsrebuttal2023-08-11

Dear Reviewer fFnJ, We sincerely appreciate your timely feedback. In light of your suggestions, we will incorporate the non-negativity constraint in all related formulations during our revision. Besides, for the experiments results, we've verified that the matrix B under optimal hyperparameters does not contain any negative entries. Furthermore, we recognize that our response in **A4** might have lacked clarity. To elucidate, in Answer 4, 'Case 1' and 'Case 2' refer to two distinct sets of hyperparameters. The terms 'With/Without Negative' are indicative of the presence or absence of the non-negativity constraint under identical hyperparameters. Thank you once again for your invaluable insights. If you have any further questions, please kindly let us know. Best, All authors

Authorsrebuttal2023-08-13

Further Response about the Non-negativity Constraint

Dear Review fFnJ, We would like to provide further details and discussions about the non-negativity constraint. In this response, we will first elaborate on how we integrated this constraint into our model. Subsequently, we will present findings to demonstrate that the absence of this constraint has not influenced our experimental results. With the incorporation of the non-negativity constraint, the optimization problem is described as: $$ \begin{aligned} & \underset{\mathbf{B}}{\arg \min }\|\mathbf{I}-\mathbf{B}\|_F^2+\lambda \operatorname{tr}\left(\mathbf{B}^{\top} \tilde{\mathbf{L}} \mathbf{B}\right) \\\\ & \text { s.t. } \quad \mathbf{B} \mathbf{T} \mathbf{1}_n=c \mathbf{1}_n, \quad\mathbf{B}\ge0 \end{aligned} $$ To tackle this problem, we employed the Lagrange Multiplier method coupled with projected gradient descent. Specifically, the iterations for $\mathbf{B}$ are: $$ \begin{aligned} & \mathbf{B}^{k+1}=\underset{\mathbf{B}}{\arg \min } L_\rho\left(\mathbf{B}^k, \mathbf{y}^k\right) \\\\ & \mathbf{B}^{k+1} = max(0, \mathbf{B}^{k+1}) \\\\ & \mathbf{y}^{k+1}=\mathbf{y}^k+\rho\left(\mathbf{B}^{k+1} \mathbf{T} \mathbf{1}_n-c \mathbf{1}_n\right), \end{aligned} $$ where $L_\rho(\mathbf{B}, \mathbf{y})=\|\mathbf{I}-\mathbf{B}\|_F^2+\lambda \operatorname{tr}\left(\mathbf{B}^{\top} \tilde{\mathbf{L}} \mathbf{B}\right)+\mathbf{y}^{\top}\left(\mathbf{B} \mathbf{T} \mathbf{1}_n-c \mathbf{1}_n\right)+\frac{\rho}{2}\left\|\mathbf{B} \mathbf{T} \mathbf{1}_n-c \mathbf{1}_n\right\|_2^2$ is the augmented Lagrange function. From previous discussions, we've made several observations: 1. Only when the hyperparameter **c** is extremely small does $\mathbf{B}$ contain a few minor negative entries. 2. As evidenced in **A4**, introducing the non-negativity constraint has a negligible impact on performance under the same hyperparameter settings. 3. In our experiments, the optimal $\mathbf{B}$ was found to be non-negative under the best hyperparameters. We further test whether the non-negative $\textbf{B}$ would be changed after applying the non-negativity constrain. Here is the finding: 4. For non-negative $\textbf{B}$ derived without the non-negativity constraint, reintroducing the constraint didn't alter $\textbf{B}$, ensuring consistent performance. These findings indicate that the absence of the non-negativity constraint has not substantially influenced our experimental results. Of course, the non-negativity constrain is essential, and we will add it into our revision. Thanks again. We hope this provides clarity on the matter. If you have any further questions, we are happy to discuss. Best, All authors.

Reviewer VK132023-08-15

The reviewer appreciates the responses from the authors, which have addressed most concerns from the reviewer. Further, based on the responses, I want to add a new more question: Since unfair LPS could get better results, why do we need to alleviate label position bias and take it as a metric to show the advantage of GNNs?

Authorsrebuttal2023-08-16

Dear Reviewer VK13: Thanks for your prompt reply and insightful question. The label position bias, where nodes distant from labeled nodes tend to underperform, is a significant concern especially when GNNs are applied in real-world scenarios. For instance, in applications like fraud detection where labeled data might be scarce, users far away from these labeled nodes are at a higher risk of being misclassified due to the label position bias, leading to potential unfairness. Therefore, while it's true that fair LPS doesn't mean better results, it's crucial to ensure fairness across the graph for the broader applicability and trustworthiness of GNNs. There is usually a trade-off between the fairness and model performance [1]. Improving fairness can sometimes come at the cost of performance. However, our experiments on multiple datasets show that our proposed method not only improves fairness but also maintains or even enhances overall performance of various GNNs. We hope this provides clarity on the matter. If you have any further questions, we are happy to discuss. [1] Dong, Yushun, et al. "Fairness in graph mining: A survey." IEEE Transactions on Knowledge and Data Engineering (2023). Best, All authors

Reviewer VK132023-08-18

Dear Authors, I have raised my score due to the better clarification.

Authorsrebuttal2023-08-18

A Gentle Remind to Reviewer Lj4a

Dear Reviewer Lj4a, We would like to express our sincere gratitude to you for reviewing our paper and providing valuable feedback. Could we kindly know if the responses have addressed your concerns? If there are any further questions, we are happy to clarify. Thank you. Best, All authors

Authorsrebuttal2023-08-18

A Gentle Remind to Reviewer tP9u

Dear Reviewer tP9u, We would like to express our sincere gratitude to you for reviewing our paper and providing valuable feedback. Could we kindly know if the responses have addressed your concerns? If there are any further questions, we are happy to clarify. Thank you. Best, All authors

Reviewer Lj4a2023-08-18

Thanks for the detailed clarification from authors. I have carefully read the rebuttal. The response answers most of my concerns, and thanks for additional experiments on more backbone GNNs to verify the universality of label position bias. I decided to raise my score.

Authorsrebuttal2023-08-18

Thanks for your response

Dear Reviewer Lj4a, Thanks for your response and support. We appreciate the insights you have provided and will incorporate them into our revised manuscript. Best regards, All Authors

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC