Improvements on Uncertainty Quantification for Node Classification via Distance-Based Regularization

Summary

This paper focuses on uncertainty quantification in node classification. The authors examine the drawbacks of UCE. Inspired by our theoretical analysis, they introduced a distance-based regularization technique to obtain a better representation network for the uncertainty quantification. Experimental results show the superior performance of the proposed approach compared to the current state-of-the-art methods in out-of-distribution (OOD) detection and misclassification detection.

Strengths

1. This paper is well motivated, it discusses the limitations of GPN model on semi-supervised node classification tasks, and it provides detailed theoretical analysis to reveal the limitations of GPN for detecting OOD nodes 2. This paper proposes a distance regularizer to overcome the above limitation. 3. This paper provides extensive experimental results that demonstrate that the proposed method can achieve state-of-the-art performance on both out-of-distribution (OOD) detection and misclassification detection.

Weaknesses

This paper is not in my area, I'm not familiar with existing work about both node classification and uncertainty qualification. So I cannot assess the contribution of this work reliably.

Questions

The authors should better discuss the limitation of their work.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This manuscript studies the uncertainty quantification for the non-independent node-level prediction problem. The proposed framework is built upon the graph posterior network (GPN) with minimizing uncertainty cross-entropy as the main loss function. A distance-based regularization technique is further proposed to learn representational space mappings in order to handle OOD samples.

Strengths

The topic of quantifying the uncertainty of GNN predictions is timely and interesting. The overall presentation is easy to follow. I checked the overall theoretical contributions and haven't found major flaws. The proposed two regularization terms are validated by extensive experiments.

Weaknesses

I am not seeing major drawbacks, except there are some points related to the problem definition.

Questions

1. How can Eq. (1) and (2) derive the aleatoric uncertainty and epistemic uncertainty? The authors do not mention the rationale behind this derivation, which makes it hard to understand the whole process.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I am not seeing major drawbacks.

Summary

The paper proposes a distance-based regularization to improve the OOD detection and misclassification detection tasks on the graph node classification. The regularization enforces the clustered OOD nodes to be close in the latent space.

Strengths

- The paper includes comprehensive experiments on multiple commonly used benchmark dataset and detailed ablation study. - The paper provides theoretic analysis on the limitation of the GPN.

Weaknesses

- Equation 2 defines aleatoric and epistemic uncertainty. Why are they defined in this way? Is there any reference or reason? - The writing is not clear. In section 4.1, there are a lot of theorems, but a sentence is missing to summarize what is the limitation of the UCE. Although at the beginning of section 4, there is a summary of those theorems, but it fails to present the overall idea of what the limitation is. Also, it is hard to connect Section 4.1 to 4.2. Why would the limitation of UCE loss mentioned in section 4.1 motivate the distance minimization on the graph? - The proposed model improves AUPRC but has worse AUROC. It is not easy to tell whether the model is better. How does the model manage to improve AUPRC by 50% while remaining AUROC unchanged in Table 2? How could a model with 95% AUPRC has only 80% AUROC? Could the authors plot out the ROC and PR curves?

Questions

The authors should improve writing in the revision by providing more contexts on the claims and definitions. It is also questionable on the results in the Table 2. I hope the author can clarify that.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

3 good

Limitations

Yes, the authors addresses limitations in the theoretic analysis.

Summary

## Post Rebuttal Update I have engaged with the authors for the rebuttal, and found their responses informative, prompting me to increase my score from a 7 to an 8. ## Original Review The paper proposes theoretical results for Graph Posterior Networks (GPNs) that use the uncertainty cross-entropy loss (UCE), and shows that it is possible to trivially set the uCE loss to zero under certain conditions. The paper then proposes a way to regularise the UCE loss using distance, ensuring that OOD nodes that are clustered together are clustered together in latent space.

Strengths

I like the theoretical analysis section of this paper. It constructs a way of obtaining zero UCE for a pathological example, showing that there are improvements required to be made to the GPN+UCE framework. Furthermore, it incorporates the fact that OOD points are not explicitly modeled in the UCE framework, and suggests ways to incorporate this. I also find the baselines considered pretty comprehensive, and the authors have done a good job covering a range of techniques. I also like the further ablations the paper has, such as selecting activation functions as well, using a validation CE loss, and I like that this has been properly ablated against. In general, I found the paper well motivated and written. I went through the proofs of Theorem 1-6 and found them accurate to my admittedly limited knowledge.

Weaknesses

I have a few criticisms of the paper that I would like to get addressed - 1. I would like a thorough description of all the assumptions made for Theorem 1, and where these assumptions would be violated or not, maybe with a toy example. For example, I'm not sure how strict the assumption "If the underlying distribution of feature vectors belonging to class k, denoted by $X_k$ disjoint to each other," is, and would like a discussion of the drawbacks/limitations of the assumptions made in Theorem 1. 2. Is the ball $B\left(\mathbf{z}_k, r_k\right)$ guaranteed to have finite volume necessarily? There are pathological behaviours in neural networks that use ReLU activations, where it is possible to obtain high softmax scores (i.e. class probabilities) arbitrarily far from the training manifold [1], and this would make the volume of the ball very large potentially in some cases. [1] https://arxiv.org/abs/1812.05720

Questions

In Table 2, it seems like the regularisation consistently gives lower AUROC but higher AUPR compared to the GPN baseline. Can you explain why this might be the case?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A.