HC-GAE: The Hierarchical Cluster-based Graph Auto-Encoder for Graph Representation Learning

Summary

This paper proposes a novel Hierarchical Cluster-based Graph Auto-encoder (HC-GAE) for unsupervised graph representation learning. HC-GAE can reduce the over-smoothing problem and generalize to multiple downstream tasks.

Strengths

1. The motivation is clear and easy to understand. 2. The proposed method seems to be simple and effective. 3. This paper is easy to follow.

Weaknesses

1. The novelty is limited. (1) Both the motivations are not novel. Over-smoothing is a classic problem on GNNs and has been discussed for many years. Multi-task ability has also been discussed in previous literature [1], which has much better multi-task performance than the proposed method. (2) The method is not very novel either. The method is just a combination of existing methods (DiffPool, VGAE) with simple modification. 2. One of the motivations of this paper is to enhance the performance of GAEs for multiple downstream tasks, and the authors also emphasize it for the proposed method. Thus the authors are encouraged to conduct experiments on link prediction task, which is also very important in graph learning and very different from node classification and graph classification. If HC-GAE is not designed for this task or cannot perform well on it, the authors should discuss the limitation on downstream tasks and highlight the discussion. 3. The loss function requires to compute multiple KL-divergence, and may require a lot of time to compute. The authors are encouraged to analyze the time and space complexity, and also conduct experiments to show the computational cost. 4. The baselines are not enough and kind of old. The authors are encouraged to include more recent baselines, including auto-encoder based methods (e.g., GraphMAE2 [2]) and contrastive learning based methods (e.g., CCA-SSG [3], GGD [4] and [5]). 5. The authors are also encouraged to include experimental results on more datasets. 6. There is no sensitivity test on the hyper-parameters. [1] Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization. In ICLR 2023. [2] GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In WWW 2023. [3] From Canonical Correlation Analysis to Self-supervised Graph Neural Networks. In NeurIPS 2021. [4] Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination. In NeurIPS 2022 [5] SEGA: structural entropy guided anchor view for graph contrastive learning. In ICML 2023.

Questions

Please refer to "Weaknesses".

Rating

3

Confidence

4

Soundness

1

Presentation

3

Contribution

1

Limitations

Please refer to "Weakness 2".

Summary

The paper presents a Hierarchical Cluster-based Graph Auto-Encoder (HC-GAE) for improved graph representation learning. HC-GAE uses hard node assignment for encoding and soft node assignment for decoding; thus, it enables hierarchical compression and expansion of graphs. The authors argue their method can reduce the over-smoothing effect by limiting message-passing to individual subgraphs. HC-GAE effectively improves performance in node and graph classification tasks on real-world datasets.

Strengths

The paper is well-written and includes all the necessary details. The proposed model solves a significant problem in the graph representation community and numerical results are promising.

Weaknesses

The code should be released at the review stage to check reproducibility. Especially for empirical work, releasing codes is a prerequisite for acceptance to me. The proposed HC-GAE is similar to existing models that employ graph auto-encoders with hierarchical (multi-level, multiresolution) pooling (or coarsening). The authors should include a discussion comparing HC-GAE with these models, both qualitatively and quantitatively. Without this discussion, the paper's claim of novelty cannot be highlighted. 1. Graph U-Nets 1. Mesh Variational Autoencoders with Edge Contraction Pooling 1. Masked Graph Auto-Encoder Constrained Graph Pooling 1. GRAPH AUTOENCODER FOR GRAPH COMPRESSION AND REPRESENTATION LEARNING 1. Multiresolution equivariant graph variational autoencoder 1. Hi-GMAE: Hierarchical Graph Masked Autoencoders It is unclear whether the over-smoothing effect does not exist in HC-GAE truly, as there are no experimental results against the number of layers. If GNN is applied to the coarsened graph, message-passing between coarsened nodes will occur, potentially leading to over-smoothing. To support their argument, the authors should empirically validate the claim that HC-GAE reduces over-smoothing (i.e., how performance varies by the number of layers).

Questions

- I think “compressing procedure during the decoding process (line 9 – 10)” should be “compressing procedure during the encoding process”. Is it a typo?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper develops a novel GAEs, namely the Hierarchical Cluster-based GAE (HC-GAE) model, to learn effective features for either node classification or graph classification. To extract the bidirectionally hierarchical structural features of the original graph, this paper first utilize the hard node assignment to transform the original graph into a family of coarsened graphs, and then utilize the soft assignment to reconstruct the original graph. During the encoding process the convolution operation is restricted within each separated subgraph, so this HC-GAE can address the shortcoming of over-smoothing problem. This new model has superior performance on both node classification or graph classification tasks.

Strengths

1.The idea of this work is interesting, and the bidirectionally hierarchical structure learning based on the adaptive node assignment seems novel to me. 2.The proposed HC-GAE model is flexible for either node classification or graph classification. 3.The paper is clearly written and easy to follow, and the experimental results also demonstrate the effectiveness of the new HC-GAE model.

Weaknesses

Although this paper introduces a novel graph representation learning method, but some problems still need to be addressed or be clearer. 1.Why the HC-GAE model utilizes the hard and soft assignment for the encoder and decoder respectively? As I see, the author can only use any of these assignment strategies, right? 2.If the hard assignment can help the new model to reduce the over-smoothing problem, how about the soft-assignment? 3.Why the reconstructed features are effective for node classification? Why do not use the middle features? 4.Although this paper is clearly written, but the writing of Section 3 can be more polished.

Questions

See the weakness raised above.

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

yes

Summary

The authors propose a new GNN-based representation learning method (HC-GAE), that can abstract effective local node features and global graph features. These features can be used for both node and graph classification. The new HC-GAE method consists of two main computational modules, they are the encoder associated with the hard node assignment, and the decoder associated with the soft assignment. Moreover, the deep representation in the middle layer can be seen as the global graph features, and the output in the last layer can be seen as the local node features. All these features encapsulate bidirectionally hierarchical structural features of the original sample graph based on the hierarchical strategy. Finally, the authors also propose a new loss function for integrating the information from either the encoder or the decoder.

Strengths

1. The idea of this work seems novel and interesting for me, performing the local convolution operation within the separated subgraphs through the hard node assignment not only addresses the over-smoothing problem, but also forms hierarchical representation for the graphs. 2. The descriptions are clear, the experiments demonstrate the effectiveness, and the new model is technical sound.

Weaknesses

Overall, the writing is easy to understand, but I see minor typos or grammar mistakes in Section 3 and 4. I didn’t check these in details, the authors should carefully correct them for the final manuscript. I feel a little strange for the loss function. Because the function tends to minimize the differences of either the structures or the node features between the input (for encoder) and the output (decoder). As I see, if the authors use the output for the node classification, why you do not directly use the original input node features? Or, foe the node classification you eliminate the effects from the node features for the loss function. This is not very clear. Appendix B, what do you mean for Ghazan? I didn’t see any explanation of this word. Is it a kind of alternative method or others? The authors discuss some theoretical advantages of the proposed method in Sec 3.4. But they don’t discuss any reason about why the proposed method performs well in Sec 4. They just simply show the accuracies are better. Some more analysis is needed.

Questions

See the questions I asked in weakness.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes