Summary
This paper presents GraphACL, which aims to tackle limitations of other graph contrastive learning works which have implicit or explicit homophily assumptions, and suffer in learning effective representations for heterophilic tasks. The approach is designed to leverage the principle of monophily, and the authors evaluate their work on several homophilic and heterophilic datasets, achieving at-par performance on the former and improved performance on the latter, on downstream node classification tasks.
Strengths
- The approach presented here is intuitive once the monophilic principle is understood, and the implementation seems straightforward.
- Results suggest the approach is strongly effective (e.g. Table 1), and hte improvement on heterophilic datasets without any material performance loss / delta on homophilic datasets is compelling.
- The paper is well-written, and the figures help clarify the principles motivating the design.
Weaknesses
- Lines 69-80 could greatly benefit from a toy example with nodes illustrated; it is confusing to understand the principles of monophily and the design of GraphACL without such an example early in the paper.
- "we are faced with the challenge of simultaneously capturing the one-hop neighborhood context and monophily in the contrastive objective" (line 74) -- it's not clear why this is a particular challenge; is there something about capturing these two together that is difficult to reconcile in an objective? If so, I didn't understand this from the text.
- Lines 40 and 97 rely quite a bit on the (in paper) reference [22] to reference issues with contrasting view-based GCL approaches on heterophilic graphs; it would be helpful to include some more formal claims or concrete examples in this work to make it more self-contained, since the work strongly seems to draw motivation from [22]'s findings. Without this, it is difficult to fully appreciate the supposed homophilic biases in those GCL approaches.
- The section in Related Work from line 137 onwards seems to categorize BGRL as a contrastive method. BGRL is not conventionally contrastive due to its lack of utilizing negative examples. There are other such non-contrastive methods (several mentioned in [1]) which are used in graph SSL which should probably be mentioned in the related work in this paper. In fact, I believe the authors' intent to pursue a graph SSL approach which does not use augmentations but instead uses induced asymmetry is actually the objective of many "non-contrastive" methods. I would encourage they look into the literature and evaluate whether they indeed think their method is contrastive or not, given it's lack of need of negative samples.
- The words "context" and "preference" are used quite often throughout the paper (I think sometimes to mean the same thing). It would be helpful to formalize these words and just use the same word frequently, as it can be confusing to disambiguate which representation is which.
- Can the authors clarify if/why the loss in Eqn 4 is required? Methods like SimSiam, BYOL, BGRL etc. which use this kind of predictor structure and stop-gradient show they avoid collapse through the asymmetric predictor and the EMA procedure on target weight update. Does GraphACL not naturally avoid collapse for the same reasons? Does it require this uniformity regularization?
- Figure 3 could go earlier to help introduce monophily and the intuition behind what existing GCL methods prioritize and what this method prioritizes.
- There are a few very strong self-supervised GNN approaches missing from baselines, e.g. [2] or the approach mentioned in [1].
- There are opportunities to better evaluate the quality of self-supervised representations using link-level tasks (e.g. link prediction), or several of the tasks mentioned in [2], which may offer improved understanding beyond node classification and node clustering (referenced in Sec 6.1)
- Evaluation on some larger datasets with greater data diversity and lower homogeneity would have been more compelling, e.g. ogbn-arxiv or ogbn-products. Flickr might also be a useful moderate-size heterophilic dataset to try out.
- The node clustering results should be included in the main paper in my opinion -- the evaluation on just 1 downstream SSL task is fairly "light" and could use greater support to evidence that GraphACL is a generally strong SSL method, rather than "SSL for node classification" method. This is one of the biggest gripes in this paper -- the results demonstrating that GraphACL is a very effective SSL method are sparse, with only 1 demonstrated task in the main (evaluated) paper in terms of node classification. Other graph SSL work typically considers several downstream tasks and evaluates across them.
[1] Link Prediction with Non-Contrastive Learning (Shiao et al, ICLR'23)
[2] Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization (Ju et al, ICRL'23)
Questions
Please see "Weaknesses" for general comments / concerns which would be helpful during rebuttal.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
These are addressed in Appendix E.