Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks

Summary

This work found that more data does not necessarily improve the pre-training of GNNs and then propose a method to adaptively select data for pre-training GNNs. Experimental results show the proposed method is effective.

Strengths

1. this work is well motivated by the observation that more data does not necessarily lead to better pretrained graph encoder 2. that the experiments seems to be convincing since APT generally outperforms other baselines

Weaknesses

There are several weaknesses mainly about the method: 1. The optimization problem in equation 5 is confusing, what is the variable you are optimizing, a series of binary variable indicating whether to select a graph or not? Based on description in subsequent part, you are basically selecting graphs with the highest score, in that case it is unnecessary to formulate an optimization problem, just state your scoring function. 2. The description is confusing, eg, what is M in line 296? The number of subgraphs being selected? 3. Efficiency is important in data selection, but the authors do not discuss it in the main body of the paper. I assume the method is slower than baseline pretraining method but how slow is it? 4. The motivation of the other four property criteria is unclear, especially when they are highly correlated with the network entropy. Can we just drop the other four criteria? 5. The authors claim in the conclusion that APT can enhance the model with a fewer number of input data, but I can't find evidence for that. Is APT only trained on fewer data than baselines? Overall, the description of the proposed method could be improved. In addition, the authors claimed they are not curriculum learning but I don't think so. Time-adaptive selection strategy is just curriculum learning in a broader sense, I think easy-first is just one type of curriculum.

Questions

See weakness above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The authors do not state limitations in either the appendix or the main body of the paper, please add them.

Summary

The paper introduced an novel graph pre-training pipeline from a data-centric view by proposing a graph data selector. The proposed approach involved sequentially feeding representative data into the model for pre-training. These representative data were determined by a data selector, which utilized prediction uncertainty and graph statistics to suggest the most relevant examples. This novel methodology offers a fresh perspective on graph pre-training and demonstrates the importance of data selection in the process.

Strengths

1. The idea is novel and interesting. 2. The proposed training pipeline is tested on many tasks from different domains.

Weaknesses

1. The utilization of predictive uncertainty based on pre-training loss lacks convincing evidence. 2. The proposed training pipeline requires input graphs to be processed order-by-order, which may introduce unnecessary bias and is not scalable for large graph datasets.

Questions

First, there are several questions about the predictive uncertainty: 1. The loss used for graph pre-training may not effectively align with the actual loss of the downstream task [1,2]. Consequently, the predictive uncertainty based on the current definition may lack meaningful interpretation. A more promising approach would be to ensure that the uncertainty of the pretraining loss accurately reflects the difference between the pre-training loss and the downstream task loss. 2. Prediction uncertainty has been extensively explored in previous research [3,4]. While many existing methods have been proposed with ground-truth labels, it would be beneficial to have discussions (and even experiments) that shed light on the connection and distinctions between different approches of uncertainty modeling. For experimentds 3. The experimental setups lack clarity. It is unclear whether data from all domains are used to pre-train a single model or if separate models are pre-trained for different domains. Others 4. Please check descriptions in lines 215~217. Smaller gamma seems leading to important roles of the predictive uncertainty term. 5. Choosing subgraphs seems to be a crucial component of the pre-training algorithm. However, the paper lacks a thorough explanation or detailed descriptions of this process, which would greatly improve the understanding of this important aspect. 6. Recently, there are many related work focusing on the data-centric approaches [2,5,6], espeically for graph pre-training [2,5]. It would be great and helpful to readers if the authors could provide a brief discussion on the connections and differences between their work and these existing approaches. Ref. [1] Does gnn pretraining help molecular representation? NeurIPS 2022. [2] Data-Centric Learning from Unlabeled Graphs with Diffusion Model. Arxiv. [3] Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ICML 2016. [4] Single-model uncertainties for deep learning. NeurIPS 2019. [5] Analyzing data-centric properties for graph contrastive learning. NeurIPS 2022. [6] Data-centric ai: Perspectives and challenges. Arxiv.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

See above.

Summary

This paper presents a data-centric framework, abbreviated as APT, for cross-domain graph pre-training. APT is composed of a graph selector and a graph pre-training model. The core idea is to select the most representative and informative data points based on the inherent properties of graphs, as well as predictive uncertainty. For the pre-training stage, when fed with the carefully selected data points, a proximal term is added to prevent catastrophic forgetting and remember all the contributions of previous input data.

Strengths

1.This paper proposes that big data is not a necessity for pre-training GNNs. Instead of training on a massive amount of data, it is more reasonable to select a few suitable samples for pre-training. This approach can also reduce the amount of data and computational costs. Compared to pre-training on the entire dataset, selecting a more carefully selected subset of data for pre-training can indeed achieve better results. 2.This paper provides theoretical justification for the connection between uncertainties by establishing a provable connection between the proposed predictive uncertainty and the conventional definition of uncertainty. The predictive uncertainty is defined in the representation space, which enables the identification of more challenging samples that can benefit the model training more significantly. 3.The entire framework seems very reasonable, and the process is described clearly. From the experiments, it appears that good results have been achieved.

Weaknesses

1.This paper mentions both "data-active" and "data-centric" concepts. It may be helpful to clarify the relationship between these two to avoid confusion. Maybe only using "data-active" in the paper. 2.There is no individual ablation experiment about the graph selector to demonstrate the effectiveness for each of the five graph properties. 3.There is a spelling error in the title, 'Prespective' should be corrected to 'Perspective'.

Questions

Q1 Are all the Graph properties used in this paper useful when dealing with graph-inherent features? Also, what is the computational complexity since multiple properties need to be calculated? Q2 Why did the performance of the "dd" dataset decrease after fine-tuning compared to the frozen parameters of APT in table 2? Q3 I am slightly concerned that if all the five graph properties are useful for pre-training. Q4 In existing works, is there any works that define model uncertainty in the representation space? And what is the difference between theirs and this work?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper proposes a novel approach to pre-training graph neural networks (GNNs) called the data-active graph pre-training (APT) framework. This framework introduces a unique method that involves a graph selector and a pre-training model. These two components work together in a progressive and iterative way to choose the most representative and instructive data points for pre-training.

Strengths

1. The key insight of the paper is that using fewer, but carefully selected data can lead to better downstream performance compared to using a massive amount of input data. This approach challenges the common practice in machine learning of using large datasets for pre-training, a phenomenon the authors refer to as the "curse of big data" in graph pre-training. 2. The paper is well-motivated, well-written and easy to understand. 3. The experimental results demonstrate that the proposed method outperformed GCC, which is the previous SOTA and a direct ablation to the proposed APT framework.

Weaknesses

1. The proposed method is incremental. The proposed “predictive uncertainty” is adapted from curriculum learning, and “graph properties” are common statistics. 2. The proposed methods are not able to handle node features. It only provides structural information which does not align with most real-world scenarios. As a result, performance is not comparable with a supervised model trained on node features. 3. The node classification performance on homophily graphs (which is an important family of graphs) are much lower than ProNE. The improvement in graph classification tasks seems limited. 4. Selection based on graph properties might cause potential test data leakage because the graph properties are selected according to test performance (as shown in Figure 2).

Questions

In line 67, section 2, the authors present an example of a triangle to show what kind of knowledge the pre-trained model can learn. However, the proposed subgraph instance discrimination task can break down that knowledge. For example, ego-graph sampling can break down the triangle. Can the authors provide some insights or justification for this issue?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Post rebuttal

The authors have addressed my concerns, and I would raise my score from 5 to 6. I would be glad to see this paper get accepted, and if so, please revise the method part to make it clear.

Thanks for the authors' response.

Thank you for the authors' thorough response. Most of my concerns have been addressed. I look forward to seeing the revised paper, and I hope all the key points will be appropriately covered.

Thanks for the rebuttal

Thank you for the detailed reply and additional experiments. Most of my concerns have been addressed and I increased my score to 6.