Data-Centric Learning from Unlabeled Graphs with Diffusion Model

Summary

The paper discusses the problem of graph property prediction tasks, where there is a limited amount of labeled data and a large set of unlabeled graphs available. The conventional approach is to train a model using self-supervised tasks on the unlabeled graphs and then fine-tune it for prediction tasks. However, the knowledge gained from self-supervised tasks may not align with the requirements of the prediction tasks. To address this issue, the authors propose a data-centric approach that extracts knowledge from the unlabeled graphs to enhance each property prediction model. They introduce a diffusion model to utilize the unlabeled graphs fully and two new objectives that guide the model's denoising process and generate task-specific graph examples and labels. Specifically, the first objective minimizes an upper bound of mutual information between the augmented and the original graphs in the graph space. The second objective maximizes the probability of the predicted label of augmented graphs being the same as the label of original graphs. Experimental results demonstrate that the proposed data-centric approach outperforms fifteen existing methods on various graph property prediction tasks. The improvement in performance is attributed to the generated labeled examples, which are more effective than self-supervised learning. The authors also discuss related work in graph property prediction and learning from unlabeled data, highlighting the limitations of existing methods and the need for a data-centric approach. They present their framework (DCT) in detail and provide results showing its superior performance compared to baseline methods used for comparison.

Strengths

1. The proposed method offers a novel approach to graph property prediction by employing a diffusion model, setting it apart from traditional techniques. 2. Empirically, the introduced method surpasses other techniques in various areas including self-supervised learning, semi-supervised learning, and graph data augmentation.

Weaknesses

1. The discussion of the graph diffusion model is insufficient. The proposed method relies on the graph diffusion model as its backbone technique. A clear discussion of the similarities and differences between this work and other graph diffusion model methods [1,2] can clarify the contribution of this work. 2. The interpretability of "visible" knowledge transfer from unlabeled graphs is interesting, but only several case studies cannot support the claim that "concepts such as some chemical rules from the unlabeled graphs are successfully transferred to downstream tasks" (lines 318-319). [1] Jo, Jaehyeong, Seul Lee, and Sung Ju Hwang. "Score-based generative modeling of graphs via the system of stochastic differential equations." In *International Conference on Machine Learning*, pp. 10362-10383. PMLR, 2022. [2] Vignac, Clement, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. "DiGress: Discrete Denoising diffusion for graph generation." *arXiv preprint arXiv:2209.14734* (2022).

Questions

1. In lines 46-48, the generation of labeled examples is part of the property prediction task pipeline. Please describe how to use these generated task-specific labeled examples to bridge the gap between the final prediction and the generated graph. 2. In lines 60-61, it is described that the trained predictor model f is used in the first objective. How is the predictor trained? If it is trained on a small amount of labeled data, then why is the not-well-trained predictor (as stated in Line 119) able to help with graph generation? 3. Can other graph diffusion model methods be applied to the graph property prediction task with trivial adaptation? If not, why?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes a new data augmentation method called DCT based on unlabeled graphs. The proposed DCT generates new augmented graphs using a diffusion model from unlabelled graphs and also with labeled graphs from down-stream tasks as guidance. Two specific objectives are proposed to guide the model's denoting process to generate task-specific graph examples and their corresponding labels. Experimental results demonstrate the superior performance of the proposed method.

Strengths

1. A new graph augmentation method is proposed based on diffusion models developed in the computer vision field. The exploration of diffusion models on graphs is relatively new. 2. The proposed method adopts two task-specific objectives, enabling to learn task-specific knowledge and generate task-related labeled graphs. This mechanism has been shown effective in addressing the shortcomings of self-supervised learning self-training methods. 3. The experimental evaluation and ablation studies demonstrate the promising performance of the proposed method on graph property prediction tasks.

Weaknesses

1. The idea of using diffusion models to generate graphs is borrowed from the field of computer vision, where the diffusion and reverse process is more intuitive. However, it remains unclear how it can be applied to both graph structure and node features. 2. Some relevant literature review, e.g., on pseudo-labelling, is missing in related work.

Questions

1. Does your method suffer from overfitting? In Figure 4, the performance of the proposed method seems to decrease as the number of selected labeled graphs for data augmentation increases. Although it is beneficial to consider downstream tasks when augmenting graphs, the potential overfitting problem should be discussed. 2. How is the diffusion and reverse process applied to both graph structure and node features? Do you perturb different types of noise to graph structure/node features? More detailed discussions are necessary. 3. In Definition 4.2, how to define \bar{G}?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have not discussed potential limitations including societal impact of their work.

Summary

The paper leverages diffusion SDE to approach the hidden chemical space and help the prediction tasks of molecular properties.

Strengths

1. It is a good idea to leverage mutual information for the alignment of the hidden chemical space within both labeled and unlabeled graphs. 2. The results presented in this study seem to exhibit superior performance when compared to other methods examined in the paper, which is an encouraging observation.

Weaknesses

1. The paper could benefit from greater originality. The primary concepts echo closely those found in references [1] and [2]. For instance, [1] utilized generative models to boost the prediction performance for an array of graph datasets, while [2] applied a score matching method to uncover the hidden chemical space, thereby facilitating active learning in the predictive task of molecular properties. I recommend an expanded discussion within the paper to clearly delineate the distinctions between the current study and the methods in references [1] and [2]. This would more effectively highlight the unique contributions of this work. 2. The paper appears to have an insufficiency in its literature review section. It would greatly benefit from an evaluation of self-supervised learning-based molecular pre-training models as discussed in references [3], [4], and [5]. These studies could provide significant context and further inform the study's approach. 3. The paper reports an incremental advancement in performance. Upon reviewing the OGB leaderboard [6], the results presented for ogbg-molhiv do not appear to supersede those achieved by SOTA methods currently leading on the leaderboard. Moreover, it seems that the performance does not even align favorably with the results presented in references [3], [4], and [5]. The authors are encouraged to provide an explanation or delve into the possible reasons behind this observed discrepancy in performance levels. [1] Liu, Songtao, et al. "Local augmentation for graph neural networks." International Conference on Machine Learning. PMLR, 2022. [2] Zhou, Kuangqi, et al. "Jointly Modelling Uncertainty and Diversity for Active Molecular Property Prediction." Learning on Graphs Conference. PMLR, 2022. [3] Zhang, Zaixi, et al. "Motif-based graph self-supervised learning for molecular property prediction." Advances in Neural Information Processing Systems 34 (2021): 15870-15882. [4] Rong, Yu, et al. "Self-supervised graph transformer on large-scale molecular data." Advances in Neural Information Processing Systems 33 (2020): 12559-12571. [5] Xu, Minghao, et al. "Self-supervised graph-level representation learning with local and global structure." International Conference on Machine Learning. PMLR, 2021. [6] https://ogb.stanford.edu/docs/leader_graphprop/#ogbg-molhiv

Questions

See above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

This paper studies leveraging unlabeled graphs to aid the supervised graph learning tasks with limited labeled graphs. This paper starts with a motivation that previous self-supervised methods may conflict with the downstream task and lead to poor performance. This paper proposes to model the graph distribution with diffusion models and train the diffusion model jointly with the downstream prediction model. The proposed method optimizes the prediction loss of the augmented graphs and maximizes the difference between the augmented graph and the original labeled graph. This paper evaluates the proposed method on 15 graph property prediction tasks, including seven molecule classification, three molecule regression tasks, four polymer regression tasks, and one protein function prediction task. Compared to previous self-supervised learning, semi-supervised learning, and data augmentation approaches, the proposed method shows significant improvement over previous methods.

Strengths

- This paper leverage diffusion models to learn the distribution of unlabeled graph and use the model to augment labeled graph for downstream tasks. In contrast to previous self-supervised learning methods, the proposed method is more effective in terms of adapting to each downstream task. - Empirical evaluations are extensive. This paper provides a broad range of experiments that demonstrate the advantage of the proposed method over 15 previous methods on various datasets. This paper also presents ablation studies and case studies that show the working of the proposed method.

Weaknesses

Details of the proposed method need to be clarified, such as the training of the diffusion model and implementation details. Please see Questions for more details.

Questions

- The proposed method uses a diffusion model to first learn graph distribution from unlabeled graphs. Is the diffusion model trained for each task? Or is a joint diffusion model trained and applied to every task? - What dataset is the diffusion model trained on in the experiments? - Is the prediction model trained on both originally labeled graphs and augmented graphs in the downstream tasks? How would the performance be if the method also trained on the original labeled graphs? - What would be the perturbation scale between originally labeled graphs and augmented graphs, which leads to the best performance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

This work has discussed the limitations of the proposed method.

Thanks for the response!

I have read the authors' responses. The responses have addressed my concerns. Therefore, I will keep my score.

The comparison to SOTA methods are missing

I appreciate the author's detailed response, particularly where answer 1 and 3 have addressed the concerns I raised. Nonetheless, I harbor reservations regarding the 2nd answer. Firstly, the work from NeurIPS 2020 did not surpass the SOTA. The author also made comparisons with two studies published in 2019, i.e. Hu et al., 2019 and Velickovic et al., 2019. Secondly, each experiment has its own distinct settings and datasets. Thus, the better performance of MGSSL than GROVER[4] reported in MGSSL paper does not obviate the necessity for comparative analysis in alternative settings or datasets. The apparent lack of specific SOTA methods in this comparative experiment prompts the question: Are there substantial differences between the proposed method and the omitted method that lead to cherry-picking of the compared methods? In light of this, I strongly advocate more exhaustive comparisons with all existing SOTA methods in experimental evaluations.

Summary of discussion

Dear reviewers, We appreciate all the valuable feedback from four different aspects: novelty, literature review, method detail, and experiment. To facilitate the discussion, we have prepared a summary for your consideration. Here we outline how we have addressed the concerns raised by each reviewer. ### Reviewer 6AKL 1. Method detail: We explained the training details of the diffusion model. 2. Experiment: We provided a detailed explanation of the dataset used in our experiments. We also analyzed the impact of the perturbation scale on model performance. ### Reviewer T9FT 1. Novelty: We have highlighted a major novelty: knowledge transfer from unlabeled graphs by data augmentation. This sets our work apart from the two referenced papers as mentioned by the reviewer, which focused on different tasks - one on node-level tasks and the other on active learning. 2. Literature review: We showed that we have covered two referenced papers as mentioned by the reviewer. We also showed that the literature review on baselines is broad and sufficient. 2. Experiment: We have addressed concerns about the performance and introduced experiments comparing a new baseline, demonstrating that our baseline selection and comparison do not lead to cherry-picking presentation. ### Reviewer trSH 1. Literature review: We discussed more work related to pseudo-labeling. 2. Method detail: We explained the details of using diffusion for graph generation tasks. We also clarified the definition of $\bar{G}$. 3. Experiment: We explained that the augmented examples can introduce noise as the top-$n$ hyper-parameter increases, potentially causing a performance drop in Figure 4. ### Reviewer 3gzg 1. Literature review: We discussed more work related to the diffusion model. We also discussed how the proposed framework can adapt to various diffusion models. 2. Method detail: We explained how to generate desirable graphs in downstream tasks and how to use them with the downstream predictor model. 3. Experiment: We presented more evidence to support our claim on the visible knowledge transfer. We are encouraged by successfully addressing the concerns raised by Reviewer 6AKL and T9FT through discussions. We sincerely apologize if we missed any important concerns that the reviewers believe were not adequately summarized or addressed. We respectfully request the opportunity to further address these concerns during the remaining discussion period. Your feedback is highly valued.

Official Comment by Reviewer 3gzg

Having received and reviewed the authors' responses to my queries, I'd like to amend my original score from 4 to 5 for this submission.