Unifying Generation and Prediction on Graphs with Latent Graph Diffusion

Summary

This paper proposes a Latent Graph Diffusion model that unifies multiple tasks such as classification, regression, and generation in a generative task. Additionally, this model achieves feature generation in multiple categories, including node-level, edge-level, and graph-level.

Strengths

1. This paper attempts to address a crucial task in the field of graphs, which is to propose a model capable of handling various tasks such as classification, regression, and generation at different levels, including node-level, edge-level, and graph-level. 2. The writing in this paper is relatively clear.

Weaknesses

1. The evaluations conducted by the authors are insufficient. For example, in the generation task, they avoid comparing their proposed method with classical diffusion-based graph generation models such as DiGress [1], HGGT [2], and DruM [3], which perform better in terms of metrics like Validity on QM9. Additionally, they avoid comparisons on larger datasets like MOSES [4] or GuacaMol [5]. Furthermore, they do not compare with commonly used metrics for graph generation tasks, such as FCD and NSPDK. 2. Regarding the approach of treating classification and regression tasks as generation tasks, the authors should provide a simple baseline (which can be seen as an ablation study) to demonstrate that they have not unnecessarily complicated simple and intuitive tasks. For example, they could use condition features as input and train a baseline prediction model with the same architecture as the denoising model, using the corresponding masked global features as supervised signals. This would help assess the benefits brought by the generation model. 3. The authors aim to develop a model that can address graph learning tasks at all levels (node, edge, and graph) and types (generation, regression, and classification), which is hard to achieve. The authors are suggested to provide a clear table to include detailed information about the source and size of the training dataset, as well as the number of model architecture and parameters for reference. 4. The explanation of how the decoder of generation tasks reconstructs the structural information, i.e., A, of the graph is not clear. 5. Figure 1, mentioned in the Section 5, is displayed in the Appendix. [1] C. Vignac, I. Krawczuk, A. Siraudin, B. Wang, V. Cevher, and P. Frossard. DiGress: Discrete denoising diffusion for graph generation. In International Conference on Learning Representations, 2022. [2] Y. Jang, D. Kim, and S. Ahn. Graph generation with K2-trees. In The Twelfth International Conference on Learning Representations, 2023. [3] Jaehyeong Jo, Dongki Kim, and Sung Ju Hwang. Graph generation with diffusion mixture., ICML 2024. [4] D. Polykovskiy, A. Zhebrak, B. Sanchez-Lengeling, S. Golovanov, O. Tatanov, S. Belyaev, R. Kurbanov, A. Artamonov, V. Aladinskiy, M. Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:565644, 2020. [5] N. Brown, M. Fiscato, M. H. Segler, and A. C. Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096–1108, 2019.

Questions

1. Whether the Encoder and Decoder are permutation equivariant? 2. Will the authors release the source code and checkpoints?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes.

Summary

The paper proposes Latent Graph Diffusion to generate node, edge, and graph-level features to meet the need of different tasks under the unified framework. Extensive experiments show the competitive performance across various graph-based tasks.

Strengths

1. It is good to reformulate regression and classification problems with diffusion generation task. 2. The paper is technically sound, and the introduction of diffusion in latent space is well-motivated. 3. The unified framework has theoretical support.

Weaknesses

1. Readers have to jump to Appendix over and over again to have a general understanding of your proposed model. 2. Despite the unified formulation as generation, other parts of the paper lack novelty. For example, there is a lack of new breakthroughs in specific model design and diffusion models. The underlying diffusion and denoising models largely build upon existing methods. 3. No code or demo available.

Questions

1. How does the LGD framework scale with very large graphs? 2. In the task of node classification from Planetoid, is the LGD model capable of generating a new graph almost the same as Cora or PubMed? 3. The graph will be too dense when considering no link as a special type of edge? How to deal with this problem

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

1. The lack of detailed analysis of time and space complexity makes the performance of the model in practical applications less transparent. 2. The sensitivity analysis of parameters in the conditional generation task is insufficient, especially since the influence of important hyperparameters is not studied deeply.

Summary

This paper proposes a model framework, LGD, which addresses multiple types of graph tasks and can simultaneously handle both generative and predictive tasks. Specifically, the LGD framework employs a method similar to stable diffusion, using a pretrained graph encoder to obtain latent representations and then training a diffusion model in the latent space. To use the diffusion model for predictive tasks, the authors incorporate the label in regression and classification tasks as a partially masked feature, using the rest of the training set as a condition to predict the masked label. Finally, the authors validate their approach through experiments for prediction, classification, and regression tasks.

Strengths

- The idea of using a conditional diffusion model for predictive tasks is novel. Unlike traditional representation learning approaches, LGD can directly generate labels using the diffusion model, providing a new perspective for the community. - The experiments cover a comprehensive range of tasks, datasets, and baselines. The model shows promising results on most datasets. - The authors provide theoretical support for using latent diffusion models in regression tasks and discuss the advantages of converting deterministic tasks into generation tasks.

Weaknesses

- The model functionality described in the introduction is somewhat misleading. In lines 28-29, the authors claim to construct a graph model capable of solving all task types. However, the implementation does not use a single foundation model to accomplish all tasks. Since the proposed framework still cannot solve the feature misalignment problem, specific tasks require training specific encoders and decoders. This makes the introduction seem like an overclaim, as LGD is just a framework that can handle multiple tasks but includes many components that need task-specific training. From a training cost perspective, this is similar with using separate models for different tasks (and potentially more expensive due to the need for GT and encoder-decoder). It is noteworthy that the authors mention this drawback in the limitations section. - The experiments lack details on training and potential unfairness. (1) In Appendix C, the authors describe that the encoder $\mathcal{E}$ was pretrained with both unsupervised and supervised methods. However, the main text does not specify which datasets were used for the encoder's training and whether they are the same as those used for training the diffusion model or if additional datasets were used. For instance, in the Conditional generation task, if the encoder was supervised-trained on the QM9 dataset, the results in Table 2 would lack fairness. This is because other baselines do not have access to supervised training information when providing conditions, giving the encoder a clear advantage. The authors should clarify the training data details for the encoder. (2) In line 762, the authors state that the latent diffusion model is shared across different tasks, but the paper does not provide details on what data this shared diffusion model was trained on. Was it trained on the entire dataset across all tasks? This also lacks detailed description. - Some of the authors' statements are overclaimed or inaccurate: (1) In the analysis of the unconditional generation task (QM9), the authors claim that generation based on 3D information is easier, highlighting their method's effectiveness. This premise is incorrect. First, RNN-based models [1], which generate based on SMILES, can achieve a validity of 99% and high uniqueness without 3D information. 3D generation actually targets a different need—conformation generation—related to molecular dynamics and has broader research and practical value, hence the use of 3D information. It is not that generation based on 3D information is easier. Moreover, the three metrics the authors compare are not the best indicators of molecular generation quality. I suggest the authors use more effective metrics such as QED and TPSA. Previous works, especially those focusing on molecular generation tasks, do not limit evaluations to these two metrics due to their limited persuasiveness. (2) In line 767 of the appendix, the authors claim their work is the first to achieve in-context learning on graph pretrained models. In fact, there are several prior works on graph prompts, such as PRODIGY [2]. [1] Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics. [2] PRODIGY: Enabling In-context Learning Over Graphs. NeurIPS 2023. - The discussion in lines 111-112 about the capabilities of generative models lacks citations.

Questions

- Did the authors use a shared pretrained diffusion model for different tasks? If so, what data was it trained on? - What do the authors think about the feature misalignment problem in graph pretraining?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the Weakness and Questions above.

Summary

This paper presents Latent Graph Diffusion (LGD), a graph generation framework adept at handling a variety of tasks including generation, regression, and classification at different objects—node, edge, and graph levels. LGD approaches regression and classification tasks as conditional generation challenges and employs a latent diffusion model within the latent space to concurrently generate features for nodes, edges, and graphs.

Strengths

1. The paper introduces the Latent Graph Diffusion as a novel approach to address the challenges of using a unified framework to achieve classification, regression and generation tasks on graph. 2. This paper clearly shows its idea.

Weaknesses

1. While GDSS and DiGress are designed to handle various types of graph data beyond molecular diffusion, it is unclear if LGD has conducted similar graph generation experiments. The article lacks relevant general graph generation experiments and evaluations using metrics like MMD (Maximum Mean Discrepancy) to assess the effectiveness of graph generation. 2. The authors have not included some successful practices of diffusion models. Recent studies [1,2] have demonstrated better performance than the proposed method on the QM9 dataset. 3. The QM9 dataset used for graph generation is relatively small, and it would be beneficial for the authors to validate their proposed model on larger benchmark datasets, such as Moses [3], which also provide a wider range of evaluation metrics. 4. The description of the model's architecture details is unclear and could benefit from further clarification. 5. The submission does not include an open-source implementation of the proposed model. [1] DiGress: Discrete denoising diffusion for graph generation. ICLR2022. [2] Graph generation with diffusion mixture., ICML 2024. [3] Molecular sets (moses): a benchmarking platform for molecular generation models. 2020.

Questions

1. It would be valuable to compare the computational efficiency of the evaluated methods during training and sampling to provide a comprehensive analysis.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A.

Thank you for your response.

After reading the response, I decide to raise my score. The additional result should be included in the revised version.

Updated experimental results and we are looking forward to your reply

Dear reviewers, We now update the results on the large MOSES dataset for molecule generation at scale. $$ \begin{array}{|cc|cccc|} \hline \text{Model} & \text{Class} & \text{Validity} & \text{Uniqueness} & \text{Novelty} & \text{FCD} \\\\ \hline \text{VAE} & \text{SMILES} & 97.7 & 99.8 & 69.5 & 0.57\\\\ \text{JT-VAE} & \text{Fragment} & 100 & 100 & 99.9 & 1.00\\\\ \text{GraphINVENT} & \text{Autoreg.} & 96.4 & 99.8 & – & 1.22\\\\ \text{ConGress} & \text{One-shot} & 83.4 & 99.9 & 96.4 & 1.48\\\\ \text{DiGress} & \text{One-shot} & 85.7 & 100 & 95.0 & 1.19\\\\ \hline \text{LGD (ours)} & \text{One-shot} & 95.9 & 100 & 95.9 & 1.42\\\\ \hline \end{array} $$ The results show that LGD has superior performance compared with DiGress in terms of validity and novelty metrics. Apart from DiGress and ConGress, we are the only model to scale to this large dataset, bridging the gap between diffusion-based one-shot graph generation model and other traditional methods like SMILES-based, fragment-based, and autoregressive models. Combining with other new results in our global response and the extensive experiments in the original submission, it is convincing that LGD reveals strong capability in both generation and prediction tasks. In addition, we sincerely appreciate reviewer suUP and a5WK's positive reply. We are looking forward to reviewer XBTL and PVEb's reply. We are happy to answer any further questions regarding the paper. Thank you.