Summary
The authors address the gap in unsupervised graph representation learning by exploring the use of diffusion models. They propose directional diffusion models that incorporate data-dependent, anisotropic, and directional noises in the forward diffusion process to better handle anisotropic structures in graphs. Experiments on publicly available datasets showcase the superiority of their models over state-of-the-art baselines, demonstrating their effectiveness in capturing meaningful graph representations. Overall, the paper presents a compelling approach that contributes to the advancement of unsupervised graph representation learning.
Strengths
1. **Motivation**
The introduction is well-motivated, providing a thorough explanation of the challenge and task at hand. The authors go beyond textual descriptions and use simple visualizations on both real and synthetic data to demonstrate their points effectively. This approach enhances the clarity and understanding of the presented research, making it accessible to a wider audience.
2. **Method**
The authors present a straightforward yet effective solution for incorporating directional noise into node embeddings. This approach effectively addresses the challenge posed by anisotropic structures in various domains, including graphs and potentially text data. Their proposed method demonstrates promising results in handling directional noise and enhancing the quality of node and graph embeddings.
3. **Architecture**
I appreciate the authors intention to adapt the well-known and effective U-Net architecture from the image domain to the graph domain. The incorporation of skip connections in the U-Net is particularly relevant for denoising tasks. This thoughtful adaptation enhances the model's ability to handle graph-related denoising effectively.
4. **Experiments**
The authors conduct a comprehensive comparison with numerous baselines across multiple datasets. Furthermore, their evaluation settings, which involves 10-fold cross-validation with standard deviation after five runs, are robust and reliable. This rigorous evaluation methodology ensures the validity and statistical significance of their results.
Weaknesses
1. **missing releted work**
There are existing works in the intersection of graphs and diffusions are missing, contradicting the authors statement "To the best of our knowledge, there have been no works for diffusion-model-based graph representation learning.". Some for example:
- Niu, Chenhao, et al. "Permutation invariant graph generation via score-based generative modeling." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.
- Xu, Minkai, et al. "Geodiff: A geometric diffusion model for molecular conformation generation." International Conference on Learning Representations, 2022.
- Vignac, Clement, et al. "Digress: Discrete denoising diffusion for graph generation." International Conference on Learning Representations, 2023.
Furthermore, some simple techniques can be applied to create a smoother SNR curves over the different diffusion steps. For instance:
- Chen, Ting. "On the importance of noise scheduling for diffusion models." arXiv preprint arXiv:2301.10972 (2023).
The same problem was presented in it over the image domain (on high-resolution images), and the solution was to use different noise schedulers. Why not simply try this trick?
2. **missing details**
There are missing details in the paper that makes it hard to fully understand and reproduce the papers results. For example:
- "µ and σ are calculated using graphs within the batch.” -- what is done during inference? it is EMA over what was been seen in training time? does it handle only batch inference?
- They paper did not state how exactly the entire graph-level representation is obtained, it is sum/mean of all node representations?
- Algorithm 2 in appendix, line 6: what is exactly $A_t$? should it be just $A$?
Questions
1. "µ and σ are calculated using graphs within the batch.” -- what is done during inference? why not just pre-calculate those on the entire train split?
2. "allowing the latent code to contain meaningful compressed knowledge while preventing the decoder from degrading into an identical mapping” -- why will that happen that requires a dedicate MLP? what is this different from the simple U-Net used in the image domain?
3. Algorithm 2 in appendix, line 6, what is $A_t$?
4. In some cases, the SNR goes even higher for larger diffusion steps (like in the Computer dataset for instance), why is that?
5. How is the entire graph-level representation obtained?
6. Referring to figure 2, shouldn't the SNR be 0 in the last diffusion steps? It doesn't reach zero as the directional noise keeps them separated, is that a wanted outcome? If so, why?
7. Why not apply just a different noise scheduler? instead of linear, coisne for instance. Similar observation was shown on high-resolution images and different schedulers where proposed [1]. I know it is still isotropic noise, so at least show results as a baseline/ablation.
___
[1] Chen, Ting. "On the importance of noise scheduling for diffusion models." arXiv preprint arXiv:2301.10972 (2023).
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors address the limitations of their work, and support their claims with experiments and ablations.