Summary
The work presents a two-step approach to graph generation. First, a vector representation is generated, which conditions the subsequent graph generation.
These vector representations are obtained using a graph encoder, jointly trained with a generative diffusion model that transforms noise into the representations.
Graph generation is performed by a conditional diffusion model, parameterized by a message-passing transformer, that generates graphs conditioned on representations.
The model is evaluated on several graph generation benchmarks and shows competitive performance compared to existing methods.
Strengths
The paper is well structured and easy to follow. Figure 2 provides a clear overview of the proposed method and its components. The work introduces some interesting ideas, such as the joint training of the graph encoder and the generative diffusion model, and alignment losses during both representation and graph generation training. The empirical performance of the proposed method is demonstrated on several graph generation benchmarks, showing competitive performance compared to existing methods. Additionally, the ablation study offers insights into the significance of the various components of the method.
Weaknesses
First of all, there appears to be a problem with the latex rendering in the paper, as a substantial portion of symbols in math equations are missing. So the following comments are based on what could be inferred from the text and figures.
1) **Contributions and differences from previous work**:
The paper adapts the previous work RCG [1] from image to graph generation. However, there are key differences: while RCG employs a pretrained image encoder to obtain representations, the proposed method jointly trains the graph encoder with a diffusion model. Additionally, the proposed method introduces alignment losses during both representation and graph generation training. Furthermore, during graph generation, the diffusion model is conditioned on a noisy version of the representation, rather than the "clean" one, with noise strength aligning with the current denoising step. The paper should clearly delineate which aspects of the method are adapted from [1] and which are novel.
2) **Theoretical justification**:
The joint training, alignment losses, and noisy conditioning convolute the modeling approach. For instance, if the graph generation diffusion model were conditioned on the clean representation, it would be clear to see that the model distribution p(G) is decomposed as P(G) = P(G | R) P(R), where P(R) means the distribution of the representation R. However, with the noisy conditioning, it is not clear what the model distribution decomposes into, and what the role of the conditioning noise is. The alignment losses add further complexity. Although the ablation study demonstrates empirical benefits of these components, there is neither theoretical justification nor intuitive explanation for these design choices, making it difficult to grasp the underlying principles of the proposed method. Similarly, the joint training of the graph encoder and the representation diffusion model lacks theoretical justification or intuitive explanation for why it would yield meaningful representations. Additionally, as the representation distribution evolves during generator training, the convergence of this procedure should be discussed.
3) **Fair ablation study**:
The ablation study highlights the empirical benefits of the method's components, but the increase in model capacity and training time is ignored. For instance, the total number of parameters for the generative model is X (parameters for the representation diffusion model) + Y (parameters for the graph generation diffusion model). It would be insightful to see how the performance of the proposed method compares to an unconditional graph generation diffusion model with the same number of parameters, i.e. X + Y. Furthermore, for a fair comparison, the training time of the proposed method (joint training of the graph encoder and the representation diffusion model + conditional graph generation training) should be compared to other graph generation methods.
4) **Related work**:
Some overlooked works on graph generation should be discussed in the related work section: [2-8].
[1]: Li et al., Return of Unconditional Generation: A Self-supervised Representation Generation Method
[2]: Goyal et al., GraphGen: A Scalable Approach to Domain-agnostic Labeled Graph Generation
[3]: Grover et al., Graphite: Iterative Generative Modeling of Graphs
[4]: Davies et al., Size Matters: Large Graph Generation with HiGGs
[5]: Haefeli et al., Diffusion Models for Graphs Benefit From Discrete State Spaces
[6]: Diamant et al., Improving Graph Generation by Restricting Graph Bandwidth
[7]: Qi et al., SwinGNN: Rethinking Permutation Invariance in Diffusion Models for Graph Generation
[8]: Bergmeister et al., Efficient and Scalable Graph Generation through Iterative Local Expansion
Questions
1) Figure 1:
From this figure, one could infer that the representation and graph generation diffusion models are trained separately and only during graph generation is the conditional information incorporated. This interpretation contradicts the description in the rest of the paper.
Additionally, it is unclear what the term *capture* in this context means and how it differs from "learning".
I suggest revising the figure to better reflect the overall approach.
2) Validity of planar and sbm graphs:
Previous studies (SPECTRE and DiGress) report the fraction of valid planar and sbm graphs generated. Given that Table 2 includes results from these studies, why has the validity of planar and sbm graphs been omitted? How does the proposed method compare in this regard?