Strengths
- The paper studies an intersectional topic: leveraging graph learning techniques for image generation, which is a creative application and an area which deserves more focus.
- The authors' use of qualitative examples (e.g. Figure 5 and 6) is commendable and helps articulate visual improvements.
Weaknesses
Please see questions and concerns below. My general feeling is the paper is fairly incremental in its introduction of a mechanism to encode graph condition into the conditioning for generation. Many design choices for graph conditioning are not discussed well and the quantitative results for some of these choices are missing which hurts the overall impact of the work.
Questions
- Typos:
- Line 18: "graph-structued"
- The motivation proposed in lines 28-30 is a little bit confusing, since the scenario the authors discuss here (e.g. virtual artwork creation based on nuanced styles of artists and genres) seems like it could be well-handled by text rather than explicitly using graph structure.
- There is limited prior work in multimodal graph learning, as the authors mention. The authors may want to reference and position their work with respect to the recent [1] which offers multiple datasets and focuses on utility of GNN methods for node/link/graph-level tasks rather than generative tasks.
- Nit: the notation is a bit awkward compared to conventional graph literature which typically uses $\mathcal{V}, \mathcal{E}, \mathcal{X}, \mathcal{F}$ or something similar to indicate node-set, edge-set, node-features, and edge-features. The authors proposed notation in line 72 seems to define P and D as different sets of images / documents compared to the nodes V, but then mentions that each node has some textual information and image information corresponding to P and D (it should be made clear whether this information is just features, or actual node relationships -- if the latter, it seems that P and D should be contained within the nodeset V).
- The process described in line 115 around introducing tokens to help condition the representation using graph structure is also explored in some related works, e.g. [2]. Perhaps the authors could consider adopting a similar approach if it makes sense in this task, since the tokenization scheme as the authors of [2] point out is key in injecting the right level of information to the model.
- Comment: the notation in pages 3-5 is quite heavy and would benefit from a symbol table.
- Section 3.2 proposes a heuristic solution for neighbor selection. I'd encourage exploring solutions designed for learnable importance of multiple edge types similar to [3].
- Can the authors discuss what sort of techniques were used to incorporate graph conditions for the baseline models like InstructPix2Pix and SD?
- Is there a quantitative understanding or experiment for the PPR neighbor based sampling approach? It seems this is one of the more heuristic parts of the paper where the design of the sampling procedure (two phase PPR + semantic re-ranking) is less conventional and deviates from other aggregation mechanisms explored in previous literature like attention-based selection, learnable importance of multiple edge types, etc. The qualitative experiment is helpful but not terribly convincing in terms of the actual performance impact in aggregate.
[1] Multimodal Graph Benchmark (Zhu et al, 2024)
[2] LLaGA: Large Language and Graph Assistant (Chen et al, ICML 2024)
[3] Pathfinder Discovery Networks for Neural Message Passing (Rozemberczki et al, WWW 2021)