Summary
Gist:
The paper presents a framework, called CommonScenes, for generating 3D indoor scenes given scene graphs as inputs. CommonScenes is a dual-branch framework where one branch generates the scene layout using a VAE and the second one generates what the authors call "compatible" 3D shapes using latent diffusion. The claim is that having this second branch (where compatible 3D shapes are generated for populating the generated layout from the first branch) allows capturing global scene-object and local inter-object relationships, something that prior works cannot capture (I am not convinced about this, but more on this later).
The generated scenes can be manipulated by editing the input scene graph, as well as sampling noise in the diffusion process.
The paper also constructs a scene graph dataset using an off-the-shelf 3D scene dataset.
Dataset Used:
3D-FRONT is the base dataset used, which is augmented with scene graph labels and this augmented dataset is termed in the paper as "SG-FRONT" dataset.
Training Mechanism:
Supervised in the form of triplet network setting and latent diffusion models
Evaluation Metrics:
To measure the fidelity and diversity of generated scenes, FID, KID scores at 256x256 pixel resolution between the top-down rendering of the generated and real scenes is used.
To measure shape diversity, each scene is generated 10 times, and the changes in shapes is evaluated using CD.
Baselines and Comparisons:
Three kinds of baselines are compared against:
1) First, a retireval-based method, namely, 3D-SLN from CVPR 2020
2) Second, a semi-generative SOTA method, Graph-to-3D from ICCV 2021, and
3) Third, a text-to-shape generation methods that follows layout generation (this is not cited, so I am not informed by the paper if this was implemented by the authors on their own or if any specific algorithm was re-implemented)
Strengths
+ Conceptualizing layout generation using graphs is a nice concept, although this is not the first time it has been addressed. A structured input modality gives rise to many applications, such as scene editing and modification, as demonstrated in the paper.
+ As seen from Figure 4, the proposed method seems to produce plausible outputs given a scene graph as input. This is also validated quantitatively, although I would only consider Table 1 to be more representative of such quantification than other tables.
Weaknesses
- Not really a concern but this is something that people will find about this paper: the paper is trying to do too many things at once. While this may also be a positive aspect in the era of today's models, a reader cannot clearly discern what design aspect leads to layout improvement and what leads to shape improvements. One may even argue: why not use the shape generation scheme employed as an independent approach and submit a paper if enough novelty exists? You get my point.
- L96-97: How are the edge predicates (like spatial relations "left/right", "closeby" etc.) obtained? Is the dataset manually annotated with semantic scene graph information? If that is the case, then, the problem formulation is weak. What would have been interesting is to automatically extract meaningful semantic scene graphs (especially that ground spatial relations to a reasonable extent) and then use these graphs to generate a 3D scene.
- L3-6: I do not understand the message in these lines. Do you mean to say that existing methods use retrieval-based mechanisms to populate the generated layout, because of which scene-object and object-object relationships are inconsistent? It is not true. So, first, I think it is important to rephrase this sentence. It is conveying an altogether different meaning.
- There is mention of a triplet graph network, triplet-GCN in L46, 129, 148, 159. However, there is not much detail about how the positive and negative examples to train this triplet network are obtained. This is quite important since the training data plays a key role in obtaining meaningful and discriminative embedding spaces in the context of contrastive learning setups.
- I fail to understand why the initial node embeddings, c_i^{node}, which are obtained from one-hot semantic encodings (as per info in L93), should be passed through a pre-trained and frozen CLIP text encoder. It makes sense to pass the initial __edge__ embeddings through it as spatial information needs to be captured and the CLIP text encoder does a good job of mapping the initial English text to a meaningful embedding space. But I cannot understand why the node embeddings c_i^{node} need to be passed through the text encoder from CLIP.
- L37: It is not trivial to obtain text information from input scene graphs. This alternative solution is not so straightforward, unlike what is mentioned in this line.
- L65-66, the sentence is incomplete
- L87-88: LEGO-Net from CVPR 2023 is the first work to leverage diffusion models for scene generation. Even though LEGO-Net is designed for scene rearrangement, I would still place it in the generative model category since it is inherently doing denoising to provide a plausible output. So, these two lines are not correct.
- Figure 4: Again, my question is how are the input scene graphs obtained? I am interested in knowing how the spatial relations on the graph edges are obtained. If this is done in a heuristic manner, it is prone to errors, and I do not think this is trivial to obtain in the presence of multiple objects. Obtaining such spatial relations is a challenge, as widely acknowledged in the community (see pre-LLM works on 3D indoor scene generation using language/text input, such as from Chang 2014, 2015, Ma 2018).
- There are quite a few typos and punctuation errors in the paper (one such example is pointed out below). Need to be corrected.
One Missing period/full-stop symbol in line 24.
Questions
Please see the Weaknesses section above
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper does not discuss the limitations of the proposed approach.
There exist many questions (please see the Weaknesses section above) that can critically limit the application of the proposed approach, starting from the way the input scene graphs are obtained. At the least, a discussion on how this work can address or alleviate such challenges using additional processing should have been discussed.