DeBaRA: Denoising-Based 3D Room Arrangement Generation

Generating realistic and diverse layouts of furnished indoor 3D scenes unlocks multiple interactive applications impacting a wide range of industries. The inherent complexity of object interactions, the limited amount of available data and the requirement to fulfill spatial constraints all make generative modeling for 3D scene synthesis and arrangement challenging. Current methods address these challenges autoregressively or by using off-the-shelf diffusion objectives by simultaneously predicting all attributes without 3D reasoning considerations. In this paper, we introduce DeBaRA, a score-based model specifically tailored for precise, controllable and flexible arrangement generation in a bounded environment. We argue that the most critical component of a scene synthesis system is to accurately establish the size and position of various objects within a restricted area. Based on this insight, we propose a lightweight conditional score-based model designed with 3D spatial awareness at its core. We demonstrate that by focusing on spatial attributes of objects, a single trained DeBaRA model can be leveraged at test time to perform several downstream applications such as scene synthesis, completion and re-arrangement. Further, we introduce a novel Self Score Evaluation procedure so it can be optimally employed alongside external LLM models. We evaluate our approach through extensive experiments and demonstrate significant improvement upon state-of-the-art approaches in a range of scenarios.

Paper

Similar papers

Peer review

Reviewer sfxx3/10 · confidence 5/52024-06-16

Summary

The authors present DeBaRa, a diffusion-based framework for indoor scene generation, and a self-score evaluation strategy to select conditioning input. They demonstrate the effectiveness of their approach on scene synthesis and several downstream tasks.

Strengths

+ The empirical results are promising.

Weaknesses

+ Insignificant Architectural Difference: The architectural differences between DeBaRa and other diffusion-based baselines (e.g., DiffuScene) are not substantial. The results benefited architecture difference is not sufficiently supported by experimental evidence. To be honest, we **do not** care about final performance, but the impact of the differences. + Lack of Implementation Details and Ablation Studies: The paper does not provide sufficient implementation details, making it difficult to reproduce the results. Additionally, ablation studies are missing, which are crucial to understanding the contributions of individual components of the model. + Missing Experiments: The paper claims advantages in downstream tasks like rearrangement and completion but lacks comparative experiments with baseline methods (ATISS, LEGO-Net, and DiffuScene) both qualitatively and quantitatively. + Unclear Writing and Missing Sections: The writing is unclear in several parts of the paper. Additionally, **Section 4.4 ("Additional Results") is missing**, which further reduces the clarity and completeness of the work. + Typing Errors: There are numerous typing errors throughout the paper, such as "biaises" instead of "biases" in Line 39. These errors affect the readability and professionalism of the manuscript.

Questions

+ The inference time is reported to be significantly faster than DiffuScene (about 100 times faster). Are these advantages due to the use of EDM over DDPM? Please elaborate on how the choice of EDM contributes to the improved inference time.

Rating

3

Confidence

5

Soundness

2

Presentation

1

Contribution

2

Limitations

N/A

Reviewer sfxx2024-08-12

Response to the author rebuttal

Thank you for your response. While some of your points partially address my concerns, I am still inclined to recommend rejecting this paper. + Novelty: Using EDM to replace DDPM for indoor scene modeling does not represent a significant contribution, which does not offer much new insight into the field of indoor scene generation. + Experiments: Since the focus is on the training framework, its effectiveness is not validated on a sufficiently scalable dataset. To my knowledge, the synthetic data (3D-FRONT) used contains only around 5k scenes for training (with subsets like library being even smaller), which is relatively small compared to current image datasets. + Comparisons: The lack of comparison with many autoregressive model works, such as COFS [1], and scene-graph-based methods like SceneHGN [2] and GRAINS [3], is concerning. + Evaluation Metrics: The metrics used in this study, such as FID, KL, and CAS, seem rather generic. They might not fully capture the essential aspects of scene generation, including diversity, complexity, symmetry pattern discovery, object interaction, and object concurrences. + Rendering Quality: The quality of the rendered indoor scenes does not appear to meet the standards expected by artists. For better examples, the authors may refer to [2, 3]. Questions: + It would be insightful to investigate whether this diffusion-based method can produce more diverse scenes, compared to autoregressive-based methods. + The evaluation protocol is not clear. How many generated/GT scenes do you use for evaluation? References: [1] Para et al., "COFS: Controllable Furniture Layout Synthesis," SIGGRAPH 2023. [2] Gao et al., "SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation with Fine-Grained Geometry," TPAMI 2023. [3] Li et al., GRAINS: Generative Recursive Autoencoders for INdoor Scenes, TOG 2019.

Authorsrebuttal2024-08-12

Response to reviewer comment (2/2)

To summarize the key points: 1. We evaluate our approach on the same standard dataset as done in recent state-of-the-art methods [1,2,3]. 2. The baselines that we compare against supersede the methods suggested by the reviewer. Nevertheless, for completeness, in the final version we will be happy to add comparisons against methods, which can be evaluated on our test set. 3. We use the same presentation pipeline as in recent published works, ensuring consistent comparisons. We thank reviewer `sfxx` for their constructive feedback. We are confident that all of the requested changes can be addressed in a minor revision. Furthermore, we remain confident of the contribution of our work, as also confirmed by, e.g., reviewer `Fsdh`. [1] DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis, Tang et al., in CVPR 2024. [2] Lego-net: Learning regular rearrangements of objects in rooms, Wei et al., in CVPR 2023. [3] COFS: Controllable Furniture Layout Synthesis, Para et al., in SIGGRAPH 2023. [4] Generalization in diffusion models arises from geometry-adaptive harmonic representations, Kadkhodaie et al., in ICLR 2024.

Authorsrebuttal2024-08-12

Response to reviewer comment (1/2)

We thank reviewer `sfxx` for increasing their initial scores and for taking the time to engage in the discussion period. **We want to highlight that most of reviewer's additional concerns and questions were not raised in their initial review**. While we would have been happy to address them in our rebuttal, we do so in the following response: > Novelty We refer the reviewer to the list of contributions mentioned in our rebuttal. Although we do not consider the use of EDM alone to be one of our major contributions, we don't employ it as a simple *drop-in* replacement for DDPM and discuss the relevance of this design choice in the context of indoor scene synthesis in our main submission, L132-139. > Experiments We evaluate our method on the standard dataset for 3D indoor scene synthesis, **which is the one used in the most recent baselines** [1, 2], including the COFS [3] paper mentioned in the reviewer's comment. **We also want to emphasize that we actually view the performance of our method** (demonstrated by our qualitative and quantitative evaluations), **and its ability to generalize to complex / unseen floor plans** (as shown in Figure 2 of our rebuttal `PDF`) **as additional strengths in the light of the limited number of training samples**. Finally, **generative diffusion models** are known to **scale favorably well** to larger training datasets [4]. > Comparisons As stated in our main submission L219, we chose to evaluate DeBaRA against established baselines from **different model *families*** (ATISS, i.e., autoregressive ; DiffuScene / LEGO-Net, i.e., denoising-based ; LayoutGPT, i.e., LLM-based). We can read in the COFS paper: "*Our model thereby **extends the baseline ATISS** with new functionality while **retaining all its existing properties and performance***", which can be verified in their experimental evaluations. On the other hand, **our method largely outperforms ATISS**, as quantitatively verified in **Table 1** and **Table 2** of our submission, and observed in **Figure 3**. The GRAINS method **only supports rooms having four walls** in its predicted layouts, while an important feature of our approach is that it takes into consideration complex (i.e., non square) input floor plans. As a result, it is not applicable to a significant number of scenes from our test set. In contrast to DeBaRA, SceneHGN proposes to generate scenes at **the object part-level**, which requires a custom dataset. However, we are willing to evaluate the 3D layout generation capabilities of the method in the revision of our manuscript. > Evaluation Metrics Unlike what is stated in the reviewer's comment, KL Divergence is not measured in our study as object's semantic categories are not part of DeBaRA's prediction space. For the same reason, measuring object concurrences may not be a relevant addition to our paper. Also note that **FID** and **KID** are known to evaluate **both** the **diversity** and the **fidelity** of the generated content. Non-diverse generation results wouldn't comprehensively capture the distribution of real scenes, which would directly penalize the FID and KID scores. Additionally, a better **SCA** score reflects more plausible layouts, as they are *harder* to distinguish from *real* ones. The superiority of denoising-based approaches in capturing symmetry / alignment patterns compared to e.g., autoregressive methods in the context of scene synthesis has been extensively studied by previous work [1, 2]. We also include in our submission's appendix additional indicators measuring the **validity** of generated layouts w.r.t. the provided floor plan (Table 5). > Rendering Quality Our scene renderings, with objects colored according to their semantic categories, have been included to help readers appreciate the quality, diversity and validity of generated layouts, while easily **distinguishing** different objects. We do not claim our renderings to be at the level of those produced by an artist. We used the rendering pipeline provided by the official implementation of DiffuScene (which builds upon the one of ATISS). Our rendering quality is therefore on par with the one of this recently published work. However, also note that our method could be employed along more advanced rendering engines. We will follow your suggestion and include additional qualitative results featuring **textured objects and floors**, which can be obtained using our current rendering pipeline. > It would be insightful to investigate whether this diffusion-based method can produce more diverse scenes Please see our previous answer regarding FID / KID. > How many generated/GT scenes do you use for evaluation? As stated in our main submission, evaluation metrics are computed across each **test** subset (L244), following the splits described L225. For each test subset, we generate the same number of scenes as the number of *real* ones.

Reviewer Fsdh7/10 · confidence 4/52024-07-12

Summary

This paper studies 3D room arrangement/layout generation. It proposes DeBaRA, a diffusion-based generative model, which can generate layouts given the list of furniture and the floor map of the room. The proposed method provides good results and is able to work in various scenarios: layout generation, LLM-guided text-to-scene generation, scene completion, etc.

Strengths

- The proposed method is a novel diffusion-based generative model, which is effective in the aimed task. - The results generated are high-quality both qualitatively and quantitatively. - An LLM-guided alternative pipeline is provided to simplify the input format, which adds more functions to the model.

Weaknesses

- There is no ablation study of the design choices. It is unclear whether and how each design choice and component helps the results. - It would be better if the ablation results could be provided for the following designs: EDM v.s. DDPM/DDIM, designed 3D spatial objective v.s. simple MSE, different classifier-free guidance scales, with v.s. without SSE. - Some of the designs are not clearly described in the paper. Besides, some of the assumptions are not very clear. - I wonder whether adding noise to the scene will cause the noisy layout to contain overlapped objects, objects of unreasonable sizes/rotations, etc. I did not see any of these in Fig.1. It might be better if a visualization of the process of scene being denoised could be provided. - I wonder how the size is determined by the generative model. For example, a sofa can be long or short, a table can be large or small, and a cabinet can be as tall as the wall or as small as a bedside table. How will the model decide the sizes of these objects? Can the user indicate the rough size of the object? - I wonder if it is assumed that most objects should be located in a rigid way that each edge is parallel to one wall? These are observed in most of the results of the proposed model in Figs. 3~6. If so, will these limit the diversity of generation results? - It seems that all the floor maps are relatively simple. I wonder how the model can work in more complicated floor maps, e.g., a floor with many rooms, a round room, or a triangle room. - In contrast to the claim in Q4 of the Checklist, many implementation details are not revealed, e.g., training settings (e.g., lr and iterations) and classifier-free guidance scales. - (Minor) The reason why this diffusion model can work might be that it can be regarded as an extension from some point cloud diffusion models, which also directly apply denoising on coordinates or some explicit presentations. However, this direction was not discussed in the related work. - (Minor) There are some minor math presentation issues. For example, multiple $min$s should be corrected to $\min$s (e.g., L166, L178).

Questions

Please see "Weaknesses".

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors mentioned some limitations in the paper and did not indicate the societal impacts. One possible societal impact might be that "the generated layouts may lead to unsafe constructions, and therefore the model should given warnings about actually using it for constructing real-word rooms".

Reviewer xNz46/10 · confidence 2/52024-07-14

Summary

This paper proposes a method for room layout generation given objects and a floor plan using score-based EDM. First, three encoders are used to encode objects, the floor plan and the noise respectively. These encoded latents are then given as input to a noise based scene encoder which decodes into the output latents. The output latents are then finally decoded to their respective categories and the decoded output is pushed to be closer to the input as measures by a proposed semantic aware Chamfer distance loss. Once trained, the model is able to generate novel layouts and scene in addition to being able to perform completion, rearrangement and retrieval

Strengths

1) Qualitatively, the layouts generated seem far more coherent and spatially aligned than prior work. 2) Quantitative results provide further evidence that this model performs well. 3) The paper is well written and easy to follow.

Weaknesses

1) While using a permutation invariant Chamfer loss is intuitive, I believe it should still be ablated w.r.t just the standard chamfer loss

Questions

N/A

Rating

6

Confidence

2

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes

Reviewer Fsdh2024-08-12

I sincerely thank the author for their detailed and well-organized rebuttal. All my concerns are addressed. I would like to increase my rating from 6 to 7. I would suggest the authors revise their paper according to the rebuttal.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC