Language-driven Scene Synthesis using Multi-conditional Diffusion Model

Summary

This paper approaches the task of predicting a location and orientation of furniture, conditioned upon a person’s motion sequence, existing furniture, and text. By conditioning on text, which prior work does not do, the proposed method enables users to actively specify furniture location. In addition, experiments show it enables object replacement, shape alteration and displacement. This work introduces captions to prior human motion-furniture dataset PROXD, and shows SOTA performance given captions.

Strengths

This paper adds a useful contribution to the task of human-guided scene layout - Prior work generates furniture from motion. This work enables users to specify location using text, making it much more applicable to real-world scenarios - Gathering captions to a standard dataset PROXD enables the proposed method to significantly outperform prior work across metrics. The paper contains good analysis of text prompts in Supplemental. - The method also can leverage text from HUMANISE, again enabling significant improvement - The approach also enables object editing, which is experimentally evaluated The method proposes the intuitive approach of “guiding points” to this conditional 3D diffusion task, which it shows is highly effective in experiments. - The idea of conditioning on a weighted combination of predicted locations from each conditioning component is intuitively more powerful than conditioning on latent encodings - The paper provides theoretical guarantees guiding points explicitly contribute to denoising - Experiments show conditioning upon guiding points is more effective than translation vectors alone, or unconditional - Experiments also show correlation between accuracy of guiding points and final performance, empirically confirming theoretical findings.

Weaknesses

Edit: after rebuttal, my concerns are well-addressed. There are several missing details and comparisons that make full assessment of the paper challenging. This includes missing limitations. - I do not understand how the training works for scene synthesis, which makes it hard to fully assess the importance of guiding points. My understanding is the LSDM denoises a point cloud given the output of the guiding points network. I assume then, the guiding points network is trained jointly, end to end with the LSDM, at each denoising step? And that no networks are pretrained, including text encoder (I’d assume this is pretrained)? This would mean S is not actually trained to predict final position, but rather consists more of geometric features? Based on Figure 6, it is hard to determine if S is directly supervised. - It feels like another reasonable design choice would be to initialize diffusion with guiding points, and denoise these, as opposed to (or in addition to) conditioning upon them. Testing this choice could perhaps more directly validate the theoretical findings that using S specifically for conditioning is helpful. - In qualitative results, a single human location is used. However, the proposed task is to consider a vector of human locations (“motion”). How does text work given the input is not a single location, but a set of locations? Text is sensitive to location e.g. “Place a desk in front of me”. Is time assumed to be the last timestep? In this case, does the dataset generate text descriptions based only on the last location of the human? In reality, I would imagine users would like to specify text based on any number of timesteps throughout the trajectory e.g. “place the desk in front of me [at frame i<N of N]” - The comparison to multi-conditional modeling is not fully satisfying. The method compares to itself without F, but otherwise keeps the same geometric-based architecture. Namely, it combines weights linearly using w. A more standard conditional diffusion approach would be to concatenate features or combine them through nonlinear (e.g. transformer) layers. This comparison would make the argument for the proposed method stronger. - Is there a breakdown into in-contact vs. not in-contact objects? Prior work specifically uses this; as this paper claims to outperform in not-in-contact objects, it would be a helpful metric to report. Contributions feel slightly niche (minor weakness) - The central contribution of conditioning upon a geometrically transformed linear combination of feature distances of objects and humans is a cool contribution. However, it feels specific to the task of furniture placement conditional on human motion and text. Is there a wider reason this method is important? - Saying scene synthesis has gained significant attention in the past few years and citing one paper from last year with 3 citations (L16) is not convincing the task is very important - Text-conditioned diffusion models predicting position and orientation already exist in the near subfield of human motion generation (Tevet et al. Human Motion Diffusion Model, ICLR 2023). The application in the near subfield of furniture position and orientation on its own feels like a relatively modest step.

Questions

- L4: “combing” -> combining - L51: “a new challenging” -> “a new challenge”

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No. I would recommend particular focus on the ability to utilize a sequence of human motion (see weaknesses), assuming this is a limitation. Others could include the assumption one knows the object of interest and further has a 3D model of it.

Summary

This paper focuses on language-driven scene synthesis, a new task integrating text prompts, human motions, and existing objects as multiple conditions. The proposed task is challenging as it requires a strategy for encoding the multi-modal conditions into a unified space. To solve the problem, the authors introduce a novel guiding points concept to combine multiple conditions, which can explicitly contribute to the denoising process. They also introduce three scene-editing applications based on the text prompt input. They demonstrate the approach empirically and theoretically; the intensive experiments show that the proposed approach achieves significant improvements over the state-of-the-art methods.

Strengths

1. Extending the scene synthesis to a language-driven setting, incorporating text prompts and human motions as input, holds great promise and significance in bridging the gap between research and real-world applications. It also enables downstream real-world scene editing applications. 2. This paper proposes a somewhat novel method to handle such a multi-conditional setting. The authors introduce guiding points that explicitly guide the reverse process of the diffusion model, offering a departure from the implicit unification approach used in previous multi-conditional diffusion models. 3. This paper's theoretical demonstration and experimental analysis are comprehensive, especially the ablative experiment, which demonstrates the impact of different modalities and how the proposed modules contribute to the overall performance.

Weaknesses

1. In comparing MIME to your approach, I notice that MIME focuses on generating 3D scenes based on 3D human motion, whereas your method takes a human pose as input. Considering this distinction, is it fair to make a direct comparison between MIME and your approach? 2. It has come to my attention that Proposition 2 and Corollary 2.1 are included in section 3.2. However, it may be more suitable to relocate them to the supplementary materials. I am uncertain about the significance of these components within the section, and it seems that the author included them primarily to showcase their expertise.

Questions

1. Could you please elaborate on the methodology used to convert the point cloud into an object mesh shown in your figures and video? 2. I'm curious about the performance of directly learning the unification of multiple conditions with a diffusion model. It appears that the proposed method outperforms others, but what about the performance gap between these two methods? 3. Based on my comprehension, does your model exclusively utilize a single-frame human pose as input rather than a motion sequence? If that is the case, I'm curious about the rationale behind this choice, considering that a motion sequence could potentially provide more comprehensive information about the scene distribution. By the way, how do you represent the human body? Minor fixes: - The video in the supplementary material is excellent, but you can provide some visualization of raw point cloud results. - Authors are advised to provide a limitation and future work section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have not discussed this paper's limitations and societal impact.

Summary

This paper targets to generate 3d scene by conditioning on text prompts and other inputs, e.g., room layouts. For this purpose, they operate on 3d point cloud representation and propose a multi-conditional diffusion model to generate guiding points to achieve 3d scene synthesis purpose. The experiments are evaluated on synthetic indoor dataset.

Strengths

- Adopting human pose into 3d scene generation process is a novel condition to consider during generation.

Weaknesses

- The authors did not motivate well on why a diffusion model is necessary or better for this task. Given the large amount of prior work in scene layout, is there any advantage of diffusion model, such that it can do something prior method cannot? - Their dataset is too simple. On one hand, the authors deal with 3D point cloud representation, which usually noisy and sparse in real-world scanning data. On the other hand, they only test the solution on synthetic dataset, which seems to be in a different distribution with real-world scan. An evaluation on real-world dataset can make the world more solid. - Their video supplementary is confusing in terms of what kind of application that are aiming at. Is the audio cut off accidentally in the mp4? As for the application, is that the authors hope to leverage human pose to generate 3d objects in the indoor scene?

Questions

1. Could you justify more on the method choice of diffusion model in this task? 2. The citation format is not the one suggested by NeurIPS 2023. Please check submission website to adjust in later revision. 3. What application is the proposed technique aiming to realize?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

good.

Summary

This paper deals with scene synthesis with human pose, room layout, and text prompts. The main architecture is a multi-conditional diffusion model, which performs progressive generation, where a new object is synthesized and conditioned on the existing scene point cloud and the language description. The key contribution is a guiding point network, which first generates a reference point cloud as a weighted sum of existing objects and human pose, then the reference point cloud is used as a condition to guide the denoising processing for a new object. The trained network allows scene generation guided by language and can produce semantically meaningful scene edits.

Strengths

The progressive generation of scenes guided by language makes it much easier to interact with the synthesis process and alleviates the control burden from the designer's side. The experiments show the effectiveness of the proposed pipeline and the learning objective. The ablation study is interesting.

Weaknesses

The architecture is quite intuitive, but the derivation is not quite clear and seems disconnected from what the author wants to do. Equations (2) and (3) are standard, but starting from equation (4), when the guiding point is introduced, some unsureness kicks in. For example, why do you assume that x_0 is a uniform distribution over a domain S, how do you define S in the first hand, and why x_0 be uniform is a good assumption? Also, why q(y|x_0) is non-zero uniform over S? What is the difference between S and S_hat, and what do you mean by sampling set of x_0? Why then q(x_0) becomes uniform? How do you infer q(x_0|y) is also uniform over S? how? If q(x_0) is uniform, then \mu_0 is the center of the region S? why this is a meaningful quantity in your consideration? Why is S_tilt a sampling set of S_hat? Eq. (10) is very intuitive, why do we need all the previous derivations?

Questions

See above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No limitation is discussed.

Summary

In this paper, the authors propose a new task named language-driven scene synthesis. This new task takes text prompts, human motion, and existing objects to generate the next object in the scene. To handle the multiple conditions, they design a guiding points strategy to unify them. It first explicitly predicts a "pseudo" target point cloud from the conditions and then uses these predicted points as a guide for the diffusion model to predict the "truly" target point cloud. They demonstrate their approach is theoretically supportive. In the experiment, they show that their method outperforms the state-of-the-art baselines. Furthermore, they introduce three scene editing tasks that are useful for application.

Strengths

- The proposed language-driven scene synthesis task integrates text prompts, human motion, and existing objects as conditions. It is an interesting direction that injects user preference for scene synthesis and thus enables real-world scene editing applications with text prompts. - To handle the multiple conditions, the authors revisit point cloud representation and propose a guiding point concept to use the conditions explicitly. They first predict a "pseudo" target point cloud from the conditions and then use these predicted points to guide the diffusion model to predict the "truly" target point cloud. This explicit strategy injects a strong inductive bias to utilize all the conditions for placing the next object. - The experiment part is intensive and demonstrates the proposed method with text prompts, human motion, and existing objects as conditions achieve the best results compared with baselines.

Weaknesses

- I have a question regarding the application of the proposed new tasks. When we take only human motion as a condition for scene synthesis, MIME (Yi et al., 2023) treat this as "turning human movement in a "scanner" of the 3D world." In your proposed task that uses human motion and text prompts, I understand it is useful when we want to place a table in the VR setting. However, what is the use case if the human motion is setting down and using "put a chair under the human" as a prompt? We can not use this in VR since we can not sit without a real chair. - I don't like the presentation of this paper. The reasons are as below. 1) Section 3.2 seems to break the flow of the whole paper. After reading this subsection, I need to back to Section 3.1 multiple times to remind myself of the notation for Section 3.3. It is suggested to make the theoretical support in the main paper shorter and at a high level and move the others to the supplement. 2) Since Section 3.2 uses too much space in the main paper, the authors make Section 3.3 short and unclear. However, this is the main contribution of this paper. It is messy for the audience to read the operations with only unclear text descriptions (also without any shape information for the variables). It is suggested to add equations or pseudocode to describe the operations. 3) Figure 2 is also unclear. For example, for the text, it is stated that "the input key is the text embedding e′, the input queries are the given scene entities." However, in Figure 2, the text embedding e′ and the scene entities are concatenated and then fed to the attention modular. 4) I read the supplement. The motivation part deserves to be moved to the main paper. The implementation details also need to be clarified. Especially, Table 2 is unreadable. Why not list the equations of the operations? - For the baselines, can you add text prompt conditions to MIME for your proposed task for a fair comparison? Considering their method is transformer-based, it should be easy to add text conditions. - For the editing tasks, is the target object necessary to be the M+1 object? In the text prompt, the target object is already indicated. In this case, it seems that we can change any object in the scene instead of only the last one. - In Line 171, you claim that "we extract spatial information from the text prompt by utilizing the off-the-shelf multi-head attention layer." What is the meaning of "off-the-shelf" here? - In Table 2 of the supplement, the output of the text encoder is 1D. I remember that the output of the CLIP text encoder is a list of tokens. Do you apply pooling here?

Questions

Please refer to the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

It seems that the authors do not properly discuss the limitation and the broader impact of their work.

Thanks for the reply

Thanks for your response. I have no further questions.

Looking forward to your reply

Dear Reviewer uwsL, Thanks for your hard work in the reviewing process! Please let us know if you have any further questions before the end of the author-reviewer discussion phase. Thanks, Authors.

Let us know if you have any further questions

Dear Reviewer aT28, Thanks for your efforts in the review! Please let us know if you have any further concerns before the end of the discussion phase. Thanks, Authors.

Looking forward to your response

Dear Reviewer RVaZ, Thanks for your endeavors in the reviewing process! Please let us know if you have any further questions before the end of the author-reviewer discussion phase. Thanks, Authors.

Please let us know if you have any further concerns

Dear Reviewer YsUe, Thanks for your constructive efforts in the reviewing process of our paper! Let us know if you have any further concerns before the end of the discussion phase. Thanks, Authors.

Reviewer Response to Rebuttal

After reading the rebuttal and other reviews I change my rating to 6 - weak accept. I believe the paper should be accepted as (1) I agree with the other authors adding text conditioning to human-guided scene layout is an important contribution; (2) contributions of the method, such as the interesting "guiding points" are well-defended in experiments; and finally (3) clarifying my concerns about training details, denosing design choices, etc.

A friendly reminder

Dear Reviewers **uwsL**, **aT28**, **RVaZ**, We sincerely appreciate the time and effort throughout the reviewing process of our submission. As the author-reviewer discussion is due in about 1 day, please let us know if you have further questions about our submission. Once again, thank you in advance, and we look forward to your feedback. Best regards, Authors.

Response to rebuttal

Thanks for the detailed rebuttal and the revision of the manuscript. The revision makes the presentation more clear now. I would like to raise my score to WA.