SceneScape: Text-Driven Consistent Scene Generation

Summary

The paper present a method for text-driven perpetual view generation. It takes as input a text-prompt and carema trajectory as input, optimizing a unified mesh representation for rendering coherent video sequence accordingly. The key idea is straight-forward by connecting text-driven inpainting and depth warping with mesh-based representation to ensure 3D consistency.

Strengths

- The paper proposes the first method for text-driven perpetual view generation with promising results, i.e., diverse scene textures and inter-frame coherence. - As an emerging direction with limited open-source work for comparisons, the authors have tried their best to compare the proposed method with others from different perspective, i.e., video generation, text-drive vid2vid, text-to-3d, for extensive evaluations. - The paper is well-organized and easy to follow.

Weaknesses

- Only bounded indoor scene as the model is constrained by accuracy of depth map. However, outdoor unbounded scene is difficult to generate since depths has infinite ranges and discontinuities. - Moreover, the proposed method has the strong tendency to generate tunnel-specific scenes, i.e., a narrow hallway or tunnel through the space. It might be caused by the inpainting+depth continuity assumption. Because the surrounding walls acts as strong geometry cues for view consistency as camera moving backward. - Prompt engineering is required. Keywords like “POV” and “walkthrough” is critical for the generation results. The author should discuss their effects and design choices carefully as the paper claims using “free-vocabulary text”. - I may question the second contribution in the introduction, which claims the proposed method as the first zero-shot scene generation method. Unlike zero-shot text-to-3D object generation (DreamField, Dreamfusion, etc.) where a 3D representation is generated and camera can freely move, the proposed method binds the camera movement with the optimization process, which limits the camera trajectory after the optimization done. Therefore, it is only a 3D-aware video generation instead of 3D scene. - This is a personal one. The general idea (text-to-image + depth inpainting) of this method is not completely novel, with lots of engineering tricks to make it work. And I believe some details are missed to evaluate other technical parts like depth & decoder finetuning. Please check the Question section. - I believe the work miss some important literature for reference, in terms of zero-shot text-driven synthesis and indoor scene synthesis. - zero-shot text-driven synthesis: For optimization-based one, there are StyleCLIP and etc. For training without text-image pair, there are LAFITE. StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery, Patashnik et al, ICCV 2021. LAFITE: Towards Language-Free Training for Text-to-Image Generation, Zhou et al, CVPR 2022. - scene synthesis / language-driven scene synthesis: Learning spatial knowledge for text to 3D scene generation, Chang et al, EMNLP 2014 Language-driven synthesis of 3d scenes from scene databases, Ma et al, SIGGRAPH Asia 2018. Text2Light: Zero-shot Text-driven HDR Panorama Generation, Chen et al, SIGGRAPH Asia 2022. - typos: - lacks a full stop in #L126 - Figure or Fig. is not consistent in the paper. For example, #L195. - ‘mesh’ instead of ‘meth’ in #L285

Questions

- Why finetune depth prediction model at test time? Finetuning may overfit/forgetting when the generated video is too long. Plus, the error would accumulate along with time steps. When I come to the ablation study of depth finetuning, I found that the RGB texture dose not jittering across video frame. In that case, does the depth map finetuning matter? - I’m not clear how depth finetuning really work as g is reverted at each time step. - The initialization of mesh M_0 is not clear to me. Does the number of point cloud is consistent with the nubmer of pixels? Moreover, how to get faces and edges of mesh from point cloud? - Online mesh update: the merging in Eq. 6 would lead to conflict. How to solve conflict is not introduced. - Figure 7 shows the failure cases resulted by error accumulation. Here comes the question, can authors evaluate the robustness and error rate of the method? For example, given a fixed set of text prompts, how many scenes can be successfully generated? - Moreover, I would like to see experiments or discussion of sensitivity to hyperparameters.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations and negative societal impact are discussed.

Summary

This paper approaches the task of text-conditioned generation of a walkthrough video upon (mostly indoor) scenes. It proposes to leverage pretrained text-to-image and pretrained depth prediction models, to generate these videos without finetuning. Implementation generates videos in an online manner by repeated warping, inpainting and mesh aggregation. Contributions include test-time overfitting of depth and inpainting, and experiments show these are important to geometric and semantic performance.

Strengths

This paper uses a cool insight to generate 3D walkthroughs using 2D pretrained models - The paper leverages the power of large-pretrained models for online generation of many images in a consistent manner, without the need for finetuning. Careful implementation enables these images to create a consistent scene - Videos are long (50 frames with high movement), showing impressive consistency. They are also typically visually appealing (especially compared to a naïve implementation) Several contributions are executed to make performance high-quality - Finetuning depth and decoder modules with previously generated content meaningfully improves consistency - Dealing with floating artifacts is a clear challenge that the proposed method deals with effectively (Fig 3) - The method performs a union operation on mesh. This is not novel alone, but (correct) implementation is important to performance. - In total, the proposed method improves COLMAP fitting errors by 3-6x over naïve method

Weaknesses

The method appears to be limited to backward movement, with little rotation. This trajectory setup makes the task easier and less interesting, as it does not handle side-to-side movement, or “looking around” which are important components of walkthroughs e.g. see RealEstate10K - Consider the motivation for this paper in L15-16 “By observing only a single photo of a scene, we can imagine what it would be like to explore it, move around, or turn our head and look around”. Unfortunately, this work does not answer this problem satisfactorily. Instead, it produces nice-looking and consistent results as we move backwards through an infinitely-long corridor, where we only slightly turn our head periodically. - More “looking around” or freedom of movement would better test mesh is consistent across dramatically different views and does not have holes. As a viewer, it is also closer to the desired goal of exploring the generated scene. - If the paper were instead focused on the infinite-generation direction, perhaps a more appropriate set of experiments and direction would be those relating to Infinite Nature? Generated videos are of houses with unrealistic structure, breaking the desired illusion - The trajectory of cameras means the video moves backward with a zig-zag type pattern. Walls tend to move correspondingly, and are typically equally at both edges of the video. - This results in sequences where the viewer moves down an infinitely long hallway that goes back and forth as the camera does. This feels unrealistic; I can’t imagine an indoor structure that looks like any of the videos. - I feel prior work (e.g. GEN-1 or VideoFusion) actually may generate houses that are in some ways more realistic, as they do not always have walls at the edge of the video frame (although the difference in movement between each frame makes the comparison difficult to judge) Experiments feel somewhat ad-hoc. The small size and lack of consistency / justification for prompts make it harder to draw conclusions from results - Ablations take place on 20 videos using 10 text prompts (small set). It is also surprising to use only 10 text prompts when generating 1000 videos for the VF experiment; or 5 text prompts on 110 videos against GEN-1. Results would be much more compelling if using a different prompt for each video. I’m curious if there is a limitation I am missing that could explain this. - Prompts are not programmatically generated, and I did not find a justification for the selected prompts (lack of consistency; do not reflect some true data distribution). Programmatic generation of prompts would also more easily enable scaling to larger sets. - Prompts contain significant unique detail e.g. “indoor scene, interior, candy house, fantasy, beautiful, masterpiece, best quality” and “POV, haunted house, dark, wooden door, spider webs, skeletons” (feels as if it could be prompt engineered, doesn’t feel reflective of some true distribution). Edit: Thanks for the rebuttal. I think the results with more significant turns are better, and showing some examples without prompt engineering is helpful. I feel after reading the rebuttal and other reviews the paper is still a 3 - reject. I stand by my original comment the lack of systematic experiments in the paper make it hard to draw clear scientific conclusions from this work. In additon, while results are consistent across frames, the completely unrealistic nature of generated houses mean this contribution is not impressive enough for acceptance in my mind. The reviews of this paper are very mixed. Despite my negative review, accepting it would not be the end of the world if other reviewers end up feeling very positive given (1) the generated frame quality and cross-frame consistency and (2) a few technical contributions e.g. floaters, finetuning depth.

Questions

- Motivating the intro with single-image novel view synthesis is a strange choice, and is a bit confusing, when the focus of this work is text-driven scene synthesis - Missing period L126 - Slightly strange writing to see a list (L130) enumerated into subsections. It would feel more typical to change this sentence to finish with a period, and overview the subsections, or to simply make subsections as paragraphs. - One addition metric that could be useful is CLIP R-Precision, which is used by prior work e.g. DreamFusion. By selecting some random frames in each video and ranking CLIP distance to all text prompts, one can compute precision. This could be helpful as automated visual semantic metrics can be limited in a setting such as this.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

1. This paper proposes a method for test-driven consistent scene generation using a pretrained text-to-image model and depth prediction models. 2. The updating of mesh representation and fine-tuning of the decoder and depth prediction module enable continuous scene generation.

Strengths

1. The continuous scene generation demonstrated in both the main text and supplementary materials proves the effectiveness of the method. The proposed method generates more continuous results than the baseline method, given the same initialization and camera trajectory. 2. It is interesting to see the fancy effects achieved by utilizing pretrained inpainting and depth prediction models.

Weaknesses

1. There is a lack of analysis on cumulative errors. 2. The efficiency analysis provided in the supplementary materials could benefit from further optimization.

Questions

1. The paper could provide additional analysis on cumulative errors and failure cases to further demonstrate the robustness of the proposed method.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The paper does address the limitations of the proposed method.

Summary

In this paper, the authors study text-driven perpetual view synthesis with the input of text prompts describing the scene and desirable camera poses. Instead of training models on a specific domain, they propose an online scene generation method with a pretrained text-to-image model and a pretrained monocular depth prediction model. To further improve the video consistency, they utilize a shared mesh to represent the generated scene and deploy online time-time training to finetune the monocular depth prediction model for better mesh construction. The experiments demonstrate that the proposed method can generate geometrically-plausible scenes, and the proposed modules can improve the video quality quantitatively and qualitatively.

Strengths

- The authors directly deploy a pretrained text-to-image model and a pretrained monocular depth prediction model as a strong indutive bias, enabling their method to work on diverse scenes in a zero-shot manner and eliminate training datasets. - The generated videos are consistent and geometrically plausible. With the text prompts, the users can decide on the generated scenes. - To further improve the online generated scenes, the authors deploy a unified mesh representation and adopt test-time training to finetune the depth prediction model, which are validated to improve the results in the extensive experiments.

Weaknesses

- The main concern of the proposed method is that the presented results are nearly all with camera trajectory moving back. Can the proposed method work on trajectories similar to those in Text2Room? Or does some inherited property limit it to work on those camera trajectories? - I have a question on the need for the decoder finetuning. When we want to apply the text2image diffusion model for inpainting, we can replace the existing part with the input though the decoder can not perfectly reconstruct the input. In this case, why do we need decoder finetuning? - When watching the visual ablation study, it is actually hard to distinguish the improvement of the proposed modules. The demonstrated videos are nearly all moving backward and corridor scenarios. In this case, only the warp-inpaint baseline will work well. - In Line 222, you mention the depth-aware splatting. Can you explain how you elaborate more on this baseline? - Why is the mesh so blurry in Figure 7 (left)? Worldsheet also uses the mesh representation, but their boundary is not that bad. Have you tried to split the mesh into foreground and background to help the discontinuity of the scene? - In the supplement video results, the warp-inpaint baseline is missed for visual ablation.

Questions

Please answer the questions mentions in the weakness part. I will adjust my score according to the rebuttal.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors have properly addressed the limitation and broader impact in the main paper and supplement.

Summary

This paper proposes a text-driven perpetual view generation method that synthesizes long-term changing views of a scene and its corresponding 3D mesh, given the free-vocabulary text and camera trajectories. The authors apply an off-the-shelf pre-trained text-to-image diffusion model and a pre-trained depth prediction model to achieve the baseline quality of novel-view synthesis by reprojection and inpainting, and also apply a test-time finetuning scheme to enhance the geometry consistency and textural consistency throughout the view sequence.

Strengths

1. The paper is easy to follow and provides many impressive qualitative results. 2. It is promising that this method can synthesize diverse and zero-shot scenes without large-scale training data on a specific target domain. 3. The SceneScape pipeline is technically sound to generate consistent views as its novel views are grounded in the scene geometry, where the generation of novel contents and geometrical structures can be readily adapted from off-the-shelf text-to-image diffusion-based inpainting/generation models and depth prediction model. 3. By maintaining a unified 3D mesh representation and adopting a test-time finetuning technique, this method achieves even more consistent results. Such a technique is meaningful for camera pose-conditioned or other geometry-aware online scene generation pipelines.

Weaknesses

I am almost satisfied with this manuscript. Here just list a few questions other than the limitations discussed in the paper, as follows: 1. The SceneScape seems not to be able to handle dynamic foreground content. If the textual descriptions tell the model that it has to generate ``dancing people'', what will SceneScape do? Moreover, as one possible downstream application, how to edit the perpetual scenes generated by SceneScape? 2. There are still quite a few strange distortions in the synthesized scenes. Can it be further improved by some instance-level priors? For example, knowing that there exists a chair, and we can use the prior knowledge of chairs to rectify the shape distortions? Maybe adding the prediction of instance segmentation maps into the SceneScape pipeline will be helpful.

Questions

1. The footage provided in the supplementary is all about a walk-through towards a straight corridor. Curious about if it can handle more complex situations, e.g., turning left/right/back? 2. How many prompts are used in total in the test stage? Can you give a full list of them? It is unclear if the prompts cover various indoor scenarios and further affect the metrics. 3. It seems time-consuming to synthesize one frame according to the supplementary material. A comparison of the runtime with the reference methods would be helpful.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have stated limitations and potential negative societal impact in the paper.

Rebuttal Response by R4 (Zj8B)

Thanks for the rebuttal. I think the results with more significant turns are better, and showing some examples without prompt engineering is helpful. I feel after reading the rebuttal and other reviews the paper is still a 3 - reject. I stand by my original comment the lack of systematic experiments in the paper make it hard to draw clear scientific conclusions from this work. In additon, while results are consistent across frames, the completely unrealistic nature of generated houses mean this contribution is not impressive enough for acceptance in my mind. The reviews of this paper are very mixed. Despite my negative review, accepting it would not be the end of the world if other reviewers end up feeling very positive given (1) the generated frame quality and cross-frame consistency and (2) a few technical contributions e.g. floaters, finetuning depth.