Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text

Summary

This paper presents Director3D, a text-to-3D generation framework that creates realistic 3D scenes with adaptive camera trajectories. It includes a Cinematographer (Traj-DiT) for generating camera trajectories, a Decorator (GM-LDM) for initial scene generation, and a Detailer (SDS++ loss) for refinement. Using 3D Gaussians for scene representation, extensive experiments demonstrate its effectiveness.

Strengths

The results are good with high spatial consistency. The task is interesting. It is a good idea to generate scene-level 3D GS directly. The writing is clear.

Weaknesses

1. The quantitative comparison with object-level methods seems unfair. The given prompts include sence information, which efftets clip score. In quantitative comparisons, the authors should at least compare with scene-level methods like LucidDreamer or others. 2. Why didn't the authors compare their method with camera control video generation methods?

Questions

1. Is the number of Gaussians equal to the number of pixels in each view? Why set it that way? It seems that 256*256 Gaussians might not be sufficient for a scene-level representation. What is the total number of Gaussians used? 2. Can this pipeline achieve user-specified camera trajectories? 3. Is it possible to generate a complete 3D scene and then reconstruct it? If the camera trajectory encompasses the entire scene, all 3D information is captured. How does the novel view inference ability perform once the three stages are completed? If only can generate denoised views, the contributions will be weakened.

Rating

5

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

1. What about failure cases? In what situations does the method fail to generate a consistent scene? I find it hard to believe that training on only two not large datasets can achieve such good results. The performance even surpasses object-level methods trained on Objaverse, which doesn't make sense. 2. I want to confirm how the test set is divided. For each class, is a portion of the videos divided as the test set, or are a portion of the classes divided as the test categories? And I want to confirm again that none of the prompts appeared in the training set?

Summary

The paper proposes a scene-generation method from text input. The framework utilizes three models that first generate a trajectory, then produce 3D Gaussians, and finally refine through SDS loss.

Strengths

- The design of using a trajectory generator and 3DGS diffusion is novel and impressive. - The proposed method outperforms existing object-level methods as shown in experiments.

Weaknesses

- The paper's evaluation is limited. The methods compared are object-level generation frameworks. The experiments can be improved by comparing them against scene-level generation methods, e.g. LucidDreamer[16] and ZeroNVS[1]. - I am confused about how SDS++ loss is presented. The paragraph following Eq. (9) contains numerous undefined variables, making it hard to understand how exactly SDS++ loss is formulated. Can I say it is the loss presented in [67] but integrated with learnable text embedding? What's the difference against the loss proposed in VSD[26] and how does it compare? - The proposed trajectory generator is novel. How important is generating the trajectory? Can we just assign some trajectories by retrieving the dataset? It seems that most trajectories in MVImgNet and DL3DV are very similar. How diverse are the generations? I see the results in Fig. 12. But aren't the differences coming from different sampling results of a text-based generator? Since the cameras are already normalized, the orientation of the first frame should not be restricted by the camera trajectory. It seems that the model is overfitting to the orientations in MVImgNet (flying around something on the table). Are the "randomly generated camera trajectories" coming from other objects? Is it because GM-LDM only works with one trajectory per scene? Does the model support working with multiple trajectories for the same scene? - Is the proposed GM-LDM only training the 3DGS scene on observed views? How do the authors avoid overfitting in the rendering-based denoising? Is the SDS++ loss only employed on interpolated camera trajectory? - Is the presented video only showing the views presented in the trajectory or they are novel view synthesis results? How do we know the generated 3D scene is not overfitting to these views? Sparse-view 3D reconstruction suffers from over-fitting issues a lot and I'd like to hear the authors' thoughts on this matter. [1] Sargent K, Li Z, Shah T, et al. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. CVPR 2024, arxiv 2023.

Questions

Please refer to the weakness.

Rating

5

Confidence

5

Soundness

2

Presentation

1

Contribution

3

Limitations

Please refer to the weakness. This work shows brilliant visual results. I hope the authors clarify my questions above to help me better understand how the results are obtained.

Summary

This paper presents Director3D, a novel text-to-3D generation framework designed to generate both real-world 3D scenes and adaptive camera trajectories. Specifically, the authors propose the Traj-DiT to generate adaptive camera trajectories, which treats camera parameters as temporal tokens and performs conditional denoising using a transformer model. The authors propose the GM-LDM and SDS++ Loss to generate robust 3D scenes by leveraging the 2D diffusion prior. Extensive experiments demonstrate the effectiveness of Director3D.

Strengths

1. This paper is written clearly and easy to read. 2. The idea of treating camera parameters as temporal tokens for denoising generation is novel and effective. 3. The GM-LDM and SDS++ loss functions achieve a fairly high level of realistic scene synthesis. The generated scenarios conform to the textual input as well as being reasonably realistic and consistent. 4. The proposed Director3D achieves impressive results on both quantitative and qualitative results.

Weaknesses

1. It would be more convincing to include more text-to-3D scene generation methods in the Quantitative Comparison in the Qualitative Comparison. 2. For ablation experiments of SDS++ Loss, please use more detailed evaluation metrics and experimental results to illustrate the effects. 3. Further implementation of conditionally controllable camera view generation would be helpful for the application of this technology. 4. Some important work in this area such as 3D-SceneDreamer should be discussed if they are not suitable for the experimental comparison. 5. The visual quality of the results seems acceptable compared to existing methods like 3D-scene-dreamer, the limitation of the method, including not allowing a large range of the camera movement, should be discussed.

Questions

No additional questions.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

1. The baseline methods like Fantasia3D and DreamFusion are mainly for object-level generation, not focusing on scene generation. 2. The result visualization can be improved, e.g., the imges are quite small for the reviewer to check the visual quality.

Summary

This paper proposes a framework for simultaneous text-to-3D scene and camera trajectory generation. The authors propose a 3-stage pipeline to (1) generate a dense camera trajectory from input text, (2) use multi-view latent diffusion from a sparse subset of the generated trajectory to generate the 3D scene representation (Gaussian splats), and (3) refine the Gaussian splats with a modified SDS loss.

Strengths

- The paper tackles a practical problem of generating a 3D scene representation while also synthesizing a camera trajectory from text. It has implications of potential further applications for video/movie synthesis using explicit 3D representations. + The paper is well-written, the presentation is clear, and the method description is easy to follow and understand.

Weaknesses

- While this paper tackles a new problem, I have concerns with the problem statement. First, what is a "real-world" camera trajectory? Virtually any kind of camera motion could be created in the real world (be it hand-held shakiness or really smooth orbital trajectories that could be achieved via physical equipment). It seems that the camera motions that Traj-DiT could synthesize are mostly orbital (object-centric) -- why would this be considered as a "real-world" trajectory? - I don't quite get why 3D scene generation and camera trajectory generation should be a coupled problem. NeRFs / Gaussian splats with good quality are not necessarily created via "camera trajectories", but rather via a broad range of covered viewpoints. - Using a diffusion model to synthesize trajectories, the number of frames would be fixed. How does one determine such trajectory length? How can one vary the length? - It is unclear why Cinematographer generates a trajectory with dense frames while only a sparse subset is ever used subsequently. - To evaluate the quality of the synthesized camera trajectory, I believe the method should also be evaluated and compared against with a video generation quality metric (e.g. FVD). The 3D scene representation could be a well-trained NeRF / Gaussian splat, and rendered videos under different trajectories could be quantified and compared.

Questions

Please see the weakness sections.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Interpolation is great. But is there renderings of more distant / novel viewpoints?

Thanks to the authors for the detailed response. A lot of my concerns are well resolved. I am, however, still confused about the **novel view evaluations**. I understand that "The presented video is rendered with uniformly interpolated cameras within the generated trajectories". But 3D Gaussians are known to be easy to overfit and interpolation still doesn't guarantee a reasonable 3D structure. My major question maybe is -- **is the generated 3D Gaussian showing reasonably good geometry and texture?** The current results demonstrate great visual quality on the limited views, but this can be achievable even when the 3D geometry is very poor. If the authors happen to have some renderings of more distant novel views or some renderings of the depth or the point cloud (proving the geometry is truly learned), showing some of these results can make the submission more solid. I notice in Figure 3 of rebuttal, that some novel view renderings are presented. Can the authors please elaborate on whether these cameras are obtained through interpolation? Maybe measuring through relative camera distance can quantitatively show how novel these cameras are. E.g. When the scene is normalized, how far are these novel view cameras from the given views.