4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models

Summary

4Real presents a novel framework for dynamic 3D scene generation. Given an input video, the framework selects a frame and prompts a text-to-video diffusion model to generate a "freeze-time video" that reconstructs the canonical 3D scene. Then, video-SDS-based optimization is used to optimize a dynamic 3DGS with reference to the input video. Compared to existing approaches that focus more on object-centric generation, the new method is able to generate more complex 4D scenes including multiple objects.

Strengths

1. 4Real generates photo-realistic 180-degree 4D scenes for the first time. In comparison, previous 4D generation works are mainly object-centric, restricted by the object-centric image models that they rely on. 4Real only relies on native text-to-video diffusion models, thus addressing the bias. The approach is more favourable than 4Dfy and Dream-in-4D as suggested by a user study. 2. Although the work mainly follows the previous video-SDS-based 4D generation frameworks, it proposes several effective techniques to enable modeling more complex scenes that include multiple objects. A key design is to generate the canonical 3D through freeze-time video and video extension. An additional dynamic 3DGS fitting and a heuristic loss (small motion regularization) are proposed to offset the remained motions in the freeze-time videos, which I find reasonable and novel. 3. The paper also provides empirical values to the video-SDS-based 4D generation frameworks. It reduces the video-sds computational cost dramatically by leveraging low-resolution pixel diffusion models.

Weaknesses

1. The description of context conditioning (L149) is not clear to me. "One dataset consists of static real-world objects with circular camera trajectories", is this dataset similar to [CO3D](https://ai.meta.com/datasets/co3d-dataset/), does it contain background, and how critical is the context embedding? Including this information can be important to reproduce this work. 2. The evaluation did not compare with AYG [28], whose results could also be easily accessed. AYG optimizes a dynamic 3DGS while the compared 4Dfy and Dream-in-4D optimize a dynamic NeRF, so I would say AYG is closer to this work. Although similar conclusions could be drawn for both AYG and 4Dfy/Dream-in-4D as they are all object-centric, the argument in L271 that claims "These methods are known to achieve some of the best visual quality compared to AYG and ..." is neither well-grounded nor convincing. L270 is also wrong to claim that Dream-in-4D is based on D-3DGS. 3. Generating a freeze-time video for the static 3D generation makes a lot of sense, but I wonder if it is possible that the generated freeze-time video could conflict with the reference video. Moreover, the canonical 3D generation stage seems to be solving a standalone image/text-to-scene task, it would be interesting to know if recent 3D scene generation works like RealmDreamer[A] and CAT3D[B] could be a drop-in replacement. Minor: - Editing is mentioned in the contributions, but it has not been discussed in any other places in the paper. [A] Shriram et al., RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion [B] Gao et al., CAT3D: Create Anything in 3D with Multi-View Diffusion Models

Questions

Please kindly address the weaknesses above.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Summary

The paper introduces a pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. The method begins by generating a reference video using the video generation model, then learns the canonical 3D representation of the video using a freeze-time video, delicately generated from the reference video. To handle inconsistencies in the freeze-time video, the method jointly learns a per-frame deformation to model these imperfections, then learns the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video.

Strengths

1. The paper delivers impressive visual results for 4D scene level generation. The quality is outstanding and scene level generation is more general than previous object level generation. 2. The paper is well written with elaborated technical details.

Weaknesses

1. The method seems extremely slow due to multiple steps being applied. The method hasn't provide enough latency comparisons. 2. Besides user studies, I wonder if any qualitative results can be shown, for example, FVD. Or some video sequences from multiview data (in door or outdoor) and compute reconstruction loss, but applying the lifting for reference video (using one view video in the data set) and render the other view to compare with the gt novel view video. 3. The model has a freeze time view deformation field, which indicates fitting static 3dgs scene from freeze-time video is unable to deliver good initial 3dgs field. How to balance the 3d consistency and plausibility from freeze time camera prior? More reasoning and analysis is desirable. 4. The paper uses freeze time SDS some part and multi-view SDS somewhere else. Better to improve the consistency.

Questions

1. The paper mentioned a freeze time view deformation field. Does every frame has it or it is only applied to the first frame. 2. How important is the freeze time view deformation field. 3. The motion video diffusion needs to sample with the first frame, how does the video diffusion handle different sampled fps? Using it as a input? This setting potentially limits the extension of this method to longer video.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Multiview video can be a better prior, it is limited by training data, but will improve in the future. 4D generation might be benefited from this prior and reduces the requirement of multiple stages.

Summary

This paper proposes a straightforward method to distill a 4D dynamic scene from a video diffusion model. Given a reference video (only generated videos are shown in the paper, no real video), they first generate a freeze-time video as a multi-view to reconstruct a reference 3DGS. Because they don't have a good multiview image model to capture the general scene multiview prior, they can only use a video model, which might produce small deformations. Some engineering effort is paid here to accept such deformation. After the reference Gaussians is reconstructed, an SDS stage is applied to both the frozen time multiview images and the temporal deformed frames.

Strengths

- This is an early attempt to generate 4D scenes from video diffusion models - Although very straightforward, the system works for generating some 4D videos.

Weaknesses

- Object-centric? One important constraint this paper assumes is that because they want to generate dynamic scenes, they don't use sophisticated object-centric models. However, most video examples this paper shows do not have significant background motion, so one strong baseline/alternative is to just distill a static background and put a dynamic subject (from a dynamic object-centric SDS) in the foreground. Maybe this would achieve much better results than the paper's pipeline. - How consistent? One question arises when you have a reference temporal video. Because there is a hallucination of the freeze-time view only considering a specific frame, however, the hallucinated freeze-time multiview images may not be consistent with what is observed in the temporal video, how does the system handle such in-consistency? - Failure case: when i open the static website files in the supply, there is a folder called "good results", which the reviewer guesses may be because of some case selection, it would be critical to also show failure case.

Questions

The authors also have to highlight their novelty, since the method is straightforward.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Some limitation is discussed, but no failure cases are shown.

Summary

This paper proposes a new pipeline called 4Real, aiming for more photorealistic text-to-4D generation than prior work. The method first generates a reference video, then learn a canonical 3D representation from a freeze-time video. Afterwards, per-frame and temporal deformation are learned to model the gap between the canonical representation and the targeted video. This paper compares with two baselines and includes both visualizations on 30 examples and user study. The method is also much faster than prior work.

Strengths

1. This paper has lifted the requirement of training a 3D generative model using limited synthetic data from prior work. This is important since there are much less available 3D assets than videos in the real world. This paper shows a potential way of fully unleashing the power of the abundant diverse video data in improving the text-to-4D models. 2. The model takes 1.5 hour for testing, which is much faster than 10+ hours from previous methods. Although it is still very slow compared to other text-to-X models, it's already a great improvement. 3. The paper has a good presentation and is well-written. It also shows all the testing results in the supplementary.

Weaknesses

This paper has clearly identified quite a few key limitations of prior work and aims to solve them, which is great. But I don't think all of these claims have been validated. For example: (1) this work captures the interaction between objects and environments From all the submitted video results, they do not seem to include much (or any) interaction between the object and the environment (e.g., relative global movement between them). For example, for the "bear driving a car" comparison between ours and 4Dfy, the car in "our result" is not moving on the lawn. The movement only comes from the bear inside the car. (2) prior work being object-centric while this work not. Although it's clear that the compared baselines only generate foreground object while the proposed method also creates the background environment. Since there's not much interaction between the background and foreground (as explained above), a better baseline would be generating an empty background/environment that suits the prompt/foreground and then putting the baseline-generated object properly in the environment.

Questions

1. From the results, it does seem that the baselines generate much less realistic objects than the proposed method as the paper has claimed. However, existing text-to-3D models have proved their power in generating realistic 3D assets [20, 24, etc.], so could it because the 3D generation model used in baselines are a bit outdated or the prompts were not the best ones? Maybe the realism in the baseline results can be largely improved by a bit prompt engineering (e.g., adding keywords like "realistic" in the prompt) or updating their used 3D generation model? 2. Does the deformation model also time-varying appearance or only time-varying geometry? 3. Minor suggestions: (1) maybe adding the input prompt to Fig. 4 may help the readers better understand the output content and quality. (2) Some typos: L24: ", E" -> ". E" L292: missing a period.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitations seem to have been sufficiently discussed.

Summary

The paper presents a method for 3D scene generation using video diffusion models. The main contribution is to present a method that does not require a multi-view generative model trained on synthetic data. The paper identifies the use of multi-view models as a strong limitation limiting existing works to generate high-quality scenes. The method first generates a video from the text prompt, and then also generates a second video starting from a reference frame that corresponds to a time-freezed camera scan of the scene. The second video allows for a static 3D reconstruction of the reference frame, that is then used in reconstructing the motion in the original video. The proposed method is novel and achieves better results than the state of the art.

Strengths

The paper is very well-motivated and well-written. The key limitation in existing works is very clearly and explicitly identified. The method is explained clearly. The paper does a good job explaining a complex pipeline and the supplemental provides further details. The evaluations demonstrate very clear improvements over the state of the art.

Weaknesses

The paper shows high-quality results that outperform existing work, and also present ablations. I was curious whether the identification of the use of synthetic data could also be demonstrated as an ablation, e.g., finetuning the snap video model on synthetic data and adding another SDS loss to demonstrate that it is indeed the key factor that limits high quality. The paper in the intro mentions "reliance on multi-view generative models". While the video generative model used in the paper does not take camera poses as conditioning, it is in fact trained on static multi-view data. If so, I would still call such a model capable of generating multi-view images as a multi-view model. How would the overall approach perform if the snap video model was not trained on static multi-view scans? It would be good to provide more details on the training data. What percentage of the training data consists of such multi-view recordings? The paper claims efficiency as a contribution; however, later in L225-228, it correctly points out that most of this improvement is due to the use of snap video model, and not due to any contributions of this paper. If true, the claims should be adjusted. From L161-163, it seems like the sufficient amount of pose variation is chosen manually? Is it not possible to automate this? If this step is manual, how is it accounted for in the reported processing times?

Questions

Does Colmap work for all the dynamic videos generated, even if the motion is large? The results are generally very impressive but it would be very helpful to the community to also show failure examples so that the remaining challenges are clear.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes.

Keep my positive recommendation

After reading the reviews and rebuttals, I still think this paper provide valuable contribution to the community. The authors should include the above references [1-5] and discussions in the revision to better clarify the potential concerns, also, if possible, report the failure ratio that how many bad or good results will be generated from the 100 videos. I keep my positive recommendation.