4Diffusion: Multi-view Video Diffusion Model for 4D Generation

Summary

The paper propose a 4D generation pipeline, namely 4Diffusion, aimed at generating spatial-temporally consistent 4D content from a monocular video.The authors design a unified diffusion model tailored for multiview video generation by incorporating a learnable motion module into a frozen 3Daware diffusion model to capture multi-view spatial-temporal correlations. After training on a curated dataset, the diffusion model acquires reasonable temporal consistency and inherently preserves the generalizability and spatial consistency of the 3D-aware diffusion model.

Strengths

1. The paper proposes to generate multiview video to guide the 4d generation, the direction is reasonable. 2. The paper is easy to follow.

Weaknesses

1. The paper's novelty wouldn't be its biggest strength, but training a multiview-video module is a good direction so, this point is moderately pass the bar of NeurIPS. 2. The results look temporally inconsistent (color flickering), see the frog man's eye and wolf (w/ rider) 's tail. This might due to insufficient training samples. 3. As another nerf based model, the results are not much better than consistent4D.

Questions

1. The improvement from this multi-view generation model is limited, the colors are still flickering, especially from the generated multview frames. 2. I wonder if the limited consistency is due to insufficient amount of training data.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper hasn't show the full potential of the multi-view video for 4D generation, sourcing more synthetic data is important to improve the results.

Summary

The paper proposes 3D-aware diffusion model trained on a curated 4D dataset for video-to-4D generation. 4D-aware Score Distillation Sampling loss is introduced to optimize 4D representation parameterized by dynamic NeRF. The proposed framework outperforms optimzation-based baselines.

Strengths

- A new subset of animatable Objaverse is presented and improves the model's generation ability as shown in Table 2. - The proposed 4D diffusion model outperforms optimization-based baselines.

Weaknesses

- The proposed method requires 12 hours on A100, which is significantly longer than baselines. - I am confused by the quantitative evaluation design as mentioned in L244. "we calculate FVD between the input video and synthesized videos to evaluate the video quality". What's the purpose of comparing against input videos? If my understanding is correct, the ground truth 4D objects from Objaverse can be rendered into ground truth videos and can be used to calculate FVD, right? - The proposed anchor loss seems very similar to the ones proposed in 4DGen[62] and is not properly discussed in L200. - The back views in Fig. 5 seem very blurry and contain transparent edges. What might be the reason for these artifacts? Is this because of SDS loss? - I am mainly concerned about the test set construction. Does this manually filtered training set overlap with Consistent4D test set? Many of the Consistent4D test sets are originally from Sketchfab.

Questions

- I appreciate the authors' effort in manually filtering the dataset. I'm interested in how many objects were there before filtering out the 966 objects? - How are the input videos for evaluation constructed? Are they ground truth videos from the Objaverse dataset? - What are the reference videos for calculating the FVD metrics? Are they ground truth videos from the Objaverse dataset? For table 2, are the reference videos the same?

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

Please refer to weakness.

Summary

This paper tackles the task of 4D reconstruction from monocular video. It introduces a training approach for a multi-view video generative model using a synthetic dataset of multi-view videos. The architecture uses a 3D-aware denoising diffusion model previously applied to multi-view images and extends it to accommodate multi-view videos. The model is fine-tuned using 1,000 synthetic multi-view videos from the Objaverse dataset. Then, score-distillation sampling (SDS) is used to generate a dynamic radiance field. The evaluation on videos of synthetic object-centric scenes demonstrates a slight improvement in terms of CLIP and FVD metrics over the recent Consistent4D work on the task of novel view synthesis from monocular video. Although the qualitative results show minor enhancement over baselines, concerns remain about the generalization to real-world videos and the evaluation, especially regarding potential training data leakage and significance of improvement over Consistent4D. Addressing these issues would warrant the acceptance of the paper.

Strengths

- The paper addresses the significant and timely issue of generating 4D content using diffusion models. - The architectural extension of the 3D-aware diffusion model and its fine-tuning are good contributions that would be useful to know for the community. - The technical contribution is highlighted by impressive generalization performance (assuming no train data leakage). - This also highlights the scalability potential of synthetic Objaverse dataset for fine-tuning video diffusion models to perform 4D generation. - Both qualitative and quantitative results indicate improvements over the baselines, albeit modest compared to Consistent4D.

Weaknesses

- The training requires a multi-view video dataset which is difficult to obtain. - The evaluation is limited to synthetic, object-centric toy scenes without backgrounds. - I haven’t found a description of the validation and test dataset for experiments in Section 4.1 - It is not clear whether assets in test-videos are unseen during training of all models (as some of them are also trained on objaverse). - Evaluation in Sec 4.1 is limited to CLIP and FVD metrics. Since multi-view video datasets were used for training, one could evaluate models for the novel view synthesis task using standard metrics such as LPIPS/PSNR (taking best of 10 due probabilistic nature of the task). - Minor: Given the small improvement over Consistent4D, further evaluation of statistical significance is needed. - Minor: The paper would benefit from more precise writing; particularly, the training description in lines 176-182 needs more clarity on each variable and the noising process, and Equation 10 lacks clarity regarding sampled variables used in expectations. The method description is overly complex, and the language is difficult to follow, containing several unclear sentences.

Questions

The paper assumes access to a multi-view video dataset. The rationale behind the need for SDS when a multi-view video diffusion model is already available is unclear. Could you explain why not fit the dynamic NeRF directly on the generated multi-view videos?

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Authors addressed the limitations.

Summary

The paper proposes a 4D generation method that aims to generate 4D content from a monocular video. A video-to-multi-view-video diffusion model is presented to create multi-view videos given a monocular video, a text prompt, and a sequence of camera poses. The trained multi-view-video diffusion model is leveraged to optimize 4D representation, i.e., dynamic NeRF. In addition, 4D-aware SDS loss and an anchor loss are introduced to train dynamic NeRF. Experimental results show the proposed method achieves the best performance, compared with state-of-the-art methods.

Strengths

1. The paper is well-written and easy to follow. 2. A multi-view-video diffusion model is presented to generate multi-view videos from a monocular video 3. The paper addresses an interesting problem, and 4D generation significantly impacts various applications.

Weaknesses

1. Some technical details are unclear. The paper builds a multi-view-video diffusion model by inserting a learnable motion module into ImageDream. The learnable motion module is critical to the proposed method. However, the paper does not provide detailed information about the motion module, such as the architecture and the layer information. Without this information, it is difficult to reproduce the proposed method. 2. The paper only uses 996 training data to train the multi-view-video diffusion model or the motion module, while the training takes two days on 16 NVIDIA Tesla A100 GPUs. Does such a small data set and such an extensive training cost lead to significant overfitting? How many parameters are in the motion modules? 3. Instead of the input monocular video, the anchor loss chooses a monocular video generated by the presented multi-view-video diffusion model as an anchor, due to the difficulty in estimating the camera pose of the input video. Why not use all videos generated by the multi-view-video diffusion? Would this operation degrade the 4D generation performance? In addition, the input video typically has better quality than the generated one. 4. Table 2 shows that using ImageDream achieves better CLIP-I than using the present multi-view-video diffusion model. Could the authors provide more explanations?

Questions

Please refer to my comments above

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper provides the limitations and societal impact of the proposed work