Vivid-ZOO: Multi-View Video Generation with Diffusion Model

Summary

The paper introduces "Vivid-ZOO," an innovative diffusion model designed for Text-to-Multi-view-Video (T2MVid) generation. This is a novel approach that addresses the challenges of generating high-quality multi-view videos from textual descriptions. Specifically, the authors propose a new method that leverages diffusion models to generate multi-view videos centered around dynamic 3D objects from text prompts. The T2MVid generation problem is effectively factorized into viewpoint-space and time components, allowing for the combination and reuse of layers from advanced pre-trained multi-view image and 2D video diffusion models. And the authors claim that it is the first study that explores the application of diffusion models to the T2MVid generation task.

Strengths

1. The paper introduces a novel diffusion-based approach to generate multi-view videos from text prompts, which is a relatively unexplored area in the literature.The creative factorization of the problem into viewpoint-space and time components is an innovative way to tackle the complexity of multi-view video generation.The introduction of alignment modules to integrate pre-trained models is a clever solution to address domain gaps and reuse of layers. 2. Furthermore, by providing a captioned multi-view video dataset, the authors not only support their own research but also contribute to the wider research community, facilitating future work and lowering the entry barriers for other researchers in this domain. 3. The proposed solution adeptly addresses a notable gap in the current literature by offering an efficient approach that does not rely on massive datasets, thereby making sophisticated multi-view video generation more accessible and applicable to a wider array of problems

Weaknesses

1. The visual quality of the generated videos might not match state-of-the-art single-view image generation models. Future work could focus on improving the visual fidelity and realism of the generated multi-view videos. 2. The model currently assumes point light sources, which might not always produce the most realistic renderings. Enhancements in simulating complex lighting conditions could improve the model's applicability.

Questions

Please refer to weakness

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss the limitations and potential negative societal impacts of their work in the paper.

Summary

This paper proposes a Text-to-Multi-view-Video generation algorithm capable of generating multi-view video content based on text descriptions. The method decouples the multi-view spatial and temporal dimensions, utilizing pretrained models like MVDream for multi-view generation and animatediff for video generation. It addresses domain gaps between different features through 3D-2D/2D-3D alignment. The authors trained on captioned 4D data from objverse and conducted various ablation studies to validate their approach.

Strengths

1. The method proposed by the authors achieved impressive visual results, with performance metrics showing improvements over naively using MVDream + AnimateDiff. 2. The 3D-2D/2D-3D alignment proposed by the authors reduces the number of parameters and training time needed. Given the limited training data, this approach also helps mitigate degradation in pretrained models.

Weaknesses

1The authors claim their method is the first to do text-to-multi-view video generation, but as far as I know, there are many works related to 4D generation currently, such as MAV3D, AYG, 4D-fy, including recent works like 4Diffusion and Diffusion4D. I believe the authors may want to emphasize that their work focuses on novel view synthesis rather than involving 4D representation, such as Deformable GS, Dynamic Nerf, etc. However, these are just differences at the algorithmic level; fundamentally, they are all engaged in multi-view video generation. Therefore, the authors' comparison in Table 1 is insufficient, and relevant 4D generation works should also be included for comparison.

Questions

1. Regarding the 'overall' metric in Table 1, does it refer to the results of a user study, or is it a comprehensive representation of the previous two results? 2. What is the test dataset in Table 1? How large is the dataset? 3. Will the 4D dataset proposed in the paper be made publicly available? 4. In the current framework, what are the results if MVDream and AnimateDiff are not fixed, or if fine-tuning is done with a very small learning rate? 5. Did the authors conduct experiments on generating 4D contents based on image prompt, and are there any visual results?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Currently, the results only compare against a self-designed baseline, lacking a broader comparison with other algorithms.

Summary

This paper focuses on generating multi-view consistent videos given a text prompt, specifically from 4 orthogonal views. It fine-tunes MVDream with a temporal layer adopted from AnimateDiff. To mitigate the domain gap between the two layers, some connected layers called 3D-2D and 2D-3D alignment layers are introduced. They train the newly introduced layers on a 4D subset of Objaverse. The proposed model can generate multi-view videos. The authors conduct evaluations to verify the new layers and choices of the base models.

Strengths

* This is the first paper to study the problem of text-conditioned multi-view video generation, based on my best knowledge. Perhaps [1] is a concurrent work and should be acknowledged. * The paper is easy to follow. * A multiview video dataset extracted from Objaverse is collected. [1] Kuang, Zhengfei, et al. "Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control." arXiv preprint arXiv:2405.17414 (2024).

Weaknesses

Despite the fact that the paper is the first to study text-conditioned multi-view video generation, I think there could still be room for improvement in the evaluation section in terms of the justification of the method as stated below: 1. There are very few generated videos (8 in the supplementary material) shown in the paper. Is this because the success rate is low? Since it is a forward-based method, I expect the time to generate the videos will not be too long. If so, it would be good to discuss the failure cases in the limitation section. 2. As for human evaluation, only five videos are used for comparison. 3. There is a lack of details about how many videos are used to compute CLIP and FVD scores. Also, are the prompts used to generate these videos separate from the training set? FVD was used to evaluate temporal coherence, but FVD is shown to be not good at it (see [2]). [2] Ge, Songwei, et al. "On the Content Bias in Fréchet Video Distance." CVPR. 2024. 4. The motions in most generated videos are small and are not indicated even in the text prompt. This could limit the use case of the model in generating specific motions. 5. All the generated videos contain a grayish background. I'm fine with it if the ultimate goal is to generate a 4D asset, and I think it is a reasonable goal. Ideally, like MVDream, the authors may want to compare with 4D generation methods.

Questions

1. As far as I know, MVDream is fine-tuned (fully?) from SD2.1, while AnimateDiff v2 is fine-tuned from SD1.5; this could cause an additional discrepancy when combining the two models, right? 2. In the ablation study of "Design of multi-view spatial module", what is the difference between what you do and directly using AnimateDiff v2?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors discuss the limitations and negative societal impact in the appendix.

Summary

The paper presents a novel diffusion-based pipeline designed to generate multi-view videos from text prompts. The core challenge addressed by the paper is the generation of dynamic 3D object videos from multiple viewpoints, a task not extensively explored with existing diffusion models. The authors factor the problem into viewpoint-space and time components, allowing for the reuse of layers from pre-trained multi-view image and 2D video diffusion models. This reuse is intended to ensure consistency across views and temporal coherence, thereby reducing training costs. The paper also introduces 3D-2D and 2D-3D alignment layers to bridge the domain gap between the training data of multi-view images and 2D videos. Additionally, a new dataset of captioned multi-view videos was created.

Strengths

1. The paper tackles a relatively unexplored area of diffusion models, extending their application to multi-view video generation from text prompts. The task is extremely challenging and important. 2. By reusing layers from existing models, the approach leverages the strengths of pre-trained systems while minimizing the requirement for large-scale data and extensive computational resources, addressing a common barrier in training new machine learning models. 3. The introduction of 3D-2D and 2D-3D alignment layers is an interesting solution to the issue of domain gaps between different types of training data. 4. The creation of a new, although smaller, dataset for training and testing in this niche area is crucial, as it provides a resource that can be built upon by subsequent research. However, this does not seem to be license-free?

Weaknesses

1. The paper acknowledges the challenge of not having a large-scale, captioned multi-view video dataset, which is crucial for training robust diffusion models. The authors attempt to mitigate this by constructing a smaller dataset, but this could limit the model’s generalization capabilities and performance across diverse scenarios. Further, this may be very challenging if we would like to further scale to scene-level contents. 2. The alignment between pre-trained 2D video and multi-view image diffusion models is crucial, yet challenging. The proposed 3D-2D and 2D-3D alignment layers are a solution, but the effectiveness of these layers in truly bridging the domain gap might not fully compensate for the inherent differences in training data types (real-world 2D videos versus synthetic multi-view data), potentially affecting the quality of generated videos. Since the model is trained on an entirely different domain of object-centric videos, there is very little we can tell in terms of video quality degradation.

Questions

N/A

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Response

Thank you for your time and effort in addressing the unclear details, adding additional experiments, and performing additional evaluations. I have got most of my concerns addressed and updated my rating.

Please discuss

Dear reviewer, The discussion period is coming to a close soon. Please do your best to engage with the authors. Thank you, Your AC

Please discuss

Dear reviewer, The discussion period is coming to a close soon. Please do your best to engage with the authors. Thank you, Your AC