Animate3D: Animating Any 3D Model with Multi-view Video Diffusion

Summary

This paper introduces a framework for 4D generation to animate static 3D models, consisting of two components: MV-VDM, a multi-view video generation model, and a framework combining reconstruction and 4D score distillation sampling (4D-SDS). A spatiotemporal attention module enhances consistency, using multi-view renderings to preserve the model's identity. The proposed two-stage pipeline for animating 3D models first reconstructs coarse motions and then refines them with 4D-SDS. Experiments show that Animate3D outperforms previous methods.

Strengths

1. The paper introduces a large-scale dataset for multi-view video generation, consisting of 84,000 animations and over 1.3 million multi-view videos, which addresses the challenge of limited 4D datasets. 2. Qualitative experiments demonstrate the ability to generate high-quality 4D objects. The generated shapes appear to have better quality and texture compared to other methods. 3. The paper is clearly written, with a clear motivation for using a multi-view diffusion model to enhance spatiotemporal consistency.

Weaknesses

1. The method appears to be a straightforward engineering pipeline, combining several existing techniques (e.g., MVDream and IP-Adapter in the MV-VDM stage, reconstruction + SDS in the animating stage) without significant innovation. 2. Motion diversity is a concern. The method shifts the core problem to the quality of the multi-view video. The dataset achieves this effect, but the maximum number of animations per ID is only six, which is insufficient for capturing the diverse motions of a single 3D object. This approach lacks generality. 3. Motion controllability is limited. The results shown in the paper exhibit simple, repetitive motions with small amplitudes. Although text prompts are used as conditions, they are relatively simple, and the generated motions are not complex. The method lacks a clear approach to achieving more controllable motions. 4. Experimental results are lacking. DreamGaussian4D provides CLIP-I as a quantitative comparison to assess the consistency between prompts and generated results, which is missing in the paper. Additionally, the qualitative results provided in terms of views and moments are too few.

Questions

1. Regarding time, the paper reports an optimization time of 30 minutes per object on a single A800 GPU. How much time does it take to reconstruct coarse motions and refine motions using the 4D-SDS model, respectively?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations and broader impacts of the study in the paper.

Summary

This paper proposes Animate3D, a 4D generation framework that consists of a multi-view video diffusion model (MV-VDM) followed by 4D Gaussian Splatting optimization, as well as a dataset of 38K animated 3D objects (MV-Video) that is used to provide spatiotemporal supervision to train the model. Different from text-to-4D approaches, Animate3D specifically tackles the problem of animating 3D assets which requires spatiotemporal consistency as well as identity preservation. The proposed architecture is based on a multi-view diffusion model (MVDream) and a video diffusion model (AnimateDiff) with additional spatio-temporal attention blocks. To animate a given 3D asset, the trained model is used for both reconstruction and distillation. First, a multi-view video is generated conditioned on multiple views of the asset. The video is used to optimize a 4D Gaussian Splatting resulting in a model with coarse motion. The same MV-VDM model is used for 4D Score Distillation Sampling to refine the motion. The method compares favorably to state-of-the-art 4D generation methods.

Strengths

**[S1]** The problem setting in this paper is interesting and timely, in an area that has recently attracted significant attention. The paper takes a slightly different angle compared to text-to-3D, text-to-4D or video-to-4D methods, focusing on animating 3D assets based on a textual description instead of generating them from scratch. **[S2]** This problem setting also opens up a new challenge, which is to preserve multi-view attributes during the animation. It is nice to see that the identity preservation and the animation are significantly improved compared to state-of-the-art methods. **[S3]** The authors tackle this problem by training a multi-view video diffusion model. Given the success of image (2D), video (2D+time), and multi-view (3D) diffusion models, MV-VDM (4D) is a reasonable next step to ensure spatial and temporal consistency simultaneously. Both the model and the dataset used to train it are likely to influence future work and could thus have significant impact. **[S4]** The proposed architecture aims to reuse existing multi-view and video diffusion models, which are already pre-trained on a larger scale of data, thus taking advantage of the priors built into these models. ---

Weaknesses

**[W1]** The data used in this paper are animated 3D models collected from Sketchfab (cited). The authors promise to distribute the collected dataset. However, there is not sufficient information provided about copyrights and licensing. The only piece of information provided is that models with the following clause are excluded: >NoAI: This model may not be used in datasets for, in the development of, or as inputs to generative AI programs. Based on the paper checklist, the paper is flagged for ethics review. --- **[W2]** Lack of clarity in the method section and method figure **(a)** The method section is very hard to parse. The notation is overloaded but not very well explained. The section would benefit from thorough revision to improve the overall clarity and quality. **(b)** It is almost easier to understand the method through Fig. 2, but even that is dense and inconsistent, and often the notation does not align with what is written in Sec. 3.1. For example, in L175 latent features $z$ seem to be the output of the image encoder. In Fig. 2 (middle), $z$ appears to be the input to the spatiotemporal attention module. In Eq. 1, $X_l$ and $X_r$ are instead the inputs to the same module. At the same time, in Fig. 2 (left) the input to the spatiotemporal module seems to be the sum of the output of the cross-attention layers, not directly $z$. This makes reading and understanding the method extremely difficult to the point where the reviewer cannot fully judge the technical contribution and similarities or differences to related models. --- **[W3]** Discussion with respect to 4D generation methods could be expanded Existing work in text-to-4D generation often splits the problem into two stages: a static 3D asset generation and a deformation field to model motion. This disentanglement makes it possible to learn the 3D motion from video diffusion models, without the need for multi-view video. The paper briefly touches upon this and compares to two 4D generation methods, 4Dfy and DreamGaussian4D. However, the paper could further elaborate on the similarities and differences of the proposed approach compared to existing methods and the advantages of a multi-view video diffusion model. --- **[W4]** Certain architectural components are not ablated and their contribution to the overall performance is unclear. See Questions. --- **[W5]** The temporal extent of MV-VDM is only 8 frames, which seems rather limited. --- **[W6]** In some of the qualitative examples, the faithfulness to the text prompt is questionable. Some of the animations provided in the supplementary video are not accompanied by a text prompt, while quite a few animations appear very similar. --- **[Minor]** Typos and suggestions: - L103: pioneer → pioneering - L113: All these manners aforementioned → All aforementioned work - L131-132: this sentence can be improved for clarity since the two parts sound a bit repetitive - L135: spatial → spatially; temporal → temporally - L156: It should be better explained how what $X_l$ and $X_r$ represent and how these features are obtained. - L292: levering → leveraging; manage → manages ---

Questions

**Q1. [Based on W1]** Please elaborate on the difference between _3D animation via multi-view video diffusion_ and _two-stage text-to-4D approaches_. In the latter case, the second stage could be viewed as 3D animation. I understand that a multi-view video foundation model would offer spatiotemporal consistency that other approaches can likely not reach, but I would appreciate a more in-depth discussion about the difference to and the challenges of text-to-3D-and-3D-to-4D approaches than what is currently provided in the paper. --- **Q2. [Based on W1]** Ideally, the authors should provide also empirical comparisons to additional methods (e.g., Dream-in-4D), if possible. --- **Q3. [Based on W4]** The authors state that >We find this simple cross-attention operator [MV2V-Adapter] can effectively improve the object’s appearance consistency in the generated video. The effect of MV2V-Adapter should be better demonstrated with an ablation. --- **Q4. [Based on W4]** Are the cross-attention layers described in L170-172 necessary for identity preservation and alignment with the text prompt? Their effect should be also demonstrated in the ablation studies. --- **Q5.** The proposed 3D animation approach uses both reconstruction and distillation. It first uses the generated multi-view video to optimize a 4D Gaussian Splatting with coarse motion. Then SDS is used to refine the motion. It seems counter-intuitive that SDS would be good at modeling the finer details of the animation, since SDS is typically known for its over-smoothing behavior. The authors should further discuss the motivation behind using SDS during 4D optimization and provide additional examples to prove its effectiveness. --- **Q6. [Based on W6]** Could the authors elaborate on the apparent lack of diversity in the generated motions? It may be helpful to provide some statistics from motion descriptions included in the dataset. Or is this a limitation of the model instead? ---

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are adequately discussed in Appendix B.

Summary

This work presents Animate3D, a framework for animating static 3D models. The core idea involves two main components: 1. A multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, trained on a large-scale multi-view video dataset (MV-Video). 2. A framework that combines reconstruction and 4D Score Distillation Sampling (4D-SDS) to utilize multi-view video diffusion priors for 3D object animation. The animation process involves a two-stage pipeline: coarse motion reconstruction from generated multi-view videos, followed by 4D-SDS to model fine-level motions. Quantitative and qualitative evaluations show enhancements to previous methods.

Strengths

1. Performance: The proposed method achieves state-of-the-art results. The experiments well validate the effectiveness of the proposed methods. 2. Clarity: The paper is well-written and easy to follow. 3. Technical Novelty: The main contributions of this paper are threefold: 1) The first 4D generation framework to animate any 3D objects with detailed multi-view conditions, which are incorporated by the proposed MV2V-Adapter. 2) The authors propose to animate any off-the-shelf 3D models with unified spatiotemporal consistent supervision, which can get better results. 3) The collected 4D dataset, MV-Video.

Weaknesses

1. Missing Reference [a]: It is understandable that you did not compare Animate3D with Animate124 in this paper, as 4D-fy and DreamGaussian4D both demonstrate better performance compared to Animate124. However, it is unusual that you did not discuss Animate124 at all, given its relevance in previous comparisons by 4D-fy and DreamGaussian4D. 2. Limited Comparison Scope: The authors only compare Animate3D with 4D-fy and DreamGaussian4D, which seems insufficient. It would be more comprehensive to include comparisons with 4DGen and TC4D, as both claim superior performance over 4D-fy and have released their code. Additionally, while the modification of 4D-fy to a Gaussian representation for fair comparison is understandable, the original 4D-fy results should also be included for a thorough ablation study. [a] Animate124: Animating One Image to 4D Dynamic Scene, \ [b] 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency, \ [c] TC4D: Trajectory-Conditioned Text-to-4D Generation.

Questions

1. The results of 4D-fy appear strange. When I check the 4D-fy results in TC4D and 4DGen, they look more reasonable. 2. I think the authors should also conduct ablation studies on MV2V-Adapter.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors have discussed the limitations and potential negative societal impact.

Summary

This paper proposes an animation method that animates a 3D model in a 4D one. A Multi-View image conditioned multi-view Video Diffusion Model (MV-VDM) is presented to generate multi-view videos from multi-view renderings of a static 3D object. The MV-VDM is leveraged to train the 4D Gaussian Splatting (4DGS), where As-Rigid-As-Possible (ARAP) loss, reconstruction loss and SDS are used as objectives. In addition, a multi-view video dataset is constructed to train MV-VDM. The paper conducts experiments to show the effectiveness of the proposed method.

Strengths

1. The idea is well-motivated and straightforward. 2. A large-scale multi-view video dataset is presented. 3. A multi-view video diffusion model is presented, where spatiotemporal attention is introduced to animate multi-view images with motions.

Weaknesses

1. The comparison is not very fair. The paper focuses on a new task, i.e., 3D animation, while 4Dfy focuses on text-to-4D, which is a different task. Furthermore, text-to-4D is more challenging than 3D animation since multi-view images are unavailable. The better performance gain of the proposed method may partly attributed to additional multi-view images. In other words, the worse performance of 4Dfy and DG4D may not be due to the methods themselves. In addition, the paper replaces dynamic NeRF in 4Dfy with 4DGS. However, some hyperparameters of 4Dfy are set according to dynamic NeRF, rather than 4DGS. Instead, the paper can compare the proposed method with AYG, which directly trains 4DGS. In addition, the paper can compare Animate234, which is an image-to-4D method. 2. The motivation for the proposed spatiotemporal attention block is not clear. Besides the temporal motion module and temporal encoding, the block introduces a multi-view 3D attention module. What is the multi-view 3D attention module used for? Why are temporal motion modules and temporal encoding not enough to generate motions for multi-view videos? 3. What is the influence of Alpha Blender on the performance of the proposed method? 4. Does the proposed method train all modules in the spatiotemporal attention block? It's not clear whether the "Multi-view 3D Attention" and "Temporal Motion Module (Pre-trained VDM)" are trained or frozen. 5. The implementation details of Alpha Blender are not clear. Is it implemented as a layer of MLP? 6. The temporal encoding is unclear. According to Figure 2, there is a temporal encoding module in the spatiotemporal attention block, but there is no description of what kind of temporal encoding is used. 7. How many views does each animated 3D object contain in the dataset? Only four orthogonal views or more? 8. Although the dataset contains 38K animated 3D objects, it is still much smaller compared to 2D video datasets. Can the proposed method trained on this dataset of this magnitude animate any 3D model? 9. Training a multi-view video diffusion model requires camera parameters. Are the camera parameters in the multi-view video dataset processed to ensure that they are consistent with the prior knowledge learned in the MVDream

Questions

Please refer to my question above.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have provided limitations and societal impact of their work

Summary

This paper focuses on animating 3D objects with multi-view diffusion models. To improve spatial and temporal consistency, this work builds a large-scale multi-view video dataset, MV-Video and designs an attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. To enhance animations from 3D objects, this work jointly optimizes the 4DGS through reconstruction and 4D-SDS. Experiments show the proposed method can more consistent 4D generation.

Strengths

* This work builds a large-scale multi-view video dataset to facilitate the training of multi-view video diffusion models. * The method designs an attention module to encourage spatial and temporal consistency for multi-view diffusion models. * The approach introduces 4D-SDS to leverage multi-view video diffusion models for animating 3D objects. * The paper is easy to read and understand.

Weaknesses

* This work focuses on 3D objects, however, the paper title claims "Animating Any 3D Models". It is a bit inappropriate. * The effectiveness of the proposed dataset is not fully validated. Can this dataset improve single-view diffusion models or multi-view image-to-3D generation models, like MVDream? * The diversity of the proposed method is unclear. Can one object perform different actions? For example, in Figure 3, can the frog perform swimming? * The picture quality is a bit low, it is hard for readers to distinguish the advantages of the generated objects from the proposed method. * For animating from 3D object, this work first leverages 4DGS to reconstruct coarse motions and then uses 4D-SDS for refinement. If the method uses better 4DGS algorithms, maybe the improvement from 4D-SDS will become little.

Questions

* It is better to validate the effectiveness of the proposed dataset, can it also improve single-view diffusion models (e.g., T2V) or multi-view image-to-3D generation models (e.g., MVDream). Then, can these models further improve other methods, like DreamGaussian4D? * Although the work verify the effectiveness of the S.T attention. The effect of MV2V-adapter is not validated. * If the approach leverages better 4DGS reconstruction methods, such as [1] for the coarse motion reconstruction, can the 4D-SDS still improve the coarse results? [1] Yang, Zeyu, et al. "Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting." arXiv preprint arXiv:2310.10642 (2023). * As this work designs a multi-view video diffusion model, can the authors compare it with existing single-view video diffusion models? * For Figure 4, both the ablation models show that the player touches the basketball. However, the full model does not show this interaction. I wonder if the full model can do better for this interaction.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The paper has discussed the limitations.

In-depth Discussion with respect to Previous 4D generation Methods

Due to limited space, our initial response did not permit a comprehensive discussion of previous 4D generation works, now we detail the in-depth discussion as suggested: Previous two-stage 4D generation works attempted to **disentangle motion learning** from appearance learning by adopting **different types of supervision signal**, i.e., video diffusion/monocular video for motion and image/3D diffusion for appearance. However, the motion and appearance supervisions adopted in their work are **not orthogonal**, and sometimes have **negative effect** on each other. For example, it is commonly agreed that video-diffusion-SDS usually brings unappealing visual effect to the appearance of the object [Animate124, Dream-in-4D, AYG]. Meanwhile, the appearance supervision signal prevents 4D object from updating along the direction that follows the exact score function of the video diffusion model, leading to less natural motion. Small motion amplitude in [4Dfy, Dream-in-4D] and shaky appearance in [AYG] could partly support this point. As for monocular-video-guided motion learning, previous work [DG4D, 4DGen, STAG4D] relies on 3D diffusion model (Zero123) to supervise both motion and appearance in novel view. Since Zero123-SDS is applied per-frame, temporal consistency in novel view cannot be guaranteed. Moreover, monocular video doesn't provides information about depth/distance, so moving closer to or farther away from the camera can be perceived as the magnification or reduction of the object, resulting in appearance distortion. In contrast, our method takes the **unified** supervision signal from MV-VDM for motion learning and appearance preservation. Our motion and appearance supervision signal inherently don't conflict with each other, since MV-VDM is conditioned on multi-view attributes of the 3D object to generate multi-view videos. Besides, **multi-view motion supervision** in our work enables more natural motion generation when compared with single-view motion supervision in other works. Thus, we achieve superior performance in terms of both motion generation and appearance preservation in the task of animating any off-the-shelf 3D object. Thanks for your insightful suggestion of in-depth discussion, and we'll add this in revision. If you have any further questions, discussions are welcomed.

Post Rebuttal

Thank the authors for addressing my questions. I am still advocating for accepting the paper.