Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models

Response to Reviewer UTNE

We thank the reviewer for the positive feedback and insightful comments. Our responses to the reviewer’s concerns are below: **Q1: If the viewpoints differ by only 30 degrees, can they still be considered independent variables?** Yes, although these two views may have some overlapping regions, the **conditional** independence still holds. Because the case in Figure 3 does not contain some special motions like twisting. The overlap between the future front view $I_{i,j’}$ and the current side view $I_{i’,j}$ also exists in the current front view $I_{i,j}$. In practice, even when twisting or rotation movement occurs, it is often subtle between frames, contributing negligible information gain and thus having no substantial violation on the conditional independence assumption. As demonstrated at **in the supplementary video (01:04~01:08)**, our method is also capable of handling objects with such dynamics. **Q2: More explanation for why VRS works and why it would reduce the variance.** Thanks. We have explained its motivation in Section 3.3 and provide further clarification here. The similar rollback operations have been adopted in many diffusion-based inpainting methods [1] for harmonizing potential conflicts. In our work, the utilization of this operation aims to deceive the diffusion model into denoising a noisy latent with a smaller variance at a higher noise level through the following process: before $t$-th denoising step, we add Gaussian noise with a variance smaller than $\sigma_{t+1}^2-\sigma_{t}^2$ onto the latent, and then denoise it again at noise level $t+1$. After that, at $t$-th step, the noisy latent to be denoised by the model actually resides at a level below $\sigma_t$. We provide a visualization of how variance evolves during denoising with and without VRS **in Figure 10** of the revised appendix. A more intuitive explanation for why it works is that when the latent resides at a lower noise level, the diffusion model can allocate more capacity to reconcile the conflicts brought by two scores. Additionally, a latent with less noise can provide clearer guidance for denoising, which can also help to mitigate potential conflicts. **Q3: If we set the sampling steps larger, will it achieve a similar effect with variance-reducing sampling?** We would say no. In the **revised supplementary video (00:12~00:21)**, we provide two videos generated using 500 denoising steps (10x than the default setting) without VRS. These assets still exhibit severe flickering artifacts. This may because the unresolved conflicts in the early stage of denoising will drive the latent away from the desired distribution, which cannot be rectified by merely extending the denoising schedule. > [1] Lugmayr, Andreas, et al. Repaint: Inpainting using denoising diffusion probabilistic models. *CVPR*. 2022.

Response to Reviewer rKU7

We thank the reviewer for the positive feedback and insightful comments. Our responses to the reviewer’s concerns are below: **Q1: The backside seems to be over-smoothed.** Thanks. To reconcile the scores from the two generative models without significantly impacting runtime, we adopted the relatively aggressive variance-reducing sampling (VRS) strategy, which sacrifices details of some cases in specific views that are under-constraint due to far from the reference frame and view. Our rationale for this trade-off is based on the following considerations: (i) This compromise does not affect the consistency of the generated results, it is still a seamless 4D assets that is consistent with the reference view. (ii) It does not degrade the details in views closer to the reference frame and view. Actually, as demonstrated in the video, our results often exhibit better detail in these views compared to other alternates thanks to the consistent direct photometric supervision in high resolution. In some applications, users may more care about quality in these views rather than back views without any hint. (iii) Finally, this issue can be mitigated by texture-dynamic decoupled reconstruction as shown in the ablations in Figure 6 of the main paper and 00:03~00:11 of the supplementary video. Besides, this issue could be partly ascribed to the multi-view prior. We observed that for some examples, their multi-view references already lack details in backside as shown **in Figure 8**. Further scaling of the used diffusion prior may alleviate this issue. **Q2: Whether finetuning the SVD on the object-centric data would be helpful.** Thanks for the insightful comment. The authors agree with this point, that fine-tuning the SVD on videos containing only foreground objects could improve quality. The current adopted video generator has learned a broader distribution than the desired single-view videos for dynamic objects, which may introduce some conflicts. We also raise this concern and propose VRS to mitigate potential conflicts. However, to preserve the training-free merits of the proposed framework, we have not included further fine-tuning of the video generative model as part of our contribution. We would like to attempt this direction in the future.

Response to Reviewer AvRx (Part 1)

We thank the reviewer for the detailed review as well as the suggestions for improvement. Our responses to the reviewer’s concerns are below: **Q1: Comparison with the baseline mentioned in W1.** Great suggestion. In ablation experiments of the main paper and supplementary video, we have compared with a similar baseline that only utilizes SV3D [2] for each frame. To ensure a fair evaluation of the improvements brought by score composition, we also incorporated variance-reducing sampling (VRS) into it. We further provide the quantitative comparison in the table below (also in Table 3 of the revised paper) with the suggested baseline (reconstructing images from only multi-view 3D generator with [5]): | **Method** | **LPIPS↓** | **CLIP(%)↑** | **FVD↓** | | ---- | ---- | ---- | ---- | | Baseline | 14.86 | 89.61 | 1412.00 | | Ours | 12.35 | 92.20 | 831.32 | The qualitative results from only the multi-view 3D generation model without VRS are shown **in the 00:31~00:35 of revised supplementary video**. It can be observed that the generated videos exhibit significant temporal inconsistencies due to the inherent stochasticity in the diffusion denoising process, making it challenging to be reconstructed into meaningful 4D assets, even with state-of-the-art 4D reconstruction methods [5] with superior fitting capability. **Q2: Similar results in two stages.** Thanks for the careful observation. Note that the second stage is a reconstruction stage, the almost identical results are partly ascribed to the Gaussian’s strong fitting ability for the training views. It also corroborates the high consistency of the images generated in the first stage. Besides, the multi-view references of examples in Figure 4 already have textureless back views (as shown **in Figure 8 of the revised appendix**), so the first-stage results usually do not exhibit significantly more blurry details in these views. **Q3: Quantitative results for ablation studies.** Thanks for the suggestion. We conduct quantitative ablations on the Consistent4D dataset, the results are presented in **Table 3 of revised appendix** and below: | **VRS** | **Score comp.** | **Feat. blend** | **LPIPS↓** | **CLIP(%)↑** | **FVD↓** | | :----: | :----: | :----: | ---- | ---- | ---- | | | | | 11.63 | 94.82 | 1248.49 | | √ | | | 12.79 | 91.85 | 1050.68 | | √ | | √ | 12.65 | 92.18 | 898.66 | | | √ | | 11.86 | 94.30 | 1170.96 | | √ | √ | | 12.35 | 92.20 | 831.32 | It aligns with our observations drawn from the qualitative ablations, where our result shines in video-level metric of FID due to its better temporal coherence. In the submitted version, we following common practice [3,4], only report qualitative ablations, as we believe they are sufficient to demonstrate the characteristics and differences from these baselines. More importantly, there is a lack of comprehensive quantitative metrics to evaluate the quality of generated assets in such tasks. Besides, we modified the evaluation of FVD in the revised paper by aligning the background color. While the comparison between previous metrics can also reflect the severity of temporal artifacts in the foreground object, the new results help to better align the scale with those reported in other 4D generation works.

Response to Reviewer AvRx (Part 2)

**Q4: Can the proposed method generalize to other multi-view 3D generation and video generation methods?** Yes, as long as they are diffusion-based generative models and share the same latent space. Our framework adopts an **architecture-agnostic** approach to combine heterogeneous scores from multi-view and video diffusion models. **Q5: Time consumption for stage-1 and stage-2.** We have already reported the total generation time in **Table 2** and indicated the time spent in the first stage at **L469** in the initial submission. Now we present them in the table below for more clear illustration. | | Time | | ---- | ---- | | Stage-1 | 8m31s | | Stage-2 | 3m33s | | Overall | 12m4s | **Q6: Extending to multi objects.** Good question. The multi-view generator used in our framework is trained on object-centric data, limiting its generation to scenes containing a single object. Constrained by this, we limit the scope of our method to the generation of single dynamic objects, similar to prior 4D generation works [1]. Usually, 4D results of multiple objects can be created by composing multiple dynamic objects generated independently. To better answer this question, we additionally provide an example of a composed scene with multiple dynamic objects **in the revised supplementary video (01:10~01:18)**, demonstrating scalability and generalization ability of our approach. > [1] Jiang, Yanqin, et al. Consistent4d: Consistent 360° dynamic object generation from monocular video. *ICLR*, 2024. > [2] Voleti, Vikram, et al. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. *ECCV*, 2024. > [3] Tang, Jiaxiang, et al. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. *ICLR*, 2024. > [4] Ren, Jiawei, et al. Dreamgaussian4d: Generative 4d gaussian splatting. *arXiv preprint*, 2023. > [5] Li, Zhan, et al. Spacetime gaussian feature splatting for real-time dynamic view synthesis. *CVPR*. 2024.

Looking forward to your feedback

Dear Reviewer AvRx, We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our detailed explanations and additional results solves the concerns. We have addressed all the thoughtful questions raised by the reviewer *(eg, results, time consumption, extending to multi objects)* and we hope that our work’s contribution and impact are better highlighted with our responses. As the discussion phase is nearing its end, it would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you! Best wishes, Authors

Response to Reviewer s4pt

We thank the reviewer for the acknowledgment of the innovation of our work as well as the valuable suggestions for improvement. Our responses to the reviewer’s comments are below: **Q1: Requires 8-16 A6000 GPUs for basic operation？** This is a constructive question regarding the application of our Diffusion$^2$. We first clarify the following facets: 1. **The use of 8~16 A6000 GPUs for inference is not mandatory.** We adopt it as a setting to demonstrate the advantage on maximum inference efficiency benefited from high parallelism, which is more important for the online tasks. 2. As mentioned in L469, multi-GPU is only used in the first stage, which takes less than 10 minutes. In addition, if we sacrifice parallelism and only use a **single GPU**, our framework is still **significantly faster than representative SDS-based methods** (e.g. 2x than Consistent4D [1]). We depict this trade-off **in Figure 7 of the revised appendix.** Actually, even without considering parallelism, our approach almost constitutes a lower bound on the time required for sampling $V(=21) \times F(=25)$ images with diffusion models to reconstruct 4D asset: we only perform the denoising process twice for each image without iterative refinement. However, sampling such a dense set of multi-view, multi-frame images may be unnecessary for reconstructing 4D representations. So the further reduction in the computational requirements lies in the redundancy of such a dense image array. Now we sampled 21 views for each frame to leverage an existing multi-view generator [2]. In practical applications, fine-tuning this generator to support a smaller $V=8$ (as in concurrent work [4]) may not substantially impact performance but can immediately reduce computational requirements by 2/3. Besides, in real applications, some efforts on engineering optimizations can also be made to mitigate the efficiency bottleneck introduced by multi-GPU synchronization and communication revealed in Figure 7 of the revised appendix. **Q2: Further exploration of challenging cases.** Great suggestion. The examples we currently use are primarily sourced from commonly used public 4D generation datasets [1,3]. Among them, the example *sighing frog* (shown in Figure 4 and supplementary video) retains the largest motion within the Consistent4D dataset, thus posing significant challenges for many prior methods. Both SDS-based and feature blending-based approaches struggle to establish the association between frames with substantial differences. In contrast, our proposed method demonstrates the potential to handle complex motions with global receptive fields of two score estimators. For 4D objects with rotational movements or articulated motions which pose distinct challenges, We show additional examples **in the revised supplementary video (01:04~01:08)** to provide more insights and demonstrate the effectiveness of our approach. **Q3: Extending to more diverse scenarios.** The capability of the proposed framework is mainly constrained by the foundational diffusion model. Since the multi-view generator [2] used in this work focuses on object-centric data, we limit our exploration to the generation of single dynamic objects, following existing works in this field [1,3,4]. For more diverse 4D scenarios, as long as the two diffusion models share a same latent space to model their marginal distributions, a similar sampling approach can be applied. > [1] Jiang, Yanqin, et al. Consistent4d: Consistent 360° dynamic object generation from monocular video. *ICLR*, 2024. > [2] Voleti, Vikram, et al. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. *ECCV*, 2024. > [3] Zhao, Yuyang, et al. Animate124: Animating one image to 4d dynamic scene. *arXiv preprint*, 2023. > [4] Xie, Yiming, et al. Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency. *arXiv preprint*, 2024.

Looking forward to your feedback

Dear Reviewer s4pt, We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our detailed explanations and additional results solve the concerns. We have addressed all the thoughtful questions raised by the reviewer *(eg, computational requirements, extending to more challenging cases)* and we hope that our work’s contribution and impact are better highlighted with our responses. As the discussion phase is nearing its end, it would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you! Best wishes, Authors