Score Distillation via Reparametrized DDIM

Summary

This paper introduces a novel algorithm called Score Distillation via Inversion (SDI) that enhances the quality of 3D shape generation. By inverting Denoising Diffusion Implicit Models (DDIM) at each step and incorporating the initial noise into the estimated score, SDI addresses the over-smoothing and detail loss issues associated with traditional Score Distillation Sampling (SDS) methods in 3D generation. Experimental results demonstrate that SDI significantly improves the quality of 3D shapes, closely matching the quality of 2D image generation, without the need for additional neural network training or multi-view supervision. The work provides valuable insights into understanding and improving 3D asset generation with diffusion models.

Strengths

1. The paper provides a theoretical analysis that links Score Distillation Sampling(SDS) with Denoising Diffusion Implicit Models(DDIM), offering a deeper understanding of the discrepancies between 2D and 3D generation process, which is different from the experimental enhancement made by most of the previous methods. 2. The modification to the existing SDS method is straightforward and does not require training additional networks or multi-view supervision. Furthermore, SDI significantly improves the quality of 3D shape generation, which makes it a practical solution for enhancing 3D generation. 3. This paper provides thorough analysis and present a set of convincing ablation experiments.

Weaknesses

1. According to the authors’ theory and hypothesis, the better the noise term is Eq. (8) is estimated, the better the 3D generation quality is improved. However, in the ablations about the choice of k(x), SGD optimization doesn’t show significant improvement. Also, this phenomenon can be observed from Figure.10, in which green curve and purple curve present lowest and comparable MSE with orange curve, while the orange curve get the best result. I would appreciate it if the authors can provide some explanations. 2. One of the paper's key perspective is that SDS introduces high variance. I have read another paper[1] dedicated to minimizing the variance introduced in SDS, and maybe it's also worthwhile to compare with that. [1] Tang, B., Wang, J., Wu, Z., & Zhang, L. (2023). Stable score distillation for high-quality 3d generation. ArXiv Preprint ArXiv:2312.09305.

Questions

Please refer to the weaknesses part.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the weaknesses part.

Summary

The paper introduces an effective SDS modification that replaces random noise samples in the SDS objective with those obtained with DDIM inversion. The proposed technique enhances text-to-3D generation, outperforming SDS and being comparable to more sophisticated methods.

Strengths

* The paper is well organized and provides valuable intuitions, illustrations, and discussions over reading. The additional discussion in Appendix E is also beneficial. * The proposed method is simple, reasonable and well motivated. The derivation is clear. * The experimental results are promising, and the ablation study in Sec 6.2 is highly valuable and interesting.

Weaknesses

* Inversion with negative guidance is surprising, and its rationale remains somewhat unclear. A more detailed investigation and justification would be useful. * The idea of using DDIM inversion is very similar to ISM[1]. The primary distinction is guided vs unguided inversion. Although the authors justify the guided inversion with negative guidance in Fig. 10, a thorough discussion and detailed comparisons are essential. * The quantitative evaluation relies on CLIP-IQA, which does not seem quite suitable for 3D generation evaluation. Conducting a user study would be highly valuable. If it is unavailable, some recent works [2,3] aim to propose automated evaluation metrics. For example, would it be reasonable to evaluate ImageReward[4] following [2]? * The quantitative results miss NFSD and the qualitative comparisons are presented only for two prompts. Could the authors add NFSD to Tab.1 and provide more visual comparisons with all methods? * There is no evaluation of diversity under a given prompt. It would be useful to include a few generated results for different seeds. To sum up, I am generally optimistic about the submission but have some concerns regarding the evaluation and close connection with [1], as discussed above. [1] Liang et al. LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching. [2] He et al. T3Bench: Benchmarking Current Progress in Text-to-3D Generation. [3] Wu et al. GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation. [4] Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.

Questions

* Please address the questions and concerns raised in "Weaknesses". * In Fig.5, the intermediate $x_t$ do not have noise in the background. DDIM inversion must follow the same marginal distributions as the forward process, but the presented $x_t$ do not. We observed the similar effect for images with monotonic regions if inversion is applied directly to a clean image. We believe such images are likely OOD for the pretrained DM, and injecting slight noise with $\sigma_{min}$ to an image fixed the problem. Do the authors have any thoughts on this? Would the solution above work for SDI and how does it affect its performance?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed the limitations and potential negative social impact.

Summary

This paper connects score distillation sampling (SDS) to a DDIM sampling process. The proposed method Score Distillation via Inversion (SDI) replaces the original random noise in SDS with DDIM inversion, and show significantly improved quality compared to SDS and other state-of-the-art prior methods.

Strengths

1. The connection from SDS to DDIM sampling is interesting. The motivation is also clear that it considers the change of the "sample prediction" variable to guide the 3D generation process. The method is supported by well derived theory (for 2D) and the DDIM inversion is a clever solution. 2. The results generally look good. It shows significant improvement over its baseline method SDS, and is competitive to recent state-of-the-art text-to-3D methods. The experiments are thorough. 3. The proposed method and results could provide insights for future research in the important direction of text-to-3D distillation.

Weaknesses

1. Overall, the visual results seem to be having a little bit "gray" color/style shift. Is this due to the approximation or some bias in the theory? Is there any hypothesis for theoretical / practical reasons? 2. The ddim inversion and approximation could introduce additional computation cost and training noise. 3. In "ISM as a special case", it seems the main difference of the proposed method and ISM is having the additional text condition. Is there any other key differences in method implementation or theory? More details about the contribution of this work based on ISM could be helpful.

Questions

Does the proposed method remedy Janus problem, compared to the baseline SDS method? (If yes, why) Also please check the weakness section.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors discussed limitations.