ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

Summary

The paper presents a nice distillation pipeline that can perform high-quality text-to-3D generation using 2D diffusion priors. By introducing the score on NeRF / mesh rendered images, the authors make score distillation work with a classifier-free guidance weight as low as 7.5. This approach largely improves the detail quality of the converged results and inspires the community with an exciting direction.

Strengths

- The authors propose to estimate scores on rendered image with a LoRA framework fine-tuned for the target scene. - By introducing this additional score, the distillation framework is able to work with low classifier-free guidance weight at 7.5, yielding much better appearance than the original SDS loss in DreamFusion. - Both 2D and 3D experiments validate the expressiveness of the proposed framework.

Weaknesses

- The paper starts with derivations to support the particle-based variational framework. However, using more particles doesn't seem to help with the generation results. Is using a single particle not enough? What's the benefit of multiple particles, and when do we need them? - In Eq.(8) for 3D experiments, the authors mention that there's a `c` for camera pose added for the conditioning of the diffusion model. How is this implemented? Is this camera pose necessary for the framework to work? It would be nice if there's an ablation study on removing the camera pose conditioning. - For 3D experiments, why do we need a LoRA model fine-tuned from a pre-trained Stable Diffusion? Does using a small U-Net work for 3D case? - For the mesh refinement stage, the authors follow Fantasia3D to refine the geometry and texture separately. SDS loss is used in the geometry stage that refines normal maps. I'm interested in whether the proposed vsd loss is helpful in this case. Adding a comparison experiment that uses VSD to refine the normal map will be great. - Existing methods mostly use view-dependent prompts, meaning that text embedding `y` and camera `c` are not independent. Do authors use view-dependent prompts in the experiments? - Why are the two rows in Eq. (5) equal? More derivations are appreciated. - Is v-prediction necessary for the LoRA model? Does the proposed framework work with eps prediction?

Questions

Please refer to the weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Please refer to the weakness section.

Summary

This paper proposes an interesting and novel technique for the task of text-to-3D generation. It defines a distribution of the target 3D scene, which is implemented as particles. Given the textual description, the distribution is updated using the Wasserstein gradient flow. Moreover, the paper proposes to fine-tune a diffusion model based on LoRA to calculate the score function of noisy rendered images, which is used to compute the objective function. Experiments show that the approach significantly improve the generation quality.

Strengths

1. The proposed technique is novel and interesting. Modeling 3D scenes as a distribution is insightful, which could inspire the following research. 2. It is a good idea to fine-tune LoRA-based diffusion models on noisy rendered images for calculating the objective function. 3. The results of the proposed model are impressive and obviously outperform previous methods.

Weaknesses

1. I am curious about how the number of particles affect the performance of the proposed model. 2. Can this idea be used in amortized text-to-3D generative models? Does it consumes a lot of GPU memory? 3. It would be better to perform ablation studies on the distillation time schedule and the density initialization.

Questions

Please see the weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The authors have discussed the limitations.

Summary

In this paper, a novel method for generating 3D representations from text prompts is proposed. More specifically, a novel variational score distillation (VSD) based optimisation strategy is proposed that allows for optimising NeRF- and mesh-based 3D representations by utilising pre-trained 2D diffusion models. It improves over the previously in Dreamfusion proposed score distillation sampling (SDS) by optimising a distribution over the 3D parameters instead of a single point Dirac distribution. Further, the authors made additional engineering improvements to the text-to-3d pipeline. The proposed system does not rely on high CFG weights as required for SDS and leads to impressive results that compare favourably against the state-of-the-art.

Strengths

- The authors clearly identify an important shortcoming of current text-to-3D approaches: suffering from over-saturation, over-smoothing, and low-diversity problems. They identify the SDS-based optimisation as the core problem, and propose a theoretically-sound alternative that can be trained with only a small overhead compared to SDS. - The shown results clearly look impressive and the obtained scene representations clearly improve over the state-of-the-art results. - The manuscript is (mostly) very well written, has a great structure, and a great "reading flow". - The performed experiments, including the 2D experiments, clearly highlight the authors' contributions and show the effectiveness of the individual components of the proposed methods. - The authors provide a clear and correct overview over their core contributions in Table 1 and demonstrate their familiarity with the fast-moving field of text-to-3D generation.

Weaknesses

- Number of particles: I am sceptical if n=1 or n=4 (L. 150) is really enough to capture the distribution. It would be interesting to investigate the relationship of number of particles and the quality of the results; as it seems that the prediction network for the noisy rendered images is trained with all particles, it could be that the quality overall improves with more samples. - Geometry optimisation and VSD/SDS: Accordingly to L. 247 ff., I believe that the authors explain that the mesh geometry is fine-tuned with SDS, not VSD (see also question below). If understood correctly, this part should express more clearly that it is only referring to the mesh geometry, not the overall geometry optimisation. Further, this seems to be an interesting limitation of the current method. It would be interesting to see a respective ablation figure comparing SDS and VSD for this mesh-geometry prediction stage. - Geometry Diversity: Figure 1 c.) shows the diversity of samples, and it appears that the geometry diversity is limited compared to the texture diversity. It would be interesting to discuss potential reasons for this. - Experimental Results: No quantitive metrics are reported and the only user study is "hidden" in Table 3 of the supplementary material. While it is true that the task of text-to-3D is very challenging to be measured quantitatively, metrics reported by previous methods such as Dreamfusion could also be reported, and the user study can be shown in the main paper or at least be referenced. - Camera Pose Priors: It would be very interesting to see how the camera sampling strategies affect the results; in particular for the "scene generation" experiments. Have the authors started investigating this dimension of the problem? - Related work: A good and thorough related work discussion is contained only in the supplementary. I would encourage the authors to add a discussion, if accepted, to the main manuscript. - Noisy prediction network: The scores for the noisy real and the noisy rendered images are predicted by two separate models (L. 166), and the one for the rendered images is smaller / a low-rank approximation. Is this introducing some form of imbalance or is this not a problem for the optimisation? - Training of noisy renderings score prediction network: If understood correctly from the supplementary, the network for predicting the scores of the noisy rendered images is trained simultaneously with the NeRF model. It could be interesting to study how different optimisation schemes affect the results. - Input text prompt: I believe it is not stated whether the same text prompt or different text prompts are used for different camera poses. Typos: - L 178: speical case -> special case - L 283: variation score distillation -> variational score distillation - L. 284: 3D parameter -> 3D parameters - L 637 of supp mat: an addition diffusion model -> an additional diffusion model

Questions

- Have the authors investigated whether there is a relationship between number of particles and the quality of the results? - Could the authors expand on the limited geometry variance in contrast to the texture variance, and why the use of the VSD is not improving over SDS for the mesh geometry prediction? - Could the authors expand on the importance of used camera pose priors, especially in the context of scene generation? - Are different text prompts used for different camera poses?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The authors discuss limitations both in the main paper as well as more extensively in the supplementary. I believe the discussion is thorough and good as is.

Summary

This paper proposes Variational Score Distillation (VSD), which is a new generalization of Score Distillation Sampling (SDS) that is based on

Strengths

The biggest strength of this paper is likely in its practical applicability; notably, I find it impressive that the paper is able to produce good results with a batch size = 1. Many prior works need larger batch sizes, and especially mesh generation (as in Magic3D) suffer if they do not have large batch sizes which significantly hurts performance. This is significant, as it makes text-to-3D generation much more feasible without access to a large GPU cluster / DGX sort of machine. However, it's not unclear if this is really due to VSD (more on this in the weakness section). There are also other practical benefits to this algorithm, like the additional diversity in the output that they can offer. This is also significant, as one of the practical needs in deploying a text-to-X system is being able to 'reroll' for better outputs that better align with the user's intent through the text prompt. This is something that VSD uniquely offers. The paper is also very well written, and the writing is very clear despite the complexity of what is being described. I do have a couple of more detail-oriented questions that I will ask in the questions section.

Weaknesses

I am slightly unconvinced about the effectiveness of VSD when it comes to increased quality. First, as a baseline, the generated results qualitatively do look much better than DreamFusion or Magic3D. The user studies agree, with an overwhelming margin (90+%). However, it is also the case that the generated results in this paper _without_ VSD and with just the higher resolution prior (and annealed t) already looks arguably significantly better than DreamFusion or Magic3D. Without a user study to then compare against this result with and without VSD, it's hard to make the claim that VSD itself is the factor that contributes most significantly to the improved quality. It's not entirely made clear in the paper what exactly is the factor that leads to this significantly quality increase. Empirically, high resolution NeRF training for SDS with Stable Diffusion has been difficult to make work (just even based on open source re-implementations of various papers), _especially_ with a small batch size of N=1 that they claim in the paper... so I am very curious what leads to the quality increase. But also it makes me question whether VSD really adds much in terms of quality without more examples or a user study. (Figure 5 does show an example where there seems to be perceived difference, but in my own experience, this is a minor difference that could just be caused by a slightly different hyper parameter) Figure 17, although it's only on a single example, somewhat confirms this since the high CFG weight seems to result in very similar results with the VSD results with low CFG. Although high CFG weight is a factor for over saturation, some of the results with VSD also seems to suffer from over saturation anyways. Regardless, VSD does offer other benefits like diversity. But it would be amazing to hear from the authors on their thoughts on this quality difference in the SDS baseline. Ideally a user study can compare between their baseline NeRF generation results with SDS against VSD, but I understand if that's difficult. At the very least, there should be more examples (beyond the single example they show in Figure 5) that compare between their baseline NeRF results with SDS vs VSD. This would make this paper an extremely solid contribution. (it's also possible I just missed something).

Questions

1. In Figure 3, the VSD results generate slightly more 'oversaturated' results in comparison to ancestral sampling. Similarly, the mesh results in the paper generally seem to still suffer from over saturation issues (although some of this is likely just due to not modeling accurate physically based rendering). Is there any explanation for the perceived differences between VSD and ancestral sampling and why they lead to these qualitative differences? 2. For the SDS experiments (in Figure 17 in the appendix, for example), what are the experimental settings? Do they use the same batch size = 1 with all of the same hyper parameters as the VSD? What hyper parameter differences exist between this and DreamFusion, Magic3D? 3. How does the SDS NeRF results from this paper (without VSD) compare to DreamFusion and Magic3D in terms of perceived quality? What are the factors that cause this quality difference?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The limitations are addressed, although I have a lot of questions about the actual efficacy of VSD with respect to quality improvements. It would also be nice to have a broader statement on the societal impacts (even if it's somewhere in the appendix, unless I just didn't see it) since text-to-3D models are something that can bring potentially harmful impacts to the labor market around content creation.

Summary

The authors introduce the VSD algorithm that generalizes SDS to optimizing a distribution of shapes in terms of KL divergence to the image diffusion model. Lacking analytic score of the implicit rendered image distribution, they train a surrogate by using LORA finetuning to the base image diffusion model. The resulting algorithm is demonstrated for optimizing an empirical distribution of particles and interpreted in framework of particle VI. They empirically find that a lower CFG is possible when using VSD as opposed to SDS which alleviates artefacts (even in the n=1 case). Additionally, a simple NERF initialization for scenes, annealed time schedule and higher resolution rendering are proposed as orthogonal improvements.

Strengths

Novel method: Using an auxiliary fine-tuned image diffusion model to approximate the score of an implicit distribution is a very neat idea and could have potential wider applicability in other areas of generative modelling. Presentation: The motivation behind the method, technical background and mathematical framework are presented well and the writing is clear in these aspects. Details regarding the method are also fairly clear. Strong qualitative result: The ability to use lower CFG is an interesting finding and the authors give evidence for its effectiveness in producing high quality results. Combined with the other practical additions to the training pipeline, this results in very impressive SOTA visual quality.

Weaknesses

Clearer experiments: There are a lot of different experimental settings which dilutes the evaluation of the core VSD contribution imo. The paper seems to have 4 main experimental settings: a. Single particle textured mesh objects b. Single particle NERFs (object and scene) c. Multi-particle NERF d. 2D VSD results In terms of multiple particles, these are only demonstrated for 1 2D prompt and 2 NERF prompts (and maybe 1 mesh in the appendix). Can multiple particle results be demonstrated for the prompts used in a/b. Given the lack of quantitative metrics, the contributions of VSD, annealing t and higher resolution are only evaluated qualitatively on 1 or 2 prompts. Ablating VSD for more prompts would help strengthen the case for it. Currently it is only done for one of the prompts in setting a. Lack of quantitative metrics: I understand there aren’t great established metrics in this area but currently the comparison is only done by showing a few qualitative examples and a user study with 5 prompts picked per baseline. Especially since one of the claims in the paper is that VSD allows diverse generation, quantifying this over a representative set of prompts would greatly strengthen the evaluation section. R-precision has been reported in previous papers and given the motivation of matching in KL, a conditional FID to the image diffusion model could also be considered. If computation is a bottleneck, then these could have also been obtained for a coarser version/2D case. Clarity of explaining VSD results: Although VSD was motivated from the stand point of enabling diverse generations, the empirical benefits it brings in the single particle case is surprising and remains not well explained. Hence, some of the discussions in section 3.3 (about “superior generalization” and regarding CFG on lines 185,186) seem too vague. I would recommend the authors could change the wording to make it more clear which results are empirical findings that may not have theoretical justification yet and further to write any intuition/speculation about the reason more clearly.

Questions

- On line 247 it is mentioned that for mesh finetuning, VSD was not helpful for geometry. To clarify, the pipeline for the textured mesh then is: 1. VSD for Nerf init, 2. SDS on DMTet geom finetuning, 3. VSD on Texture finetuning? How would this compare to following Fantasia3D and skipping the 1st NERF stage? Can you show the intermediate results during each stage? Does the SDS stage affect the geometry much in stage 2? - Is the ice cream ablation in Figure 12 using the textured mesh pipeline? - In the appendix, there was some intuition about VSD proving more fine/sharp update direction. What do samples from the LoRA tuned model and the corresponding update direction look like? - The paper claims that SDS is one of the main bottlenecks to scaling resolutions to 512 (line 226). Could you elaborate on this, since it seems they are orthogonal? - For the 2D experiment, you tried the small U-Net with 2048 particles. Did you also try n=2048 for the LoRA? Are there issues with scaling LoRA?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations as expanded upon in the appendix is sufficient. Maybe the authors can expand more about the robustness of the method and sensitivity to hyperparameters which is a common issue in related works.