SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D

**Q: Novelty** A: Please refer to the common response for our reply. **Q: A simple baseline (AGP-only) is missing.** A: Thanks for pointing this out. When generating the 3D results shown in the submission, AGP has actually dominated the geometry modeling in both NeRF- and DMTet-based pipelines, as we set a large weight (5x \lambda_ori) for \lambda_align at the beginning of the optimization. As demonstrated in Figure 9 of the revised version, the overall shape generated by AGP-only is almost the same as the AGP + DeepFolyed IF SDS, except for minor details. As a result, the qualitative results are the same as the Ours NeRF-based IF. Please refer to Figure 9 in the revision paper for further visual results at each stage in the generation pipeline. **Q: The metric of R-Precision.** A： We thank the reviewer for this insightful suggestion. For more discussion, please refer to the feedback of Reviewer nptq. During the rebuttal, we obtained the numerical results of appearance-related metrics. We report the R-precision as your suggestion. The results are presented below and in Section A.7 and Table 3 in the revised paper. **Q: The ablation discussion is absent.** A: Thanks for this suggestion. Please refer to the common response for more details on this point. **Q: How long does the fine-tuning for a stable diffusion model take?** A: Since we only use coarse geometries, i.e., low-resolution 64x64 CCM maps, to fine-tune the SD, the training process is highly efficient. The whole fine-tuning process takes about one day on 8 A100 GPUs for reasonably good results, and the final model used in the submission was trained for two days. We added more details about finetune in the revision A.2. **Q: How does the efficiency (inference time) of this model compare to other methods?** A: Our AGP model is designed to be as lightweight as possible, with the goal of improving the consistency of generated results. The inference time of the fine-tuned SD model is almost identical to that of the original model. And the time for the whole generation depends on the base text-to-3D model, but our model has introduced almost no additional computational time overhead. Specifically, it takes 0.5 hours for our DMTet-based pipeline and 1 hour for our NeRF-based pipeline, using 4 V100 GPUs.

Your feedback is greatly appreciated! We've strived to address your concerns in our response. As the deadline approaches, should further queries arise, we are fully prepared to answer. Additionally, we respectfully request your kind consideration regarding a potential adjustment in the rating. Your insights are invaluable to us. Thank you sincerely for your time and support!

**Q: Ablation studies and the generalization ability of the proposed method.** A: Please refer to the “More ablation studies” and “Technical novelty” in the common response for the ablation studies and generalization ability, respectively. **Q: Novelty.** A: Please refer to the “Technical novelty” in the common response. **Q: Filtering the data and not using all may constrain the generalization ability.** A: Our filtering process mainly filters out noisy 3D data, such as low-poly shapes, point clouds, unbounded scenes, etc, using some simple and heuristic rules. In addition, we also filtered out some obviously wrongly oriented samples in certain categories (e.g., characters) based on the orientation of their Axis-Aligned Bounding Box (we missed this filtering in the submission and have revised accordingly). While we agree that the data can potential affect the fine-tuning process, we also believe that inherently the key is the diversity and quality of the data rather than simply the absolute amount of data. Moreover, we only use the coarse geometries in the dataset for aligning the diffusion priors at the coarse geometry level, such “less is more” way of using the data has demonstrated its efficacy in alleviating the multi-view inconsistency issues without compromising significantly the generalizability of powerful diffusion priors in terms of highly varied appearance and geometric details, as evidenced by extensive successful results in the submission. In addition, recent works have also found that 2D diffusion models are already very powerful, and the results of lightweight fine-tuning using a small number of high-quality 3D data may even be better than training with a larger amount of data (such as Wonder3D and Instant3D). This is consistent with our experimental experience, and we have added more discussions on this in the revision (See Appendix A.5). Last, we have also added a discussion on the generalizability of our method in Appendix A.4. More results, that showcase the ability to generalize to highly varied objects unseen in the dataset, can also be found on this anonymous website. http://3dsweetdreamer.github.io/nerf-based-gallery_0.html.

Your feedback is greatly appreciated! We've strived to address your concerns in our response. As the deadline approaches, should further queries arise, we are fully prepared to answer. Additionally, we respectfully request your kind consideration regarding a potential adjustment in the rating. Your insights are invaluable to us. Thank you sincerely for your time and support!

**Q: Ablation studies between CCM and other representations.** A: Thanks for the valuable comment, please refer to the common response about “More ablation studies” for more details. **Q: Limited experiments.** A: Almost all existing text-to-3D methods, including ours, share a common limitation that the optimization-based generation process could take hours for obtaining a single 3D result. For example, DreamFusion takes 1.5 hours on 4 TPUs, Magic3D 40 minutes on 8 GPUs, Fantasia3D 1 hour on 4 GPUs, and ProlificDreamer approximately 8 GPU hours. Due to this very reason, while DreamFusion managed to present results obtained using 398 prompts, the number of results presented in most of these works is indeed limited, such as 40 prompts in MVDream and a dozen in Fantasia3D and ProlificDreamer. Yet, in our submission, we have presented around 100 results obtained using various arbitrary prompts in the video and supplementary, and compare to baseline methods on 80 prompts. Therefore, we would like to emphasize that our experimental results are sufficient and should not be considered as a weakness, as these prompts are several times more than those used for evaluating previous methods. During the rebuttal, we have increased the number of text prompts up to 150 (see anonymous links http://3dsweetdreamer.github.io/nerf-based-gallery_0.html and https://3dsweetdreamer.github.io/dmtet-based-gallery_0.html) but are indeed limited by the computing resources for more, we are happy to add more results using more prompts to the revision in the next stage. **Q: Criteria for determining 3D consistency in quantitative evaluations.** A: We agree that the definition of the 3D consistency can be vague to describe, and currently there is a lack of objective metrics that can be computed automatically to determine if a 3D object is multi-view inconsistent or not. This is why we rely on human users for quantitative evaluation of the 3D consistency of the generated results. Specifically, the human participants involved in Table 1 of the paper are researchers from academic laboratories, with highly relevant research backgrounds such as Computer Vision, Computer Graphics and Visual Computing, while the users involved in obtaining the preference rate charts are from loosely related backgrounds such as CS and EE (a more noisy evaluation process, yet our method consistently outperforms other baselines). Furthermore, we have included the user study interface in the appendix of the revised paper for clarification. **Q: Report the appearance quality.** A: We thank the reviewer for this insightful suggestion. As also mentioned in the common response, our method can preserve the high realism of the 2D results of the powerful text-to-image foundation model, as we only align the geometric priors at the coarse geometry level. This is particularly attractive, as the 3D dataset (e.g., Objaverse) in its current form falls short of covering samples as diverse as the 2D text-image dataset, particularly in terms of the appearance. During the rebuttal, we have obtained the numerical results of appearance-related metrics. As suggested by Reviewer UgSj, we use the R-precision, which is also used in related work. The results are presented below, and in Section A.7 and Table 3 in the revised paper. It is obvious that our method achieves the best performance on this metric over competing methods. | Method | CLIP B/32 R@1 | CLIP B/32 R@5 | CLIP B/16 R@1 | CLIP B/16 R@5 | |------------------|---------------|---------------|---------------|---------------| | Magic3D | 60.1 | 71.5 | 62.7 | 77.7 | | TextMesh | 51.7 | 65.1 | 55.1 | 78.4 | | SJC | 40.2 | 51.2 | 52.5 | 62.5 | | DreamFusion | 59.7 | 70.2 | 61.6 | 74.3 | | Ours (NeRF-based)| **77.5** | **84.9** | **88.7** | **92.3** |

**Q: Whether using a small amount of high-quality data for fine-tuning will improve the performance.** A: In the submission, our model is fine-tuned using 3D objects obtained by filtering the Objaverse via some simple and heuristic rules. In addition, we also filtered out some samples in certain categories (e.g., humans) simply based on the orientation of their Axis-Aligned Bounding Box (we missed this filter in the submission and have revised accordingly). After these filterings, most of the data have been canonically aligned (around 80%, estimated based on statistical random samplings), and the rest wrongly oriented objects have only a minor impact on the performance of our model, as manifested by extensive successful results in the submission. In addition, as suggested, we have also trained our model on a carefully and manually selected dataset, which consists of 20,000 objects. The resulting model is still able to generate highly varied 3D results, with detailed appearances (See A.5 and Figure 10 of the revised paper). While we have not observed significant improvement on this setting, we believe that using more well-structured data could potentially benefit the 3D generation and will perform a more thorough study in the next stage. **Q: More implementation details.** A: Ours (NeRF-based full) uses the DeepFloyd IF model first and then using SD. While DeepFloyd IF can be a more powerful model for text-to-image, it can only perform at the 64 x 64 resolution on our side due to its use of much more computational resources. For a fair comparison, Ours (NeRF-based) in the user study is conducted using only DeepFloyd IF, which is the same as other competing methods. **Q: Hyperparameters.** A: All results presented in the paper were generated using the same hyperparameters, without the need for adjusting individually for each generation. More specifically, at the beginning of the generation we guide the generation to focus more on the geometry, so the $\lambda^{align}$ is set to a larger value (5x $\lambda^{ori}$) and then decreased to 0.5x for higher-quality appearance. **Q: More details about the interface of the user study and the criteria for user selection.** A: We added more details and the figure of the interface for user study in the appendix. Please kindly see the feedback in the revision.

Thank the authors for the detailed reply. The ablation studies between CCM and other representations show the advantage of CCM. The new results measuring CLIP R-Precision show the advantage of the proposed method on image quality. Moreover, the added details would be helpful for readers to have a better understanding. One minor question: in the user study interface, have the generated objects been shuffled in each row? It seems that in the figure of the NeRF-based method, the objects in the 4th row always have the largest size.

Hi, we hope our responses have effectively addressed all your concerns. Please be informed that the author-reviewer discussion period ends today. Should there be any unresolved issues, we would appreciate your prompt feedback. If your concerns have been addressed, please kindly consider updating your score. Thank you!

**Q: Ablation studies among CCM, depth map, and normal map.** A: We appreciate your feedback. Please refer to the common response for more details. **Q: The generalization ability of the model after fine-tuning.** A: Since we only finetune and constrain the geometric prior at a coarse resolution, the powerful generalization ability of the image diffusion models is largely preserved. The appearance modeling totally relies on the original diffusion model trained on billions of high-quality images. During the geometric modeling process, we were surprised to find that the finetune model still has strong generalization ability and can generate prompts such as Einstein wearing a gray suit and riding a bicycle. This is difficult to achieve with other existing methods. We have added more discussion on the generalizability in Appendix A.4. **Q: More details about normal rendering.** A: The bumps come from the tangent space normal. For appearance modeling, we follow Fantasia3D and optimize the tangent space normal, which is a crucial component of Physically Based Rendering (PBR) materials. The normal map showcased in our paper emerges from offsetting the original geometric normal through the application of tangent space normal. This procedure generates multiple bumps in visual, thereby enriching both the hierarchical structure and realism of the appearance. It allows the representation of intricate surface details without increasing the geometric complexity of the underlying model. This efficiency is particularly useful for representing high-resolution details on low-resolution geometry. **Q: More details about the orientation.** A: Let us illustrate the concept of 'mis-oriented' with an example. In the objaverse dataset, imagine 3D asset labeled as a 'cute panda.' When this asset is rendered from an azimuth=0 perspective, deviating from the front view of the panda, we classify this as the object being mis-oriented. Conversely, if the front view aligns with azimuth=0, we deem this as the correct or consistent orientation. Following the application of a specific set of rules to filter around 270k objects from the initial dataset, we conducted random sampling and manually assessed the percentage of objects exhibiting the accurate orientation. Our findings revealed that roughly 80% of the data exhibited consistent orientations. We reveal that the orientation of objects generated by the proposed model trained with such dataset is almost always correct and has a strong generalization ability. In order to further investigate the impact of data with a higher proportion of correct orientation on the generated results, We then manually screened the top 20k data, resulting in a dataset of 18,488 instances with a uniform orientation and high correspondence with their descriptions. From the experimental results obtained by training on such a smaller dataset, we are surprised to find that even with a smaller dataset of only 20k instances for fine-tuning, our text-to-3D pipeline still exhibits strong generalization and is capable of generating diverse 3D objects. Please refer to Appendix A.5 in the revised version of our paper.

Your feedback is greatly appreciated! We've strived to address your concerns in our response. As the deadline approaches, should further queries arise, we are fully prepared to answer. Additionally, we respectfully request your kind consideration regarding a potential adjustment in the rating. Your insights are invaluable to us. Thank you sincerely for your time and support!