Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day

Response by Authors

We thank Reviewer TYx2 for acknowledging the novelty of our proposed framework, appreciating its effectiveness, and proposing constructive feedback. Below, we address the concerns raised in your review point by point. > Q1: The study employs the Objaverse dataset as the sole testbed. Following your suggestions, to further demonstrate the generalizability of the proposed approach, we set a testbed on an in-the- wild real-world dataset — the OmniObject3D dataset[a]. The OmniObject3D dataset contains 6,000 scanned objects with 190 daily categories. Both 2D and 3D sensors are adopted to capture each 3D object. We include the details of how we split the test set in the updated manuscript, as highlighted in blue color. Below we present the LPIPS score reported on the OmniObject3D dataset. | Training Steps | Zero 1-to-3 | Ours | | --- | --- | --- | | 10k. | 0.327 | **0.253** | | 20k | 0.290 | **0.248** | | ... | ... | ... | | 105k | 0.276 | - | | 300k | 0.268 | - | As shown in the Table, the proposed method Efficient-3DiM converges faster than the baseline approach Zero 1-to-3. Moreover, our method only takes 20k steps to reach the best performance, better than the zero 1-to-3 method even when the latter one is being optimized by 300k training steps. More details can be found in Appendix C. > Q2: Are all comparison methods adopting the same fixed number of viewpoints? Yes, we directly use the rendered training views provided by the authors of the Zero 1-to-3 approach. For each ablation experiment, we use the same training and testing set. > Q3: The work itself follows the line of works by (Watson et al., 2022) and (Liu et al., 2023b), and mainly studied training efficiency improvement. The contribution is solid yet not fully substantial. The topic of 3D object creation has gained growing popularity in the past year. Given the importance of diffusion modes and their applications in 3D vision, developing an efficient training approach would greatly aid researchers in hastening the development process, consequently propelling the advancement of this direction. Our work is simple yet effective. The simplicity allows the proposed technique to be easily integrated into a newly designed framework in future works. We believe our work will help researchers develop novel algorithms with limited computational resources. References: [a] OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation

Response by Authors

We thank Reviewer FP6T for acknowledging the novelty of our approach to the problem and appreciating the results of our experiments. Below, we address your questions and concerns: > Q1: Would this proposed approach be generalizable to accelerating training other image-to-3D models, such as Zero 1-to-3 and Syncdreamer? Yes, our baseline method is exactly the same as the Zero 1-to-3 method, so the proposed technique is beneficial to Zero 1-to-3 as has been verified in Figure 7 of our main manuscript. The pipeline of Syncdreamer requires finetuning from a pre-trained Zero 1-to-3 checkpoint. Therefore, accelerating the training procedure of Zero 1-to-3 indeed helps reduce the total training cost of SyncDreamer approach as well. > Q2: The evaluation is solely conducted using the Objaverse dataset. It would be beneficial if the authors could also showcase results on additional datasets for a more comprehensive assessment. Following your suggestions, we set a testbed on a real-world dataset, the OmniObject3D dataset[a], to further demonstrate the generalizability of the proposed approach, The OmniObject3D dataset contains 6,000 scanned objects with 190 daily categories. Both 2D and 3D sensors are adopted to capture each 3D object. We include the details of how we split the test set in the updated manuscript, as highlighted in blue color. Below we present the LPIPS score reported on the OmniObject3D dataset. | Training Steps | Zero 1-to-3 | Ours | | --- | --- | --- | | 10k. | 0.327 | **0.253** | | 20k | 0.290 | **0.248** | | ... | ... | ... | | 105k | 0.276 | - | | 300k | 0.268 | - | As shown in the Table, the proposed method Efficient-3DiM converges faster than the baseline approach Zero 1-to-3. Moreover, our method only takes 20k steps to reach the best performance, better than the zero 1-to-3 method even when the latter one is being optimized by 300k training steps. More details can be found in Appendix C. References: [a] OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation

Response by Authors

> Q1: I do not fully understand why LN can mitigate numerical errors, in the third part of Efficient-3DIM The DINO-v2 encoder and UNet denoiser are separately trained under two different training tasks. Directly constructing a new model using these two components may cause instability and this issue is more severe under the mixed-precision training scheme, since 16-bit may easily introduce numerical error. The normalization layer will help calibrate the gap between these two models. We choose layer normalization rather than other types of normalization techniques due to the fact that the DINO-v2 encoder follows the design of Vision Transformer, which adopts layer normalization by default. > Q2: Could the authors elaborate on how the proposed method could be integrated with distributed training, and whether its speedup benefits may diminish in the (more scalable) distributed training setup? The proposed techniques are capable of distributed training systems and will not diminish, due to our major contribution focusing on algorithm-level only. Although we claim 192 batch size per GPU in the setting, it is achieved by 4 times gradient accumulation due to memory issues. Therefore, the real batch size per GPU is actually 48 for each single step. Under this circumstance, utilizing 4 distributed nodes and avoiding using gradient accumulation will deliver theoretically the same performance but a much faster training speed. Same as all other tasks, the speedup ratio is not ideally the same as the number of nodes we are using, due to the communication cost between each node. Improving the speedup ratio is more like a system-level task, which is currently beyond the scope of this work. > Q3: Figure 8: I need help seeing how Zero 1-to-3 falls short of producing multi-view consistent visual outputs. All displayed results have valid visual quality to me. Following your suggestions, we run each comparing method and show multiple outputs with different camera parameters as conditions. As shown in Figure 10 in the updated manuscript, we run the inference procedure under 4 different random seeds and different camera parameters. The baseline method Zero 1-to-3 fails to produce multi-view consistent textures in the highlighted region, while the proposed method successfully generates consistent results in all groups of seeds. > Q4: Minor: “neurallift-360” paper was incorrectly cited twice in reference. Thanks for pointing out the issue, we have fixed it in the updated manuscript.

Response by Authors

We thank reviewer mro2 for appreciating our contributions and providing constructive feedback. Below we address your concerns point by point. > Q1: I am not persuaded by the motivation from comparing training image classifiers and generative models, which are not comparable. Moreover, we usually treat Zero123 as a foundation model which does not need retraining. Under this circumstance, I think the value of the proposed Efficient-3DiM diminishes. The topic of 3D object creation has gained growing popularity in the past year. Given the importance of diffusion modes and their applications in 3D vision, developing an efficient training approach would greatly aid researchers in hastening the development process, consequently propelling the advancement of this direction. In other words, reducing the training time can help researchers achieve rapid algorithm iterations. We also believe that our work enables more researchers to join this community to develop novel algorithms if they only have limited computational resources. > Q2: The section "ENHANCED TRAINING PARADIGM" contains several well-known tricks, such as mix-precision training. I would like to suggest the authors should not emphasize this too much in their contribution. Thanks for your suggestions, we have edited this part accordingly, as shown in the updated manuscript, highlighted in blue color. > Q3: I expect the proposed method with a more advanced 3D feature extractor to achieve better performance than Zero123. It is suggested to apply 3D reconstruction (e.g., Neus) on the generated views to compare Efficient-3Dim and Zero123. To verify whether a stronger 3D feature extractor can help improve 3D consistency, we conduct experiments by separately generating multiple novel views using different camera parameters. As shown in Figure 10 in the updated manuscript, we run the inference procedure under 4 different random seeds. The baseline method Zero 1-to-3 fails to produce multi-view consistent textures in the highlighted region, while the proposed method successfully generates consistent results in all groups of seeds. > Q4: Could you elaborate more on how to "conduct several different spatial interpolation processing"? The spatial resolution of the captured feature map from DINO-v2 is $(16 \times 16)$. However, the spatial resolution of the UNet denoiser varies across different stages. To match the resolution, we conduct bilinear interpolation before sending those features to the diffusion model. > Q5: And why do you only inject the features to the encoder of the UNet denoiser? We generally assume the encoder of UNet is where the 3DiM model understands the geometry of the input image and the decoder is for the purpose of reconstructing details. Therefore, adding a geometry-aware feature to the encoder may help strengthen its capability, as has been demonstrated in our experiments. This design philosophy also follows ContolNet [a], which tries to adopt add conditions to image generation. We leave the study of understanding the role of encoder and decoder for our future work, as it is currently beyond the scope of this work. > Q6: I also suggest including more details on how to conduct feature amalgamation (e.g., feature shape and resolution). Thanks for your suggestions. We include another section in the updated manuscript (Appendix D) to better describe the details of feature amalgamation module. References: [a] Adding Conditional Control to Text-to-Image Diffusion Models