Responses to Reviewer veoH
Dear Reviewer veoH,
Thank you for your response and the helpful comments. We respond to each of your additional questions below. Please do not hesitate to let us know if you have any additional questions.
**Discussion-Q1: Ablation studies on the choice of foundation models.**
We fully recognize the importance of evaluating each sub-task by substituting the foundational models with similar alternatives. Actually, we have conducted the ablation study to explore the effectiveness of our chosen solutions in each sub-task by comparing them with the alternatives. The results and analisis are presented in Sec.C.1 and Table 4 of the Appendix. Specifically, we conducted ablations to replace Shap·E with One-2-3-45, replace Open-CLIP with EVA-CLIP and replace Omnidata with MiDaS, observing performance degradation with the alternative models. We further visually compare Shap·E with One-2-3-45 for shape generation in Fig. 8, where the results demonstrate that Shap·E is a more robust solution for generating 3D models from 2D instances. These ablation studies validate the effectiveness of each choice made within our framework.
**Discussion-Q2: The claim of zero-shot scene reconstruction.**
Actually, the term "zero-shot" signifies that no specific data or data-driven training is required for the task of scene reconstruction from single images. Unlike previous works in this field, such as PanoRecon and Total3D, which require specific image-scene pair data, our approach does not rely on such data. The training data used in Shap·E is only capable of shape reconstruction learning, and it is impossible to train a model for single-view scene reconstruction solely by relying on the Shap·E data.
We justify that for the "zero-shot" tasks, using large-scale data from a different domain or task is not prohibited. For example, CLIP models are widely used for zero-shot image classification without requiring specific image-class pair data, though they do require massive amounts of image-text pair data for contrastive learning. Similarly, the data used in Shap·E does not undermine our claim of "zero-shot" single-view scene reconstruction.
We will provide further clarification on the tasks and claims to enhance the understanding of DeepPriorAssembly's capabilities and the data used in each foundational model integrated into our framework.
**Discussion-Q3: Performance on outdoor scenes.**
We would like to justify that none of the previous works can successfully reconstruct outdoor scenes from single images. All prior approaches are trained on indoor scenes and struggle to generalize to real-world outdoor images, which contain out-of-distribution objects such as trees, buildings, and animals. Leveraging powerful large foundational models, DeepPriorAssembly is the first to demonstrate the capability for zero-shot scene reconstruction in these complex real-world outdoor scenes. As illustrated in Fig.A of the rebuttal PDF, DeepPriorAssembly accurately reconstructs scene geometries even in challenging outdoor scenes. However, the texture produced by Shap·E may not be optimal for the outdoor shapes. In the future, we may consider replacing Shap·E with a shape reconstruction method that performs better on outdoor shapes to further enhance the performance of outdoor scene reconstruction.
**Discussion-Q4: Optimize the connection and compatibility between different priors.**
Thanks for your suggestions. In the revision, we will improve the connection and compatibility among the foundation models by conducting more ablation studies on the choice of each foundation model within our framework, supplementing the analysis presented in Sec.C.1 and Table 4.
We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Please let us know if there is anything we can clarify further.
Best regards,
Authors