Physically Compatible 3D Object Modeling from a Single Image

Summary

This paper presents an optimization method that produces 3D shapes from a single images that considers about mechanical properties and external forces. The shapes of the generated results in a state of static equilibrium are able to match the input image. The proposed methods can be used with other single-view image shape generation methods.

Strengths

The strengths of the proposed method are: - The proposed optimization framework is novel and versatile. - The method supports to produce fabricatable shapes from a single image.

Weaknesses

The weaknesses of the proposed method are: - The limitation of the proposed method is not discussed in the main paper. - The discussion with alternative approaches is not clear in the main paper.

Questions

- I am curious about why the authors decided to optimize the deformation instead of optimizing the hollow of the solid geometry, similar to the method used in Make-It-Stand [Prevost et al. 2013] and the follow-up papers. - It is unclear to me what is the limitation of the proposed method. - It is unclear to me what is the performance overhead for the proposed method. I would encourage the authors to report the shape complexity and the corresponding computational time. - It is unclear to me whether image loss is used during the optimization? In fig. 2, the pipeline invovles image loss but I did not see the discussion in Sec. 2 but only in evaluation part.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

As mentioned above, I think the limitations should be discussed more in the main paper.

Summary

This paper introduces the concept of physical compatibility into single-image to 3D generation. It proposes a post-optimization method for existing 3D generation pipelines, which takes the generated mesh as the target mesh and optimizes the deformation gradient to obtain a rest-shape geometry without external forces. This rest-shape geometry can align with the target mesh under physical conditions provided by the user. The proposed method significantly improves the results for different single-image to 3D pipelines and is demonstrated to be effective through a comparison of 3D printed results under real-world scenarios.

Strengths

The idea of introducing physical compatibility into 3D generation is very innovative, and it is crucial for the application of current 3D generation results to real-world scenarios. The authors provide detailed derivations and explanations, and propose five reasonable evaluation metrics to assess the degree of physical compatibility. They demonstrate the effectiveness of their method in real-world scenarios through 3D printed objects.

Weaknesses

This is not strickly a weakness of the proposed method. The paper mainly focuses on the optimization of a single object. However, current 3D generation pipelines can handle the generation of multiple coupled objects, such as a castle. I wonder whether this method can be applied to such senarios.

Questions

The proposed method optimizes the positions of mesh vertices while maintaining the connectivity of the vertices. However, existing 3D generation models may introduce incorrect connectivity for complex or partially occluded input image. I wonder if physical compatibility approach can correct such errors in connectivity during the optimization process.

Rating

6

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes.

Summary

This paper introduces a physical compatibility optimization framework for reconstructed objects from a single image. The approach considers mechanical properties, external forces, and rest-shape geometry, integrating static equilibrium as a hard constraint. This framework improves upon existing methods by ensuring the stability and accuracy of reconstructed objects under external influences. Quantitative and qualitative evaluations show enhancements in physical compatibility.

Strengths

1. Performance: The proposed method achieves state-of-the-art results. The experiments well validate the effectiveness of the proposed methods. 2. Clarity: The paper is well-written and easy to follow. 3. Technical Novelty: The main contributions of this paper are twofold: 1) They propose a physical compatibility optimization framework for 3D objects and decompose the mechanical properties, external forces, and rest-shape geometry. 2) They optimize the rest-shape geometry using predefined mechanical properties and external forces and ensure the object’s shape aligns with the target image when in static equilibrium.

Weaknesses

In this paper, the authors primarily apply the physical compatibility optimization framework to enhance the physical attributes of 3D models obtained from existing methods rather than reconstructing them from a single image. Therefore, I think the title "Physically Compatible 3D Object Modeling from a Single Image" may not be suitable, as the focus lies on enhancing physically compatible modeling of 3D objects derived from off-the-shelf methods.

Questions

1. All $\mathbf{x}\_{static}$ may be $\mathbf{X}_{static}$ as the defination in line 89. 2. In Some cases discussed in the paper, like the flamingo standing on one leg, should it be standable even after optimization? I think it should not be standable under gravity. 3. Why do you evaluate the different off-the-shelf methods using connected components when this cannot demonstrate the superiority of your proposed method?

Rating

5

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors have discussed the limitations and potential negative societal impact.

Summary

The paper presents a 3D mesh optimization framework to ensure physical plausible 3D object reconstructions from a single image. These 3D reconstructions should conform with global (e.g. gravity) and user-defined constraints (material stiffness) as well as match with a target image of the reconstructed, simulated object. The reconstructed mesh is considered as a solid object. The method is combined with 5 existing single-image 3D object reconstruction methods, which are evaluated on a collection of 100 Objaverse samples. Several additional metrics are introduced to test for the physical 'compatibility' of the reconstructions: mean stress, connected components, standability and the difference to the reference view.

Strengths

The motivation for physically plausible reconstruction is clearly stated. Ensuring physical constraints of reconstructed 3D meshes allows for further downstream applications, such as 3D printing, hence it is an important aspect to consider. The writing and quality of the figures (including the video) is of very high quality. The set of metrics is well chosen and the paper includes very detailed descriptions of the method. Its effectiveness is demonstrated using real-world 3D printed results with and without the proposed optimization.

Weaknesses

- While I can understand the space constraints of the submission format, the main paper should still include a complete and comprehensive overview of related work instead of deferring it to the appendix.

Questions

- Figure 3: The plot for TetSphere show minimal improvements using the proposed optimization. Could the authors elaborate on this?

Rating

8

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The checklist is complete and the paper discusses its limitations.