Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication

Summary

This paper proposes to combine physical standability with existing diffusion-based text-to-3D generators in order to synthesize objects that not only follow the text description but also can stand on their own without falling. To this end, the paper proposes a physical loss function and uses it with score distillation sampling methods that generate 3D objects from texts. The paper shows that their generated model show better standability both in simulation and after physical realization.

Strengths

- Good exposition and writing, paper is rather easy to follow. - The validation of results especially the 3D printing of generated models. - The standability and stable equilibrium losses are elegant.

Weaknesses

First, there are good methods, such as 'make it stand' that could be used as post-processing tools for making text-based generated models stand. Why should we go through the paper's approach in the light of these methods? Some important ablations are missing. For example, - Separating of how much each loss is contributing to standability. For example, what if we use Magic 3D outputs and smoothen the geometry? - What are the pros and cons of using the physical loss as a post-processing or jointly with SDS? The claims are more broad than the actual contributions. - Take "fabrication-ready" in line 62. The paper is concerned with stabdability and it should adapt the text (and even the title). In 3D printing, "self supporting" means that there is no need to support materials. The paper doesn't cover this. - The relationship to robotics is not well-founded except an example that uses robot grippers to do the standability tests, which is not really related to benefits to robotics. The paper is heavily relied on score distillation sampling methods but does not explain them very well. I understand that the paper refers to the original papers in this area but it is worth including a paragraph about SDS. I look forward to see a justification in the rebuttal.

Questions

- The paper mentions too many steps of integration and back propagation in Section 4.1.1. Does it still do this? If not, where exactly the differentiable physics simulation is used? - What is the default intimal physical state? - How does the paper figures out the upright position? - The paper mentions mesh topology (Line 246) but doesn't show it. A zoom in mesh is necessary for this. - What are the mesh sizes and dimensionalities of printed objects? - Are Figure 7 results printed? - x_{com} is not defined in Eq. 7.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

A main limitation is the mesh size. The paper has mentioned it. This is important in the light of 3D printing that require sometimes huge resolutions.

Summary

This paper introduces a method to make 3D assets produced by SDS-based generative models stand on their own. On top of the SDS loss, it proposes a “standability" loss that encourages the rigid-body simulation result to be rotation-free, and a “stable” loss to encourage the generated shape to be a “local minimum” of height for the center of mass. The evaluation is conducted both in simulation and 3D-printed shapes in the real world.

Strengths

- This paper addresses an interesting problem of making 3D-generated shapes stand. Although this problem has been studied from the perspective of 3D printing, this paper introduces new methods tailored to the new setting of SDS-based 3D generation. - The proposed method seems to be effective from the various presented results, with a significant improvement in the success rate of standing.

Weaknesses

- The proposed method involves solving an optimization that involves multiple loss terms. No ablation study is provided to prove the necessity of each term. In particular, are both standability loss and stable loss required? How are their influence on the optimization results respectively? Are both normal smoothing Eq (8) and Laplacian smoothing Eq (9) required and why is Laplacian smoothing only applied to the bottom of the shape? - The traditional balancing method in computational fabrication [59] is only discussed but not actually compared in experiments. - What are the loss weights for each term in Eq (10)? These numbers may be tricky to set, and thus make the paper hard to reproduce. - How is \(T\) set in Eq (5)? How do different choices of \(T\) influence the results? - The success rate of the baseline method is not reported in Fig. 6 for the audience to understand the improvement.

Questions

See weaknesses. These are important questions to answer for a well-rounded paper.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Seems adequate.

Summary

The paper introduces a differentiable simulation-based loss to refine the existing SDS(Score Distillation Sampling)-based text-to-3D frameworks. Concretely, it relies on a differentiable simulator Warp to provide gradients for keeping the rotation of the generated mesh unchanged after a period of time. Besides, the authors also introduce a stable equilibrium loss, which favors a flat contact surface. Experiments show that the proposed method can generate self-supporting 3D models given text prompts, verified by simulation and real-world results with 3D-printed objects.

Strengths

The paper introduces an effective and flexible method to refine the existing SDS(Score Distillation Sampling)-based text-to-3D frameworks, like Magic3D and MVDream. The authors also provide experiments on real-world validation.

Weaknesses

1. The paper lacks enough baselines to show that the task the paper tackles is non-trivial. For example, the claim that "Directly integrating these methods with 3D generative AI as a postprocessing module is suboptimal" (L55-L56) is not verified by the experiments. If a self-supporting mesh is needed, the simplest baseline can be cutting the mesh by a flat plane moderately higher than the lowest vertex, as a postprocessing step. 2. The physical constraints used in this work are relatively limited. The standability loss only keeps the rotation unchanged. It can not recover some common cases like "a standing horse" (e.g., Napoleon Crossing the Alps), although those cases may be beyond "self-supporting" defined in this paper. It seems that the simulator has not been fully leveraged, since a simple post-processing baseline (e.g., flatting surfaces that contact with the ground) may also achieve good standability and stability. 3. The current method is only applied to SDS-based methods only (already pointed out by the authors in the limitations)

Questions

1. Can the authors provide more baselines, especially any existing post-processing method with the help of differentiable simulators, to enhance their technical contributions? One example is given in the *Weaknesses* section. 2. Can the authors give a clear definition of "self-supporting"? It is a little ambiguous, since one can replace the initial pose with the pose after simulation so that Eq. (3) about standability can be satisfied, as long as the limit exists.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper has addressed its limitations.

Summary

This paper addresses the problem of generating 3D models from text that are visually appealing but often physically unstable in simulations or when 3D printed. The authors incorporates a differentiable simulation-based loss function and physically inspired regularization to ensure generated models are stable under perturbation. The proposed method involves a two-stage training process where a coarse model is first generated from text prompts and then refined with physical constraints to ensure stability. Experiments show that models created with Atlas3D maintain stability better than those from existing methods. The optimized models are also 3D fabricated to verify the stability in real world.

Strengths

The paper proposes an original and important solution to the issue of physical stability in text-to-3D generation, combining differentiable simulation-based loss functions with physically inspired regularization. This approach significantly enhances existing methods, reducing the need for manual post-processing and making the models immediately practical for various applications. The method is sound, with clear and concise writing that guides the reader effectively and should be straightforward to implement for someone with a background in physical simulation. The experiment quality is high, validated through extensive experiments in both simulations and real-world scenarios, demonstrating the method's robustness and versatility. The results confirm the method's effectiveness in ensuring the stability of 3D models.

Weaknesses

The paper is strong overall, and I did not identify any significant weaknesses. However, to further support future research in this area, I would suggest releasing the code. Providing the implementation would greatly benefit the community, enabling others to replicate the results and build upon this work.

Questions

I don't have any specific questions at this time. The paper is clear and well-executed, addressing an important issue effectively. Implementation is also described in detail.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

I am curious about the failure modes of Atlas3D. Could you provide more details on the scenarios or conditions under which the method might not perform as expected? Understanding these limitations would be helpful for future research and applications. Additionally, addressing these potential failure modes could further strengthen the robustness of your approach.