DiffuseBot: Breeding Soft Robots With Physics-Augmented Generative Diffusion Models

Summary

This work proposed a new framework that augments the diffusion-based synthesis with physical dynamical simulation in order to generatively co-design task-driven soft robots in morphology and control. The extensive experiments in simulation to verify the effectiveness of DiffuseBot.

Strengths

1. The paper is well-written and easy to follow. 2. The visualization is good and help me understand how the method works. 3. The application of diffusion is always be encouraged and diffusion model is an interesting and promising backbone.

Weaknesses

1. The novelty is limited and more like an incremental work that applied diffusion model into 3D soft body generation task. The relationship and difference with Softzoo mentioned in the paper should be more clearly illustrated. 2. The baseline for comparison is relatively weak, consisting of some outdated works from a few years ago. The paper should includes more strong and recent baselines. 3. The evaluation way is not agreed. 'To avoid being trapped in the local optimum, we run each baseline with 20 different random initializations and choose the best one. Since DiffuseBot is a generative method, we draw 20 samples and report the best' . If the baseline is sensitive to the initialization seed, more detailed results should be reported rather not choose the best one. And the 20 runs should report the mean and var rather not the best. The explanation about the generative model is not convincing enough for me. 4. The novelty is limited and more like an incremental work applying diffusion model in 3D generation domain. 5. The physics augmented component is not included in ablation studies and this will harm the convincing conclusion in the paper. 6. Why diffusion not other generative model, such as VAE or transformer? The ablation studies should be listed or the paper will be likely the incremental work.

Questions

Please give more detailed clarification and experiments results to address my concerns.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

1. The novelty is limited and the motivation of using diffusion model is not well illustrated. 2. The baseline is weak and old, the evaluation way is not agreed. The ablation studies are not enough. 3. The role and the improvement bring by physic-augmented component is not well studied. 4. The limitation part is absent in the submitted version. I hope the authors can discuss the limitations.

Summary

This paper presents DiffuseBot, a framework that uses physics-augmented diffusion models to generate soft robot designs and control strategies for various tasks. The authors propose to optimize the embeddings conditioned by the diffusion model to improve the physical utility of the generated robots, and to reformulate the diffusion sampling process as a co-design optimization that leverages differentiable simulation. The authors demonstrate the effectiveness of their method on several tasks, such as balancing, landing, crawling, hurdling, gripping, and moving objects. They also show how to incorporate human feedback and fabricate a physical robot prototype.

Strengths

The paper is well-written and clear. The proposed method is novel and interesting, combining diffusion models for shape generation, physics-based simulation and co-design optimization. The paper provides extensive experimental comparisons with baselines and ablation studies on both latent optimization and co-design to validate the proposed method. The paper also shows some fun qualitative results of diverse robot designs that function under passive dynamics, locomotion tasks and manipulation tasks.

Weaknesses

- The paper does not discuss the limitations or failure scenarios of the proposed method. In addition, discussions on design choices can be help: how hyper-parameters are chosen, such as the guidance scale, or the number of MCMC steps? - Figure 1 shows that the motivation of this work is to deploy the optimized soft robot into the real world. However, though I may miss it, I did not find discussions in the paper about the feasibility of manufacturing the resulting soft robots, such as the soft gripper. Given that the actuators are currently assumed to be muscle fibers, it can be hard for manufacturing.

Questions

- Can limitations of the current pipeline be discussed and included in the paper? - Can the paper includes discussions about manufacturing? - How design choices are set? How sensitive is the current pipeline to hyper-parameters? - How to interpret the metric reported in Table 1, 2? "We report the average performance with standard deviation in the superscript." -- Is the performance success rate?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Please include a section about limitations.

Summary

This paper introduces DiffuseBot, a physics-augmented diffusion model designed for generating and optimizing the morphologies and control mechanisms of soft robots. DiffuseBot aims to bridge the gap between virtually generated content and physical utility in the domain of soft robotics. Firstly, it combines the diffusion process with a physical simulation that serves as a performance certificate, thereby ensuring the feasibility and effectiveness of the generated designs. Secondly, it details a co-design procedure that simultaneously optimizes the physical design and control of the soft robots, leveraging insights from differentiable simulation. The paper validates the efficacy of this approach by presenting a variety of both simulated and physically fabricated robots, along with their diverse capabilities.

Strengths

1. In general, the paper is well written, with only minor flaws. Even those unfamiliar with soft robot design will find the paper easy to comprehend. 2. Although diffusion models are expressive and powerful, their performance for tasks dealing with physical tasks often falls short. Thus, injecting a physics prior or 'physics-augmented diffusion model' is crucial. I think the method proposed in this paper is interesting and promising. 3. The evaluation is comprehensive and thoughtful. The physical robot is impressive.

Weaknesses

Overall, I did not identify any major weaknesses in the paper, but here are a few points that could strengthen it: 1. While the writing is generally clear, certain sections could benefit from clearer exposition, such as: * The section on diffusion as co-design is not very intuitive, especially for audiences not familiar with soft robot design. Specifically, it should be clearer how gradient-based optimization benefits robot design and what exactly line 152's "synergy" means. * It would be helpful if the authors clarify that the "condition" in this work actually refers to text. 2. The robot's actuator and stiffness seem oversimplified, having only constant stiffness. Given that the gradient of $\Psi_{act}$ is almost zero, it appears that the actuator and stiffness are solely determined by the geometry. 3. A similar idea of tuning in the embedding space is proposed in[1]. A discussion and connection to this existing work could be interesting. 4. In general, the method the paper uses to inject a physics prior into the generation process could be applicable to more general scenarios. Works like Diffuser[2] or Decision Diffuser[3] generate state sequences with diffusion models, but the generated states can sometimes be physically implausible. A deeper discussion about the potential of the method could make the paper stronger. [1] Gal, Rinon, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik and Daniel Cohen-Or. “An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.”, ICLR, 2023. [2] Janner, Michael, Yilun Du, Joshua B. Tenenbaum and Sergey Levine. “Planning with Diffusion for Flexible Behavior Synthesis.”, ICML, 2022. [3] Ajay, Anurag, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, T. Jaakkola and Pulkit Agrawal. “Is Conditional Generative Modeling all you need for Decision-Making?” ICLR, 2023.

Questions

1. I do not fully understand how the k-means clustering is performed for actuator and stiffness generation. Specifically, what kind of feature is used for clustering? 2. In line 86, which structural biases are you referring to ?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

see weakness.

Summary

The paper introduces DiffuseBot, a system that aims to simplify and automate the design of soft robots in simulation and real-world systems. DiffuseBot uses diffusion-based algorithms to co-design soft robot morphology and control for specific tasks, combining the diversity of evolutionary algorithms with the efficiency of gradient-based optimization. The system is made possible by advancements in AI-driven content generation. However, existing generative algorithms face challenges when applied to physical soft robot co-design, such as the lack of consideration for physics and task performance. To overcome these, DiffuseBot uses physical simulation to guide the generative process of pretrained large-scale 3D diffusion models. It also develops an automatic procedure to convert raw 3D geometry into a format compatible with soft body simulation. The system optimizes the embeddings that condition the diffusion model, skewing the sampling distribution toward better-performing robots as evaluated by a simulator. It also reformulates the sampling process to incorporate co-optimization over structure and control. DiffuseBot has been tested on a wide range of tasks, demonstrating its superiority to comparable approaches. It also allows for human input in the robot generation process and has been used to create a proof-of-concept 3D-printed real-world robot. The paper contributes a new framework that augments the diffusion-based synthesis with differentiable physics simulation, methods for driving robot generation in a task-driven way toward improved physical utility, and extensive experiments in simulation to verify the effectiveness of DiffuseBot.

Strengths

This paper is robust and comprehensive in its approach. It introduces an innovative method that applies diffusion models to the co-design of robots, representing a significant contribution to the field. The authors have ensured thorough experimental coverage by testing their system, DiffuseBot, on a diverse range of tasks. This extensive testing underscores the versatility and applicability of the proposed method. Furthermore, the paper is not limited to theoretical constructs but extends to practical, real-world applications. The authors demonstrate this by providing a proof-of-concept 3D-printed real-world robot, thereby solidifying the relevance and potential of their research in real-world scenarios.

Weaknesses

The paper does not exhibit any significant shortcomings or areas of concern.

Questions

In your paper and supplementary materials, you've provided a qualitative discussion on the challenges of translating simulated results into the fabrication of real robots based on the design developed in simulation. Could you delve deeper into this issue by providing more detailed, quantitative results that highlight the discrepancies between the behavior of the simulated robot and its real-world counterpart? Additionally, could you propose potential solutions aimed at minimizing this gap between simulation and reality?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The paper effectively addresses all identified limitations.

Summary

This paper proposes a physics-augmented diffusion model that generates soft robot morphologies capable of excelling in a wide spectrum of tasks called DiffuseBot. DiffuseBot bridges the gap between virtually generated content and physical utility by (i) augmenting the diffusion process with a physical dynamical simulation which provides a certificate of performance, and ii) introducing a co-design procedure that jointly optimizes physical design and control by leveraging information about physical sensitivities from differentiable simulation. In the experiment, they showed a range of simulated and fabricated robots along with their capabilities.

Strengths

1. The paper introduces a new framework that augments diffusion-based synthesis with physical dynamical simulation in order to co-design task-driven soft robots in morphology and control. 2. The method leverages optimizing input embeddings and incorporating differentiable physics into the diffusion process for driving robot generation in a task-driven way toward improved physical utility. 3. They performed experiments in simulation to verify the effectiveness of DiffuseBot, extensions to text-conditioned functional robot design, and a proof-of-concept physical robot as a real-world result.

Weaknesses

The presentations in this paper were sometimes unclear, as asked in the following placeholder. The paper admitted that there are countless non-trivial challenges in the physical robot fabrication and real-world transfer, including stiffness and actuator design, and the sim-to-real gap. However, in my opinion, other contributions of this paper may outperform the weaknesses.

Questions

1. In Section 2 before 2.1, there was no reference to 2.2. Is it fine? 2. I did not find the explicit definition of bold c in Eq. (5). 3. L159: what does the slash mean? 4. In the experiments, I want to know why such tasks were selected (i..e, motivation for the tasks). 5. I cannot find the explanation about the performance metric such as Tables 1 and 2 (sometimes having minus values). This information is important and should be mentioned in the main text. 6. The (short) introduction of the baseline models and the reasons can be mentioned. 7. The figure at the bottom of page 7 was Table 4, but may be incorrect. And Figure 4 in L233 may be incorrect. 8. Section 3.4: The paper discusses the use of textual inputs and is very interesting. Can the authors discuss the potential for using other input formats?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

In the conclusion section, there seems to be less limited information about this work from the experimental results. In other parts, general limitations were mentioned.

Thank you and we are looking forward to your post-rebuttal feedback!

Dear AC and all reviewers: Thanks again for all the insightful comments and advice, which helped us improve the paper's quality and clarity. The discussion phase has been on for several days and we have not heard any post-rebuttal responses yet. We would love to convince you of the merits of the paper. Please do not hesitate to let us know if there are any additional experiments or clarification that we can offer to make the paper better. We appreciate your comments and advice. Best, Author

Thanks for the rebuttal.

Thanks for the authors. Novelty is currently acceptable to me. I am very grateful for the efforts made by the author, especially for incorporating the VAE-based generation method (although I was anticipating seeing the advantages of diffusion over other generative models by incorporating physics constraints or knowledge as conditions in the backbone of conditioned VAE). However, considering the limited rebuttal time window, I will not be demanding more. Furthermore, I still believe that providing mean and variance through multiple evaluations is better than reporting only the best results. For instance, in the supplementary Table A3 provided by the author, for the locomotion task and the task of moving a box, it can be observed that the variance is particularly high. There is already significant overlap with the VAE algorithm that does not incorporate physics constraints. The author should provide the mean and variance for the baseline in Table 2 of the main submission to enhance the persuasiveness of the results and conclusions. If my concerns are addressed, I will immediately increase the score.

Please read and respond to authors' rebuttals

Dear reviewers, Thank you for your reviews. The authors have posted their rebuttal. If you have not yet done so, please read the rebuttal and the other reviews, and comment on whether the rebuttal has addressed your comments or concerns.

nice work!

Thanks a lot for the rebuttal and the nice work! Authors cleared most of my concerns. Would like to keep my rating of 7 Accept.