Response to Reviewer Zvck - Part 1
We thank reviewer Zvck for acknowledging our contributions and here is our response regarding the reviewer’s questions and concerns:
1. The simulation itself would benefit from having a comparison to real-world experiments as well. Questions such as how well the dynamic behavior of soft objects matches reality (bouncing objects), or how well the contact model applies to objects sliding/being pushed under friction.
Thank you for the thoughtful feedback. We have added real-world experiments to further evaluate our system in Section A.6.2. We also added the real2sim system identification demonstrations on our website to show the comparison between sim and real for pressing, sliding, and twisting scenarios. We believe simulating objects is essential for manipulation, several works such as Pac-Nerf [https://arxiv.org/abs/2303.05512], NCLaws [https://arxiv.org/abs/2304.14369] have presented the accuracy of soft object simulation with sim-to-real comparison. However, the focus of our work is the tactile sensor simulation and since our simulation system is modulized, it is straightforward to replace one or multiple modules with improved simulation approaches such as simulating objects with FEM or using constraint-based contact models. We would like to explore the fidelity of the different object simulation methods in future work.
2. How does the simulation compare when simulating articulated bodies? For example the lid-opening task, is the hinge a soft body or a joint? How well does this compare to modeling real-world objects?
We approximate the articulated objects by using the MPM-based approach and assign different materials for different parts. Here the joint is simulated as a soft and thin body, and other parts are simulated as rigid bodies. We further demonstrated that the optimized trajectory for the case opening task can be successfully deployed on real-world settings with a real articulated object in Section A.6.2.
3. Adding some runtime reports, at least in the appendix, would be appreciated from a practitioner's perspective. MPM or PBD was used to simplify object simulation, is this correct? If so, it would be interesting to see how expensive each part of the simulation is, where simplifications are necessary when used in practice. Were any of the learned grasping or manipulation policies applied on the real robot?
We have added the running time report in Section A.5 on all four contact-rich tasks. We provide both the simulation time and gradience backpropagation time, along with each simulation module (FEM, contact, MPM/PBD) running time in Table 11.
And yes, MPM and PBD are used to simplify object simulation, we do observe fast PBD simulation speed but MPM simulation speed is rather slower because of the large amount of particles we used in simulation.
We also added two sets of real-world experiments to further evaluate our system in Section A.6.2. First we deploy the optimized trajectory from simulation on real-world robot setup for surface following and box opening tasks. Then we train a grasp stability prediction model in simulation based on tactile sensing data. We directly apply the trained model to a real-world deformable object grasp task and show that we can adaptively grasp the soft deformable object.
4. For the trajectory optimization tasks in manipulation, it would be interesting to see how many iterations/computational resources each method was given to converge, was it the same for each, or was each method run until convergence?
We have added the training curves of trajectory optimization on contact-rich manipulation tasks in Section A.4.6. To make a fair comparison with other baselines, we run all experiments with 100 iterations of optimization. From the training loss curves, we see most tasks can converge within 100 iterations.
5. Were the constraints applied to the reinforcement learning methods for trajectory planning also applied to the gradient-based optimization?
Yes. Since our system is implemented with semi-explicit solvers which require a small simulation step size, we constrain the action space to [-0.15, 0.15] for RL and set small learning rates for gradient-based methods to ensure the stability of the simulation. And we use the same initial trajectories for both the gradient-based method and RL methods.