Learning rigid-body simulators over implicit shapes for large-scale scenes and vision

Summary

This work presents a GNN-based rigid-body simulator augmented with learned signed distance fields (SDFs). The core idea is to connect surface nodes between objects by testing whether their signed distances are within a certain threshold. This way, the resultant graph networks have sparser edges than prior works, enabling larger-scale simulations of rigid bodies with contacts and collisions. The paper evaluated the proposed GNN simulator on synthetic and real-world scenes.

Strengths

The paper is well written. The main idea is very straightforward to follow and easy to implement. Obviously, using SDFs for rigid-body collision detection is not new in physics animation, but marrying it to a GNN rigid-body simulator looks new to me. For this particular problem setting in this paper (many rigid bodies with frequent contact), using SDF to prune the collision edges is very reasonable. The large-scale scenes are also nice. While classic numerical simulators can handle these scenes without trouble, I don’t know any existing GNN-based simulators that can solve such big scenes and generate visually plausible results. However, I am not an expert in GNN, so I may have missed some state-of-the-art GNN simulation papers.

Weaknesses

I have one particular concern with the overall storytelling in the paper. The paper keeps drawing comparisons with mesh-based GNN simulators in its introduction, method description, and experiments, e.g., choosing MeshGraphNet and DPI as baselines. I think the overall storytelling is a bit biased and occasionally misleading, without stating the pros and cons of mesh-based GNNs fairly. Methods like MeshGraphNet and DPI can represent deformable objects and fluids, which are more complicated physical systems than rigid bodies. The proposed method in this paper enjoys the benefits of SDFs in collision handling because it focuses on rigid bodies only. For collisions between deformable bodies like those in MeshGraphNet and DPI, a precomputed SDF is not possible because shapes deform over time. Therefore, comparing this paper with MeshGraphNet and DPI is neither fair nor necessary: one wouldn’t use particle systems like DPI to simulate rigid bodies in the first place. I think the paper needs to provide a much more balanced statement about these methods. At a minimum, it should inform readers that by introducing SDFs in this new GNN simulation framework, it is no longer obvious to resolve deformable solids and fluids with contact that previous mesh-based GNNs can typically handle.

Questions

I am not sure about one critical technical detail in the method. From what I understand, the collision edges need to be dynamically created and updated during simulation: at each time step, based on the current relative positions of these objects, one checks the vertices from one object in the SDFs of another object. If the distance is smaller than a threshold, collision edges are then created. This process has to be computed on the fly. Could you confirm whether my understanding is correct? Lines 69-71 suggest acceleration structures like BVH often rely on CPU implementations that are difficult to accelerate. BVH trees on GPUs are actually common, well-known, and efficient, e.g., “Fast Parallel Construction of High-Quality Bounding Volume Hierarchies” and “ "Maximizing Parallelism in the Construction of BVHs, Octrees, and k-d Trees." It might be good to tone down the claim here. Constructing SDFs: Did the paper construct a narrow-band SDF or the full SDF on the whole grid as the training data for the network to fit? Also, did the network encourage unit SDF gradient norm during training? Computing closest points: Equation 1 is correct in theory. In practice, due to the numerical representation of SDFs, one needs to apply Eqn 1 multiple times before obtaining a fairly accurate closest point on the zero-level set. SDF-based inter-object edges: I am not sure I get the O(K^2) complexity for mesh-based simulators. Such simulators rarely go over all O(K^2) pairs in a brute-force manner. Instead, they apply broad-phase and narrow-phase collision detections to prune many pairs.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The two large-scale scenes (Spheres-in-Bowl and Shoes) are nice, but both of them have some limitations. For the sphere scene, using a neural SDF is an overkill: collision detections between spheres can be computed in its closed form. For the Shoes, only one SDF is needed. These scenes are visually pleasing, but I don’t think they have touched the “true” large-scale rigid-body simulation with contact for the two reasons above. A “true” large-scale scene would be many rigid bodies with various shapes. Overall, I think I lean positive. Moderately rewriting certain text should address most of my concerns with the paper.

Summary

In this paper, a learning-based simulator based on graph neural networks (GNNs) is proposed that leverages signed distance function (SDF) shape representations of objects for efficient parallel processing. The GNN operates on a set of nodes and edges where the nodes are defines by the object center of masses and contact points on the object surfaces. Edges are formed between object centers and collision points on other objects within a distance threshold. The SDF is used to query distances to the object shapes and determine approximate collision points using the gradients of the signed distance fields in a single step. The approach facilitates scenes with 100s of objects with millions of sample points for collision checking on a V100 GPU. Results indicate that the method scales to larger scenes than mesh-based learning-based simulator baselines. Comparison to classical analytical simulators like NVidia PhysX, MuJoCo or bullet is not provided.

Strengths

- The proposed approach of using SDFs for learning-based simulation is novel and interesting. - The results section provides an interesting set of evaluations of scaling, run-time and accuracy performance. Results indicate improvements in scaling performance over the state-of-the-art learning-based methods. - The paper is well written and easy to follow.

Weaknesses

- l. 182ff, why not differentiate through the euler integration and shape matching steps and take these processing steps into account for the learning approach? Are these steps differentiable? - The paper should also compare with classical analytical state-of-the-art simulators such as NVidia PhysX, MuJoCo, bullet. SDFs can be converted into mesh representations for these simulators. How does SDF-Sim relate to these approaches in performance? - The run-time evaluation is unclear. The plots in Fig. 7 suggest that the simulator requires several seconds per time step. What is the simulated time per time step ? Is the simulator real-time capable? At which number of objects / collision sample nodes does the method loose real-time property? - l. 207, please provide a reference for FIGNet* on first occurence. - It is unclear if the edge pruning in FIGNet* is fairly adjusted to these datasets for the comparison with the proposed method. How does FIGNet* perform with variations in the pruning parameters? Can a similar distance threshold be used as in SDF-Sim ? How does the performance tradeoff change with such parameter variation in Fig. 7 ? Can FIGNet* outperform SDF-Sim overall ? - Sec. 4.4. how do classical analytical simulators such as PhysX, MuJoCo, bullet perform on this "large real-world scene" when transforming the extracted scene SDF to a mesh? - l. 314, DANO [10] uses NeRF representation of object shape, not SDF. - l. 327, the requirement of watertight objects can be a disadvantage too, e.g., if the objects are thin. Please discuss. - The broader impact argues that extracting a mesh for simulation from VolSDF reconstruction would be a specialized skill. This is questionable, as a simple MarchingCubes reconstruction might be sufficient to extract such a mesh.

Questions

Please address questions raised in paper weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper discusses several limitations of the proposed method.

Summary

This work tackles the problem of learning how to simulate rigid-body objects, scaling up to very large scenes that may consist in hundreds of objects and around a million mesh vertices. Graph neural network (GNN) baselines were usually used but become intractable for large scene, with high memory and computational costs. To solve this, this paper presents a new method based on the signed distance function (SDF): first, a learnable SDF is fitted to each object, which can be queried quickly and is memory efficient. Then, the SDFs are leveraged to build a graph between the objects without connecting all vertices, reducing the amount of edges needed. The method is evaluated against baselines on different simulation setups. It is noted that the proposed approach does not perform better than SOTA method based on GNN when the scene is small enough for them to be applied to. However, when scaling the scenes up, these baselines fail due to memory constraints, while the proposed method still runs. Some accompanying videos also show realistic qualitative results.

Strengths

1. Generally, I found the paper easy to follow and the ideas clearly presented. 2. The new **SDF-based inter-object edges** that project a node of object 1 to object 2 using the object 2's SDF is an interesting idea to reduce edges number and get nearly continuous information about where node 1 is approaching object 2. 1. It is also noted that the worst case complexity is reduced from quadratic to linear w.r.t. the number of nodes per object. 3. Results show a reduction in memory and computational cost regarding baselines. The experiment with growing number of objects (section 4.2) displays clearly that the proposed method scales better than the baselines and allow to simulate more complex cases where the GNNs cannot be run. 4. The predicted simulation rollouts are qualitatively realistic without intersection, as seen on the provided videos. 5. The proof-of-concept vision experiment suggests that once trained on clean objects from a dataset, the method could still generalize to "real" meshes coming from 3D reconstruction.

Weaknesses

1. As is noted in the work, the proposed method is still a bit behind the SOTA GNNs in the accuracy of the prediction w.r.t. the ground-truth simulation. Thus, the proposed method is more aimed at larger scenes only where the baselines cannot be run. 2. Learned SDFs for the objects, while fast to query, still need to be fitted first for each object and its accuracy is limited by the network's capacity. 1. In addition, every object's SDF-model need to be pushed on the GPU--or swapped in-and-out. This work seems to use mostly a smaller set of repeatable object, and thus can reuse the same models. This could potentially be solved by weight sharing between models or, as noted in the limitations. 3. The *vision* experiment, simulating in an environment extracted from a multi-view NeRF reconstruction, is a bit underwhelming. While it is indeed just a proof-of-concept, it uses only a single object in a single scene, which is directly extracted as a rigid environment. I am unsure what is the main point of this result as the pipeline seems to be mostly the same as the previous ones. 4. There is not much motivation behind needing *learned* simulators. Maybe expending on the limitations of the simulators themselves could make the work stronger.

Questions

1. See Weakness 4., why is that important to build learned simulators in the context of rigid-object simulations? Are the simulators also memory-constrained and that makes them unusable in the large scenes discussed in the work? 1. A learnable surrogate model that approximates a simulator can also be useful as a *differentiable* simulation, e.g., in order to optimize the scene or some objects. Could the presented approach be use in such a pipeline? 2. A more open-ended question: assuming no intersection, all the SDFs are used in their positive domain. Would the proposed method also work for *open surface* objects, using their UDFs (unsigned distance functions)? In practice, as there might be tiny amount of penetration, the SDF can detect it (negative output) but not the UDF. I wonder if the method is resilient to this or may fail. A minor suggestion: for Fig. 8, adding the standard-deviation or similar metric to the plots can be helpful, as they represent an aggregate of multiple simulations.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are discussed in the paper: the need to train an SDF MLP per object in the scene, that the current accuracy is slightly below SOTA (when those GNN model can be applied), and is currently limited to rigid-objects only.

Summary

This paper presents a neural network-based simulator of rigid body dynamics with contacts that is memory-efficient and thus scalable to scenes with hundreds of objects and up to 1.1 million nodes on a single GPU. The key idea is to use SDF as the geometry representation to simplify collision detection compared with mesh representations.

Strengths

The paper addressed an important and interesting problem, which is an efficient end-to-end simulator of rigid body dynamics. It has significant potential value in various fields, including robotics, AR, etc. Replacing mesh representations with SDFs allows the framework to largely reduce the computational complexity when modeling the collisions between objects. This key idea has been clearly conveyed and experiments validated this claim. Section 2 provided a nice context that educates readers about the basic knowledge required to understand the framework of this work. The methodology presentation in Section 3 is clear, compact, and easy to follow. The experiments laid out the performance of the proposed method from small-scale scenes to significant larger-scale scenes, which is beneficial for readers to appreciate the merits of this work. The authors provided extensive details in the appendix about the implementation of the geometric representation, network modeling, training, and evaluation, which made the whole paper much more informative and improved reader's understanding.

Weaknesses

I found weaknesses mainly in the experiment section, mainly in the limited explanation and interpretation of the experimental results. First, I found the wording "small-scale datasets" and "small datasets" misleading. The conventional interpretation of a "small-scale dataset" refers to the size of the training data, while in this paper, I believe it actually means the simulation scene is of a relatively small scale, i.e., involving a limited number of objects and nodes. It is a core concept in the experiment section, and the potential gap between the interpretations could lead readers to totally different takeaway messages and create inconsistency between the claim and the evidence. According to Appendix B, 1500 trajectories are used for training in both Movi-B and Movi-C datasets, which reveals the actual scale of the training data. On top of this point, there isn't enough discussion about why the proposed method performs slightly worse than the baseline methods on datasets with small scenes and outperforms in larger scenes. What is the difference between smaller scenes and larger scenes for these models? It is a key finding in this paper, but the discussion is largely omitted. Appendix E.6 seems to try to answer this question, but I didn't quite understand this paragraph. What do you mean by " train an SDF-Sim architecture using the accurate distances directly computed from a mesh"? I thought it was exactly what was done in the paper. Why do an experiment with unsigned signed distance? I think this section created more confusion. A minor issue is in Appendix E.4. and Figure S3. The figure shows the baseline in purple (blue to me) and SDF-sim in red (orange to me), while the text says the opposite. Please make them consistent.

Questions

I have a question about the surface nodes. In the paper, the authors didn't explain how the surface nodes are generated. From Appendix E.5, it seems like the surface nodes used in the experiments in the paper are the mesh vertices used for collision simulation. Is it correct? In E.5, using surface nodes as samples on the SDF can achieve similar accuracy with much fewer nodes. Why isn't it the default design in the main paper? Is there a trade-off?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

The authors adequately addressed the limitations. The authors and the reviewer did not foresee the potential negative societal impact of this work.

Thank you for the clarification

The rebuttal is helpful. I thank the authors for bringing Fig. 10 and the attached PDF to my attention. The responses to my other questions also look good. I will increase my rating.