DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering

Summary

This paper presents a new method to learn 3D particle dynamics from sparse 2D observations using inverse rendering. In constrast to previous work, it does not learn a fully unconstrained model but makes use of known physical priors. The method learns graph network kernels to model the particle interation forces in the DEA framework, which are classically designed domain experts. These graph networks are trained from 2D observations using a differentiable renderer without any 3D supervision. The method is evaluated on different scenes and using different physical materials. It surpasses the provided baseline comparisons in the test shown in the paper.

Strengths

The paper presents a novel combination of graph networks and classical particle-based simulation that, in conjunction with a differentiable renderer, allows it to recover 3D particle dynamics from images. The presented method is technically sound, well explained, and easy to follow. The evaluation is sensible and very extensive, encompassing different material configurations, and shows a siginficant improvement over the SoTA. The source code and dataset will be made available to the public.

Weaknesses

There are several things that are not clear to me, especially related to scene initialization and the initial velocities. These points are detailed in the questions below. I don't think these are fundamental problems, but I nonetheless hope the authors can clarify these aspects in the rebuttal.

Questions

- 3.3: the $v_{ij}^{n}$ or $v_{ij}^{t}$ used in equations 14 to 16 do not match any of the definitions in appendix B. - For training (3.4), how is the scene initialization done with the renderer? And how accurate is the estimated initial velocity based on this initialization? - Closely related: how sensitive is your method to inaccuracies in the initial velocity? - How long are the sequences used in your experiments? - How long is your training (number of iterations, epochs, or variable updates)? - Are the views shown in the figures the ones that were used for training or are they novel views? There are several grammatical errors throughout the paper, for example: - 291 "metrix" - 291 "cham**b**er distance" should be "cham**f**er distance" - 306 There seems to be a new paragraph missing before "Results in particle view" - 311 "we claim the reason that..." - 343 "while keep the priors regulated..." I would encourage the authors to do a complete pass over the paper to fix any writing mistakes. The tables and figure on page 9 are also very squeezed.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are briefly discussed.

Summary

This paper proposes to incorporate the neural network with Discrete Element Analysis framework for particle-based simulation. The method adopts GPF as differentiable renderer and is trained through multi-view videos. The proposed model delivers faithful rollouts and outperforms the baselines.

Strengths

* The proposed method deeply integrates the DEA and obtains robust performance. * The model can be trained through multi-view videos in an end-to-end manner.

Weaknesses

1. To simulate different scenes, such as deformable objects and fluid, it seems that one has to train different models on different cases, indicating scene-specific design with limited generalisation abilities. 2. The results of verifying the generalisation abilities are missing. For example, after trained on the deformable bunny as shown in Figure 5, can the model simulate objects of other shapes, such as a car, or objects composed of more particles? 3. The method seems to be capable of dealing with objects with limited particles and struggle to simulate large amount of particles.

Questions

1. To enable the training on different scenes through the images, are different pertained GPF needed for different scenes? 2. How is the performance of the method for long-term predictions? For example, a trajectory with 150 frames. 3. At L253, the author claims to use L2 loss. However, for equation 17, it seems the loss is L1. Is this a typo?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Please refer to the weaknesses and limitations.

Summary

The paper proposes a framework to learn 3D dynamics from 2D observations. DEM uses hand-designed kernel functions to model interaction force between particles, which may vary significantly for different material types. The paper changes these kernels to learnable GNN kernels. The time integration still follows DEM. Combined with a differentiable particle renderer, these kernels can be trained to match 2D observations.

Strengths

The framework is quite general, with force-based time integration providing a strong physics prior. The use of learnable kernels to model point interactions enables the DEM to effectively fit observations. Numerous comparisons are conducted, and the results are promising.

Weaknesses

The evaluations are conducted only on synthetic data. The learned particle interactions are black-boxes, which are not explainable.

Questions

[Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics](https://arxiv.org/abs/2304.14369) should be cited, since it solves a similar task. Are the material types in particle attribute A pre-known? Are the initial particle distributions known? The known assumptions should be listed in the paper. How are particle colors assigned? In reality, objects usually have textures. Manually assigning colors may be impractical. The particle initialization step is illustrated in Fig. 1, but no technical details are discussed in the text. It seems that the initial geometry is provided in the experiments. In that case, the known-geometry assumption should be illustrated in the figure instead. However, geometry reconstruction is important for real data applications.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed.

Summary

The paper considers the problem of physical modeling the dynamics of 3D objects in space using only 2D observations. The authors propose to solve this problem by viewing objects as sets of interacting points and using differentiable rendering of these point clouds. In this pipeline, neural models are used to directly predict the dynamics of 3D particles. Unlike existing methods, the authors propose to use a physically principled framework (Discrete Element Analysis) to develop a partially interpretable neural model that can be used to predict the target dynamics. This framework decomposes the forces applied to the points into gravity, potential, and viscous interaction forces (and further into normal and tangent components). The proposed model thus is constrained to predict certain components from these force decompositions resulting in a more principled, interpretable, and effective dynamics prediction. The authors propose new more challenging synthetic datasets containing variable materials and objects, evaluate on this data, and compare the results to several existing baselines. The proposed method shows improvements over the baselines in all the scenarios. The authors also provide an ablation study showing the importance of different components.

Strengths

* The proposed model seems like an effective combination of a physically principled framework and neural model, which potentially can be further explored and improved. * The proposed method significantly outperforms all baselines. * The authors propose a new dataset.

Weaknesses

* The written text requires some additional polishing (citations should not be treated as nouns, sometimes the layout of figures and text is too tight, and there are some grammatical errors and not correctly formulated sentences).

Questions

How are CD and EMD computed? Are any of the intermediate states used to compute the metrics or are they only computed for the final state? If the proposed model is interpretable does it make sense to compare the predicted particle forces in all the intermediate steps with the ground truth forces from the simulations? Some examples show that objects can be «torn» over the course of the considered scenario. Currently, the resulting distinct parts of the torn object will still be considered as one single object by the model, is not it? Does it cause problems? Have you considered any solutions for it?

Rating

6

Confidence

3

Soundness

4

Presentation

2

Contribution

4

Limitations

The limitations are properly addressed in the text.

Re: Rebuttal by Authors

Thank you for the clarifications. Trusting that these unclear parts will be clarified in a final version I'd be happy to still support an "accept" for this paper.