GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation

Summary

The paper proposes an improvement of PAC-NeRF for the task of estimating material properties from multiview video using 3D Gaussian Splatting (3DGS). Instead of estimating geometry solely based on the first frame like PAC-NeRF, the proposed method uses 4D Gaussian Splatting (4DGS) with reduced order modeling to construct 4D geometry, enabling the use of 3D supervision. A coarse-to-fine internal filling strategy is introduced to ensure that the simulation operates on a solid volume. 2D mask loss is used for additional supervision.

Strengths

The reduced order modeling of 4DGS is a good fit for the reconstruction task with a limited number of fixed views. Directly applying full-order 4DGS seems hard to optimize due to the high number of DOFs. The experiment results are promising. A real-world application is provided.

Weaknesses

Some symbols are not defined clearly, making it hard to follow at times. For example, $Discretize$ operator in Line 214; $\tilde{P}$ and $F$ are not defined in the text. I need to guess from Alg 1.

Questions

If the 4D geometry is accurate enough, is it adequate to only use 3D supervision? Ablation studies are needed to validate the necessity of 2D mask supervision. Coarse-to-fine density field creation: At the beginning, the reconstruction contour is much larger than the object. How does it shrink to the actual boundary? The $TrilinearInterpolation$ operator will not shrink the contour, and there is no operator to assign zeros. In the simulation, how do the Gaussian kernel scales evolve? The paper seems to assume isotropic kernels, but physical deformation can transform the sphere into an ellipse. How is this addressed?

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are well discussed.

Summary

This paper introduces a novel hybrid method that leverages 3D Gaussian representation and continuum to estimate physical properties of deformable objects. From multi-view video, the Gaussian- informed continuum can be extracted and then combined with material point method (MPM) simulation to train the whole pipeline by using both 3D shape and 2D as supervision. The experiments show that the proposed method outperforms previous approaches based on continuum dynamics or 3D Gaussian representation for dynamic reconstruction, system identification or other real-world applications.

Strengths

1. The paper introduces an efficient motion-factorized dynamic 3D Gaussian network to reconstruct the object states as a linear combination of basis motions, in which the estimated motions and coefficients share the same backbone. 2. The generated Gaussian-informed continuum consists of the density and grid size scale given by the proposed coarse-to-fine filling strategy, which is further used as supervision in training together with the MPM simulation. This tackles the issue of using quantised Gaussian particles for simulation of continuous structures. 3. The experiments show that the method can achieve SoTA performance compared to pre- vious works among various deformable objects, especially when large deformation occurs. The method is moreover applicable to real-word scenarios.

Weaknesses

1. The authors stated that such a lightweight architecture of the motion-factorized dynamic 3D Gaussian network is sufficient for complex motions rather than modeling each basis with an independent network (line 168-171) while lacking an experimental proof. 2. The representation in Section 4.3 is lacking in elaboration and should accompany with more details in the supplementary. Please give some detailed elaboration for Section 4.3 about Gaussian-informed continuum, e.g., notations used in Algorithm 1, dimensions of the variables, etc.

Questions

1. It would be nice to see a comparison between the choice of Motion-factorized dynamic 3D Gaussian network and previous architectures ? 2. It is important that the experimentation on complex motions must be performed in order to truly understand the strength of the proposed method.

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

To some extent, yes.

Summary

The manuscript proposes a novel hybrid framework that leverages 3D Gaussian representations for system identification from visual observations. The framework captures both explicit and implicit shapes using dynamic 3D Gaussian reconstruction and a coarse-to-fine filling strategy to generate density fields. These fields are used to sample continuum particles for simulation and extract object surfaces, which can render object masks during simulations to guide physical property estimation.

Strengths

1. The presentation is clear. The figures look high-quality. 2. I personally appreciate the real-world experiments. I am happy to see the proposed method also works in real life. 3. The proposed method solves one of the most interesting problem in the intersection of guassian splatting and physical simulation, where physical simulation requires volumetric representation but 3dgs outputs surfaces.

Weaknesses

1. In the real-world experiment, I found the authors switched to FEM for deformable body simulation, which conflicts with the MPM simulator used in their pipeline. I think it needs justifications. 2. Some wordings are confusing: e.g., line 166, do you mean effective instead of efficient?

Questions

1. I wonder what makes the difference between the proposed method and PAC-NeRF in the infilled particle generation. 2. If I understood correctly, the motion network only takes time as input. I wonder if it helps by combining the temporal encoding from DiffAqua [1] to further capture the low and high frequencies. 3. The infilling algorithm seems expensive since the complexity grows exponentially. Did you try using Octree [2] or similar algorithm to speed it up? [1] Ma, Pingchuan, et al. "Diffaqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation." ACM Transactions on Graphics (TOG) 40.4 (2021): 1-14. [2] Meagher, Donald JR. Octree encoding: A new technique for the representation, manipulation and display of arbitrary 3-d objects by computer. Electrical and Systems Engineering Department Rensseiaer Polytechnic Institute Image Processing Laboratory, 1980.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper presents an approach for estimating the geometry and physical properties of objects through visual observations using 3D Gaussian representations. The method employs a dynamic 3D Gaussian framework to reconstruct objects as point sets over time and a coarse-to-fine filling strategy to generate density fields. This facilitates the extraction of object continuums and integrates Gaussian attributes, aiding in rendering object masks during simulations for implicit shape guidance.The extracted geometries are then used to guide physical property estimation through differentiable MPM simulation.

Strengths

The experiments in this paper demonstrate improvements over prior works such as PAC-NeRF and Spring-Gaus. The introduction of a novel hybrid framework leveraging 3D Gaussian representations for physical property estimation is straightforward and easy to understand, and experiments have confirmed their effectiveness. Overall, the paper makes meaningful contributions to the problem of geometry + physical property estimation from multi-view videos.

Weaknesses

Some of the technical terms are misused, making the paper confusing to read. For example, "implicit shape representation" is repeatedly used to refer to the rendered object image masks, but "implicit" generally refers to using functions (parametric or neural networks) to represent shapes, where shapes must be retrieved through function evaluations, hence "implicit". Please correct this terminology, as the geometries in this work are represented using GS, which are explicit representations. Additionally, some of the results presentations are confusing and could benefit from clearer explanations and more organized presentation (see below). Clarifying these aspects would greatly enhance the paper's readability and overall impact.

Questions

1. Table 3: Can you include the ground truth values in the table so that readers understand what to expect? 2. System Identification: How do you set the initial parameters for the system identification? Are the optimizations sensitive to initial conditions? 3. Figure 4: Would mask-based supervision fail when the estimated shapes are significantly different from the ground truth? Have you observed any cases where this occurs? 4. Figure 1a: There is a typo in the figure caption. Change "caption" to "capture." 5. References: It would be beneficial to include references to works on differentiable cloth simulation for system identification and inverse problems, such as "Differentiable Cloth Simulation for Inverse Problems" by Liang et al. and "DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact" by Li et al.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of the method regarding known camera parameters, assumption of known material models, and continuum mechanics. However, I do wonder about the failure cases of the method, if there are any. Understanding specific scenarios where the method does not perform well would provide valuable insights and help guide future improvements.