PhyRecon: Physically Plausible Neural Scene Reconstruction

Summary

This paper addresses scene surface reconstruction in an object-compositional format. Unlike previous work that only considered rendering constraints, the proposed PhyRecon framework incorporates physical constraints into the model training. To identify interaction points, the paper introduces a differentiable transformation between the SDF field and explicit surface points. Using these points, the authors integrate physical constraints into previous approaches to ensure reasonable interactions between objects. To further enhance model performance, they introduce a joint uncertainty modeling design for depth, normal, and physical constraints, which increases the stability of the final reconstruction results. The experimental results demonstrate the effectiveness of the framework and its components, proving beneficial for various downstream tasks.

Strengths

1. The overall framework is mind-blowing and insightful. Extending the neural implicit representation, which is mainly supervised by rendering loss, to include physical constraints is non-trivial. This work proposes an innovative solution in this direction with well-reasoned designs, making it more applicable to real-world applications like robotics and virtual reality. 2. The physical constraint is novel and enhances the fidelity of object reconstruction within a compositional design. The proposed framework offers a comprehensive solution with several interesting components, such as SP-MC and physical-guided pixel sampling. 3. The experimental results are extensive, showing significant improvements in object reconstruction quality and stability compared to previous baselines. 4. The physical pixel-guided sampling is an interesting and important trick that enhances the effectiveness of the physical constraint.

Weaknesses

1. The overall solution is based on ObjectSDF++ and RICO, which requires hours of optimization for training. The introduction of additional constraints in PhyRecon further increases the total training time. Improving efficiency could be a focus for future work. 2. The training process involves three stages: initially applying rendering loss for initialization, then incorporating a physical simulator, and finally introducing a physical loss to stabilize object contact. This multi-stage optimization and the complexity of the loss functions make the entire system intricate.

Questions

1. The method relies on ground-truth instance labels. It would be interesting to explore whether the proposed physical solution can handle noise in the ground-truth 2D instance mask, such as erosion and dilation. While uncertainty training might help mitigate this issue, it is crucial to understand how much the physical constraint can aid in overcoming label noise. 2. In the particle simulator, how does the system perform when the points are sparse and non-uniformly distributed? Understanding the simulator's effectiveness in such cases would be beneficial.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The author mentioned their limitation and potential societal impact in the appendix.

Summary

The paper introduces a novel method for physically plausible 3D scene reconstruction. It enforces the reconstructions to be in a static equilibrium state, which is achieved by minimizing the displacement of 3D points over time in a differentiable rigid body simulator. The experiments are conducted on scannet/replica/scannet++, showing quite significant improvement over prior methods.

Strengths

- The paper is easy to follow, and the visual comparisons are adequately made. - There are good technical contributions and the method is sound. - They present a differentiable pipeline that can propagate gradients from after-simulation states to the reconstruction, which is achieved by a differentiable mapping from neural SDF to particles (SP-MC), and explicit Euler integration for simulation (which is already differentiable). - They also develop a re-weighting strategy that down-weights the loss terms with large uncertainty. The choice of uncertainty is different from term to term, measurement uncertainty is used fo depth and normal; physical uncertainty (as defined by the displacement of point over simulation) is used for segmentation as a proxy. The use of uncertainty has two major benefits: (1) the noise in the depth/normal/segmentation priors are filtered out and (2) the pixel sampling can focus on regions with high uncertainty to enhance the reconstruction quality. - The results are strong. The comparisons clearly shows the improvement over prior methods. They also compared to prior works quantitatively, showing better performance in terms of reconstruction quality, normal consistency, and object stability. - The authors ablated the design choices of physical loss, uncertainty and sampling strategy, showing the benefit of adding each component.

Weaknesses

- It only deals with static scenes with rigid objects. The physical loss presented is not applicable to complex contact interactions, dynamic scenes or soft materials. But I don't think this is a major issue as the method works well for the indoor scene reconstruction task being considered. - More explanation on SP-MC is needed. Since the gradient only comes from the refinement step, I would imagine the gradient to be fairly local and noisy. How does it work when the initial shape is far from being physically plausible? - I'd like to suggest moving appendix C to the main text, since the physical simulator plays an important part in the overall story.

Questions

- How expensive is the simulation compared to rendering? e.g., in a forward pass, how much time is consumed by simulation? Does the use of simulation introduce significant compute overhead? Relatedly, I'd like to see a sensitivity analysis of performance vs number of simulation steps. - How is the ground-plane computed for physical simulation? Does the method handle complex terrian (e.g., with a carpet on the floor, or outdoor scenes)? - Since the optimization is multi-staged, it will be good to show result after each stage. - Showing more intermediate results along the axis of optimization would help. For instance, how does the shape change from iteration 0 to final iteration?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

The paper discussed the limitations and social impact.

Summary

This paper proposes to improve implicit surface representations by considering object dynamics (physics) in their resting state. It made a clever observation that the objects should be stable with respect to their supporting planes if the surface representation is correct. This observation leads to utilizing object surface points in the physical simulator to decide if the objects are stable, by penalizing any instability, the solution encourages more plausible surface reconstruction. Physical loss and physical uncertainty were introduced, assisted by the proposed SP-MC (Surface Points Marching Cubes) to improve surface point sampling efficiency for deriving the losses in simulations. Tests on ScanNet and ScanNet++ demonstrated superior results compared to existing works.

Strengths

1. Applying physical stability loss is a rather novel idea. 2. The implementation of using particle-based physical simulation, although complicated, seems to be a plausible approach for understanding and observing any physical instability. 3. The paper is well-written with many implementation details.

Weaknesses

1. Applying this idea would require an object mask, and also knowledge of the ground. This is not a constraint in the existing work MonoSDF. 2. The construction of Physical uncertainty (equation 7), based on the trajectory of the point before it makes contact, is not actually the same as the supporting structure of the object. To be more specific, such a trajectory is often a curve unless the object is falling without any rotation. Such a loss would lead to bulkiness in the supporting structure during reconstruction (as seen in Figure 7e). 3. The author mentioned a couple of times about the supporting plane or ground, where is that information from? 4. In general, this physical loss is only applicable to contact the the supporting plane, and not applicable to other contacts (such as a cup on a table).

Questions

1. What is the ground plane, and how is it calculated/retrieved from the dataset? 2. In the losses the author used (as shown in the appendix), a combination of losses from many existing works is used, what would be the performance if the baseline is a version that combined all of the existing losses? 3. Section 3.3 mentioned the physical uncertainty $U_{phy}(r)$, which the author didn’t give a clear definition.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The author discussed certain limitations in the Appendix. See the above comments for additional concerns.

Summary

This paper proposes a framework for scene reconstruction using neural implicit function with an emphasis on physical plausibility. The key idea is to integrate differentiable physical simulation into the neural scene reconstruction such that the reconstructed shape is stable under gravity. A surface points marching cube method is proposed to convert the neural surface into particles, which is differentiable, enabling the back propagation of gradients from simulation. The authors further propose rendering and physical uncertainty to identify the regions that are not consistent across multiview images and unstable under physical simulation. These uncertainties are beneficial to capture the thin structures of the reconstructed object. Experimental results on various datasets including ScanNet, ScanNet++, and Replica are conducted to demonstrate the effectiveness of the proposed method.

Strengths

- The paper is well-written. The specific techniques are thoroughly discussed with enough details provided. The proposed method is more like a system consisting of multiple components. The authors have clearly put a lot of efforts to elaborate on each component in detail, as can be seen in the appendix. - The ablation study is nicely designed and well-controlled. I personally find Table 2 very informative, which clearly demonstrates the effectiveness of each proposed component.

Weaknesses

- My major concern of the paper is its novelty. Though the authors position their paper as scene reconstruction, the examples shown in the paper are actually object-centric. In this sense, there are actually a series of works that have explored the physical stability of the generated objects represented as neural functions. Specifically, [37] handles the robustness of slender structure in shape generation, and [38] deals with stability of the generated shape. Especially the second work, it has a very similar problem setting to this paper, as they also first converted the SDF into surface points followed by a differentiable simulator. Somehow I feel the method in [38] is neater since the gradient is propagated to the shape latent code rather than a dense voxel grid as proposed in this paper. Neither [37] or [38] is compared in the experiments. This undermines the motivation and technical novelty of the proposed approach, since compared to shape generation, addressing scene reconstruction exclusively does not pose additional challenges for physical plausibility. - With the above being said, I think the technical novelty lies in the uncertainty model. However, what confuses me is whether this modeling on the whole voxel grid is necessary. From the qualitative results in Fig. 7, the rendering uncertainty models are mainly helping to grow the shape regions that are "sparsely" covered by the multi-view images. I guess simple post-processing methods such as region growing would be sufficient for dealing with these cases. The physical uncertainty may also be unnecessary: from Fig. 7 and the failure cases in the appendix, physical uncertainty and physical losses are nothing more than changing the geometric shape so that its center of mass lies on the supporting plane of the shape's points connected to the ground. The proposed approach, especially the part of "physical uncertainty", to me is an overkill to problem. I am also not a big fan of using the term "uncertainty" here since there is really no probability estimation/inference. A better wording could be "heuristics".

Questions

See Weaknesses.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have provided sufficient limitations and failure cases of the proposed method.

Reply to the rebuttal

Thank you for the authors’ rebuttal. Unfortunately, many of my initial concerns remain unaddressed. I would like to emphasize that the review and rebuttal process aims to improve the paper and make it compelling to a broader audience. If the paper is accepted, it should clearly convey the technical aspects of the proposed method: what is novel, what is comparable, and in which scenarios the method may fail. Therefore, I encourage the authors to consider my comments as potential questions that any general reader might ask. While these comments do not necessarily lead to the rejection of the paper, addressing them can significantly enhance the paper’s impact and solidity. Detailed responses to the authors’ rebuttal: **1. Use of MLPs for SDF fields and dense voxel grids** The authors state that their method employs MLPs to model SDF fields and rendering uncertainty, while a dense voxel grid is used solely for modeling physical uncertainty, reflecting accurate 3D physical information from the simulator. However, using MLPs for SDF does not inherently preclude the use of a dense voxel grid, as querying dense voxel positions is necessary for rendering images. My primary concern was whether the “dense voxel grid” is truly necessary, particularly when compared to the shape latent code used in [38]. I would be convinced if the authors provided results or compelling arguments demonstrating the advantages of the dense voxel grid over shape latent code, such as better reconstruction quality, faster convergence, or reduced computational resources. **2. Clarification on rendering uncertainty** The authors assert that my review implied that their method was “growing the shape regions that are sparsely covered by the multi-view images,” which was not true. My original concern was whether a simpler strategy, such as growing sparsely covered regions, might be sufficient to address the issue, rather than employing a complex model for “rendering uncertainty.” **3. Claim of integrating differentiable rendering with differentiable physical simulations** The authors claim that their work is the first to integrate differentiable rendering with differentiable physical simulations for learning implicit surface representation. I find this claim invalid, as there are already published works [1, 2] that achieve similar integration. Additionally, positive comments from other reviewers do not necessarily resolve the concerns I raised. [1] RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation, ICLR 2022. [2] PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification, ICLR 2023. **4. Comparison between heuristic region growing and proposed method** The authors argue that heuristic region growing could lead to degenerated results deviating from the image observations. While this is true, it does not imply that the proposed method strictly adheres to image observations in all cases. A comparison between the two approaches would be necessary to substantiate this point. The remainder of the rebuttal has adequately addressed my concerns.