DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation

Summary

The paper presents a stable hand pose estimation method leveraging a neural network for stability estimation trained from simulation to improve the physical stability of estimated hand poses. The main idea is training a neural network via simulated results. The network can further provide smooth gradients to refine estimated hand poses. Experiments on two datasets demonstrate the effectiveness of the method and the ability of DeepSim network to provide better gradients where the overall learning framework can benefit from.

Strengths

- The idea of training a neural network as a simulator that can both provide accurate simulation results regrading stability and provide smooth gradients friendly for network training is sound. Gradients provided by the network are also demonstrated to be of higher quality compared to analytical gradients or those from finite differences. - Experiments are valid and reasonable that are able to demonstrate the superiority of the proposed method. Implementation details are also provided.

Weaknesses

- The idea of designing neural networks as a differentiable simulator is not a new thing [1,2]. The network designed in the paper does not has explicit physics priors and only approximates the stability value, which make its generalization ability towards unseen and out-of-distribution data ambiguous. - Using networks to learn the stability prediction process is not fully explored. For instance, could the current strategy generalize well towards out-of-distribution test data? Is it possible to improve the generalization ability and train a very powerful stability prediction network by creating and leveraging a large scale synthetic dataset via simulator? Is there any opportunity to fuse physical priors into the design of DeepSim for generalization enhancement? [1] Mezghanni, M., Bodrito, T., Boulkenafed, M., & Ovsjanikov, M. (2022). Physical simulation layer for accurate 3d modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 13514-13523). [2] Mezghanni, M., Boulkenafed, M., Lieutier, A., & Ovsjanikov, M. (2021). Physically-aware generative network for 3d shape modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 9330-9341).

Questions

- Is there any in-depth analysis of the DeepSim network (please see weakness for details)? - Some works in robotics propose to relax and improve contact models so that smooth gradients can be provided for optimization [1,2]. It is hard to compare with them directly since they are not open-sourced. However, is it possible to conduct toy analysis to compare the effectiveness of soften contact models and the neural simulator proposed in the paper? [1] Pang, T., & Tedrake, R. (2021, May). A convex quasistatic time-stepping scheme for rigid multibody systems with contact and friction. In *2021 IEEE International Conference on Robotics and Automation (ICRA)* (pp. 6614-6620). IEEE. [2] Jain, S., & Liu, C. K. (2011, December). Controlling physics-based characters using soft contacts. In *Proceedings of the 2011 SIGGRAPH Asia Conference* (pp. 1-10).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are stated in Section 5.

Summary

This work proposes using an external physics simulation to aid in monocular joint 3D hand and object pose estimation. By analyzing the stability of a perceived grasp inside the simulation, the proposed models learn to factor in grasp dynamics when estimating hand and object pose, producing stable and physically plausible grasps. To circumvent the problem of non-differentiable simulation, a DeepSim model is used as a proxy for learning the dynamics of the physics simulator and enables gradients to be propagated through. Quantitative and qualitative results show that the proposed method produces state-of-the-art results in pose estimation while improving physical realism.

Strengths

1. Leveraging the laws of physics effectively can benefit vision-based systems by providing physical prior. However, the effective use of physical laws and/or simulation is difficult as it creates extra overhead which may lead to intractable systems. This paper provides an effective hand/object pose estimation method that intelligently leverages a (simplified) world model (DeepSim) to learn physical prior from simulation. I find the formulation intuitive, and since the ending estimation pipeline is no longer reliant on the simulator, the pipeline is efficient. Essentially, the network is trained in a physics-aware fashion through the use of the world model. 1. I think the simplified world model (DeepSim) formulation is interesting and effective. By directly estimation the stability loss the model is easier to learn and can be directly used to optimize the objective. 2. The motivation of the paper is clear, and the proposed solution and components solve the raised issues. The experiments on the analytical gradient and numerical gradient show the necessity of learning the DeepSim model and provide a clear view of the limitation of the current differentiable physics simulator (namely, unstable gradient when dealing with complex contact geometries). 3. Results show that the learned pose and hand estimator outperform SOTA methods in terms of physical plausibility and are comparable in terms of pose estimation accuracy. Qualitative results and simulation videos also show that the proposed method is effective in estimating physically stable grasps. 4. The proposed stability analysis is general and could be applied to other base pose estimation networks as well as other domains such as stable human pose estimation.

Weaknesses

1. I find the applied adhesion force a little questionable. In the real world, humans grasp objects by applying forces, which is akin to small penetration. Applying an adhesion force is similar to having extra suction cups on the fingertips, which is not realistic. How is the force modeled? Is it a constant or is it a function of the contact forces like static friction? 2. The current formulation essentially biases the model towards firm grasps and static holdings of objects. All of the examples shown in the results are grasps that require almost all fingers. How about when the object is resting on the palm or not requiring the support of the hand? Since no ground or tabletop is modeled, would the model also provide grasping when the object is resting on the table? In that case, the object is supported by other forces and fingers do not need to apply to grasp. Would the model still bias toward a solution where all fingers are closed in on the object? 3. As dexterous manipulation is a study of motion, the current setup can be quite limited in modeling faster motion and movement of the objects. Similar to 2, the method is biased toward static and firm grasp, which is not always true when handling an object. The object's own momentum and movement can have a large effect on its stability, which a single-frame model would not factor in.

Questions

I would like to see some discussion on how he adhesion force is applied and modeled, and how well does the model handle non-grasp poses. --- After rebuttal, my question about adhesion force and some other details are addressed. I would like to maintain a positive rating of this work. ---

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

See weakness. I think the method is biased toward stable grasp and would not be able to model hand and object motion (as opposed to pose).

Summary

This manuscript presents a new approach for generating physically realistic estimates of hand and object pose during hand-object interaction. They argue that a weakness of existing approaches is that while they utilize approaches to avoid hand-object penetration or enforce contact, they do not enforce physically realistic contact that would obey e.g. gravitational forces, which they term dynamical constraints. Their approach is based on the assumption that the hand-object should be stable, that is, if physical forces were applied to an observed frame for a period of a few hundred milliseconds, the hand and object should show minimal displacement. First, following estimation of hand pose and mesh parameters and object rotation using existing approaches, they use a physics simulator to forecast displacements of the hand-object configuration forward 200 ms, and estimate a stability loss equal to the L2 norm of the displacement. They argue that the gradient of this loss relative to state in the physics simulator is unstable, and therefore they use a separate MLP they call DeepSim to approximate the stability loss, taking as input a concatenation of the hand and object signed distance field and hand-object configuration. They alternately train DeepSim and the base network. They compare their DeepSim approach on two benchmark datasets with state of the art techniques and achieve competitive, albeit weaker performance on hand and object estimation metrics, but improved performance on a set of more physical criteria – the penetration depth, distance of object displacement and fraction of frames with contact. They motivate the chosen DeepSim architecture through ablations, and show decreases in the magnitude and variance of the gradient using the appproximation approach. Overall I found the manuscript fairly complete, although of a fairly narrow scope that may mean it is more suited to a specialized computer vision conference. I haven concerns about some of the assumptions made about the physical environment and number of evaluations performed. If some of these questions are addressed I would consider moving up the score.

Strengths

• The manuscript is well motivated, clearly explained, and well contextualized within the field. There are evaluations across multiple datasets, set of ablations. The conclusions for the most part match the results. This should be published somewhere. • The architecture and approach are novel to me, and the approach of having a network learn to smooth the state gradient of a physics simulator could be more broadly useful. • The qualitative examples are quite compelling and there are seemingly significant increases in the physics scores for the hand-object interactions.

Weaknesses

• The method does not achieve state of the art in hand and object pose performance on the chosen datasets. • While the approach of smoothing the state gradient could be quite general, the application is narrow in scope. Not only is it limited to hand-object interactions, but the subset of interactions where the grasp is stable. It would be a bit awkward to use this method in practice since it is restricted to the subset of stably grasped frames and those would have to be pre-detected. • There are assumptions about contact force strength and the orientation of the gravity axis that are made and it is not clear to what degree they impact the results.

Questions

• How were hyperparameters for the losses chosen? • Can you provide accuracy comparison values (eg MJE) for the chosen examples? It would be nice to see if they are representative. Alternatively, it would help to see examples of what the failure mode of the techniques were. • Why are results from only a single random seed presented? • L258 “We set the gravity acceleration as 9.8 m/s2 in the y direction of the camera frame. For the adhesion force, we empirically set the gain as 100 and the force as 10N, so that a simple touch is not sufficient to stably grasp the object. “ To what degree is it accurate to assume the gravity axis in these datasets is aligned with the images? To what degree do slight variations in the gravity axis produce large changes in the object pose? Does this affect the comparison between techniques in Tables 1 and 2. • It is disappointing that the hand ground truth is not available in Table 2 and it makes it difficult to evaluate the technique as there is really only a single dataset with joint errors. Are there other splits or datasets that could be used? • Can you report the performance of networks trained with NimblePhysics and FiniteDifference gradients? It is unclear, especially with the Finite Difference network, if the networks simply don’t converge well or if there is a more catastrophic reason why they cannot be used for training. This was a key motivation for the DeepSim approach and it would be nice to verify that they are problematic. It would be interesting if they produced e.g. more physically plausible results.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Summary

This work presents a novel pipeline for 3D hand-object pose estimation, focusing on improving arbitrary base hand/object pose estimators with applying physical simulation and its induced physical loss (fitted as a neural network). The performance is tested on DexYCB and HO3D data sets and the method is compared against many SotA baselines. Results show improved performance mostly on the physics metrics.

Strengths

- The proposed idea and method are reasonable, interesting, and valid. - The performance on the introduced physics metrics gets improved over baseline methods. - The paper writing is good and the paper is easy to read and follow. - Qualitatively, the proposed method does generate more physically realistic hand/object interaction poses.

Weaknesses

- My major concern is that the performance regarding hand/object pose estimations is worse than the baselines [32, 52] in Table 1 & 2. Since the proposed method is particularly emphasized on optimizing physical realism, I think it's under the expectation that the method performs better in terms of the proposed physical metrics. But the whole point of doing this should be to improve the hand/object pose estimation results, which is not achieved as shown in the tables. - I'm confused by Fig. 1. The gt object pose looks quite different from the input image. Is this the dataset annotation issue or any visualization issue? Why is the gt object pose annotated so off? It's hard to say it's the problem of the baseline [32] if the gt is so off. - Why the related work [47, 24] are not compared in Table 1 / 2? I think it's important to compare against them as they also considered physical realism for the same task and the authors have discussed them in related work that they have disadvantages. It's better to show them in numbers. - In additional, I feel the idea of using physical simulation as loss to supervise interaction-oriented perception tasks is not new, as the many papers cited by the authors in the related work section and more others.

Questions

see weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

no issue found

Summary

Authors propose an approach for 3D pose estimation for hand-object interaction from a single image. Unlike prior work which is purely data driven (except for some works in robotics literature) and focus on visual quality, this work aims to also grasp stability. To this end, authors use physics based simulation to model grasp stability. Since PBS is typically non-differentiable, authors propose to train a neural network (with full supervision from the simulation) to emulate the simulation. This allows them to differentiably approximate the simulation. Authors outperform baselines on DexYCB and HO3D datasets.

Strengths

+ Most existing works focus on visual plausibility. It is interesting to see this work also reason about physical stability. + The idea of approximating non-differentiable simulation has previously been used () but it is still under explored. It would be better if authors provide more context about prior work in this space. + Authors outperform the baselines.

Weaknesses

[Technical] 1. Eq. 2: Can the authors provide a bit more motivation/intuition around the equation. What does it do? Why is it correct? Is it a general formulation or does it have specific applications. Current reference ([14]) is a 650+ page book on Classical Mechanics, which does not help the reader much. Maybe point to specific portion of the text that elaborates on the equation. It is not clear how did the authors arrive at this equation from Euler Lagrange equation. This derivations should be included in supp. mat. at least. 2. Sec 3.3: Approximating non-differentiable functions with a neural network is a well known problem. The most straightforward solution is to train a MLP with supervised data. Isn’t this exactly what DeepSim is doing? This can be traced back to “Approximation of functions and their derivatives: A neural network implementation with applications”, Nguyen-Thien et. al. Applied Mathematical Modelling, 1999 Other domains such as cloth simulation have also used similar techniques to train a neural network to predict the outcome of a physical simulation. (Holden et al. Eurographics’19) Can the authors elaborate how is their proposed DeepSim different? This is an under explored area so it is okay to have some similarities with prior work but authors should provide more context and flesh out what are the new insights here. 3. L142: How are the physical properties of the hand and object eg. mass, coefficient of friction etc. obtained from the input image for simulating forces? Does this generalise? 4. Eq. 7: Do we need to mark 8 corners on all template meshes? Isn't this restrictive? Why not put the loss on object mesh vertices directly? 5. Eq. 3: What about rotation? A grasp is still not stable if the object rotates due to a static grasp? Why consider only translation? 6. L140: If M is a matrix, what does M(\cdot) mean? [Minor] - Missing related work: TOCH. Zhou et al. ECCV’22. They learn to predict stable grasps from data. Jiang et al. ICCV’21: They learn to predict stable contacts on object and synthesise hand to match the contacts.

Questions

Some key formulations are unclear to me (see pt.1) this is important to clarify as it is one of the main contributions of the work. More clarity around design choices would also help the manuscript (see pts. 2-6).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors discuss potential limitations and broader impact in the paper.

Thank you, slightly raising the scores

I appreciate the authors' response. I like this paper and while I acknowledge that the use of physics simulators is not entirely novel, I do think there is a lot of value in the current approach. Because of this I am going to slightly bump my score, but I would not strongly advocate for acceptance. I am somewhat less concerned on a re-read about the method not hitting SOTA for object and pose keypoint detection but I also think the physics metrics could be presented in a realistic use case to be more convincing. I don't feel like two of my points are really addressed. I don't see how this method will apply to cases where the stability levels are unclear , eg during walking phases where your heel is off the ground higher stability can be achieved by increasing surface area. This is also noted by bTxd. I also don't see sufficient consideration of just guessing the gravity axis.

Post rebuttal update

Thanks authors for the rebuttal. It clarified my doubts. After reading other reviews and rebuttals, I maintain my positive overview of the work. The the discussion around integrating rotation in the stability loss is interesting and can be briefly incorporated in the "limitations/ future works" section.

Reviewer Response

I thank the authors for the detailed response. My concerns about "Bias Towards Grasping" and "Generalize to Fast Object Motion" has been addressed. The only remaining question is still centered on the adhesion force. What is the "gain" (L259) in the context of adhesion force? Is the adhesion force also applied to other SOTA methods in the supplementary video?

Thanks for the rebuttal

Thanks for your clarification. After reading the rebuttal and other reviewers' comments, I think the paper is of some value to be published somewhere. But I still have some concerns about its significance and the potential impact. In short, the paper does not conduct a deep and insightful discussion on an interesting and valuable problem. **The DeepSim network:** The network and the representations used are not designed under careful thinkings or after rigorious explorations. There are many works on learning simulation that are carefully calibrated to inject priors to the network architecure like [PINNs,NCLaw]. In this work, the authors are expected to conduct thorough discussions w.r.t. how to design a network to predict the physical stability of a grasping pose. It involves the netowrk architecture, input and output, what to predict, and so on. Presenting the designing process to readers either in the ablation study or in the supp would make the work more inspring to others. The current content in the paper cannot fully convince readers that the MLP structure is the most suitable one for the stability regression. Seemingly it is not designed under careful considerations. Besides, how to represent the grasping is expected to be discussed in depth. The current approach simply leveraging signed distances from hand to object and from object to hand. An intuitive illustration through figures to demonstrate it's effectiveness in representing the grasping (sometimes are incorrect with penetrations) would be helpful to let others get its insights. **The simulator:** Is the current simulator enough to calculate provide correct simulation? Mujoco was published in 2012. After that, simulators for graphics or robotics have undergone a fast development. A comparison is presented in [Dojo]. Besides, RK4 is used as the integrator in Mojuco. However, it seems that explicit Euler is used here (Eq. 2)? Have you tried other simulators? A discussion w.r.t. which offline simulator should be leveraged to provide GT stability scores is expected to be covered in the paper as well. [PINNs] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. *Journal of Computational physics*, *378*, 686-707. [NCLaw] Ma, P., Chen, P. Y., Deng, B., Tenenbaum, J. B., Du, T., Gan, C., & Matusik, W. (2023). Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics. *arXiv preprint arXiv:2304.14369*. [Dojo] Howell, T. A., Le Cleac’h, S., Kolter, J. Z., Schwager, M., & Manchester, Z. (2022). Dojo: A differentiable simulator for robotics. *arXiv preprint arXiv:2203.00806*, *9*.