Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos

Summary

This paper proposes a method to capture physically plausible human motions from monocular videos. The paper builds on top of NeuralPhys and introduces Kalman filtering to alleviate noise from kinematic motions. The framework combines Kalman filtering and physics simulation to be fully differentiable, thus enabling training in a supervised manner. The experimental results show that the proposed approach can achieve competitive performance.

Strengths

- The proposed method uses Kalman filtering to produce targets for PD control, which is more robust to the noise from pure kinematic poses. - The framework can be trained in a supervised manner due to its fully differentiable implementation.

Weaknesses

- The simulation and filtering refine the initial kinematic poses with dynamic constraints, which do not consider the image observations. The simulation may remove some high-frequency motions; thus, the refined poses may not be consistent with 2D poses. The authors should overlay the 3D pose onto 2D images and show more visualization results so that we can evaluate the discrepancy. - Although the method predicts contact information, the framework cannot prevent penetrations between humans and the floor since scene dynamics are not considered. - The optimal pose is predicted from Kalman filtering, while the velocity is obtained from simulation. In previous works, the final body pose is integrated from simulated velocity as in Eq. 2, which can guarantee physical plausibility. However, the optimal pose in this work is further updated with Kalman filtering, which may not be consistent with simulated velocity. Therefore, the final poses may not obey physical laws. - The work relies on external force, which may result in some unnatural states (e.g., body leaning).

Questions

- Do $q_t$, $\dot{q}_t$ in the output still follow rigid body dynamics as in Eq. 1? - The work builds on top of NeuralPhys with additional Kalman filtering. What is the performance of applying Kalman filtering on simulated or kinematic motion only? - Humans may penetrate the ground plane due to depth ambiguity. How does the method address penetration between humans and the ground plane? - The body shape in this work is fixed. Can the trained models be generalized to different skeleton shapes?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are discussed.

Summary

The paper presents a novel physics-based human motion capture method that is physically explainable, conforming to the PD control theory and rigid body dynamics. The key designs of the method involve an integration of a kinematic Kalman filter and Newtonian equation-based physics simulation, and learnable Kalman gains, PD gains, external forces and robot inertia biases. The method outperforms existing kinematics-based and physics-based motion capture methods on keypoint accuracies.

Strengths

(1) The method is physically explainable without unrealistic approximations of the control process and the robot dynamics. (2) Under the paradigm of using physics simulation to capture human motion, the method provides novel insights about which physical properties should be modeled by neural networks. (3) The method is superior to previous kinematics-based and physics-based methods in the accuracy of joint predictions. (4) The writing is clear and easy to follow.

Weaknesses

(1) The contact modeling only considers foot-ground contact, ignoring full-body contact that commonly appears in human-object and human-scene interaction scenarios. Besides, the contact on each foot is represented as a force vector on a pre-defined contact point, ignoring changes in the contact point and the resultant torque of the contact. (2) The method updates the inertia matrix $M$ online. However, the inertia matrix is the attribute of the robot and should be fixed values during the whole motion capture process for better physical interpretability. (3) To fully examine the generalizability of the proposed method, existing physics-based methods should also be compared on datasets Fit3D and SportsPose.

Questions

Typos: * In Figure 2, "$q_{t|t-1}$" -> "$q_{t+1|t}$" * In the caption of Figure 2, "performs" is redundant * In Equation 6, "$\sum_c^2$" -> "$\sum_{c=1}^2$" * In line 295, "HMDCap" -> "OSDCap" * In line 478, "the" is redundant

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

One limitation is that the human dynamic model is formulated as a connection of circles and cylinders, which neglects the modeling of geometric details and wearings of humans.

Summary

This work focuses on tackling the problem of single-person motion estimation from a monocular video. Current approaches produce temporal artifacts such as jittering. Most approaches are entirely kinematic while others that combine physics, do it by re-simulating the kinematic inputs by using automatic PD controllers. These methods, however, require simplifying assumptions that compromise the motion realism.This method combines information from two sources to produce an optimal estimate of single-person 3D human motion. One source comes from an off-the-shelf kinematic motion estimation pipeline and the other from a differentiable physics formulation. These two "measurements" about the motion are combined in a Kalman-filter to generate an optimal output. Here, the authors propose to selectively incorporate the physics models with the kinematics observations in an online setting, taking as inspiration the neural Kalman-filter. The method uses a meta-PD controller and a physics-based simulation step. The Kalman filter is realized via a recurrent neural network which aims to balance the kinematic inputs with the simulation. The authors propose an end-to-end model for this purpose which is not trivial to accomplish. The method is capable of capturing accurate global trajectories and, at the same time, producing physically plausible human poses.

Strengths

* The paper is very well written and the experiments are well presented which makes the paper easy to follow. * Authors present extensive experiments comparing several SoTA methods and include in-domain and out-of-domain test data for these. * The setup and the method are sound. * I would say that this is the first paper that successfully combines information from kinematic estimates and a differentiable physics simulation step in an end-to-end manner. It is not trivial to refine kinematic estimates with physics simulation (or physics informed estimates) outside the RL framework. It seems that the neural Kalmal filter is a promising direction to bridge the gap between kinematics and physics. I believe that this work is of high significance for the field.

Weaknesses

### **Presentation** The qualitative results could be better presented as it is sometimes hard to have a good sense of the pose estimated by the kinematic approach (TRACE) both in Fig. 6 and in the supplementary .gif images. The way the poses and the original video are visualized can be improved. First, I would suggest making the kinematic skeleton more visible as it is “obscured” by OSDCap results. Even better, it would be nice to have SMPL visualizations of GT, kinematics and OSDCap as it is presented in Fig. 1. In my opinion, changing visualization styles within the paper can reduce the presentation quality. I also advise the authors to focus on video results with several examples. If possible I would like to see this as part of the rebuttal, if not, then this should be present for the camera ready version of the paper. ### **Physics-based metrics** Sec 4.3: It would be interesting to show the results for more physics-based metrics other than the Acceleration metric, for example, foot skating and ground penetration, which should be corrected by the physics formulation. As authors are modeling contacts and have physics-based losses (e.g., friction, velocity), it would be interesting to know these metrics in comparison with the SoTA or at least the baseline used (TRACE). ### Minor - (Typo) Fig.2: 4th line: performs contains-->contains. - (Typo) L295: HMDCap -->OSDCap? ### References I would advise the authors to include most recent papers that combine kinematic and physics estimates for pose/motion estimation, either in the introduction or related works. For example: - Ugrinovic et. al., “MultiPhys: Multi-Person Physics-aware 3D Motion Estimation” (CVPR 2024). - Zhang et. al. “PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos” (CVPR 2024).

Questions

- L38: To be clear, what the authors refer to as a "differentiable physics simulation" is the use of rigid body dynamic equations or is it an actually differentiable simulator, e.g., Tiny Differentiable Simulator from PyBullet, similar to Gartner et. al? - L116: Authors create a proxy character (shown in appendix B). I wonder if it is possible to use the humanoid generated by SimPOE or KinPoly for the same end? This latter humanoid seems to be the most realistic representation of a 3D human body for simulation. I would like what the authors think about this and why they chose this specific form. - Sec. 3.2: How are the contacts modeled? Are they modeled directly with the GRU and data annotations or is there also a specific physics formulation to account for these contacts?

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

Yes.

Summary

This paper introduces a novel method to estimate 3D human motion from a single camera, aiming for physical plausibility and accuracy. The authors use a differentiable Kalman-filtering approach in an online setting that balances kinematics estimated from an off-the-shelf algorithm (TRACE) and the physics simulation. They built OSDNet, which consists of fully connected and GRU layers, for the given optimal state human dynamics capture task. They show their method outperforms the existing methods in the same online estimation setting while each technical component was validated through ablation studies. In general, the method sounds reasonable and novel. However, there are several things that need to be cleared.

Strengths

They propose the approach inspired by the neural Kalman filtering approach. This method seems to be novel. Compared to their baseline approach (TRACE), they provides significant performance improvement, which is not the case for the other physics-based approaches (if I remember correctly).

Weaknesses

It is not clear how they constructed the character. They mention that they used metadata of Human3.6M dataset. Does it mean they use the ground-truth bone lengths of the subject? – If so, they should state their method with automatic character generation (similar to what SimPoE did) for a fair comparison. It is not clear if their outperformance is coming from the known limb length assumption or the method itself. – If not, please state more clearly how they generate the character in the physics simulator. It is not clear why they selected TRACE as a default method. From Table 1, TRACE seems to significantly underperform VIBE, which has been widely used as a baseline kinematics estimator (e.g., SimPoE). The major advantages of using TRACE are its capability to track a person's identity and the robustness of the camera motion. However, all training and evaluation were done in a fixed camera coordinate system. In the paper, they mention that since TRACE's output is expressed in the first frame global coordinate, their method is agnostic to the initial calibration (line 473). I disagree with this line. Isn't the initial pose of camera essential to know the gravitational direction? Will this method be robust to the camera which is highly tilted? In Table 3, conventional PD methods are showing catastrophic performance. I am doubtful about this, maybe the training configuration was not set properly? This is telling us that neural PD control is destroying the performance which does not align with existing literature. Many of the state-of-the-art video-based kinematics estimation models are not stated in the paper including HybrIK (CVPR 2021), CLIFF (ECCV 2022), PMCE (ICCV 2023), ReFit (ICCV 2023), MotionBERT (ICCV 2023), and HMR2.0 (ICCV 2023). To the best of my knowledge, all these methods (not limited to) have better performance on Human3.6M dataset.

Questions

[Some are duplicated with the Weakness section] – How did you construct a physics-based 3D person model? – Why did you use TRACE? IF you value the potential of future use case with moving camera, this needs to be stated. – Why do neural PD controllers perform so badly in your test case? – I guess the GRU unit is uni-directional, am I correct?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

–

Summary

The kinematic motion estimation suffers from inconsistencies of frame-wise predictions while the physics-based method suffers from the gap between the simulator environment and the real-world ground truth. This paper proposes a method to taking advantages of both by connecting them by a learnable kalman filter network. The proposed method OSDCap is a new physics-based human motion and dynamics estimation method. The proposed Kalman filter takes the simulated motions and the 55 noisy 3D pose estimation as inputs, combines them, and produces an optimal state prediction as the 56 output. It also has a learnable inertia prediction for weights distribution, that produces plausible motion as well as valuable estimates of exterior forces and internal torques.

Strengths

- The proposed method is very straightforward and intuitive. Considering the noise pattern of the frame-wise kinematic method (frame independent) and the noise of physics-based method (temporal propagated), the idea matches with the design motivation of kalman filter very well. - The presentation is good. I could understand the high-level idea and the technical design without trouble.

Weaknesses

- Some related works are missing in the experiments, especially considering that kalman filter is widely used for human pose tracking. Moreover, as the authors proposed a learnable module to replace the role of Kalman filter (by predicting the gains), it would be helpful to compare the proposed method and a baseline with the classic kalman filter parametric model. - There are some missing baselines, especially the kinematic ones in the Table 1, for example PointHMR (CVPR’2023) and KTPFormer (CVPR2024), considering that the papers are available before the submission of this work, is there any reason that some recent works are not listed in the comparison?

Questions

I am overall satisfied with the novelty of the proposed method with no previous work having implemented the straightforward but well-connected idea as presented in this work as far as I know. However, I will need more evidence about the experimental significance to adjust my rating. Or, could the authors clarify the reason of the missing of more recent baseline works? I would need more evidence to recognize that the kalman filter (at least the learnable one in this work does work generalizability and is more superior than the classic parametric KF).

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations in the paper. They use the widely distributed datasets with human faces but they are controlled in the studio environment and under consent.

Feedback

Thanks for the authors’ response and the rebuttal document. I have some additional comments/questions. #### **Presentation Improvement.** I see what you did in Figure*2, in my opinion this looks much better. This is more of a nuance, I would further suggest breaking each sequence in two images: one image corresponding to the baseline vs. GT and the other for the proposed method vs. GT, this way the comparison is even clearer. Otherwise you could be masking failures of either the baseline or your method behind the clutter. #### **Physics-based metrics.** The results presented in Table*2 of the rebuttal look good. The use of the GD metric makes sense to me as it is true that predicted motions that contain floating can falsely show very good GP metrics. Also, the results presented are quite good for the proposed method which makes the work potentially more impactful. Will authors release the code so that these results can be reproduced? #### **Differentiable simulation in OSDCap.** I see, thanks for the clarification. This is more of a curious inquiry: I would like to know the authors’ opinion/intuition on how would these fully featured differential simulators work with their approach, if they use it to replace the rigid body dynamics equations and Euler's integration? Could this framework allow for such a change? #### **Selection of proxy characters.** For clarity, I would like to know what do the authors exactly mean by “dynamic predictions” in this context.