Universal Humanoid Motion Representations for Physics-Based Control

Author Response to Review arMx

We thank the reviewer for helpful, constructive, and positive feedback. We are glad that you find our work "well-motivated", "well-demonstrated", and "outperform baselines on downstream tasks by a large margin". Here, we try to address the questions and concerns: **Comparison with Baselines on Imitation** As ASE and CALM do not use a motion-tracking objective for training, we cannot compare motion imitation results directly. We show that they do not have good motion skill coverage in the VR controller tracking task, which is a generalization and arguably harder task than full-body motion imitation. The VR controller tracking task only has partial body observations, but requires the policies to produce the same full-body motion as the full-body imitation task. Between PULSE and imitate \& repurpose, the main difference is how the latent space is trained, where PULSE is through distillation and imitate \& repurpose is through RL. Imitate \& repurpose also does not use a Gaussian prior or learnable prior, though both PULSE and Imitate \& repurpose use the AR1 prior. We report the result of training using RL and no distillation in Table 6 of Appendix C.2. We can see that training without distillation can learn a large amount of motion in AMASS (76\%), but it cannot scale to the performance of PULSE. We hypothesize that random sampling for variational bottleneck and random sampling for RL hampers the learning process. This ablation can be viewed as imitate \& repurpose implemented in our framework and humanoid, and we show that it does not scale as well in the imitation task. **Comparison between PHC and PHC+** Both PHC and PHC+ are evaluated on the same set of modified training and testing sequences in Table 1. After removing sequences with issues, there are 11313 and 138 AMASS motion sequences for training and testing, and we re-run all PHC's evaluations on the modified dataset. **Plot Content Visibility** We have updated the plot for better visibility. Thanks! **Answer to Questions** - As sim2real from physics simulation [1, 2, 3, 4] continues work, we believe that a similar methodology should transfer to real humanoids. Although modifications to the humanoid kinematic tree, reward designs, and domain randomization will be required, the methodology should be applicable. PHC has also demonstrated motion imitation on diverse body shapes, which could also be distilled to a shape/dynamics-aware latent space. [1] Cheng, Xuxin, Kexin Shi, Ananye Agarwal, and Deepak Pathak. 2023. “Extreme Parkour with Legged Robots.” arXiv [Cs.RO]. arXiv. http://arxiv.org/abs/2309.14341. [2] Zhuang, Ziwen, Zipeng Fu, Jianren Wang, Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, and Hang Zhao. 2023. “Robot Parkour Learning.” arXiv [Cs.RO]. arXiv. http://arxiv.org/abs/2309.05665. [3] Fu, Zipeng, Ashish Kumar, Ananye Agarwal, Haozhi Qi, Jitendra Malik, and Deepak Pathak. 2021. “Coupling Vision and Proprioception for Navigation of Legged Robots.” http://arxiv.org/abs/2112.02094. [4] Radosavovic, Ilija, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. 2023. “Learning Humanoid Locomotion with Transformers.” arXiv [Cs.RO], March. https://doi.org/10.48550/ARXIV.2303.03381.

Thanks for your suggestions!

Thanks again for your questions and discussion. It will help us improve the paper and mention the pure RL-based experiments in the main paper. We appreciate the discussion and constructive comments. Best, Authors

Author Response to Review UBzx

The authors thank the reviewer for constructive and encouraging feedback. We are glad that you find our method a "fundamental representation of humanoid motion", our method's effectiveness "well presented", and ablation "compelling". To address questions and concerns: **VR Motion Tracking Results** We acknowledge that the quantitative result on VR motion tracking is less performant on joint position errors though better on success rate. Similar performance degradation on joint errors can also be observed in Table 1, where after distillation with the variational latent space, the imitation performance is worse than the teacher's. This is a known trade-off between prior regularization and achieving good reconstruction performance in VAEs, where the reconstruction error usually does not reach 0 during training. As shown in Appendix C.3, if a non-variational latent space is used (e.g. spherical), a better distillation performance can be achieved (100\% success rate and 28.1 $E\_\text{mpjpe}$). However, such a latent space does not provide a variational prior, and random samples from it do not generate coherent motion. Thus, while our latent space has good coverage of the motor skills required to perform AMASS, it is not full coverage. Downstream task such as VR motion tracking is searching in a constrained solution space and can lose performance on precise motion tracking (though gains faster convergence). Future investigation is needed to achieve better motion imitation results while retaining the generative capabilities of the latent space. **Failure Case not Using Prior R** We believe that this refers to the cases "one can also supervise $\pi\_{\text{task}}$'s distribution using prior $\mathcal{R}$", which leads to an "adverse feedback loop". We apologize for the confusing statement. Here, we are referring to an alternative to using the residual action formulation. In Sec. 4.3, we propose to let downstream policy produce a residual with respect to the learned prior $\boldsymbol{a}\_t = \mathcal{D}(\pi\_{\text{task}}(\boldsymbol{z}\_t | \boldsymbol{s}^{p}\_t, \boldsymbol{s}^{g}\_t) + \boldsymbol{\mu}^{p}\_t)$. The alternative is to directly output the latent code rather than the residual, and supervise the output distribution with the prior $\mathcal{R}$. We **do not use** this alternative due to the described issue. The paragraph is included as a justification for the residual action formulation. This failure case only applies to the alternative approach and does not apply to the residual action one. The typos and the reference section are cleaned up in the revision. Thanks!

Author Response to Review a8Fw 1/2

We thank the reviewer for the positive feedback and constructive comments. To address the questions and concerns: **Comparison PhysicsVAE and ControlVAE** We acknowledge that PhysicsVAE [1] and ControlVAE [2] and ours all strive to learn a compact latent space using a variational bottleneck. While our structure resembles a VAE, we do not use auto-encoding/reconstruction as a learning objective. Our decoder outputs actions $\boldsymbol{a}\_t$ instead of the input state $\boldsymbol{s}\_t$. As a result, we do not require learning any world model during training. This design allows us to directly distill motor skills from a pretrained imitator that has achieved strong performance on a large-scale (40 hours) motion dataset. PhysicsVAE[1] has tested on the CMU dataset (a subset of AMASS) to further understand its scalability. They "observed that the model learned from the CMU dataset does not perform well when compared to other models learned from either the locomotion dataset or the dancing dataset" (Sec.6, third paragraph, page 11 in PhysicsVAE). The learned latent space from PhysicsVAE also requires an additional adaption layer to solve some of the harder tasks such as terrain traversal, which PULSE does not require. Similarly, ControlVAE [2] stated that "experiment on a large-scale dataset composed of 3.3 hours of motions" "converges to a lower reward", and the "learned skill embedding can still recover input motion with a larger visual discrepancy, but the performance on the downstream task are significance degrenrated" (Sec.5.7, last paragraph, page 14 in PhysicsVAE). Compared to PhysicsVAE and ControlVAE, we learn from a much larger dataset and demonstrate that our latent space can (1) scale to the motion dataset after distillation, (2) perform well for downstream tasks without any additional adaptations. PhysicsVAE and ControlVAE also use different humanoids and simulation environments (Bullet and ODE) from ours (SMPL humanoid and Isaac Gym), and the gap between environment and humanoid choices hinders the reimplementation and fair benchmarking. **Benefit of scaling to the AMASS dataset** Our main motivation for scaling to AMASS is to have a latent space that covers a wide range of human motion. Thus, we include the VR controller tracking task, which requires tracking three 6DOF points in the global space. This is a generalized version of the full-body motion tracking task, where only partial observation is given to the humanoid to create full-body motion. This task requires the latent space to contain motor skills to match **arbitrary** user input in a continuous and streaming fashion. A latent space that has only seen small-scale locomotion datasets would not have the motor skills required to perform these free-form motions. Our supplement video (Sec. Motion Tracking Downstream Task) shows that the controller that uses our latent space can handle motions such as squatting, punching, dancing, running, rolling, etc. We also quantitatively demonstrate the tracking performance on the AMASS dataset in Table 2. Notice that the joint errors reported in Table 2 are **full-body** joint errors, and a low value means that the full-body motion generated from three-point input is close to the ground truth. Table 2 and Fig. 5 show that using our latent space, one can efficiently solve this task and achieve a high success rate on training, testing, and real-world data capture. Previously proposed latent spaces (ASE and CALM), when also trained with the AMASS dataset, do not obtain the coverage demonstrated in our latent space. We also note that few prior latent space methods have attempted a motion-tracking task due to the diversity and coverage required for free-form motion tracking. On generative tasks, we test simple ones ranging from reach, speed, and strike, but also challenging ones like complex terrain traversal. The terrain traversal task requires the humanoid to handle stairs, slopes, rough surfaces, and obstacles. Prior art [3] on this task has relied on GAIL to achieve natural motion. Using our latent space, we no longer require GAIL but can still learn human-like behavior. Note that previous latent space approaches either fail on a stairs-only environment [4] or require an adaptation layer to traverse rough terrains/slopes [1], while our latent space can be used to solve the task without modification, thanks to the wider range of motor skills it can draw upon.

Author Response to Review a8Fw 2/2

**Answers to Questions** - We design the VR motion tracking task to show that our learned latent space can cover a large amount of motor skills. A controller using our latent space is able to achieve good full-body motion tracking performance on the AMASS train and test and also transfers well to real-world data captures using commercial VR headsets. If the latent space has only been trained on a handful of locomotion sequences, it will only have the skills to perform simple actions. Our result on complex terrain traversal also shows that scaling matters, where the humanoid will employ skills like jumping when traveling fast on stairs. - We will include failure cases for PHC+ from the AMASS test set and other data sources. In general, while PHC can imitate a large number of motion sequences, unseen highly dynamic ones (e.g. badminton jump smash) or noise motion estimated from videos can still cause the humanoid to fall. - We agree that motion naturalness can be hard to evaluate and subjective, and we included a large number of video results for qualitative evaluation. The motion quality of a generative method is hard to quantify since there is no ground truth to compare against. Thus, we follow the prior art in the field and provide reward comparisons and video demonstrations. For the VR tracking task, we include pose estimation metrics, which are quantifiable ways to judge the quality of estimated motion. For unnaturalness in motion, we acknowledge that PULSE does not strictly constrain itself to certain types of motion when solving tasks. Since we train on a large-scale motion dataset, certain styles of motion that may be deemed unnatural for the speed task are also available in the latent space. For instance, AMASS contains motions for "running fanatically" or "limping", whose motor skills are also learned in the latent space and can be used to solve the speed task. While training from scratch will often result in inhuman behavior, PULSE solves tasks using human-like behavior even when there are no constraints on the strategies used. To further constrain the solution space to natural behavior defined for certain tasks, one can enforce symmetry or incorporate additional regularizations. - The discrepancy on baselines (ASE and CALM) mostly comes from the training data scale. For a fair comparison, all baselines are trained on the AMASS training set, the same as PULSE. The original ASE is trained on a specialized dataset that only contains locomotion or character movement (eg. strike). By construction, a latent space trained on specialized datasets will only include motor skills that are deemed natural for the downstream task. However, datasets like AMASS contain diverse full-body motion that are not specialized. Thus, motor skills for sequences such as "crouch walking" or "running fanatically" could also be learned in the latent space of ASE. When using these motor skills for downstream tasks, the resultant motion can appear unnatural, though still human-like (in the sense that a human can recreate these motions if desired). [1] Won, Jungdam et al. “Physics-based character controllers using conditional VAEs.” ACM Transactions on Graphics (TOG) 41 (2022): 1 - 12. [2] Yao, Heyuan, Zhenhua Song, Baoquan Chen, and Libin Liu. 2022. “ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters.” arXiv [Cs.GR]. arXiv. http://arxiv.org/abs/2210.06063. [3] Rempe, Davis, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. 2023. “Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion.” arXiv [Cs.CV]. arXiv. http://arxiv.org/abs/2304.01893. [4] Hasenclever, Leonard, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. 13--18 Jul 2020. “CoMIc: Complementary Task Learning \& Mimicry for Reusable Skills.” In Proceedings of the 37th International Conference on Machine Learning, edited by Hal Daumé Iii and Aarti Singh, 119:4105–15. Proceedings of Machine Learning Research. PMLR.

Additional Experiment for showcasing the benefit of scaling to AMASS

We would like to thank the reviewer again for raising the concern over the benefit of scaling to AMASS. Here, we would like to add some quantitative evidence. We study the effect of training our framework on a locomotion-only dataset (a subset of AMASS, used in PACER [3]) that contains 275 motion sequences of a total length of 38 minutes. The dataset contains mainly walking, running, and jumping. After distilling, the encoder-decoder imitator can reach an imitation result of: | AMASS-Locomotion | | | | | | ----------- |----------- |----------- |----------- |----------- | | $Succ \uparrow$ | $E\_{g-mpjpe} \downarrow$ | $E\_{mpjpe} \downarrow$ | $E\_{acc} \downarrow$ | $E\_{vel} \downarrow$ | | 100% | 25.5 | 20.1 | 2.9 | 4.3 | which shows that the latent space has successfully learned the training sequences. Random sampling from the latent space also results in human-like walking and running motion. However, when applying this latent space to the VR controller tracking task, we obtain the following result on the AMASS train and test, as well as the real-world data capture: | | AMASS-Train | | | | | AMASS-Test | | | | | | ----------- | ----------- |----------- |----------- |----------- |----------- | ----------- |----------- |----------- |----------- |----------- | | | $Succ \uparrow$ | $E\_{g-mpjpe} \downarrow$ | $E\_{mpjpe} \downarrow$ | $E\_{acc} \downarrow$ | $E\_{vel} \downarrow$ | $Succ \uparrow$ | $E\_{g-mpjpe} \downarrow$ | $E\_{mpjpe} \downarrow$ | $E\_{acc} \downarrow$ | $E\_{vel} \downarrow$ | | Locomotion-latent | 56.8 % | 111.6 | 75.4 | 4.5 | 8.8 | 15.2 % | 138.2 | 90.1 | **5.7** | **10.9** | | Ours | **99.5 %** | **57.8** | **51.0** | **3.9** | **7.1** | **93.4 %** | **88.6** | **67.7** | 9.1 | 14.9 | | |Real-World | | | | | | | -----------| ----------- |----------- |----------- |----------- |----------- |----------- | | | $Succ \uparrow$ | $E\_{g-mhpe} \downarrow$ | $E\_{acc} \downarrow$ | $E\_{vel} \downarrow$ | | Locomotion-latent |5/14 | 117.5 | **5.5** | 9.5 | | Ours |**14/14** | **68.4** | 5.8 | **9.0** | Both ours and the locomotion-latent model have the same training data and procedure, with the only difference being the training data when forming the latent space. From the result we can see that scaling to AMASS significantly increases the VR controller tracking task's performance on the real-world dataset, showcasing its usability in real-world scenarios. Scaling to a large dataset increases the diversity and coverage of the latent space and improves its applicability to downstream tasks. We would love to further discuss and provide explanations if the reviewer has any additional concerns or questions. Thanks!

Thanks for your constructive comments!

Thanks again for your constructive comments and discussion, which will help improve our paper and our presentation on the importance of scaling up to a bigger motion dataset. We appreciate the discussion and raising the score. Best, Authors