InsActor: Instruction-driven Physics-based Characters

Summary

This work proposes InsActor, a framework for instruction-driven character animation. Given human instruction and/or way points, InsActor first leverages a diffusion model to generate state sequences of the character. Next, it uses a skill embedding model to convert the state sequence into physically plausible trajectories with state and actions.

Strengths

1. The paper presents the motivations, setups, and methods quite well. The diffusion model for state sequence generation and conversion from state sequence to physical trajectories are described in sufficient details. 2. The paper provides sufficient experiments and comparisons to show it out-performs the baseline method. The hierarchical design is also justified with ablation studies.

Weaknesses

1. The works appears to have limited novelty, as it is somewhat a straightforward combination of character motion synthesis with diffusion methods such as [13] and a low-level trajectory optimization framework. 2. This work seems to miss a comparison with a straightforward method for the low-level policy, since the second stage can be formulated with a standard trajectory optimization problem. Since Brax is also a differentiable simulator, trajectory optimization should be easy to set up and in principle, it can be used to obtain an optimal solution in terms of distances between the states generated from the first stage and states from a physically plausible trajectory. I think this work could include results and analysis from trajectory optimization or explain why it is not feasible here.

Questions

Suggestion on writing: - line 92: I believe angular velocity $q$ of a link is not the derivative of rotation, and the symbol $q$ clashes with the notation of joint position $q$. This description needs to be clarified to avoid confusion.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The paper has provided adequate discussion of limitation.

Summary

This paper tackles the problem of generating text-conditioned character animation that is physics-based. It proposes a two-stage approach that first generates a kinematic motion conditioned on text using diffusion, and then tracks this motion with a physics-based motion VAE in the learned latent space. Experiments on the common KIT-ML and HumanML3D benchmarks show improved performance over prior work in physics-based motion from text, and show the ability to specify goal waypoints for the motion to hit while following human text prompts.

Strengths

Physics-driven text-to-motion is an important problem and conditioning on free-form text input has not really been tackled in the literature, so the paper is novel in that respect. Diffusion has shown promising results recently, but these are all kinematic. The proposed idea of using high-level diffusion followed by physics-based tracking is simple and solid. It would make a good first baseline for future work in this area. Technically, InsActor uses a physics-based motion VAE to track kinematic motion which is novel, and it uses differentiable physics to train rather than a learned world model or RL. If this approach really does work to track general motions in the HumanML3D dataset, it would contribute an alternative to recent RL approaches that can be difficult to generalize. The proposed method is evaluated on HumanML3D and KIT, which are the most relevant benchmarks for the text-to-motion task. Also, the DReCon baseline in Tables 1 and 2 is an important baseline that uses the alternative state-based approach to tracking. The supplementary video and figures are visually pleasing, and I appreciate that the supp video is extensive and shows many results. The shown demo is also cool, and demonstrates fast generation capabilities (relative to other diffusion approaches).

Weaknesses

To better motivate the need for physics in text-to-motion, there should be a comparison between the proposed InsActor and a state-of-the-art kinematic diffusion model like MDM [33] added to Tables 1 and 2. Currently some numbers in these tables, e.g. FID and diversity, are worse than those reported in MDM, and it’s not clear why that’s the case since the high-level planner in InsActor is very similar. Does this high-level planner perform worse than MDM? Or does the physics-based motion tracking somehow have a large effect on motion quality and diversity? The high-level policy ablation reported in Table 4 takes a step in this direction, but it is trained on rollouts from the motion VAE instead of directly on mocap data as done in MDM and other text-to-motion diffusion models. Looking at video results at 0:46 and 01:54, I don’t think the high-level diffusion planner is on par with recent models like MDM. Both with and without waypoint guidance there are some significant artifacts like jittering and skating, some of which seem to be affecting the final motion from InsActor (e.g. some noisy popping of limbs and unnatural sliding). I understand this planner is not necessarily the main contribution, but I think poor kinematic motions from the planner undermines the comparison to the target-state tracking policy DReCon, which may perform better when operating on, e.g., outputs from MDM that better reflect realistic motion. Since the low-level motion VAE model for tracking (Sec 4.2) is a key contribution of the work, it’s very important to justify that it is necessary by showing that the DReCon baseline is still inferior when operating on more reasonable kinematic inputs. The methods Sec 4 is missing some details that could improve understanding and reproducibility: * The tasks states described in L92 are all local, so how is the global root trajectory modeled in the diffusion Sec 4.1? * L154: if the pose state is in the local frame, how is inpainting performed to ensure motion meets a global target waypoint? * What is the architecture of the diffusion model? Is it using 1D convolutions as in Diffuser or a transformer as in other human motion diffusion models? $\mu$ and $\Sigma$ in Eqn 2 are never defined. In general, I’m wondering why not use a SOTA motion diffusion model out-of-the-box for this high-level planning component? * L185: is the low-level motion VAE trained directly on outputs of the diffusion model or on mocap data from the dataset? If on mocap, why is the encoder (Eqn 4) expected to produce reasonable results when operating on noisy and unrealistic pose transitions? * Similarly, Sec 5.3 shows robustness to perturbations from boxes, but is this kind of perturbation seen in training of the low-level policy too? If not, how does this robustness arise without using RL for training (i.e. without some exploration). An evaluation on the low-level tracking component by itself would be very helpful. E.g. reporting tracking errors for the latent policy from InsActor compared to DReCon for both motion-captured and diffusion-generated motions. The current metrics in Tables 1 and 2 were designed for kinematic text-to-motion models, and I would think are mostly influenced by the diffusion planner which is the same for InsActor and DReCon, so a tracking-only evaluation could help parse the difference in performance. There are also open-source RL physics-based trackers that may be worth considering, e.g. [Luo et al., Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation, NeurIPS 2021].

Questions

Overall, I think a general physics-based text-to-motion model is an important and novel direction, and the hierarchical approach of diffusion planning with physics-based tracker could be a strong baseline going forward. But I’m mainly concerned that the quality of the output from the high-level diffusion planner is compromising the comparison to DReCon and therefore the need for a latent motion VAE tracker has not been fully justified. I would really like to see how InsActor performs when plugging in a SOTA diffusion model like MDM [33] as the planner. Moreover, an evaluation of standalone tracking performance would make the comparison between the two tracking approaches (latent vs state-based) much more clear. Some other comments and suggestions that didn’t have an influence on my rating: * The related work (Sec 2) is missing relevant physics-based human animation methods and a discussion of why they are difficult to scale up to the general text-to-motion task. E.g. [Peng et al., ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters, SIGGRAPH 2022] [Won et al., A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters, SIGGRAPH 2020], etc.. * Concurrent work PhysDiff [Yuan et al., arxiv 2022] is an alternative approach to adding physicality to text-to-motion diffusion and could be discussed in future revisions of the paper. Also related, concurrent work Trace and Pace [Rempe et al., CVPR 2023] gives controllability over physics based characters with guidance of a diffusion planner. ===================== After Rebuttal ============================ After considering other reviews and discussions with authors and between reviewers, I have decided to slightly raise my score and am leaning towards accept. I think the paper lays out a compelling kinematic diffusion + physics-based tracking idea that can serve as a baseline and inspire improvement in each component of the system including differentiable simulation, motion diffusion, and physics-based tracking. The evaluations show that this hierarchical approach works better than state+action diffusion, and that tracking with a latent model is more robust than direct state-based tracking for planned motions from MotionDiffuse. However, I am still quite concerned about the qualitative results and would really encourage the authors to update the paper text to discuss these qualitative issues such that future work can pursue important directions (e.g., the choice of Brax and differentiable simulation in general rather than RL, and the noisy plans from MotionDiffuse especially in the waypoint setting). It would also be good to clarify why in Table 2 of the rebuttal doc, motion quality (FID) drops significantly from kinematic “Planner” (b) output to full physics-based InsActor (c), indicating that adding physics-based tracking is not necessarily improving motion realism despite it being physically constrained (unlike in PhysDiff). I encourage the authors to show some video results of the planner output vs InsActor on regular text-to-motion (not the waypoint setting) to demonstrate the difference and potential advantages/disadvantages of using the latent tracking technique vs RL and more reliable simulators as in DReCon. ==========================================================

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Limitations are sufficiently discussed in Sec 6.

Summary

This work proposes InsActor, a physics-based character control framework that enables controlling agents to follow high-level instructions. It employs a high-level diffusion motion planner and a low-level character controller to achieve mapping between text-based instructions and physics-based motion control. By creating waypoints and desired state transitions, the diffusion motion planner specifies the motion plan that the low-level motion controller follows. The low-level motion controller consists of a pre-trained VAE encoder to translate diffusion state into actionable latents for control.

Strengths

- I find the idea of using a higher-level motion diffusion model to specify waypoints and states, and then using a lower-level motion controller to follow them, intuitive and easy to understand. This formulation is flexible and creates the ability to have both high-level control of motion through text and slightly lower-level control through waypoints. - The low-level skill discovery module based on diffMimic itself seems like a significant contribution, as it has the potential to be used in many tasks and enables character control. Using a VAE to translate invalid state transitions into actionable latent codes for stable control has great potential in other downstream tasks. - Experiments show that the methods outperform the in-house implementation of the state-of-the-art methods PADL and DReCon.

Weaknesses

- The main weakness I find in this work is its qualitative results. The simulated character is jittery, floaty, and appears to have foot sliding, which is unexpected for a physics-based method. For instance, at the 1:43 mark of the video, InsActor's feet quickly shuffle in a way that should be impossible in a physics simulator. The character seems overall drunk when walking, and there are occasional high-frequency jitters in the root (or camera?), for instance, at 00:48. I am not sure what could have caused this issue: is it the policy? or the setting/fidelity of the Brax simulator? - Similar to the previous point, the visual results of the implemented PADL and DReCon are far inferior to their original shown results. The movement is unnatural and jittery. I understand that there is no official implementation provided, but similar scenarios could be recreated in InsActor to create visual comparisons. PADL's generated motion is stable and physically realistic, unlike the ones shown in the provided video. As motion generation is largely visual, the provided quantitative results do not really provide that much insight into the real performance of the method. - Visualization should be provided for the high level diffusion planer. Since states are directly generated by the planner, they should be visualized and compared with existing models such as motion diffusion model (MDM). The lower level controller, combined with the latent skill decoder $p_\psi$, forms an imitator that follows specified states. How well does the diffusion model generates the state and how “invalid” are they? - The claim of "long, unstructured human commands" is overstated. The tested text instructions are still short, clear, and close to the ones in the dataset. - There are many missing details about the performance of the lower-level skill discovery module. If the whole HumanML3D dataset is used for training, how well can the skill embedding encapsulate the motion described in the dataset?

Questions

My main concern is the fidelity of the generated motion. Why does the character appear to be floaty and jittery? Is it the learned policy or the simulator settings? --- After rebuttal, my main concern about the fidelity of the generated motion returned. However, after discussion with other reviewers and the authors, I believe that the this issue could be a problem of the underlying simulator and should not undermine the overall contribution of the framework. Thus, I raise my score to borderline accept. ---

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are addressed.

Summary

The paper proposes an approach for generating physically plausible human motion from open text prompts. The approach combines high-level trajectory generation with guided diffusion and a low-level skill model encoded in a latent space to correct for physical plausibility during execution in simulation with PD control. The proposed approach is compared against existing motion generation approaches with a set of quantitative metrics and qualitative evaluation.

Strengths

Overall this work soundly presents an improvement to SoTA in the space of human motion generation from text prompts. The use of guided diffusion to produce high-level trajectories is not surprising, nor is its effectiveness when combined with low-level skill models as a method of correcting physical implausibilities and errors. The paper is well written and clearly describes the architecture, training procedure, and use of guidance for waypoint following. Evaluation is sound and clearly demonstrates the strengths of the approach. I appreciate the use of multiple quantitative metrics and stochastic evaluation of many samples. This work benefits from the existence and availability of its component parts (Brax, ControlVAE, motion diffusion models), but connects them to achieve fairly good results. I’m always happy to see motion generation connected to physics simulation instead of remaining kinematic. The results of this work seem fairly robust to perturbation, which is a good signal for the usefulness of future iterations of the approach.

Weaknesses

While the resulting motion is fairly good, it isn’t yet achieving the level of fidelity to make it generally useful in generative contexts (e.g. game or animation characters). This work achieves its success from connecting other existing techniques, though the overall system is a sound contribution. The stack is trained considering only the character’s joint space and a specific character’s physical constraints.

Questions

Should cite Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion for similar architecture (guided-diffusion trajectories + low-level physical controller) https://arxiv.org/abs/2304.01893 Table 3 ordering of metrics is different from Metrics text section. Swap Multimodal Distance and FID in one or the other. Is it feasible to generalize this stack to different body shapes, dynamics, to include objects or other kinds of waypoints? Would it be possible/better to have some model-based or explicit low-level skill experts for specific sub-tasks (e.g. locomotion) to achieve higher fidelity in addition to general/open motion skills?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Discussion of limitations was pretty good and I appreciated the limitation notes throughout the paper. I did not see any mention of generalizing to other character morphologies (even different human body shapes). I currently believe this work is using a single embodiment as a strong assumption for achieving accuracy and physical plausibility. I would be interested to hearing more about the feasibility of generalization for applications.

Summary

The paper presents an approach for generating animations using a language-conditioned neural network in particular a diffusion model. The underlying goal is to generate physics based human character animation in an easy and intuitive fashion. The approach leverages a diffusion model to generate high-level states of the animated character which are then refined in a lower level module. This module is realized via an auto encoder which generates skill embedding representing the state transition in a compact fashion. A decoder is then used to synthesized the final action of the agent. The approach also uses an inpainting strategy to ensure that the agent goes through critical phases or points, i.e., this seems similar to the difference of keyframes and in-betweens in traditional animation. The paper compares the introduced approach to other recent methods on the topic, e.g., PADL and concludes among other things that more faithful animations (wrt to the original language query) are generated. The contributions of the paper is the combination of Diffusion and autoencoder models for motion generation, the inpainting strategy and waypoint system.

Strengths

The paper addresses a really interesting and important problem that is very exciting. It is also one of the first papers to use diffusion models in this context and the results seem promising. The combination of in-painting with diffusion is also very clever (but details are missing). A BIG plus compared to other similar systems is that it does not require RL. For example, PADL requires multiple policies to be learned and later combined with each policy requiring 7 simulated years to be trained! That does not seem to be the case here since this is a purely supervised approach - even though substantial computation is required for all deep learning of course. I really appreciated the use of contrastive models to evaluate the faithfulness of generated models. Diversity was also an interesting measure since in animation it is critical that the character shows a range of motions. Generally the paper is written in a way that can be followed (with caveats see below). ** Update after rebuttal ** I have increased my score to "weak accept". I appreciate the details provided by the authors - this helped better contextualize and understand the paper. What is clear from the rebuttal process is that it is hard to judge the visual fidelity alone, despite the substantial effort the authors put into the evaluation section. Especially the choice of visualizing the resulting animations via retargeting to a robot (in Blender) may have negatively impacted the reviewers' opinion. The animations in the latter part of the video are actually more convincing in my opinion. However, as pointed out before, the animations are not well-adapted to the environment, i.e., foot skating, actions on objects, etc. In future submissions on the topic, I strongly advise addressing this topic even if only partially.

Weaknesses

A major question that the paper is not addressing is the generation of actions on objects and surroundings. A critical element of any character animation is to be able to specify an object acted upon, e.g., "push the red object on the table". That is not addressed at all in this paper, while it is addressed on other papers in robotics (the manipulation papers) but also animation (PADL). In PADL, a module specifically identifies the target object and animations are synthesized accordingly. In the robotics papers [Lynch2020, Stepputtis2020] the actions of the agents are also conditioned on an image of the surroundings and target objects are visually identified in the image. Potentially that could be included into InsActor through the waypoint system (maybe?), but it is so far not addressed in the paper. Personally, this is the biggest flaw of the paper, since at the moment it would not be appealing to me or many others to use this system. Along these lines, the presented example results are not very convincing with regards to the complexity of the generated motions. Another weakness of the paper is that it seems important details are left out or not well justified. For example, the waypoint encoding is interesting but should be explained with an example. I am particularly unsure what this would entail for the user - do we have to place the character at those waypoints and set all of its joint values? This would require the user to actually become an animator and generate keyframes. That could be a serious limitation if true. Right now I am assuming we only need to position the basic character at the waypoint (i.e. modify position and orientation of agent) not moving limbs (i.e. modify joint angles). The paper states "...replace the Gaussian noise in the first and last 25% frames with the noisy states of the character standing at the starting position and target position respectively." Personally, I would really have liked to see a long sequence in which the agent performs a number of different actions in a row. At Siggraph they often have these cute teaser figures in which a humanoid walks, crouches, jumps etc. over a period of time.

Questions

- What exactly is the transition function? I am assuming that is the differential simulator, right?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

As described above, the lack interaction with the environment and linguistic goals is a major limitation that has not been brought up in the paper. The system is not able to execute commands like "lift the green cup" or "move to the lion statue". That substantially limits the applications of the approach. It is also surprising that the authors mention problems like foot skating at the beginning but do not describe how that is resolved later in the paper. While I understand that in a physics-based system the dynamics of the environment can make some of these considerations obsolete, there is still room for broken animations if this not adequately addressed. Hence, I would ask the authors to address this point in the paper.

Followup question

I would like to thank the authors for their thorough response to my points and questions. Could you please expand on why the results from MDM/MotionDiffuse (in Table 1 of the attached document) are not directly comparable to the output of "Planner" in Table 2? Both methods output a set of joint positions/rotations that can be used to compute the metrics, correct? Is it because InsActor use a different skeleton topology than the SMPL body used in MDM? Thanks!

Reviewer Response

I thank the authors for the detailed response! My concern about the motion quality remains. I understand the artifacts such as penetration, foot sliding, or wrong collision could be a result of Brax simulator, but the ones shown in the videos are very obvious and jarring. At a bare minimal the collision and penetration needs to properly modeled to ensure physically plausibility, otherwise what is the purpose of using a simulator? Claiming that the motion is physically plausible while the generated motion has large non-physical artifacts seems disingenuous. If the physics simulator (Brax) is the issue, then more investigation needs to be made or a different simulator used. I understand that DPS is under rapid development, but enjoying its benefits (being differentiable) while not considering/discussing its drawbacks seems to be misleading for the community. Purely diffusion-based methods for motion generation seems to be generating much more smooth and good-looking motion overall. Results shown from the MotionDiffuse paper seems to be better than the result shown here (00:45 and 01:54). What could be the cause? For the tracking error, I think acceleration error and velocity error, in addition to a MPJPE is needed for better a better picture. How many sequences can be tracked successfully?

Response to rebuttal

I would like to thank authors for the explanation. I agree that the main contribution of the work is a scalable pipeline for character motion synthesis, and I would keep my original **accept** recommendation. However, after discussion with other reviewers, I think the quality of character motion could still be improved (penetrating and jittering artifacts). Without improving the quality, I do not think computational efficiency or flexibility need to be prioritized. Therefore, I still encourage including trajectory optimization (potentially adding stronger contact constraints to reduce artifacts) to see if it would help improve the synthesis quality.