InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint

Summary

This paper focuses on text-to-motion synthesis, specifically, to generate motion of multiple interacting people. The method can potentially work with an arbitrary number of people and models inter-person interactions as pairs of joints that can be either in contact or not, separated by a certain control distance. The method is composed of two main stages: a LLM planer and a diffusion model based on Human Motion Diffusion Model (MDM) that incorporates a motion controller, called Motion ControlNet, and an inverse kinematics (IK) module to have finer control over the joints of generated motion sequences and, thus, generate interactions between synthesized characters. The authors show that the distance between pairs of joints for interactions can be generated with off-the-shelf LLMs. A key feature of this work is that it is zero-shot, thus, the models used do not need to be trained with multi-person motion datasets. However, as it uses a motion controller (based on the ControlNet paradigm) which is fine-tuned to generate global pose sequences (global motion) as opposed to relative motion typically generated by SoTA methods such as MDM. This would be the first method to enable a single-person text-conditioned motion generation for generating interactions for multiple people. The paper includes comparisons with the SoTA using standard metrics (FID, R-precision, Diversity, Foot Skating, Trajectory error, Location error and Average error). Authors compare with single-person motion generation methods (MDM, GMD, OmniControl). For the two-person setting the paper includes a comparison of Spatial Errors and a user study comparing with PriorMDM.

Strengths

* In part, the paper is clear and well-written. * The proposed InterControl framework is meaningful. * Although comparisons with multi-person motion generation models is not comprehensive mostly due to differences in training datasets used (which is understandable), the paper clearly shows an improvement compared to the priorMDM model, both quantitatively and qualitatively.

Weaknesses

### **Method section needs a bit of re-writing** The method section is somewhat confusing especially in the beginning. There are some definitions that the reader needs to make assumptions about in order to continue reading. For example: - There is no overview of the method in this section and even if Fig. 2 gives an overview, there is no link to it in the text. I would advise the authors to add this after the formulation and preliminaries so that the reader can easily understand the overall architecture. - L154: It is not so obvious to me what this discrepancy between relative and global motion is precisely and why it makes priorMDM less controllable in global space. I understand that generating global motion poses different challenges than relative motion, but could the authors explain this further? Doesn't it boil down to generating the global trajectory and then coupling/refining the global motion so that it is coherent with this trajectory as they do in [1]? - L157: While it may be considered as concurrent work, I suggest the authors discuss a bit [1] which also addresses this issue. - L174: At this point it is not clear to me what exactly are the spatial conditions. Authors could clarify with a simple example. - L175+: At this point, it is not clear why the control signal c is potentially sparse? Is it because it comes from text? This is hard to understand as there is no overview pointing out that the method uses a “Contact Plan” based on a LLM. - L177: I don't understand why the control will be desired for only a select few frames. Don't we want this control all the time? But again, authors may need to first clarify what exactly is this control signal and where does it come from. ### **Additional baselines** I believe that authors did not include other multi-person motion generation baselines because those explicitly learn from multi-person data. However, I am curious to know how different the results from the InterHuman method differ from this approach? References: [1] Zhang, Siwei, et. al, RoHM: Robust Human Motion Reconstruction via Diffusion (CVPR 2024)

Questions

Aside from the questions posted in the Weakness section, I have the following: ### **Need for clarifications** - I don't fully understand how the model generates the motion for all people. Is it done first individually and then merged somehow? Does the method need to run one MDM+Motion ControlNet instance per person? - From the qualitative results, it seems to me that I see the effect of an optimization post-processing step after the generation? Do the authors use a post-processing step with the IK or is this the effect of the guidance by the IK module? - Is the IK module guidance applied both at the end of each denoising step and also after $x_0$ is obtained from the diffusion model? - To apply the IK module guidance, does the method use the gradient information from the optimization steps to influence or does it apply directly on the estimated poses in a similar fashion as in PhysDiff? ### **Additional comments** I would suggest including a discussion on the Inter-X dataset at the first paragraph of the introduction. Even though this work is concurrent and this is not strictly necessary, this can make the paper stronger and more up-to-date. - L34: Could the authors expand a bit about which methods ignore a good design? - L38: Could the authors include a counter-example for such interactions that do not require 'additional interaction data'. Are these interactions closer interactions such as hugging or, the opposite, more subtle interactions such as talking or holding a meeting? - L108: I would suggest adding a comment about the method proposed by InterHuman which is also based on the MDM architecture. ### Minor (typos and grammar) - L155: "is not able" should be “are” instead of “is”. - L160: is proposed--> proposes - L166: is “almost” a random noise to be sampled. Almost? Why almost? $x_T$ should be an isotropic Gaussian at T. - L187: desperately needs a comma, otherwise the sentence completely changes meaning!: "(FK) to convert" -->"(FK), to convert"

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes, the authors include limitations.

Summary

This paper tackles the challenge of generating human interaction motions involving a flexible number of characters. To simplify the representation of these interactions, the authors propose a joint-pair contact/avoid representation. Given an interaction description, a large language model (LLM) generates motion instructions and identifies contact joint pairs. These outputs serve as inputs for the subsequent motion diffusion model. The Motion ControlNet is trained to produce motion sequences based on the motion instructions and to optimize the joints to align with the specified joint contact pairs.

Strengths

1. This paper is addressing an important human interactions learning task that is not well studied due to the lack of multi-person interactions datasets, and this paper suggests a framework that only leverages the single-person dataset to produce reasonable multi-person interactions. 2. The joint-pair distance representation of human interactions are simple and the IK-guidance shows to be effective. 3. The manuscript is well-written and easy to follow.

Weaknesses

1. It is not clear how does the contact joint pairs for each frame are generated by LLM. A joint jump usually presents when the contact between two humans happens, and what can be this unsmooth motions artifacts result from? 2. I still observe that there are still severe human collision happens in some generated human interactions. The joint pair contact type produced from LLM might not be fine-grained enough to produce natural and plausible interactions. The followed physical simulator tracking step might mitigate the collision artifacts a bit, but still it needs better spatial control over body roots to avoid severe collisions during motion. 3. The iterative IK guidance based optimization happens at each diffusion step, and this might result in less efficient framework. Might directly optimise the noise like done in DNO [1] or a single-step post-optimisation also work? [1] Optimizing Diffusion Noise Can Serve As Universal Motion Priors, CVPR 2024

Questions

This paper is addressing an important task - human interaction generation, by only leveraging single-person dataset. In addition to some of my questions presented in the weakness section, I have a few general questions here, and I am glad to hear some feedback or insights from authors: 1. If we want to scale up to more involved human characters * How about the efficiency at inference time of the proposed pipeline? * Would LLM planning still be effective and fine-grained enough for multiple-person interactions? 2. It would also be insightful to see the comparison with Intergen or fine-tuning on InterHuman dataset as well in the two-person interaction scenario.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have addressed the limitations.

Summary

The paper introduces InterControl, a method designed to address the task of controllable human motion generation and zero-shot human interaction generation. By leveraging the prior knowledge of LLM, InterControl can generate human interactions involving any number of people in a zero-shot manner. Specifically, the authors utilize LLMs to convert textual descriptions of interactions into contact plans, transforming the task of multi-person interaction generation into single-person motion generation. The InterControl model, which is based on Motion ControlNet and IK Guidance, is then used to achieve controllable single-person motion generation. Experimental results demonstrate the superiority of InterControl over previous controllable motion generation methods and its ability to produce realistic human interactions.

Strengths

- The proposed task(zero-shot human interaction generation) is interesting, important, challenging, and meaningful. - The related work section is great, providing an excellent summary of relevant research. - The proposed method makes sense and is logically sound. - The results are reasonable. I particularly appreciate the application section, where the character interactions in the simulation are executed very well.

Weaknesses

- The qualitative results are not really good. a) The supplementary video shows some cases where some joints have abrupt velocity changes, which conflicts with L232: "IK guidance can adaptively modify velocities from the start to frame n." b) There are some cases of body penetration and unrealistic interactions in the three-people-fighting/two-people-fighting scenarios. - It might be beneficial to incorporate some visualization of the contact plan. According to Sec 3.1, contact should include both contact and avoid forms, described by distance d. However, in the supplementary demo, I only observe d=0 contact. I would like to see some examples of avoidance and how non-zero distance contact plans guide motion generation. - Maybe more discussion is needed on the differences between omnicontrol and intercontrol, as they are similar methods (both of InterControl/OmniControl use ControlNet-like framework and classifier guidance). The author may want to discuss if using OmniControl as the control yields better results in the task of zero-shot human interaction generation? - The paper didn’t compare InterControl with data-based methods such as InterGen. The author may want to demonstrate the superiority of InterControl as a zero-shot method, possibly by highlighting interactions that InterGen cannot generate (2 people interaction). - It might be beneficial to include some qualitative comparisons between InterControl and state-of-the-art methods(e.g. OmniControl) in the task of Single-Person Controllable Motion Generation to better demonstrate the superiority of InterControl.

Questions

- What is the corresponding text for the supplementary material? - Can the LLM really provide a reliable contact plan?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Although the authors included a "Limitations'' subsection in Sec.5, from my perspective, they did not clearly claim the limitations of the method. The authors may want to point out that the realism of the interactions generated by InterControl heavily depends on the LLM's correct interpretation of the text, which cannot be guaranteed.

Summary

### Summary The paper "InterControl: Generating Human Motion Interactions by Controlling Every Joint" aims to generate interactions between multiple people based on text descriptions, with precise joint control. It leverages a pre-trained single-person motion diffusion model and extends it to multi-person scenarios using a large language model (LLM) to guide joint interactions. The methodology involves: - Using an LLM planner to map out interactions and distances between joints. - Combining priorMDM with ControlNet, conditioned on LLM-extracted distances. - Optimizing interactions with LGBFS to enhance performance. Evaluations are conducted using Text2Motion datasets and user studies, focusing on joint control and positional accuracy. The approach also assesses 100 interactions from the InterHuman dataset. ### Contributions 1. **Dynamic LLM Guidance**: Introduces LLMs to guide joint interactions dynamically, making the process scalable and suitable for large-scale data generation. 2. **Precise Joint Control**: Allows precise control over any joint, improving flexibility compared to methods requiring predefined control signals. 3. **Multi-Person Interaction**: Extends a single-person model to handle multi-person interactions, demonstrating scalability. 4. **Enhanced Visualization**: Suggests using human body models like SMPL or GHUM for better visualization and interpretability of interactions. 5. **Comprehensive Evaluation**: Provides thorough evaluation and comparison with previous models, addressing issues like interpenetration and alignment between control and text prompts. While the paper presents promising results and tackles the challenging task of generating realistic multi-person interactions, it also identifies areas for further improvement, such as better handling interpenetration and improving alignment between spatial control and text conditions.

Strengths

- The use of large language models (LLMs) to dynamically guide joint interactions introduces a scalable and innovative approach to multi-person motion generation. - The method allows for precise control over any joint at any time, enhancing flexibility and improving upon previous methods that relied on predefined control signals. - The approach successfully extends a single-person motion model to generate realistic interactions among multiple people, demonstrating its scalability and applicability to complex scenarios. - The paper is well-written and easy to follow, with a clear presentation of previous works and a strong introduction that provides good context. - The simplicity of the IK guidance and LLM-generated contact points helps automate the generation process, which is beneficial for large-scale data generation pipelines. - The paper includes comprehensive evaluations and comparisons with existing models, showing that the results for single-joint control and multi-person interactions are indeed better in several aspects. - The integration of ControlNet and LGBFS optimization with LLM planning is a novel contribution, pushing the boundaries of 3D human motion generation.

Weaknesses

- The contributions of the paper seem limited, with similarities to existing works like OmniControl and GMD. - The generated motions often suffer from interpenetration, which reduces the realism of the interactions. - There are discrepancies between the spatial control and text conditions, suggesting that the alignment between them needs improvement. - It is unclear how ControlNet is finetuned, especially regarding the use of the HML3D dataset and the extraction of necessary control features. - A simpler baseline for generating each person’s motion and optimizing interperson distances is missing, which could highlight the necessity of additional modules. - The qualitative results lack sufficient visualization, making it difficult to assess the plausibility of interactions from the provided videos. - The method focuses on multi-person motion generation but only evaluates single-person text-to-motion datasets, missing evaluations on available multi-person datasets. - The use of 3D joint locations without a human body surface makes it hard to judge motion plausibility and perceive contacts between people. - The presentation lacks clarity in some sections, particularly in explaining complex steps and figures, such as the Gaussian noise issue and Figure 2. - The approach relies on LLMs for joint distances, which may hallucinate content and lead to errors in generating plausible contact maps. - The work does not include suitable multi-person motion capture data during training, limiting the robustness of the interactions modeled. - The focus on joint-to-joint contacts overlooks other types of human contact, such as grabbing an arm, which are not modeled effectively. ___ Small notes - The teaser is misleading as it suggests conditioning on images for interaction generation. - The claim that TEMOS performs worse than MDM is not substantiated; TEMOS actually performs better in some cases.

Questions

### Questions and Suggestions for Improvement (importance sorted) **Evaluation and Results**: - Expand evaluations to include more comprehensive datasets, especially those involving multi-person interactions, and provide a detailed performance analysis. - Provide an analysis of the LLM's error rates in generating plausible contact maps and discuss how this impacts the model's performance. - Evaluate the model on available multi-person datasets and collect text descriptions for these datasets to provide a thorough assessment. **Methodological Improvements**: - Address the issue of interpenetration in generated motions by refining IK guidance and adding constraints to improve realism. - Elaborate on the process of fine-tuning ControlNet on the HML3D dataset and extracting necessary control features. - Implement and compare a simpler baseline that generates each person’s motion using existing methods and optimizes interperson distances using LGBFS. **Clarification of Contributions**: Clearly distinguish the paper's contributions from existing works like OmniControl [49] and GMD [26]. Highlight the novel aspects and how this work advances the field beyond these prior studies. **Visualization and Presentation**: - Use human body models like SMPL for better visualization of motions, facilitating better perception of contact and interaction. - Include mesh-based visualizations and use original AMASS skeletons for training to enhance interpretability and clarity of results. - Improve the clarity of the presentation, particularly in explaining complex steps and figures, such as the Gaussian noise issue and Figure 2. By addressing these points, the paper can provide clearer contributions, ultimately strengthening the overall quality and impact of the work.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The Limitations are discussed.

Feedback

- ### Comparison with InterGen InterGen is dataset-based method, so it may not be able to generalize to unseen interactions. Is this interaction within its scope and training set? Since you are not training in their dataset, it would be nice to provide some more details in this comparison. Your generalizability is clearly superior, but comparisons should be fair. - ### OmniControl Indeed your method performs slightly better than OmniControl. However, the IK-guidance could results in such better interactions. OmniControl results seems close to yours. Are you visualizing the results before or after IK-LGBFS optimization? What happens if you apply the same optimization to OmniControl? - ### Teaser I have read the rest of the reviews and haven't seen any agreement on the teaser's relevance/appropriateness. Showing images is not so scientific since this is neither your contribution or input to the model. The input are text descriptions and your not guiding your interactions through poses. Hence, this is not even in the scope of this work. It could be a potential application after some retrieval method. Teaser images are normally used as a gist of a method and should avoid overstating things. - ### Ablating LLMs I manually checked your plans and seem quite accurate in general. However, it would be nice if you added some extra analysis on failure cases or some -at least empirical- discussions on it. - ### 3D joint locations without a body surface makes it hard to judge motion plausibility and perceive contacts. I kindly disagree that joint locations is common representation for human motion. The largest database (AMASS), which most of the works benefit from adopts SMPL rotations as a representation which enables using a full body mesh. The adoptions of joint positions and non-SMPL rotations from users of datasets such as HML3D and the post-processing optimization they commonly performs is a technique used by such works for text-to-motion, but there are other works which predict SMPL rotations and meshes directly. TUCH (CVPR 2021) highlights the importance of the human body surface for self, and person-to-person interactions. I would expect retraining MDM with SMPL rotations - done in STMC(https://mathis.petrovich.fr/stmc/) - and using such features to control the generated motions. ### Final Remarks The zero-shot manner that is used for the interactions is nice. The proposed LLM-based planning is clear and seems to work fairly well. - The surface of the human body is the appropriate way for this to be shown and achieved. - Except from qualitative comparisons, is there any quantitative comparisons on the datasets proposed in InterGen? - I suspect that OmniControl which such IK guidance could perform similarly. Is that the case?