Social Motion Prediction with Cognitive Hierarchies

Summary

This paper proposes a novel approach to address the social motion prediction problem by introducing a new large-scale multi-person 3D motion dataset featuring intense and strategic interactions among participants. The authors formulate the problem using a multi-agent reinforcement learning perspective and incorporate behavioral cloning and generative adversarial imitation learning to boost learning efficiency and generalization. They also take into account the cognitive aspects of the human social action planning process and develop a cognitive hierarchy framework to predict strategic human social interactions. The proposed approach outperforms state-of-the-art methods in challenging long-term social motion predictions.

Strengths

The main strengths of this paper include: + Introducing and analyzing the limitations of existing datasets for multi-person motion prediction tasks. + Proposing the use of GAIL to improve the generalization ability of the learned policies. + Provide a demo video.

Weaknesses

**Good motivation and interesting methods, but not sufficient efforts to support the claim.** I will state weaknesses as follows: + Distribution comparison in Fig. 2 shows the comparison with CMU-Mocap, which is an old dataset. For claiming the dataset novelty, I suggest authors compare your dataset with more datasets here. Only several sub-figures of several joints are not sufficient to reflect the distribution of the full dataset. + For dataset comparison, [1] has 2.2 hours of data and [2] is 6.5 hours. These two datasets should be compared. + **Over-claim: I think a dataset of less than an hour cannot claim to be a large-scale dataset.** + In lines 29-32, why [56, 1, 25] cannot be used for interactive tasks? CMU-Mocap can be used to perform motion prediction and generation tasks. + Authors mentioned NTU-RGBD in Section 2.1 but did not compare with it in Table 1. + Writing: + orphan row (lines 233, 251, Table 3). + Line 236: Table 4 -> Figure 4 [1]: Umpm benchmark: A multi-person dataset with synchronized video and motion capture data for evaluation of articulated human motion and interaction. [2]: InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions.

Questions

**According to review guidelines, 'if the contribution is a dataset or model, what steps did you take to make your results reproducible or verifiable?' The authors did not claim it in the main paper or abstract (It is important. Do NOT claim it in the appendix). Will authors only make a subset of data public or a full set?** + Any ablation of $\lambda$s? + Multi-person motion capture datasets can be used not only for prediction but also for motion generation. Recent research (HumanMAC: Masked Motion Completion for Human Motion Prediction) use generative frameworks for motion prediction, and it will be great to discuss motion generation. + w/o GAIL and w/o mid GAIL ablation shows that the design works little for global pose and local pose. I would like to discuss this with the authors. + "This dataset surpasses the existing ones in scale, diversity, dynamics, and interaction, thus posing new challenges to the social motion prediction problem. " Little experiments to verify how the (1)scale, (2)diversity, and (3)interaction benefit the task? + Description of annotation, SMPL or SMPL-X? How many joints? + For motivation, why do authors build a dataset with intense, strategic interactions? Why are intense, strategic interactions essential for the task and the community? Intense, strategic interactions property is a characteristic of your dataset, but a lack of motivation. It will be great to show how these properties benefit the task in experiments. **Although I have many concerns about the dataset, the author's method of technological innovation is still interesting. I provide a borderline score here. If the author could discuss and address my concerns, I would improve my score.**

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are discussed.

Summary

The paper addresses the problem of social motion forecasting utilizing a multi-agent reinforcement learning and combines behavioral cloning and generative adversarial imitation learning. Social and strategic interactions are modeled in a “cognitive hierarchy framework”. The paper further introduces a new large-scale 3D multi-human motion dataset based on no-dribble-3-on-2 basketball.

Strengths

The paper is, in general, well-written and the method well-motivated. The novel dataset is highly relevant in the under-explored area of multiple human motion forecasting.

Weaknesses

In general, 1s of forecast for such complex social interactions is too short, as players mostly just continue their motion, e.g. swing of an arm. In basketball there is the 5s rule, I would thus suggest to at least forecast 5s of future motion to get a good understanding of the interactions and of the game. Instead of MPJPE metrics for long-term motion forecasting could be employed, e.g. [b,c]. The method results in the supplementary video exhibit a lot of foot sliding, suggesting a generalization problem for walking motions. I would assume that this is due to using 3D velocities as actions, which are translation invariant but crucially not rotation invariant. The choice of a rotation-variant actions is surprising as the introduced dataset (almost) plays out on a circle, with 3 players on the circle and 2 players within. Did the authors try to alleviate the issue by augmenting the data? The authors say that "Human motions exhibit greater dynamism in terms of pose diversity and movement speed, making motion prediction more challenging than in previous datasets” (LL42-44) but then say that players are not allowed to “dribble” (L111 & L115) - does that not significantly reduce the range of motion, i.e. there is little global translation? The authors should elaborate how they obtained body joint angles (L144), Figure 2. In Figure 1 it seems that the authors extract SMPL parameters as pose representation but it is never specified - however, extracting angular representations from 3D joint locations is non-trivial and the authors need to explain how those are produced. The lack of this information makes Figure 2 unconvincing: assuming SMPL was used, the skeletal structure is quite different from the parametric model used in CMU [1], we thus would expect significant differences in angular distributions. For example, hinge joints in CMU are zero everywhere except at the hinge dimension - is this the case in the authors representation as well? The paper should replicate the MRT [59] experiments - at least for the available datasets CMU Mocap and MuPoTS-3D. HRI [32] originally was trained on local (normalized) motion and not on global coordinates: in this work, are the global coordinates passed to HRI for training and evaluation? How would HRI perform with normalization such as subtracting the root location at the last input frame? Such normalization can be undone trivially and should greatly improve the (short-term) motion prediction results. The evaluation as is is unfair for HRI if global coordinates are passed. **Minor issues** LL54-55: While CHT [9] is cited and put into context (LL98-105) a short overview would be helpful, as it is not clear from the paper what the authors mean by “k represents the depth of strategic thought” (L55). An interesting dataset for evaluating the approach would be the Haggling dataset [a] where persons play a triadic haggling game, an adversarial game between two players to sell a product to a third person (buyer). This work should be at least cited and compared against in Table 1. In the context of 3D human motion forecasting planning or “intention” has been addressed as intermediate signal in previous works [c,d] and should be cited. [a] Joo, Hanbyul, et al. "Towards social artificial intelligence: Nonverbal social signal prediction in a triadic interaction." CVPR 2019. [b] Gopalakrishnan, Anand, et al. "A neural temporal model for human motion prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. [c] Tanke, Julian, Chintan Zaveri, and Juergen Gall. "Intention-based long-term human motion anticipation." 3DV 2021. [d] Diller, Christian, Thomas Funkhouser, and Angela Dai. "Forecasting characteristic 3D poses of human actions." Proceedings of the CVPR 2022. **Rebuttal** I appreciate the effort made by the authors to answer my questions. However, I still believe that 1s is too short for what the authors claim their dataset is capable of demonstrating: intricate "strategic" social interplay.

Questions

I wonder if it would be possible to evaluate the combined motion of all persons, e.g. not just of the per-person poses but also the global configuration of the poses is sensible or not. Imagine, for example, one person throwing the ball to their left-hand person but the right-hand person generates the "receive" motion - which would be unlikely. It is unclear which version of PoseTrack the authors address in Table 1: PoseTrack17 [23] contains 556 sequences (and not 13) and PoseTrack18 [5] contains 1138 sequences. Could the authors elaborate if this version is a subset of PoseTrack? I would assume this is an encoding error but in the supplementary video every 4 to 5th frame “freezes” making it a bit more difficult to check the results frame by frame in video (QuickTime Player, OSX). What was the motivation of color choice for Visual representation of the Cognitive Hierarchy Visualization: I would suggest to use warm colors for the offense and cold colors for the defense as especially purple and rose are too close to each other.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors address some limitations, notably the assumption that a certain cognitive hierarchy is present in the data. It would be interesting to see how far into the future the model can predict without producing failing poses or freezing up.

Summary

The paper has two major contributions to the field of multi-person motion detection: 1) A new open source dataset that is sufficiently large scale as compared to those already available. But more importantly is dense in terms of strategic interactions and more diversity of action distributions for reinforcement learning algorithms. This is very relevant to the field. 2) To demonstrate the efficacy of the datasets, they also propose a new initiation learning framework combining behavior cloning as well as GAIL.

Strengths

1) The paper is very well written with clear figures to make the concepts easy to understand. 2) The dataset is a welcome addition to the space of close space multi agent interaction modeling. 3) The experimental section is very detailed with an ablative section to help understand the implications of each of the contributions of the MARL framework.

Weaknesses

The paper suffers from two major issues. The authors ignore a large body of work that are not directly related to the close space interactions but highly relevant to the field of multi-agent trajectory prediction. This includes both datasets as well as methods available to get state of the art results in the domain. 1) Lack of survey of related work with respect to the dataset: Many work in the autonomous driving domain feature highly complex social scenarios with many agents interacting with one other. The datasets include industrial benchmarks like Argoverse, nuScenes, Waymo Open Motion Dataset [1] amongst others. While these datasets including not just pedestrian trajectories but also other road users like vehicles and cyclists, a comparison must be made about how the complex multi-agent crowd level interactions are different and similar to the close space interaction modeling that is being studied in this paper. The crowds of pedestrian in these scene perform complex social maneuvers of working cooperatively ( cross walks) and destructively (jaywalking) as examples and how these strategies are relevant to this field is an important comparison. Given the existence of these other much larger dataset in terms of number of subjects available, it will be important to understand what benefits the new dataset adds to the field. 2) Lack of survey of related work with respect to the methods: On the datasets mentioned above a large body of work exists involving both simple deep learning transformer and non transformer based Bayesian methods as well as reinforcement learning frameworks [4] (as an example) that the MARL framework needs to be compared against. [1] Waymo Open Motion Dataset [2] Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation

Questions

Would it be possible to include the relevant recent work in the field of multi-agent human trajectory prediction?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

NA

Summary

This paper aims to predict multiperson human motion. The main contributions of this paper are: 1. The paper presents a large-scale multi-human 3D motion dataset with intense, strategic interactions. 2. The paper formulates the multiperson human prediction problem as MARL and presents a hierarchy framework to model complex social interactions.

Strengths

The paper has several strengths: 1. Overall, the paper is well-written with a clear and well-motivated introduction. The hierarchy framework is introduced to consider the recursive decisions in the interaction. 2. The paper proposed a large-scale multi-person motion dataset 3. Evaluation on the proposed dataset demonstrates that the method outperforms state-of-the-art methods.

Weaknesses

1. In L32-34, the paper claims that most existing methods employ end-to-end supervised training which overlooks the cognitive aspects. Considering the state (past motion) and action (future velocity) space defined in this paper, could the author provide more insight on the difference between the BC, GAIL used in the paper and GAN-based supervised learning in related work? 2. While this work qualitatively demonstrates the utility of the proposed framework for tasks specific to motions like sports, it would be beneficial to provide additional comparisons on existing benchmarks in addition to Figure 4. 3. Other multi-person datasets in sports also exist, such as KTH Multiview Football datasets. The paper may compare the proposed dataset and method with them.

Questions

Please check 1. - 3. in Weaknesses. In addition, why was the best performance achieved when K=3? if the claims of the cognitive hierarchy are true, I was wondering what happened after having more levels of decision.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The visualization of the output from each level shows some existence of cognitive hierarchies, but it should be more clear if the paper and the demo video could include the visuals in meshes (with ball movement if possible) like Figure 1. Overall, the author's response to the concerns in the Weakness section is needed to make the final decision. I am happy to increase the rating if my concerns are addressed.

Summary

--

Strengths

--

Weaknesses

--

Questions

--

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

--

References (also appended to the PDF)

[R1] Van der Aa, N. P., et al. "Umpm benchmark: A multi-person dataset with synchronized video and motion capture data for evaluation of articulated human motion and interaction." ICCVW, 2011. [R2] Dang, Lingwei, et al. "Msr-gcn: Multi-scale residual graph convolution networks for human motion prediction." ICCV 2021. [R3] Yan, Rui, et al. "Social adaptive module for weakly-supervised group activity recognition." ECCV 2020. [R4] Li, Yixuan, et al. "Multisports: A multi-person video dataset of spatio-temporally localized sports actions." ICCV 2021. [R5] Ibrahim, Mostafa S., et al. "A hierarchical deep temporal model for group activity recognition." CVPR 2016. [R6] Ramanathan, Vignesh, et al. "Detecting events and key actors in multi-person videos." CVPR 2016. [R7] Felsen, Panna, Pulkit Agrawal, and Jitendra Malik. "What will happen next? forecasting player moves in sports videos." ICCV 2017. [R8] Kazemi, Vahid, et al. "Multi-view body part recognition with random forests." BMVC 2013. [R9] Ettinger, Scott, et al. "Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset." ICCV 2021. [R10] Igl, Maximilian, et al. "Symphony: Learning realistic and diverse agents for autonomous driving simulation." 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022. [R11] Chang, Ming-Fang, et al. "Argoverse: 3d tracking and forecasting with rich maps." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019. [R12] Caesar, Holger, et al. "nuscenes: A multimodal dataset for autonomous driving." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020. [R13] Joo, Hanbyul, et al. "Towards social artificial intelligence: Nonverbal social signal prediction in a triadic interaction." CVPR 2019. [R14] Liang, Han, et al. "InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions." arXiv preprint arXiv:2304.05684 (2023). [R15] Hanyue Tu, Chunyu Wang, and Wenjun Zeng. "Voxelpose: Towards multi-camera 3d human pose estimation in wild environment." ECCV 2020. [R16] Jianfeng Zhang, Yujun Cai, Shuicheng Yan, and Jiashi Feng. "Direct multi-view multi-person 3d pose estimation." NeurIPS 2021. [R17] Tevet, Guy, et al. "Human motion diffusion model." ICLR 2023. [R18] Tseng, Jonathan, Rodrigo Castellon, and Karen Liu. "Edge: Editable dance generation from music." CVPR 2023. [R19] Rempe, Davis, et al. "Humor: 3d human motion model for robust pose estimation." ICCV 2021.

References (also appended in the General Response)

[R17] Tevet, Guy, et al. "Human motion diffusion model." ICLR 2023. [R18] Tseng, Jonathan, Rodrigo Castellon, and Karen Liu. "Edge: Editable dance generation from music." CVPR 2023. [R19] Rempe, Davis, et al. "Humor: 3d human motion model for robust pose estimation." ICCV 2021.

[EMERGENCY] Reviewers can not see the rebuttal.

Dear PCs, ACs, and reviewers, We are writing regarding a potential issue with the visibility of our submitted 'Author Rebuttal' in OpenReview (Submission Number: 534). **One of the reviewers, Reviewer yiis, has mentioned in a recent communication that they were unable to see our rebuttal.** **We have confirmed on our end that the 'Author Rebuttal', including a general response as well as individual responses to each reviewer's comments, has indeed been submitted and should be visible to the 'Reviewers Submitted' group.** We kindly ask for your assistance in resolving this matter. It is essential for us to ensure that all reviewers have access to our responses, as we believe the rebuttal provides important context and clarification to our paper. We understand that these are technical issues that can occasionally arise and we appreciate your support in addressing this. Please do not hesitate to contact us if you need any further information from our end. Thank you for your attention and assistance in this matter. Best regards, Paper 534 Authors

I read other reviews and comment here

I read other reviews and list my comments to authors, other reviewers, ACs, and SACs. **For Reviewer Nb5i** + For predicting 1s concern, I do not think this is a weakness. Prediction of too-long motions is a generation task exactly. therefore, I suggest authors perform generation results in the experiments, which is a better pretext task. + I have the same concern on annotation, like SMPL(-X), ball, and equality (foot sliding). **For Reviewer 8iac** + For `Lack of survey of related work with respect to the methods`, can these works be compared with your methods as baselines? --- Moreover, I check that the authors clicked the Reproducibility as YES, and I am not sure whether authors should provide (inference) codes in the submission. If not provided, it might be reported to the ethics reviewer.

General Response (Repost)

We sincerely thank all reviewers for their meticulous reviews and constructive remarks. It is heartening to note that all reviewers have acknowledged the motivation behind our work, as well as its main contributions — especially the proposed method. We will incorporate a discussion of relevant works and additional experiments in the final version accordingly. In the following, we will first address some shared issues on the proposed dataset, then respond to each reviewer's comments respectively. **Q1: Dataset comparisons** A1: We discuss the existing multi-person motion datasets in Section 2.1 and provide a comparison with some of them (Table 1 of the submitted manuscript). The table contains comparisons with _'existing multi-person motion datasets employed by previous works on the multi-person motion prediction task'_ (table caption). In other words, we primarily compare with the utilized datasets within this domain as referenced in previous works [2, 3, 58, 59, 19, 54, 40, 63]. We recognize that there are other multi-person datasets that are relevant to our work, as recommended by the reviewers. Consequently, we present a brief discussion on the differences between our work and each of these datasets below. **These works will also be cited and discussed in Section 2.1 and Table 1 of the revised manuscript following your suggestions.** - **Multi-person Sports Datasets (X2WW).** There exists a variety of multi-person sports datasets for group behavior understanding, e.g. NBA [R3], MultiSports [R4], Volleyball [R5], NCAA [R6]. However, these datasets are primarily designed for action recognition from RGB videos and lack pose annotations or 3D information. Felsen _et al._ [R7] collect a water polo dataset and a basketball dataset for group trajectory prediction, but they only contain trajectory-level data rather than human poses. The KTH Multiview Football dataset [R8] consists of two subsets (2D and 3D), with the 3D subset containing merely 800 frames (32 seconds) for two isolated players. Therefore, it might be more suitable for testing or few-shot adaptation. - **Multi-agent trajectory prediction datasets in autonomous driving (Reviewer 8iac).** As suggested by the reviewer, there is a large body of work on multi-agent trajectory prediction that is _'not directly related to'_ our task, but _'highly relevant'_. We briefly discussed this research area in Line 20-26. Given the different task objectives, the datasets also exhibit different characteristics. The trajectory datasets [R9, R10, R11, R12] possess advantages such as larger scale and diverse classes of agents. Nevertheless, these datasets only contain coarse-grained interactions at the trajectory level (mainly collision avoidance) and fail to capture the rich and fine-grained human actions. Furthermore, our dataset features intense and strategic interactions, such as frequent sudden direction changes and deception maneuvers, which are largely absent from existing datasets. Thus, our dataset can serve as a valuable supplement to the existing ones, posing additional challenges to this research field. - **Haggling dataset [R13] (Nb5i)**. The Haggling dataset shares relevance with our work as that it also invloves social interactions and body gesture prediction. The main difference is that they focus on a triadic conversational scenario, therefore the interactions are relatively simpler, and the body motions are more static. - **UMPM benchmark [R1] (yiis).** The UMPM dataset is a multi-person mocap dataset. As per the paper, 2/9 of the scenarios contain interaction between the subjects and is suitable for our task: _(6) a conversation with natural gestures, and (7) the subjects throw or pass a ball to each other while walking around_. Compared to it, our dataset is advantageous in terms of a larger number of subjects and more strategic interactions. We also conduct experiments on an extended version of UMPM following [40] and the results are reported in Table 1 of the rebuttal PDF. - **InterGen [R14] (yiis).** This concurrent work introduces a multimodal dataset called InterHuman for text-to-multi-human motion generation. Consequently, their primary objective is to cover a diverse range of interactions and annotate with extensive textual descriptions. In contrast, the focus of our dataset is intense and strategic human interactions for social motion prediction. We believe that both our dataset and InterHuman offer valuable additions to the research community, albeit with different task focuses. **Q2: Data format** A2: Our work primarily focuses on 3D body keypoint sequences following previous studies [59, 19, 54, 57, 40, 32], as discussed in supplementary material Line 26-28. The 3D skeletons are also the standard product of markerless multi-view motion capture methods [64, R15, R16] which are more suitable for our data collection scenario than marker-based ones. We use $J=15$ skeleton joints with the same definition as previous works [40, 59].

Response to Reviewer EKS7 (Repost)

Thank you for taking the time to provide your feedback. However, we noticed that the specific details of your review were not included. To better understand your viewpoint and address any potential issues, it would be immensely helpful if you could provide more detailed feedback.

Response to Reviewer X2WW (Repost)

We sincerely appreciate the reviewer's constructive and positive feedback. We are grateful that you recognize our main contributions and find it 'well-written with a clear and well-motivated introduction'. In the following, we aim to address your questions and concerns: **Q1: Provide more insight on the difference between BC, GAIL and GAN-based supervised learning.** A1: Wang _et al._ [57] initially introduces an imitation learning formulation for single-person human motion prediction with BC and GAIL compared to end-to-end supervised training. Our work extends conceptually from [24, 57] as it adapts BC and GAIL for a multi-agent imitation learning context. Our approach is distinct from GAN-based supervised learning in two primary ways: - The GAIL objective focuses on making the state-action pairs produced by the joint policies of agents and that of experts indistinguishable. This is in contrast to the GAN framework (_e.g._, [59]), where the objective is to make the predicted future motion indistinguishable from the real ones. - Our MARL formulation enables a more principled integration with the cognitive hierarchy theory, while existing works overlook the cognitive aspects of human social action planning. By modeling the motion prediction process with chained policy networks, we derive the BC and GAIL regularization for each level accordingly, and propose to share parameters for policy networks $\phi_{(1)} \dots \phi_{(K)}$, both of which lead to performance improvements as shown in Table 3 of the submitted manuscript. **Q2: It would be beneficial to provide additional comparisons on existing benchmarks.** A2: Following your suggestion, we train and test our method on the CMU-Mocap (UMPM) dataset [40]. Please refer to Table 1 of the rebuttal PDF for the results. Our approach outperforms the previous methods [32, R2, 59] and is comparable with concurrent work [40]. Notably, our method exhibits particular strength in predicting long-term root trajectories and global motions. **Q3: Other multi-person datasets in sports also exist, such as KTH Multiview Football datasets.** A3: Please refer to General Response Q1 for the discussions. We will make it more clear in the revision. **Q4: Why was the best performance achieved when K=3?** A4: Our cognition-inspired framework aims to learn joint policies from real-world expert demonstrations and to discover the explainable cognitive hierarchies. The finding that $K=3$ provides the best fit for the real-world dataset may suggest that this particular depth of strategic reasoning most aptly encapsulates the human decision-making process as captured in our collected data. As we increase the number of hierarchical levels ($K>3$), we continue to observe the cognitive hierarchy visualizations among different levels, though there is a slight increase in the mean prediction error. **Q5: Could include the visuals in meshes (with ball movement if possible).** A5: Thanks for your suggestion! - **Body Mesh**: As discussed in General Response Q2, we use 3D poses in the experiments. While the 3D poses can certainly be fitted to a parametric body mesh (such as SMPL, as shown in Figure 1) for a more intuitive visualization, this process may introduce additional errors. To avoid these potential complications, we chose to present the 3D skeletons as they are -- the direct output of the model, free from potential distortions introduced by fitting. This approach makes it easier to identify potential issues. However, in response to your suggestion, we will include the mesh fitting and visualization scripts in our code release. - **Ball movement**: We implemented a 3D ball trajectory estimator using an object detector and multi-view triangulation. However, we chose not to include the ball information in model training for two reasons: 1) to ensure a fair comparison with previous methodologies, and 2) due to the fact that the ball moves significantly faster than humans, which sometimes result in inaccurate estimations caused by motion blur and detection failures. We are committed to improving our ball detection algorithms, and in response to your suggestion, we will include the 3D ball trajectory estimator and visualization tools in our code release.

Response to Reviewer 8iac (Repost)

We sincerely appreciate the reviewer’s constructive and positive feedback. We are happy that you find our paper 'well-written', our proposed dataset 'a welcome addition', and our experiments 'very detailed'. In the following, we aim to address your questions and concerns: **Q1: Lack of survey of related work with respect to the dataset.** A1: Thank you very much for the suggestion! We provide a discussion on the multi-agent trajectory prediction datasets in General Response Q1. We will cite and discuss these works in Section 2.1 in the next revision following your suggestion. **Q2: Lack of survey of related work with respect to the methods.** A2: Thank you very much for the suggestion! We referred to the relevant recent work in the field of multi-agent human trajectory prediction in Line 20-26. We will cite and discuss more research works in this area in Section 2.2 in the next revision following your suggestion. By the way, is part of the review references ([4]) missing?

[1/2] Response to Reviewer Nb5i (Repost)

We sincerely appreciate the reviewer’s constructive and detailed feedback. We are grateful that you find our paper 'well-written', our proposed 'well-motivated', and our dataset 'highly relevant in the under-explored area'. In the following, we aim to address your concerns: **Q1: 1s is too short.** A1: We appreciate the reviewer's insightful suggestions. We understand the importance of longer-term predictions in understanding complex social interactions and the dynamics of the game. However, we would like to provide the following points to clarify our choice of a 1-second forecast horizon: - Basketball is characterized by its swift pace, whereby a significant amount of action can occur within a 1-second timeframe. In full-court matches, possession often concludes within few seconds. Our research scenario is even more intense and compact compared to conventional basketball games, and it operates at a faster pace. - Our video demonstration reveals that the 1-second future doesn't merely continue past motions. Instead, it involves defensive switches and sudden changes in moving directions. As a matter of fact, our level-1 policy is primarily motion continuation, but our policy network learns complex strategic behaviors at higher levels through the proposed cognitive hierarchy design. - We opted not to use a longer prediction horizon primarily because future motions heavily depend on the uncertain outcomes of passes. For instance, if player A is passing to player B, the subsequent motions will be greatly influenced by whether the ball is successfully received by B or stolen midway. Predicting such outcomes based on current information poses a considerable challenge for any model. We plan to extend our datasets to include regular basketball games with more information and explore longer prediction lengths for future research. **Q2: Foot sliding.** A2: Foot sliding might be partly due to our model not explicitly regularizing foot contact velocities as previous works have done [R17, R18, R19]. Our research primarily targets predicting the strategic action decisions rather than the quality of the motion itself. Hence, we did not initially design our model to address these detailed aspects of the motion quality. Based on the reviewer's suggestion, we plan to explore training techniques such as foot contact loss and data augmentation to enhance the visual quality of the motion predictions. We believe these techniques could potentially alleviate the foot sliding issue. **Q3: Little global translation?** A3: The no-dribbling rule in this practice enforces the players to make more strategic decisions. While dribbling is prohibited for offensive players, it doesn't necessarily result in a significant decrease in the overall range of motion. It's important to note that defensive players are not under the same restriction. They are required to continually move back and forth to maintain adequate defense coverage. This requirement leads to substantial global translation, as evidenced by the video we have provided. Therefore, while the scope of certain specific motions is reduced, the overall diversity and dynamism of human motions are preserved, and the challenge of predicting those motions remains high. The large root error (especially for the 'frozen' baseline) in Table 2 of the submission also demonstrates this idea. **Q4: Body joint angles.** A4: As discussed in General Response Q2, our dataset uses 3D pose representation. Previous work MRT [59] converts the CMU-Mocap dataset to a format of 15-joint 3D poses. We strictly followed its definition and perform data analysis on the aligned skeletal structure. We directly computed the angle of intersection formed by the hinged limbs since we do not have the limb twists of the 3D pose representations. **Q5: Replicate the MRT experiments.** A5: Following your suggestion, we train and test our method on the CMU-Mocap (UMPM) dataset [40]. Please refer to Table 1 of the rebuttal PDF for the results. Notably, our method exhibits particular strength in predicting long-term root trajectories and global motions. **Q6: Details on HRI.** A6: When training and evaluating HRI, we pass in normalized motions following the original settings. Therefore, it achieves respectable short-term motion prediction accuracy in Table 2, as supposed by the reviewer. **Q7: CHT overview.** A7: We provide an intuitive explanation of CHT and level-k reasoning in Line 51-55 and Line 195-197. We will add a short overview to Section 2.3 following your suggestion. **Q8: Haggling dataset.** A8: Please refer to General Response Q1. We will cite and compare with it following your suggestion. **Q9: Intention as intermediate signal.** A9: Thank you for the information, we will add them to Section 2.

[1/2] Response to Reviewer yiis (Repost)

We sincerely appreciate the reviewer's constructive and positive feedback. We are grateful that you are interested in our 'method of technological innovation' with 'good motivation'. In the following, we seek to address your concerns: **Q1: Compare with more datasets and joints.** A1: We primarily compare with CMU-Mocap [1] as it is the main dataset used by previous works in this area. We will add more dataset comparisons with other datasets and joints following your suggestion. **Q2: UMPM and InterHuman dataset.** A2: Please refer to General Response Q1 for comparisons. We will cite and compare with the datasets following your suggestion. **Q3: Overclaim data size.** A3: Thanks for your valuable feedback on our dataset size claim. Nonetheless, it is important to note that the dataset is larger than any previously used in the field of multi-human motion prediction. Our dataset provides 3D skeleton ground truths with intense and strategic human interactions with professional athletes. From such a perspective, we have greatly enlarged the available dataset in this research field for academic uses. We understand how the term "large-scale" might be misconstrued given the temporal length of the dataset. Therefore, in our revised manuscript, we will carefully modify the wording to more accurately represent the size and value of our dataset. **Q4：Why cannot be used for interactive tasks.** A4: While it's true that these datasets can be used for motion prediction and generation tasks, their utility for tasks involving interaction - a key focus of our research - is limited. This is due to the relatively low level of interactiveness, as we mentioned in Line 30-32. For instance, the individuals in these datasets are often recorded while moving casually or interacting randomly with others, rather than participating in strategic or purposeful social interactions. **Q5：NTU-RGB+D.** A5: NTU-RGB+D contains some action classes with two interacting humans. We provide a comparison in Table 3 of the rebuttal PDF. We will add the comparisons to Table 1 in the revision. **Q6: Dataser release.** A6: The entire dataset, as well as our code and models, will be publicly released. **Q7: Ablation of $\lambda$s.** A7: We provide an ablation study on $\lambda$ in Table 2 of the rebuttal PDF. **Q8: Discuss motion generation.** A8: Thanks for your suggestion, and our dataset can also be potentially used for motion generation. We will cite and discuss HumanMAC and other related works on motion generation in _Introduction_ and _Related Works_ sections of our revised manuscript. **Q9: GAIL effects.** A9: Thanks for your insightful comment. In our manuscript, we have briefly addressed this observation in Line 275-276. Training without GAIL tends to increase the errors in long-term trajectory predictions, while it marginally reduces the errors in local pose prediction. Interestingly, we notice that models trained without GAIL tend to generate motions with smaller magnitudes, often resembling a "freezing" effect. This results in visually less appealing and less realistic animations. Moreover, an important aspect to note is the impact of GAIL on the emergence of cognitive hierarchies within the model. In our experiments, the absence of GAIL leads to a lack of discernible cognitive hierarchy. **Q10: Verify how the (1)scale, (2)diversity, and (3)interaction benefit the task.** A10: Thank you for bringing up the important point about the benefits of the scale, diversity, and interaction of our dataset for the social motion prediction task. We are pleased to clarify these aspects: - Scale: Larger datasets are generally considered beneficial for research tasks as they provide a more extensive base for training models, improving their ability to generalize to unseen data. The advantageous size of our model makes it a valuable resource of the task for the research community. - Diversity: As shown in Figure 4 and discussed in Line 235-242, our dataset exhibits a high degree of diversity. The results demonstrate that the proposed dataset serves as a more convincing benchmark for social motion prediction and introduces more challenges to the field. - Interaction: As evidenced by Table 2 and Line 245-249, while the HRI [32] model performs exceptionally well on short-term motion prediction, the global context becomes crucial for long-term motion prediction. Our dataset, with its highly interactive nature, offers strong cues for motion prediction with a global context. These interactions are vital to understanding and predicting complex social behaviors. We appreciate your feedback and will revise our manuscript to articulate these points more clearly. **Q11: Annotation.** A11: Please refer to General Response Q2. We will make it more clear in the revision.