Social-Transmotion: Promptable Human Trajectory Prediction

Thank you for your valuable feedback. Below are the responses to your comments. > Utilizing pose estimates from real detectors. In response to your interesting suggestion, we conducted an experiment involving an off-the-shelf pose estimator [1], on the JRDB dataset (ground-truth 3D pose is not available). The obtained estimated 3D poses were then used as input cues for our model. As demonstrated in the table below, our model exhibits a notable capacity to leverage even imperfect pose estimations, resulting in a performance improvement. Note that while the gain achieved with pseudo-ground-truth pose (here on the JRDB dataset) is slightly lower compared to utilizing ground-truth pose (as showcased in the main paper with a gain of about 10.1\% and 8.58\% on the JTA dataset), the model's adaptability to real-world pose estimators underscores its robustness and practical utility in scenarios where accurate ground-truth information may be challenging to obtain. | | T | T + estimated 3d P | |--------------------|-----------|-------------------------| | Social-Transmotion | 0.40/0.77 | 0.36/0.72 (8.70%/6.80%) | As you mentioned, in table 2 of the paper, we evaluated the performance using different percentages of available cues and introduced Gaussian noise, providing a comprehensive analysis of the model's robustness under various conditions. We add this discussion to the paper too. > Qualitative comparison with baselines. Following your suggestion, in the appendix, we have provided visual comparisons between our model and the best baseline, EqMotion, on the JTA dataset. Both models successfully capture interactions with neighboring agents. However, our model exhibits a higher level of progressiveness, leveraging interactions across a greater number of agents and more modalities (pose and trajectory). This distinct feature results in predictions that are closer to the ground truth. > Investigating multi-person full-body motion prediction. While our primary focus in the paper centered around the trajectory prediction task, it opens future explorations into extending our model to pose prediction. In response to your comment, we conducted a preliminary experiment, modifying our model to predict poses as a new output head. In this initial exploration, given the constraints of time, we focused on the single-person pose prediction task using the Human3.6M dataset. Following the settings of EqMotion (from CVPR 2023), we assessed the prediction results in terms of Mean Per Joint Position Error (MPJPE) across different prediction horizons, as presented in the below table. | Model | 80ms | 160ms | 320ms | 400ms | |----------------------------------------|------|-------|-------|-------| | EqMotion | 9.1 | 20.1 | 43.7 | 55.0 | | Social-Transmotion + pose decoder head | 9.2 | 21.4 | 48.5 | 61.2 | The findings indicate that although our model performs well in pose prediction, it currently does not outperform the state-of-the-art. This underscores the need for further refinement and optimization in future investigations. > Table notations change. Thanks. We changed the notations in the tables, and the PDF will be updated accordingly. [1] Grishchenko et al, Blazepose ghum holistic: Real-time 3d human landmarks and pose estimation, CVPRW 2022

Thank you for your insightful review. The following are our responses to the points you have raised. > The performance on other datasets: Argoverse, Waymo. In addition to the JTA dataset, our paper incorporated various common datasets in human trajectory prediction, including multi-person multi-modal robotic scenarios (JRDB), multi-person single-modal traffic scenarios (ETH-UCY), and single-person multi-modal traffic scenarios (Pedestrians and Cyclists in Road Traffic dataset). To enhance our analysis of multi-person multi-modal traffic scenarios and following your suggestion, we added the Waymo Perception dataset. This dataset was chosen for its extensive multi-modal annotations, including 2D/3D bounding boxes and 2D/3D pose keypoints of humans, as well as frame conversions and 2D-3D label associations. Due to the substantial size of the dataset and the time constraints, we conducted training and evaluation on selected subsets (25,000 random samples from the training set and 5,000 random samples from the validation set). Our model leverages the various visual labels and their associations provided by this dataset whenever they are available, and masks them out when not. For comparison, we selected two recent top-performing models based on the results presented in Table 1 of our paper: EqMotion (from CVPR 2023) and the transformer-based Autobots (from ICLR 2022). The detailed results can be found here: | | ADE/FDE | |--------------------|-----------| | Autobots | 0.40/0.85 | | EqMotion | 0.44/0.88 | | Social-Transmotion | 0.39/0.82 | It shows that our model exhibits better performance compared to previous works. We acknowledge that with further hyperparameter tuning, optimization, and training on the entire dataset, our model's performance could be further enhanced. We have added this experiment to the paper. > Computational costs comparison. Thank you for the suggestion. We have analyzed the computational costs of our model with various visual cues in Appendix A4. Upon your recommendation, we now include a comparison with the Eqmotion and Autobots baselines in the table below, detailing inference times based on over 5000 samples using a Tesla V100 GPU. Notably, our dual-transformer model achieves faster inference speeds than these baselines, even when processing 3D Pose data, which is advantageous for resource-limited environments. This efficiency gain is partly due to our use of CMT for encoding modalities and information compression, allowing the subsequent transformer, ST, to operate on a compacted learned representation. | | EqMotion | Autobots | Social-Transmotion (T) | Social-Transmotion (T+3d P) | |---------------------|----------|----------|------------------------|-----------------------------| | Inference time (ms) | 8.28 | 7.50 | 5.74 | 6.73 | It is important to acknowledge that our approach relies on additional estimation methods within its pipeline. Hence, when considering the complete processing time, especially for potential real-world deployments, these factors should be taken into account. However, these estimation methods have significantly improved in speed recently, enabling real-time usage without substantial concerns. > How's the model's performance with inaccurate or noisy pose estimations? Regarding the performance with less accurate or noisy pose estimations, we refer to the first part of the Discussions in Section 4.4, titled "What if we have imperfect input?". There, in table 2 of the paper, we evaluated the performance using different percentages of available cues and noisy pose, providing an analysis of the model's robustness under various conditions. In response to you and another reviewer, we conducted another complementary experiment involving an off-the-shelf pose estimator [1], on the JRDB dataset (ground-truth 3D pose is not available). The obtained estimated 3D poses were then used as input cues for evaluating our model without re-training. As demonstrated in the table below, our model exhibits a notable capacity to leverage even imperfect pose estimations, resulting in a performance improvement. | | T | T + estimated 3d P | |--------------------|-----------|---------------------------| | Social-Transmotion | 0.40/0.77 | 0.36/0.72 (8.70\%/6.80\%) | Note that while the gain achieved with pseudo-ground-truth pose (here on the JRDB dataset) is slightly lower compared to utilizing ground-truth pose (as showcased in the main paper with a gain of about 10.1\% and 8.58\% on the JTA dataset), our generic model's adaptability to real-world pose estimators underscores its robustness and practical utility in scenarios where accurate ground-truth information may be challenging to obtain. One way to lower this gap is to train the model with the estimated/inaccurate poses.

Thank you for your helpful input. Please find our answers to the raised points below: > Ensuring a balanced related work section for all supported prediction modes. We would like to clarify that our work does not encompass multiple prediction modes; rather, it is exclusively focused on trajectory prediction. Consequently, in our related works section, we have a subsection dedicated to the human trajectory prediction task and another subsection addressing the ways in which the utilization of visual cues can enhance it. These input modalities are not predicted but utilized to augment trajectory prediction accuracy. We have endeavored to cite all relevant studies on the use of each input modalities and are grateful for any additional references we may have overlooked. > Deterministic vs. probabilistic ADE/FDE? Indeed, as you pointed out, some of the models in the main table had a probabilistic approach and some deterministic. Notably, some of the probabilistic models have implemented a deterministic scenario and have reported ADE/FDE metrics for both deterministic and probabilistic results (e.g., EqMotion, Trajectory Transformer). To facilitate consistency in reporting, we have adjusted the prediction mode to 1 for probabilistic models (as they provided in their opensource public codes), asking them to predict the most reliable trajectory as the deterministic scenario and report deterministic ADE/FDE metrics. > Performance changes with different data quantity in training. We appreciate your recommendation to investigate the impact of varying amount of data in training on the performance. Following your suggestion, we conducted experiments with models trained on incrementally increasing data quantities. The outcomes of these experiments, as detailed below, indicate a marked performance enhancement up to the 40\% data volume threshold. Post this threshold, we observe steady improvements up until the 80\% data mark. Post-80\%, there is a noticeable plateau in performance enhancement. | Training data quantity | 10\% | 20\% | 40\% | 60\% | 80\% | 100\% | |------------------------|-----------|-----------|-----------|-----------|-----------|-----------| | ADE/FDE | 1.20/2.41 | 1.14/2.32 | 1.01/2.06 | 0.95/1.95 | 0.90/1.85 | 0.88/1.80 | We added this experiment to the paper. > Visual cues or spatial cues? In our manuscript, we used the term "visual cues" to refer to information extracted visually, such as the coordinates of 2D pose keypoints appear within the image frames or . Explicitly in this paper we refer to visual cues, whether they are poses, bounding boxes, or a combination thereof. We recognize the potential for ambiguity and have therefore added the definition in the manuscript. The exploration of additional input cues, including but not limited to scene context, is put for future research. > Not all traditional predictors have limited performance, because of reliance on a single data point per person as input. Thanks for pointing it out. We wanted to express that single-modality input is a limitation for predictors that relies solely on x-y coordinates, not for all of them. We updated the manuscript and added the missed citation.

Thank you for your comments. Our responses to each point are outlined below. > Using transformers for trajectory prediction lacks novelty, echoing established studies like Wayformer. As we mentioned in the paper, transformers have been widely utilized recently in trajectory prediction. However, it is essential to emphasize a distinctive contribution in our study. Our primary innovation lies in the introduction of a generic model with our modality-masking technique. This technique enhances the generality and robustness of the model, enabling flexible utilization of visual cues (we train **only one** model with all the available cues and at test time generate predictions only based on a subset of these cues), as demonstrated in Table 1 of our manuscript. Regarding the architecture, our proposed dual-transformer is modified from the language model BERT, which is different from the encoder-decoder structure used in Wayformer. The main similarity of our model and Wayformer for example, is that we both use latent queries to encode features from different modalities. However, in the case of Wayformer and most trajectory prediction architectures, the modalities represent only the locations of agents and map information. Our model differs in its focus on broader visual cues beyond that, leading to distinct learning processes for agent interactions. In Wayformer, the model learns the agent-agent interaction based on the location. In contrast, our model, designed for the task of human trajectory prediction, necessitates learning not only location-wise interactions but also pose interactions across agents. This distinction becomes particularly crucial in complex, crowded scenes where each agent may exhibit multiple modalities. Consequently, to address the challenge of modeling these intricate interactions, we employ distinct transformers for cross-modal feature extraction and social interactions, which enables the model to learn how neighbours' poses, bounding boxes and trajectories affect the agent's future trajectory. > Additional datasets like Argoverse and Waymo for benchmarking against other transformer-based models. In addition to the JTA dataset, our paper incorporated various common datasets in human trajectory prediction, including multi-person multi-modal robotic scenarios (JRDB), multi-person single-modal traffic scenarios (ETH-UCY), and single-person multi-modal traffic scenarios (Pedestrians and Cyclists in Road Traffic dataset). To enhance our analysis of multi-person multi-modal traffic scenarios and following your suggestion, we added the Waymo Perception dataset. This dataset was chosen for its extensive multi-modal annotations, including 2D/3D bounding boxes and 2D/3D pose keypoints of humans, as well as frame conversions and 2D-3D label associations. Due to the substantial size of the dataset and the time constraints, we conducted training and evaluation on selected subsets (25,000 random samples from the training set and 5,000 random samples from the validation set). Our model leverages the various visual labels and their associations provided by this dataset whenever they are available, and masks them out when not. For comparison, we selected two recent top-performing models based on the results presented in Table 1 of our paper: EqMotion (from CVPR 2023) and the transformer-based Autobots (from ICLR 2022). The detailed results can be found here: | | ADE/FDE | |--------------------|-----------| | Autobots | 0.40/0.85 | | EqMotion | 0.44/0.88 | | Social-Transmotion | 0.39/0.82 | It shows that our model exhibits better performance compared to previous works. We acknowledge that with further hyperparameter tuning, optimization, and training on the entire dataset, our model's performance could be further enhanced. We have added this experiment to the paper.