LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning

Author Responses to Reviewer yD2w

We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further. **1. Difference from HumanTomato** Thanks for this insightful comment. Our method differs from HumanTomato in terms of model architecture and training objectives. While HumanTomato employs the motion encoder and text encoder from TMR [1], LaMP utilizes a transformer-based motion encoder and text encoder similar to that of CLIP. Furthermore, our training objective is not limited to contrastive loss; we designed four proxy tasks including contrastive learning, matching tasks, motion-grounded text generation, and text-grounded motion generation. This design enhances the alignment effectiveness of LaMP. As a result, we can obtain motion-aware text embeddings as well as language-aware motion embeddings, making it easier for us to finetune a motion captioning LLM. We have added the differences from previous methods in the appendix. (L831-848) **2. Inference Time** Thanks for this valuable suggestion. We have added a figure of inference time as Figure 7 in the appendix (L868). Following MoMask, we assess the efficiency and quality of motion generation compared to some methods in Table 1. The inference cost is quantified as the average inference time across 100 samples executed on a single Nvidia 2080Ti device. Our findings indicate that LaMP achieves a more advantageous balance between generation quality and computational efficiency when compared to baseline methods. This may be attributed to MoMask's multi-layer Residual-VQ structure, along with the refinement of the inference process through the Residual Transformer, resulting in reduced time consumption on our part. We have added the inference time experiment in the appendix. (L804-812) | Method | AIT (s) | FID ↓ | | -------------- | ------------- | ---------- | | T2M | 0.03 | 1.09 | |TM2T | 0.76 | 1.50 | |MLD | 0.22 | 0.47 | |MotionDiffuse | 10.89 | 0.63 | |MDM | 18.20 | 0.54 | |T2M-GPT | 0.38 | 0.12 | |MoMask | 0.12 | 0.04 | | LaMP-M2T (Ours)| 0.08 | 0.03 | Table 1. Comparisons on FID and Inference Cost. [1] TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis

Author Responses to Reviewer ZgrR

We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further. **1. Difference from previous work** Sorry for being unclear. To the best of my knowledge, there are currently two existing works titled MotionGPT, both of which focus on finetuning large language models (LLMs) for motion generation. However, a critical observation from these studies is that MotionGPT demonstrates suboptimal performance in motion generation. This limitation arises from the unresolved gap between motion and language. In contrast, we utilize a smaller model specifically designed for motion generation, employing a text encoder capable of producing motion-informative text embeddings to bridge the gap between motion and language. This approach yields significantly improved results in motion generation. Additionally, while MotionGPT requires instruction tuning of the LLM for the motion captioning task, our method for motion captioning does not necessitate such an operation, as our motion embeddings are inherently compatible with the LLM’s understanding. Consequently, we have effectively addressed the intermodal gap, and our proposed method, LaMP, can be integrated with MotionGPT. This integration would involve using LaMP to obtain motion embeddings, followed by instruction tuning of the LLM, potentially leading to enhanced performance on motion captioning tasks. We have added the differences from previous methods in the appendix. (L831-848) **2. BERT vs. LLM** Thanks for this insightful question. I believe your question revolves around why LaMP is based on BERT rather than a Large Language Model (LLM). LaMP consists of a motion transformer and a text transformer, both of which are structured based on BERT with shared parameters. If we were to use an LLM as the backbone, we would face several challenges: firstly, the number of parameters that would need fine-tuning would be substantial, resulting in significantly slower inference speeds. Our goal is to ensure that LaMP bridges the gap between motion and language, as well as facilitates motion-text retrieval; hence, a lightweight module is preferable. Otherwise, using it as a text encoder would lead to inefficiencies in processing speed. Secondly, akin to the first concern, motion embeddings cannot be directly understood by LLMs and may even compromise the LLM's inherent capabilities. LLMs are not well-suited for training with representation tasks as their objective. Consequently, the four proxy tasks we utilize for training would not be effective. Thus, we choose BERT as the backbone for this purpose. To address your concern, we also attempt to train using LLMs; however, due to their large number of parameters, they present significant challenges in terms of training feasibility. **3. Text caption model** Thanks for this constructive comment. In the field of natural language processing, decoder-only model architectures are generally considered more suitable for text generation tasks, such as GPT. BERT, as an encoder-only model, employs a bidirectional attention mask, making it more adept at text understanding and inference tasks. Thus, you are correct that BERT does indeed have its limitations. Furthermore, during the first stage of training LaMP, we also utilized motion-grounded text generation as one of the proxy tasks. To address the reviewer's concerns, we conduct an evaluation of LaMP's own motion-to-text capability, as illustrated in the first row of Table 1. | Method | Top 1 ↑ | Top 3 ↑ | MultiModal Dist ↓ | BertScore ↑ | Bleu@1 ↑ | Bleu@4 ↑ | Rouge ↑ | Cider ↑ | | -------------- | ------------- | ---------- | ----------------- | ----------- | -------- | -------- | ------- | ------- | | LaMP | 0.397 | 0.663 | 4.076 | 17.8 | 30.1 | 6.87 | 26.6 | 16.2 | | LaMP-M2T (Ours)| 0.547 | 0.831 | 2.808 | 32.7 | 47.8 | 13.04 | 37.1 | 28.9 | Table 1. The quantitative results of motion captioning on the HumanML3D. **4. CFG scale** Sorry for being unclear. We did not make any further adjustments to these two parameters and simply followed MoMask. Additionally, I apologize for not specifying the CFG scale of HumanML3D in the text. In HumanML3D, CFG scale is 4, and in KIT-ML, CFG scale is 2. We have corrected this in the revised manuscript (L459).

Author Responses to Reviewer ZgrR

We thank the reviewer for the valuable feedback again! We hope our responses adequately address the questions raised about our work. Please let us know if there is anything we can clarify further.

Author Responses to Reviewer S3ZK

We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further. **1. Comparison with DLP on motion-to-text generation** Thank you for your reminder! We have added the comparison with this work in the revised manuscript. Our approach fundamentally differs from DLP in that DLP utilizes a Large Language Model (LLM) as the backbone and leverages the LLM’s in-context learning capabilities for instruction tuning. In contrast, our method inputs motion into LaMP to obtain language-informative motion embeddings, facilitating a better understanding of the motion by the LLM. Additionally, there are differences in the training datasets: DLP constructed a new dataset, MoCap, and trained the LLM using this dataset, resulting in an inconsistency in our baselines. However, our approach could be integrated with DLP by extracting language-informative motion embeddings using LaMP and then applying DLP for instruction tuning to achieve improved results. Unfortunately, we find that DLP has not released their code, which prevents us from validating this quantitatively. We will realize this after they release the code. The comparison results are shown in Table 1. We have added the brief of **DLP** in the related work. (L151) | Method | Top 1↑ | Top 3 ↑ | MultiModal Dist ↓ | BertScore ↑ | Bleu@1 ↑ | Bleu@4 ↑ | Rouge ↑ | Cider ↑ | |-------------------|---------------------|---------------------|-------------------|-------------|----------|----------|---------|---------| | DLP | 0.551 | 0.832 | 2.813 | 35.0 | 51.1 | 15.5 | 41.9 | 36.2 | | LaMP-M2T (Ours) | 0.547 | 0.831 | 2.808 | 32.7 | 47.8 | 13.04 | 37.1 | 28.9 | Table 1. The quantitative results of motion captioning on the HumanML3D, using different training datasets. **2. Failure cases** Thanks for your insightful suggestion, it can enhance the comprehensiveness and robustness of our work. We identify two primary categories of failure cases. The first category arises from issues inherent in the dataset's textual annotations. As depicted in Figure 8 (L892), the textual prompt is "A person raises his left arm." However, the dataset itself contains incorrect motion that leads to failures in our model’s outputs. Similarly, for the prompt "A person kicks with his right leg and then kicks with his left leg," the ground truth does not include the motion "kick with left leg," which results in failures in our generated results. The second one is in the context of our experiments with cartoon characters, we encountered a challenge. Cartoon characters frequently perform motions that are impossible for humans, such as flying. As illustrated in Figure 9 (L937), when we use the text prompt "A person stands and flies up," the results does not meet our expectations. This shortfall is attributed to the current dataset's relatively limited size and lack of generalizability. Moving forward, it is crucial to integrate the motions of cartoon characters with those of humans to construct a more extensive dataset to address this issue comprehensively. We have added the analysis on failure cases in the appendix. (L812-828) **3. Details about encoder-decoder and reconstruction metrics** Sorry for being unclear. There is no text encoder-decoder. For the motion encoder-decoder, we follow T2M-GPT and utilize the vanilla VQ-VAE. We employ a straightforward convolutional architecture consisting of 1D convolutions, residual blocks, and ReLU activations. We have added the framework of VQVAE as Figure 5 in the appendix (L702). Temporal downsampling is achieved via convolutions with a stride of 2, while nearest neighbor interpolation is utilized for upsampling. We also evaluate the reconstruction metrics on the HumanML3D dataset and KIT-ML dataset, as detailed in Table 2. | Dataset | FID ↓ | MPJPE ↓ | Top 1 ↑ | |-----------|---------------|----------|----------------| | HumanML3D | 0.068 ± 0.001 | 63.06 | 0.504 ± 0.002 | | KIT-ML | 0.468 ± 0.011 | 57.64 | 0.426 ± 0.005 | Table 2. The reconstruction results of VQVAE on HumanML3D and KIT-ML.

Author Responses to Reviewer PasF

We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further. **1. Generalization ability of LaMP** Thank you for this insightful question. Firstly, it is indeed true that CLIP demonstrates superior generalization capabilities. However, due to the inherent differences between motion and images, CLIP has not been able to effectively align text with motion. We agree that current text-motion datasets are generally quite limited, with HumanML3D being the largest dataset available at this stage. Our method could be applied whenever a larger dataset is available. We are also actively exploring constructing larger and more comprehensive text-motion datasets, and we plan to retrain a version with enhanced generalization capabilities on a larger-scale dataset in the future. We believe that although LaMP's generalization performance is relatively lower due to the small dataset size, its performance in the domain of motion surpasses that of CLIP. To address the reviewer's concern, we have conducted additional experiments to validate the generalization of LaMP. Specifically, we utilize the LaMP model pretrained on the HumanML3D dataset as a text encoder and re-train the text-to-motion task on the KIT-ML dataset. The results are presented in Table 1. | Methods | Top 1 | Top 2 | Top 3 | FID↓ | MultiModal Dist↓ | Diversity → | |-------------|----------------------|----------------------|----------------------|-------------------|------------------------|---------------------| | HumanML3D-LaMP | 0.423±.006 | 0.657±.005 | 0.771±.005 | 0.226±.012 | 2.768±.022 | 10.536±.098 | | KIT-LaMP | **0.479±.006** | **0.691±.005** | **0.826±.005** | **0.141±.013** | **2.70±.018** | 10.929±.100| Table 1. The quantitative results of text-to-motion generation on the KIT-ML dataset with LaMP pretrained on HumanML3D (the first row) and KIT-ML (the second row). **2. LaMP-BertScore** Thank you for this constructive comment. We acknowledge that the current training dataset is relatively small. However, this is a common drawback for all evaluation metrics in this field, e.g., FID and R-precision, as they are all trained on the same small datasets (i.e., HumanML3D and KIT-ML) as ours. Therefore, the scale of the dataset is a significant issue in this field. We initially propose LaMP-BertScore with the aim of evaluating the semantic information of generated motions. We consider this to be an important metric, as a single text prompt can correspond to multiple different motions, provided that the generated motion is semantically accurate. Reviewer S3ZK and yD2W also believe that the introduction of LaMP-BertScore offers a valuable metric for assessing the quality of generated motions, particularly the semantic fidelity of textual descriptions. This enhances the evaluation process in the domain of motion generation. **3. Dataset errors** Thank you for this insightful question. The training of LaMP is based on the premise that motion and text are correctly paired in a one-to-one manner. However, we admit that we have not specially addressed this issue. We believe that the ratio of such errors in the dataset should be minimal and would not significantly impact the model's performance. **4. Some typos** Thank you very much for pointing out these errors! We have corrected them in the revised version.

Author Responses to Reviewer PasF

Thank you for this suggestion! - For Motion-X, the precise data in Motion-X (motion capture) is identical to that of HumanML3D. However, the other data in Motion-X have been extracted using video-based algorithms, which are less accurate. We experimented with these datasets but did not achieve satisfactory results. - For "Large Motion Model for Unified Multi-Modal Motion Generation", the text-to-motion data in this dataset comprises HumanML3D, KIT-ML, Motion-X, and BABEL. The only difference is the BABEL dataset, which uses data from AMASS and is the same as HumanML3D. Of course, we plan to train a LaMP model based on SMPL-X on MotionVerse, but the complete dataset has not yet been released. In addition, we will cite these two papers in the related work section and provide a brief overview. Thank you again!