MotionGPT: Human Motion as a Foreign Language

Summary

This paper proposes MotionGPT, an approach that unifies language modeling with human motion modeling by treating each new motion token the same as a language token. To achieve this, a motion tokenizer based on VQ-VAE is first learned. Then, a pretrained language model is fine-tuned to learn from a unified vocabulary of motion and language and then instruction-finetuned to perform tasks such as text-to-motion, motion completion, and motion captioning. Experiments show that the proposed model achieves state-of-the-art performance for the proposed tasks.

Strengths

- The idea of treating motion as discrete tokens for language models is novel and sound, and can enable a large number of possible applications. The showcased tasks are also comprehensive and demonstrate the flexibility of the proposed model. - The provided quantitative and qualitative results show state-of-the-art performance and generate natural and good-looking human motion. The multi-task ability is impressive and shows that the model can handle multiple different tasks and natural language instructions. - Extensive ablation is provided.

Weaknesses

- I do find the lack of failure analysis a bit concerning. It is difficult to gauge how well the model learns the relationship between motion and language. - Similarly, I find the evaluated motion in the provided video and supplement to be relatively simple and clear instructions, which do not really conform to the objective of "generalizing effectively to unseen tasks or data". For instance, in the text-to-text demonstration, the word "praying" is showcased, but I wonder how well the motion generator part can handle words that have semantic meaning but could be unseen. Minor: Error T2M-GPT and MDM’s citation are mixed in Table 1.

Questions

Do TM2T, T2M, and poseGPT capture all human motion in their training dataset's discrete latent code? How is the reconstruction loss on ALL the training data? --- After rebuttal, my main concerns about failure analysis and text complexity are addressed and I would like to maintain a positive rating of this work. ---

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

More failure cases should be discussed.

Summary

In view of the idea that motion could be perceived as a form of body language, the authors propose to fuse motion and language to perform a unified motion-language pre-training. In detail, motion is quantized into discrete tokens in the same form as natural languages. Then, language modelling is performed on both motion and text in a unified manner. Furthermore, prompt tuning is adopted to fine-tune the pre-trained model. Experiments demonstrate the impressive performance on multiple motion tasks.

Strengths

The idea of unifying motion and language into tokens for uniform pre-training is interesting and novel. The uniform motion-language model manages to provide a solution for a wide range of tasks. The provided demo is rather impressive and convincing. Extensive ablation studies provide a detailed analysis on the effectiveness of different design choices.

Weaknesses

Though the quantized representation provides the ability to unify motion and text, it also imposes constraints on the motion representation due to the sequence-level encoding. In other words, when the operation granularity is smaller than the down-sample rate, I'm not sure whether the method could provide satisfying performance. For example, motion in-between seems to be designed as a token-level in-between. If only given a start frame and an end frame as the in-between input (which could be a more practical application scenario), would the model perform well? Both HumanML3D and KIT are limited in the vocabulary size and the overall dataset size compared to language datasets as I know. Therefore, I understand the limited performance on KIT and when increasing the scale of the model. While in view of the recent success of LLMs, I think the authors should pay attention to unifying current available datasets to exploit the scalable potential of language models when processing large scale data besides increasing model size.

Questions

I'm interested in the vocabulary that VQ-VAE learned. Is it possible to visualize some of the tokens? Or directly generate description on each single token? How is the down-sample rate chosen? It is a fundamental hyper-parameter that decides the overall granularity of the model.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors provide their discussion on the limitation of the paper.

Summary

This paper presents a motion-language model via a shared vocabulary, where the texts are represented by original tokens, and the motions are encoded by a trained discrete tokenizer. Based on a pre-trained encoder-decoder framework, i.e., T5, the authors fine-tune the T5 with masked modeling on motion-language paired data. Finally, the obtained model is finetuned with specific instructions for the target job (text-to-motion, motion-to-text, motion prediction, motion in-between in the paper). The experiments demonstrate the MotionGPT does such tasks well.

Strengths

- this paper has a clear and interesting motivation, i.e., jointly learn motion-language on a token-to-token model, enabling the trained model to be aligned with the text instructions. - the main paper, along with the supp., provides solid and comprehensive experimental results.

Weaknesses

- It seems the work is finished in a rush. For example, the instruction tuning is performed on each task independently, acting like a simple finetuning (the experiments also give some pieces of evidence, where it depends on the task-specific tuning). In this way, is it just a pretrain-finetune scheme? It is interesting to see any difference with the previous pretrain-finetune pipeline, such as enabling more abilities like reasoning, and zero/few-shot learning. - The motion prediction lacks comparison with border models like T2M-GPT, since the auto-regressive models are better at prediction. - Although the authors use a strong pre-trained language model, and finetune it on each task independently, it shows a limited performance gain compared with previous works.

Questions

- Why do you choose T5 as the base model? which is an encoder-decoder architecture. Have you tried a decoder-only model like LLaMA since it is a more straightforward solution for instruction tuning? - How do you implement the MDM on the motion prediction and in-between tasks? the numbers of MDM exist a large gap with text-to-motion. I mean, for diffusion models, there are lots of details in implementation. For example, reset the prefix 20% motion at each diffusion step. Lacking the details makes the numbers not fully convincing. - How do you merge the text vocab and motion vocab in detail? concatenating them together? - For tuning on each task, do you tune the entire model or just part of it? Table 2 is redundant since all the information is duplicated in the latter tables.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper introduces a motion generation pipeline called MotionGPT, which is based on GPT. MotionGPT utilizes VQ-VAE to discretize human poses into tokens and combines them with language tokens to create a unified codebook. The model is initially pre-trained on motion language data and subsequently fine-tuned on prompt-based tasks to enable it to perform various motion-language tasks.

Strengths

1. This is the first work that explores the application of Large Language Models (LLMs) in the field of text-driven motion generation. The proposed prompt finetuning method further extends the scope of applications by including 10 different tasks. These methods provide inspiration for future research in this area. 2. The performance in the Motion-to-Text task shows a significant improvement on Bleu@4 and Cider compared to TM2T. 3. The paper is well-written and effectively conveys information in a clear and understandable manner.

Weaknesses

1. MotionGPT exhibits poorer performance in the crucial text-to-motion task, with a significant gap in FID metrics compared to T2M-GPT. Particularly on the KIT-ML dataset, there is a considerable difference in R Precision compared to T2M-GPT, MLD, and MotionDiffuse. 2. The demo video provides limited comparisons with other examples. Apart from the "crouch down" example, the performance of MotionGPT is not noticeably superior to T2M-GPT in the provided examples. These examples do not sufficiently demonstrate an advantage in terms of generation quality. 3. The overall technical contribution is limited. The major distinction from T2M-GPT lies in the combination of motion modality and language modality for modeling, along with subsequent prompt finetuning. However, in the current version, they appear more like shared intermediate layers rather than truly integrated. For instance, a potential approach could involve language-based motion editing, where given a reference motion sequence and a desired text modification, the algorithm produces the edited result. Such task types would better illustrate the advantages of unified modeling. Additionally, considering the significant performance gap between MotionGPT and T2M-GPT in standard text-to-motion tasks, MotionGPT seems to sacrifice accuracy in exchange for additional functionalities. Such technical contribution does not meet the bar set by NeurIPS conference.

Questions

1. Can MotionGPT perform motion editing or motion composition similar to MotionDiffuse and MDM? 2. The supplementary material states that there were only 15 users in the user study for Motion-to-Text. The number of testers for Text-to-Motion is not mentioned. However, 15 users may be considered an insufficient sample size for a reliable evaluation, especially for Motion-to-Text. It is recommended to have a larger sample size, preferably around 50 or more, to provide a more credible assessment. 3. What is the reason behind the significant difference in performance between the KIT-ML and HumanML3D datasets?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations have been well discussed.

Reviewer Response

I thank the authors for the detailed response. I find most of my concerns resolved and would maintain a positive score.

Post-rebuttal Comment

I express my gratitude to the authors for their comprehensive rebuttal. I am pleased to note that my initial concerns have been satisfactorily addressed. I have also taken into account the input from other reviewers, and it appears that no significant additional concerns have been raised. I am inclined toward recommending acceptance and will revise my score after reviewer discussion period.