Summary
In this paper, the authors introduce $M^3$GPT, a multimodal multitask framework designed for both motion comprehension and generation. Utilizing discrete vector quantization, $M^3$GPT establishes a discrete semantic representation space for various modalities. To avoid information loss during discrete de-tokenization, $M^3GPT$ jointly trains the large language model (LLM) and motion de-tokenizer, thereby optimizing the LLM within both the discrete semantic space and the continuous raw motion space. Furthermore, the authors construct paired text descriptions for music and devise two supplementary tasks music-to-text and text-to-dance, which facilitate the alignment of music and dance within the text embedding space. The efficacy of this approach is demonstrated through experiments across a range of motion-related tasks, and showcases zero-shot generalization capabilities for highly challenging tasks.
Strengths
1. The motivation behind investigating the integration of text, music, and motion for motion comprehension and generation is convincing.
2. The proposed joint optimization of the large language model (LLM) and motion de-tokenizer, along with the synergy learning strategy incorporating auxiliary tasks, demonstrates potential in enhancing multimodal motion generation.
Weaknesses
1. The effectiveness of combining text, music, and motion modalities in the $M^3$GPT framework is not thoroughly evaluated. Although the authors present the $M^3$GPT capabilities in handling zero-shot tasks. However, there is still doubt about how can text-to-motion or music-to-dance tasks benefit from the multimodal framework.
2. Some key experiments are missing.
a. In Table 3, the authors evaluate proposed $M^3$GPT on text-to-motion dataset Motion-X on 4 tasks. However, MotionGPT[1] can also perform the tasks while the comparison between MotionGPT[1] is missing.
b. Besides motion prediction lacks comparison with models like T2M-GPT[2], since the auto-regressive models are better at prediction.
c. Motion inbetweening lacks comparison with the state-of-the-art models like MoMask[3] capable of temporal inpainting which can handle motion inbetweening tasks.
3. The qualitative comparison of generated motions by the proposed $M^3$GPT framework from text or music is absent. The inclusion of detailed qualitative results is crucial for providing a comprehensive understanding of the naturalness and overall realism of the generated gestures.
Minor:
1. In Figure 2, there is an error in drawing graphics in the Motion Codebook part.
[1] Jiang, Biao, et al. "Motiongpt: Human motion as a foreign language." Advances in Neural Information Processing Systems 36 (2024).
[2] Zhang, Jianrong, et al. "Generating human motion from textual descriptions with discrete representations." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.
[3] Guo, Chuan, et al. "Momask: Generative masked modeling of 3d human motions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Questions
1. I am curious about the long-duration dance generation mentioned in Page 6, especially if the LLM can maintain coherence and consistency in generating long sequences or this long-duration generation is achieved by the consistency during de-tokenization.
2. In Table 4, the authors evaluate music-to-dance on two datasets AIST++ and FineDance, $M^3$GPT achieves state-of-the-art performance on AIST++, but the performance on FineDance is not that competitive. Could the authors provide more insights into the reasons behind this performance gap between the two datasets?
3. Have authors evaluate different sizes of the $M^3$GPT model? Given the added modalities and tasks, it would be interesting to understand how the model scales with respect to performance and computational requirements.
Limitations
The authors have discussed limitations.