MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models

Summary

This paper explores the application of state space models (SSMs) to co-speech gesture generation. The authors identify the computational challenges and jittering issues associated with the direct application of SSMs to gesture synthesis. To address these, they propose a two-stage modeling strategy with discrete motion priors and hybrid fusion modules. The first stage involves learning discrete holistic gesture priors with multiple VQVAEs, and the second stage refines latent space representations using local and global scans. Experiments demonstrate that MambaTalk outperforms state-of-the-art models in generating natural, rhythmic, and contextually appropriate gestures.

Strengths

1. The paper is well-structured and clearly written. 2. It seems that MambaTalk is the first to explore the potential of the selective scan mechanism for co-speech gesture synthesis. 3. The methodology is thoroughly validated through extensive experiments, including both subjective and objective evaluations.

Weaknesses

1. Limited novelty. Integrating the local and global scans seems not novel. More discussion on the comparison between MambaTalk and baseline methods is needed. 2. Limited datasets for method evaluation. Only the BEATX-standard dataset is adopted to evaluate the performance of the method. How about the zero-shot quantitative comparison on other datasets, such as the BEAT dataset or the RAVDESS dataset, which take emotion into account?

Questions

See the "Weakness" section.

Rating

6

Confidence

2

Soundness

4

Presentation

4

Contribution

3

Limitations

See the "Weaknesses" section. If possible, I suggest adding more visualization results and videos to the supplementary materials. Additionally, please note that the final score for this study is not solely determined by the peer reviewers' discussion. If the authors can address my main concerns, I would be willing to raise the score.

Summary

This study explores the use of state space models (SSMs) to enhance gesture synthesis, addressing challenges such as diverse movement dynamics and unnatural jittering in generated gestures. Through a two-stage modeling approach and the introduction of MambaTalk with hybrid fusion modules, the study demonstrates superior performance compared to existing models in subjective and objective experiments.

Strengths

1. This work is the first to explore the potential of the selective scan mechanism for co-speech gesture synthesis, achieving a diverse and realistic range of facial and gesture animations. 2. The writing of this work is fine, there are no obvious typos. 3. The workflow pipeline figure is clear to understand.

Weaknesses

My concerns and suggestions of this manuscript are listed as blew: 1. The motivation of this work sounds unclear and looks a little incremental. As claimed in the abstract section 'the high computational complexity of these techniques limits the application in reality'. This statement is not clear. Does computational complexity mean more model parameters? or training time? or inference time? or more GPU memory cost? The authors didn't have a clear statement. 2. Moreover, in the abstract, 'which stem primarily from the diverse movement dynamics of various body parts.' are actually the general difficulties of the co-speech gesture generation task, not the specific one of how to employ the SSMs to this task. 3. The introduction section is not clear. The authors pay much more attention to the related works. Not the motivation and the high-level technical contributions of their work. This led me to feel that the entire work lacked technological innovation after reading the Introduction. 4. As for the methods, it is just directly imply the SSMs to this work. I cannot see any design on how to effectively solve the problem of 'computational complexity'. Actually, the technical contribution is poor, and the overall pipeline is very similar to the previous work EMAGE[1]. 5. Could the authors explain why they only experimented on the BEATX dataset? As far as I know, TED[2] and TED-expressive[3] are also two commonly used co-speech gesture datasets. 6. The author did not conduct experiments to demonstrate how to effectively reduce the computational complexity by using SSMs, which directly led to the unclear motivation of this work. [1] Liu, H., Zhu, Z., Becherini, G., Peng, Y., Su, M., Zhou, Y., ... & Black, M. J. (2023). Emage: Towards unified holistic co-speech gesture generation via masked audio gesture modeling. arXiv preprint arXiv:2401.00374. [2] Yoon, Y., Cha, B., Lee, J. H., Jang, M., Lee, J., Kim, J., & Lee, G. (2020). Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6), 1-16. [3] Liu, X., Wu, Q., Zhou, H., Xu, Y., Qian, R., Lin, X., ... & Zhou, B. (2022). Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10462-10472).

Questions

please refer to Weaknesses

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

please refer to Weaknesses

Summary

The paper explores the selective scan mechanism for gesture generation. First, it trains VQVAE to reconstruct faces and body parts using discrete latent space. Then, It uses local and global scanning mechanisms to improve the latent representations of various body parts for the purpose of gesture generation.

Strengths

1) The paper explores Mamba for gesture generation. 2) The objective function includes features related to gesture acceleration and velocity, which are important to capture. 3) The paper contains a user study where even the ground truths are evaluated. I think that is the right direction for gesture-related studies.

Weaknesses

1) The related work contains many acronyms not introduced or known to a wide community (e.g., VQVAEs, HiPPO, and LSSL). 2) How the authors used Mamba in their work is unclear. The explanation in section 3.1 is vague. 3) Audio feature extractor is not strong. The authors should have explored stronger speech representations that can be used to improve speech prosody. 4) The technical contributions of the paper might be limited to implementing Mamba architecture for gesture analysis.

Questions

1) Can you please dedicate section 3.1 to how you used Selective State Spaces for gesture generation? The explanation given is very generic. The reader can not follow this generic description without prior knowledge of Mamba. 2) How are the motion priors (the codebook) initialized? Also, how often is this codebook updated? 3) Why are the results reported in "EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling" differs from what you report in Table 1. FGD in the EMAG paper is 5.512, while in your paper, they are reported as 5.423. If you are using two different backbone models (and hence features), then how fair is this comparison if you fine-tune your models with the best parameters?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

One limitation related to gesture diversity across speakers and cultures is not addressed. The mentioned limitations are mainly related to technical details, which I think are already being addressed using Transformer and Diffusion based models.

Summary

This paper focuses on the problem of co-speech gesture generation, particularly aiming to address the challenges of jittery movements, long motion sequences, and holistic gesture generation (including both face and body movements). To tackle these issues, the authors present a new method that combines diffusion models with state space models and incorporates both local and global scans. The proposed method has been evaluated on the BeatX dataset.

Strengths

This paper introduces novel methodological aspects for co-speech feature generation. For instance, combining diffusion models with state space models and local and global scan approaches has not been explored before, to the best of my knowledge. The experimental results clearly demonstrate the effectiveness of the proposed approach. Moreover, the paper presents both quantitative results and qualitative results through user studies, providing a complete and strong evaluation. Additionally, the proposed approach is faster compared to the state-of-the-art methods, as presented in the appendix. However, there are some weaknesses that need to be addressed. Please see below for detailed comments.

Weaknesses

Major Comment: The main weakness of the paper is its limited evaluation. While the paper excels in generating facial, body, or holistic gestures, evaluating on only one dataset is substandard. To be competitive, the authors should consider additional standard benchmarks such as those used in the GENEA challenge [1]. There is also a long list of papers using diffusion models for co-speech gesture synthesis [1], none of which are mentioned in the paper. While I appreciate the authors' work and acknowledge the scarcity of datasets with both face and body motions, the proposed approach handles face and body separately. Therefore, at least body motion generation performance could be compared on existing datasets to demonstrate generalisable benefits. Another weakness is the insufficient details regarding long-term motion generation, which is claimed as a key contribution. The paper lacks sufficient details on the length of the sequences considered in training and evaluation. Moreover, there is no comparison to support this claim. Minor Comment: The lower part of Figure 2 is helpful for understanding the workflow. However, including pseudo-code would be beneficial to increase the reproducibility of this method. [1] https://genea-workshop.github.io/2023/challenge/

Questions

Why is only one dataset considered? Please explain the rationale behind evaluating the method on only one dataset. How are these results generalisable and competitive given this limitation? Generalizability and Competitiveness: How do you ensure that the results obtained from the BeatX dataset can be generalized to other scenarios? Discuss any potential limitations in terms of generalisability and how your method addresses them. Training and Evaluation Sequence Lengths: Please elaborate on the length of the motion sequences considered during training and evaluation. Impact on Co-Speech Gesture Generation Performance: Discuss how the length of motion sequences impacts the performance of co-speech gesture generation. What are the implications of using different sequence lengths, and how does your method handle long-term motion generation effectively? Including comparisons or experiments that highlight these aspects would be beneficial.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have satisfactorily discussed the limitations and broader impact of their work.

Response to authors

Dear authors, Thanks for your efforts and responses to my questions. Although most of my concerns are addressed by the authors, I still have some main concerns about the motivation and writing of this work. Therefore, I raise my rating from 3 to 4. Thank you. Best regards

reply: Rebuttal by Authors

Thank you for the additional information. After reviewing all the feedback and responses, I have decided to maintain my initial score. While the paper introduces some novel methodological aspects, the primary concerns remain: the lack of experimental results and the need for further clarification, particularly regarding the motivation for long-term and real-time motion synthesis.