Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs

Summary

The authors identify two major issues of text-to-motion generation as overemphasis on action names and the coarseness of sentence-level representations. To this end, hierarchical semantic graphs are adopted to factorize coarse sentences into fine-grained action concepts, thus refining the generated motion from coarse-to-fine. Furthermore, the hierarchical design allows flexible control of the generation procedure. Extensive experiments show the efficacy of the proposed method.

Strengths

The proposed three tiers of node as motions, actions and specifics are interesting. The hierarchical diffusion is well-designed in disentangling different level of granularity. Also, the experimental results show reasonable improvements with more and levels. The performance is impressive. The motion refinement application is attractive and promising. The analysis on distribution of diffusion steps might be interesting in identifying the performance bottleneck among different semantic levels. Extensive ablation experiments provide a nice view of the effectiveness of different components.

Weaknesses

Qualitative comparison of different hierarchies is missed. It would be helpful to visualize motion generated from different level of latent embeddings, since fundamentally human is the only gold standard to evaluate motion generation, given the rather close numeric metrics of different levels. The ablation on the design of the semantic graph is missed. A more detailed description of the motion refining procedure, like a figure, an algorithm, or some equations might be added for better clarity.

Questions

Is it possible to generate with Action level or Specific level only? It would be interesting to see the corresponding results, given there is a trend that the more diffusion steps for Specific level brings better performance. The hierarchical nature of the method might be suitable for long sequence generation. Comparison on this would be an interesting thing to do. For motion refinement, is it possible to modify the graph beyond changing edge weights, i.e., adding/deleting nodes? Or tuning some nodes' weight to zero. What would happen?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author described some limitations of the methods, while it would be more helpful to discuss whether the current top-down paradigm could be extended in a bottom-up manner.

Summary

This work decomposes the motion description into three levels including motion, action and specifics, and proposes hierarchical semantic graphs to achieve fine-grained control of motion generation. Experiments with the proposed method on HumanML3D and KIT datasets demonstrate better motion generation and more sensitive to subtle differences in texts than existing techniques.

Strengths

The motivation is clear, and the proposed solutions are to the raised issues. The results seem to be impressive and the experimental setup is somewhat reasonable. The ability to continuously refine the generated motion is meaningful and helpful to the community.

Weaknesses

1. The paper states several times that Transformer may overemphasize action names, but I think that the action level in the hierarchical semantic graph is also another kind of emphasis. Therefore, if possible, I hope the authors can experimentally demonstrate the "overemphasis" of the Transformer and prove that the hierarchical semantic graph does not have this problem. 2. Why is the last one in the action level of Figure 1 resumes instead of walking? How does it handle "squats to picks" and distinguish "walks and walking"? Also, the hierarchical semantic graph is built on a valid semantic parsing tool, and I'm not sure whether the tool will greatly affect the overall robustness of the model. 3. What are the results in the first row of Tables 3 and 4 in what configuration? By my understanding, these two rows should be the complete ablation of their respective parts, so they should be numerically different. 4. Is the modification in Figure 5 implemented in other latest works as well? Does it only modify the weights of the edges and can it modify the nodes?

Questions

See the above weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors fully explain the social impact of the work in the appendix, but only elaborate on the shortcomings of the used models (e.g., diffusion model and VAE) in terms of limitations, and do not well explain their own limitations and failure cases.

Summary

This paper proposes a coarse-to-fine diffusion model coupled with a hierarchical semantic graphs to address the text-to-motion generation problem. To preserve the fine-grained control signals from captions, three-level textual features are extracted through GAT. Then, three diffusion models are adopted to recover the latent motion representations, which is decoded into motion sequences via VAE.

Strengths

1. Semantic graphs are introduced to model the coarse-to-fine textual descriptions. 2. Hierarchical diffusion model is technically sound to learn the latent motion distributions. 3. The experimental results outperforms sota methods. 4. Fine-grained controllability of the proposed model is interesting.

Weaknesses

1. The motivation behind the imbalance problem needs to be more convincing. In line 31-33, deficience of the existing method boils down to imbalance and coarseness. I am curious about if there are some examples could prove that imbalance problem exists in the other methods. 2. Many descriptions in this paper are misleading or incorrect, and it needs further explaination. In line 176, the term ``codebook size'' is ambiguous because it is generallly associated with VQVAE. In line 180-181, the purpose of diffusion process should be learning a mapping from gaussian noise sampled from $\mathcal{N}(0,1)$ to motion latent representation $z^m$, and the condition is the global motion node $V^m$. In line 183, $\beta_t$ is not step size, it is the noise schedule.

Questions

Q1: Please present more examples to prove that imbalance problem exists in the other methods and how your method addresses it. Q2: According to Table D and E in your supplementary material. R-Precision of your proposed VAE models outperforms Real motion, does this mean that R-Precision is not suitable for evaluating these two motion datasets?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have discussed the potential negative societal impacts.

Summary

This paper presents a novel motion generation pipeline that utilizes a 3-level hierarchical semantic graph. The entire reverse process of the motion diffusion model is divided into three stages: overall motion, local actions, and action specifics. The semantic graph is extracted through semantic role parsing and further enhanced using a Graph Attention Network. The obtained node features are subsequently input into a single transformer based on their semantic level.

Strengths

1. The proposed GraphMotion approach demonstrates excellent performance in terms of numerical metrics on two datasets, showing significant improvements compared to existing methods. Particularly noteworthy is the detailed comparison of GraphMotion at different diffusion steps with other state-of-the-art methods, which adds more persuasive evidence to the results. 2. The motion refinement aspect is intriguing, as users are able to achieve a certain degree of motion editing by modifying the content of nodes. This adds an interesting and interactive element to the method. 3. The motivation and objectives of the entire approach are well-explained, making it easy to understand the rationale behind the proposed method.

Weaknesses

1. It is recommended that the authors conduct user studies to quantitatively compare their method with other existing works based on qualitative results. The current demo video lacks sufficient comparisons, and additional comparisons should be included to provide a comprehensive evaluation. Furthermore, it would be beneficial to showcase the qualitative results of motion refinement in the demo video to provide a more complete demonstration of the method's capabilities.

Questions

1. Why is the video length of MotionDiffuse shorter than other methods in the demo video? 2. Why is there a significant decrease in the FID metric on KIT-ML when the number of steps increases from 50 to 150? 3. Can GraphMotion perform low-level motion editing or motion composition similar to MotionDiffuse and MDM?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations have been well discussed.

Some concerns that have not been addressed

Thank you for taking the time to comment. I am satisfied with some of the answers provided. However, there are still some concerns that have not been addressed: (1) Why are the first rows of Table 3 and Table 4 numerically the same? I think that they are ablation experiments done from different two aspects, so their baselines should not be the same. (2) Regarding limitations, the supplementary material only elaborates on the shortcomings of the used models (e.g., diffusion model and VAE) in terms of limitations, and does not well explain their own limitations and failure cases.