CigTime: Corrective Instruction Generation Through Inverse Motion Editing

Summary

The paper introduces a novel approach to generating corrective instructional text based on motion editing and generation models. This work is particularly motivated by the applications in sports coaching and motor skill learning. The authors propose a method that takes a user's current motion (source) and the desired motion (target) and generates text instructions to guide the user towards achieving the target motion. The approach leverages existing motion generation frameworks to compile datasets of triplets (source motion, target motion, and corrective text) and employs a large language model (LLM) fine-tuned on this data to produce corrective instructions.

Strengths

1. This paper properly uses LLM and focuses on an important and novel application. 2. The author formulated this problem as an inverse problem of motion editing and used the pretrained editing model to create a data model.

Weaknesses

1. The inputs of this work are motions, which are hard to obtain in real life. The off-the-shelf motion estimation algorithm may cause estimation errors, which limits the application range and fields. 2. The data-generating pipeline heavily depends on the motion-editing model, which may lead to error accumulating and overfitting.

Questions

1. Could the authors try to compare with vision LLMs, like GPT4O, InternVL [1], and MotionLLM [2]. 2. The way of using text pure LLM for comparison is in-context learning, which is not fair. How does an LLM understand the confused motion index tokens without fine-tuning? 3. The method and dataset heavily depend on the generalizable abilities of the motion editing model. How does the author make sure that the pretrained motion editing model can take flexible instructions generated by Chatgpt as input and achieve satisfactory editing results? 4. The authors take discrete token indexes as special tokens. This is hard to capture the detailed spatial and temporal information. Did the authors try to use continuous representations? Just like in LLaVA [3] and MotionLLM [2]. 5. The authors use a pretrained motion editing model to generate data, then finetune an LLM using the data, and finally evaluate using the same motion editing model. The data-generating pipeline and evaluation use the same motion editing model. This may lead to heavy overfitting, which can be seen in Figure 4. [1] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Chen, Ling-Hao, et al. "MotionLLM: Understanding Human Behaviors from Human Motions and Videos." arXiv preprint arXiv:2405.20340 (2024). [3] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

The limitations have been discussed in the paper.

Summary

This paper introduces an innovative method designed to generate corrective instructional text, guiding users to achieve desired motions. The approach utilizes existing frameworks to create a dataset of motion pairs and corresponding corrective texts. A new motion-language model is introduced, efficiently generating corrective instructions based on the datasets. By fine-tuning large language models with these motion-text paired datasets, the method demonstrates precision in instructional generation. Both qualitative and quantitative analyses experimentally confirm the superiority of this approach over traditional methods, which often require extensive manual annotations and struggle with dynamic motions. The researchers’ contributions include a novel data collection pipeline and the application of advanced language models to effectively bridge the gap between motion understanding and natural language instruction generation.

Strengths

1. The paper introduces a novel task—generating motion corrective instructions—which is both interesting and potentially impactful. 2. The authors compile a corresponding dataset (it is unclear if the authors plan to make this dataset publicly available). 3. The writing is clear, and the method is easy to follow.

Weaknesses

1. If I understand correctly, the authors rely on GPT-4 to automatically generate instructions and use MDM to obtain modified motions. How is the quality of this collected data evaluated? Due to the hallucination in large models, the generated instructions may not always be reasonable. Additionally, as a classical method, MDM may not produce natural and controllable motion quality in some corner cases. How do the authors ensure the generation quality of their pipeline? 2. Is the in-context method used by the authors to score the model fair and reasonable? Given the input context length limitation, the number of examples that can be provided to the large model is restricted. 3. Does the authors' division based on upper and lower body introduce any bias? For instance, the instruction "try to touch your toes with your fingertips" cannot be generated. 4. Can the authors provide detailed experimental settings for Llama-3-8B-LoRA? From the results in Table 1, the fine-tuning results based on LoRA are even less effective than the original Llama-3. Do the authors have any explanations or analyses?

Questions

The task proposed by the authors is innovative. My primary concerns are based on the weaknesses. Please address the following in the rebuttal: the quality of the pipeline's generation, comparison strategies with baselines, prompt settings, and experimental setups. This will help illustrate the value of the authors' method.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors' method does not address social impact.

Summary

This paper works on generating corrective instructional text from source and target human motions. Specifically, it utilized existing motion editing framework to collect a dataset for this task. Then, an LLM instruction generation method was proposed to generate text from source and target motions. This paper also shows better performances than baseline methods.

Strengths

1. This paper is well-written and easy to follow, and the supplementary video effectively enhances the content. 2. The motivation of generating corrective instruction from source and target human motions is very interesting and has potential in real-world applications. 3. The proposed method shows better performance than previous methods in the evaluation.

Weaknesses

1. The overall technical contribution is limited. The presented method seems to be an extended application of existing text-to-motion methods, and the adopted components are basically from previous methods. 2. Some recent works (e.g., [1]) are missing to discuss or compare. [1] AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond, CVPR 2024 3. MotionGPT used T5-770M as its LLM backbone, while the proposed method uses ChatGPT-4, so a direct comparison with MotionGPT using different LLMs doesn't seem to be fair.

Questions

To resolve my concern regarding the technical contribution. I would like to know what are the main challenges of this addressed task compared to existing text-to-motion tasks?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations have been discussed in Section 5.

Summary

This paper presents CigTime for generating motion corrective instructions. The key idea is to leverage motion editing to create datasets of motion triplets and use a fine-tuned language model to generate precise and actionable instructions.

Strengths

(1) Introduces a approach for converting motion discrepancies into actionable textual guidance, with applications in sports coaching, rehabilitation, and motor skill learning. (2) Proposes a motion-editing-based pipeline to efficiently generate large datasets, reducing the dependency on manual annotations. (3) Demonstrates the effectiveness of the method through comprehensive evaluations, outperforming existing models.

Weaknesses

(1) For prompting ChatGPT-4 for corrective instruction generation like in Fig.2, I think text description for upper/lower body motion is not enough, prompting ChatGPT4 to give details on how to achive the desired motion is need for "corrective instruction". (2) For corrective motion, blending source and target motion could generate not human-like motion, why don't you try to directly generate corrective motion by masking source motion with editing text prompt. (3) Although corrective motion instruction generation is helpful for lots of applications, I believe the proposed method don't show its potential to enhance any application, just like a toy-level verification. (4) For LLM finetuning, do you extend text vocabulary, or just use a number(index) to represent motion token and directly put it into LLM? (5) For comparison with ICL of other LLMs, it's not fair. The motion index list space is so large and LLM don't see possible mapping between it and text in pretraining, LLM could only succeed when you put similar index list and get similar text like few-shot example. When you try some different index list, it's likely to output similar text. This is because the mapping between index list and text is not established using few-shot examples. So maybe you should provide all these LLM trained with a PEFT(like LoRA) method. (6) What is the result on KIT? And to support finger motion, Motion-X is an optional dataset.

Questions

(1) For tokenizer training, it lacks the loss for codebook. (2) In line 445, FID from 1.44 to 0.24 is not subtle if you are fimiliar with t2m task. Also, why is FID for Fit3D better than HumanML3D? It's quite different from the changing of other metrics.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

As described in questions and weakness.

Response to rebuttal

Thanks for the author's detailed response. I really appreciate the authors' effort to answer my questions. The authors solve most of my problems. The model architecture can be seen in other models, such as [1]. MotionLLM still can input motion instead of videos. However, this paper still introduces an interesting and useful task. Thus, I decided to increase my rating. [1] Chen, Ling-Hao, et al. "MotionLLM: Understanding Human Behaviors from Human Motions and Videos." arXiv preprint arXiv:2405.20340 (2024).