Reply to reviewer ep7Y (R1)
**“Are there motion representations other than the adopted one from HumanML3D tested for the proposed method? How do you expect the motion representation could affect the generation performance?”**
SinMDM is experimented with three different representations: (1) The HumanML3D representation, including joint rotations, global locations and velocities; (2) The Mixamo representation, which includes rotations and global location for the root joint only; and (3) The Zoo representation, which requires extra degrees of freedom and includes rotations + xyz relative location for all joints.
For all datasets, we use rotations of the 6D format, which has been proven by prior work to yield optimal results (See Table 2 in the GANimator paper).
Overall, we showed that our approach is robust with different motion representations.
**“In what circumstances does the method fail?”**
In addition to the limitations mentioned in the conclusions section, we observe a failure case that is common to prior works as well - sub-motions are played in an arbitrary order, and this can be a disadvantage when a sequence of sub-motions needs to be executed in a precise order. For example, while doing the bird dance, one should clap hands immediately after shaking the hips. However, single motion algorithms will place these two sub motions in a temporally arbitrary order.
This can be addressed by enlarging the receptive field (at the cost of lower diversity). We added this limitation to the conclusions section (see in blue).
**“Results of some editing applications... Similar results can be achieved by simple manipulations”.**
Many of the applications enabled by SinMDM may also be implemented using designated algorithms. However, these designated algorithms are neither simple nor general, while ours provides a one-stop-shop for all applications, and yields better quality results when compared to simple manipulations.
For spatial composition, using cut-and-paste may yield a motion where the hands and the legs are un-synched, for example, in a walk motion, we expect that when the left hand is at the front, the right hand will also be at the front. In addition, when using cut-and-paste there is no balance compensation when the body leans to a specific direction, for example, when the upper body leans to the right, the pelvis should balance by leaning to the left. The motion generated by cut-and-paste is merely a copy of the upper/lower part of the inputs, hence has no flexibility to adjust them to match each other. On the other hand, the motion generated by SinMDM can be different from the one on which it was trained, thus there can be seamless synchronization between the upper and lower body parts.
In the context of the style transfer application, while manually synchronizing the pace of steps may yield comparable results, SinMDM achieves this straightforwardly, offering a solution that would otherwise demand a designated, complicated approach, as shown in prior works (Menardais et al., 2004; Lee et al., 2020; Panagiotakis et al., 2013). Temporal synchronization requires the identification of both motions' rhythms, a non-straightforward task by itself. Moreover, stretching one motion according to the rhythm of another is naively done by a linear interpolation, which can cause motion blur, and may not be linear by nature (say, accelerating a leg motion at the beginning of the step and slowing it as it continues). Our model has no such challenges. It generates the required motion such that it is natural, maintaining the motion motifs of the learned input.
[Stephane Menardais, Richard Kulpa, Franck Multon, and Bruno Arnaldi. Synchronization for dynamic blending of motions. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 325–335, 2004.]
[Juyoung Lee, Myungho Lee, Gerard Jounghyun Kim, and Jae-In Hwang. Effects of synchronized leg motion in walk-in-place utilizing deep neural networks for enhanced body ownership and sense of presence in vr. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology, pp. 1–10, 2020.]
[Costas Panagiotakis, Andre Holzapfel, Damien Michel, and Antonis A Argyros. Beat synchronous dance animation based on visual analysis of human motion and audio analysis of music tempo. In International symposium on visual computing, pp. 118–127. Springer, 2013.]