MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence

Summary

The authors proposed a novel framework to address the motion-to-music and music-to-motion tasks. They leveraged aligned latent spaces between motion and music, a multi-modal diffusion transformer, and a cross-guidance sampling strategy. Experiments were conducted to demonstrate that their approach outperforms their counterpart models.

Strengths

The paper proposes several interesting concepts, including rhythmic contrastive learning to produce aligned latent spaces across different modalities and transformer-based diffusion models to address music-to-motion and motion-to-music synthesis tasks. Additionally, it offers the possibility to jointly synthesize music and motion. The user study and ablation study are also well conducted.

Weaknesses

Please refer to the questions section.

Questions

- The authors discussed the architectures of the multi-modal VAE in Section 3.1. Could the authors provide more specific implementation details, including - Whether the motion vectors in shape $\mathbb{R}^{T_m\times J\times 2}$ are first of all flattened to $\mathbb{R}^{T_m\times(J\times 2)}$ for VAE? If not, how are they processed? - What exactly are $T_a$ and $T_u$ in Line 105 for the music VAE? Does this imply that the music VAE has a lower downsampling ratio than the motion VAE in the temporal dimension? - Is normalization needed for the autoencoder representations of the both modalities so that they can be further processed by the diffusion model? - The authors went through how they constructed the motion-music latent clip for VAE training which incorporates the kinematic amplitude indicator as depicted in Eq. 2. Could the authors further elaborate on why incorporating this is important by e.g., showing a histogram of kinematic amplitude in the dataset and how it is sampled?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately address the limitations.

Summary

MoMu-Diffusion is a motion-music co-generation model with an aim of improved temporal synchronization of the generated motion and music sequence, based on two major components: 1. a bidirectional contrastive rhythmic VAE (BiCoR-VAE) that provides aligned latent space of the motion and music through joint training, and 2. a multi-modal diffusion transformer (DiT) the model the latent space with cross-guidance sampling.

Strengths

The proposed BiCoR-VAE's rhythmic contrastive objective using a kinematic amplitude indicator is well inspired and interesting approach to extract key frames that contain motions of interest. It is based on the valid assumption that the motions in the video data likely align with the beat present in the music. The beat matching metrics in Table 2 justifies the improvements made in the proposed approach, and the ablation studies has been done thoroughly in Table 7. Human evaluation in Figure 6 shows strong prererence to the proposed model.

Weaknesses

I would like to see the qualitative samples of the baselines and the ablation models to the demo page as well to form the readers opinion on the claimed synchronization of this work. While the presented results in the manuscript looks convincing, having direct comparisons in the demo will be helpful for the readers to have a better idea of the improvements.

Questions

* (line 121-122): I think it would be helpful to add a specific example of the framerate differece between motion and music frames. Currently it might read like the motion framerate is always higher (by integer multiple) than the mel spectrogram framerate. * Assuming that the framerate design is defined by the mel framerate (followed by the vocoder), is resampling the motion sequence to match the mel framerate trivial enough to assume that we have the time aligned motion and music frames in the pre-processing stage?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have mentioned the limitations in this work, including limited computational and data budget to verify the idea to scaled up dataset that potentially include video clips that do not adhere to the assumptions made in the rhythmic contrastive objective.

Summary

The paper propose MoMu-Diffusion, that generates both music-to-motion and motion-to-music videos. MoMu-Diffusion achieves SOTA results and is able to generate joint distribution of music and motion instead of one way.

Strengths

1. Generate joint distribution of music and motion 2. Propose rhythmic contrastive learning to align music and motion. 3. Demo included, reasonable evaluation metrics and great experiments.

Weaknesses

1. I might miss it, but that would be nice if there is an ablation study of the hidden size and layers of Diffusion Transformer.

Questions

1. How long can you generate the video in the joint distribution generation setting? Is that possible to use DiffCollage (Zhang et al., 2023) for longer video generation?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. Maybe can explore longer video generations.

Summary

This paper proposes a framework that enables the generation of music from motion, motion from music, or both simultaneously while maintaining synchrony between the two. To achieve this, kinematic amplitude is extracted from the motion, and motion audio clip segments corresponding to different kinetic amplitudes are sampled. Contrastive loss is applied to the audio and motion embeddings of these sampled segments to ensure that the embeddings align with each kinetic amplitude. The resulting embeddings are then used for joint diffusion, enabling audio-to-motion, motion-to-audio, or joint generation. A method for applying cross guidance is also proposed. Quantitative evaluations were conducted based on the beat information of the generated music, as well as the FID and diversity of the generated motion. A user study demonstrated that the generated outputs were preferred over those from previous models.

Strengths

The paper proposes an embedding method using kinetic amplitude and contrastive learning to ensure that the intensity of motion and music/motion are well-aligned. This approach effectively leverages prior knowledge of motion. I believe this methodology can be applied to problems requiring synchronization of intensity and timing beyond just motion-music synchronization. The use of diffusion for multimodal generation, incorporating DIT and introducing a cross-guidance sampling strategy, also holds potential for future research in multimodal generation.

Weaknesses

The proposed methodology leverages the relationship between changes in motion and beats in music as the primary element of synchronization, with evaluations mainly focused on this aspect. However, there can be instances where the beatness in motion or music is not clear (especially during certain periods). A discussion on how to handle such cases, or considerations beyond beatness, would be beneficial for achieving more natural generation.

Questions

Suggestion: Section 3.2 is a crucial part that explains BiCoR-VAE, but the process of sampling clips was difficult to understand. Referring to Figure 2(a) made it easier. Including an example to explain this process would make it more comprehensible.

Rating

9

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

-