Recent progress in music generation has been remarkably advanced by the state-of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet, sampling with the MusicLM requires processing through these LMs one by one to obtain the fine-grained acoustic tokens, making it computationally expensive and prohibitive for a real-time generation. Efficient music generation with a quality on par with MusicLM remains a significant challenge. In this paper, we present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that generates music audios of state-of-the-art quality meanwhile reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling 10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for semantic modeling, and applies a novel dual-path diffusion (DPD) model and an audio VAE-GAN to efficiently decode the conditioning semantic tokens into waveform. DPD is proposed to simultaneously model the coarse and fine acoustics by incorporating the semantic information into segments of latents effectively via cross-attention at each denoising step. Our experimental results suggest the superiority of MeLoDy, not only in its practical advantages on sampling speed and infinitely continuable generation, but also in its state-of-the-art musicality, audio quality, and text correlation. Our samples are available at https://Efficient-MeLoDy.github.io/.
Paper
Similar papers
Peer review
Summary
This paper introduces an LM-guided diffusion model designed for the generation of music audios. The proposed system follows a three-step process, starting with the extraction of audio-text embeddings (MuLan tokens) using a pre-trained MuLan model. Next, semantic tokens (wav2vec tokens) are generated with the aid of a language model. Finally, the synthesized music audio is produced using a dual-path diffusion model. Experimental results highlight the superiority of the proposed model, showcasing its practical advantages in terms of sampling speed and the ability to generate music seamlessly.
Strengths
1. The paper exhibits clear and well-written sections before the experiments, ensuring the reader’s comprehension and engagement. 2. The paper addresses a highly meaningful topic and task, aligning with the current trends and hot topics in language modeling. 3. The proposed method in this paper is well-designed, effectively integrating domain knowledge in music and demonstrating a deep understanding of the subject matter.
Weaknesses
1. The overall structure of the paper’s writing could benefit from optimization. For example, important experimental results should ideally be included in the main body rather than being relegated to the appendix. 2. The paper lacks a comprehensive objective evaluation of the main experiments, particularly in comparing the proposed method with the baseline approaches instead of only using objective metrics in the ablation study. 3. Based on the findings in Table 3, the proposed method exhibits a noticeable drop in musicality and text correlation compared to the baseline approaches. This raises questions about the motivation behind the proposed method. Is the trade-off in performance for the sake of efficiency justified? Does the efficiency gained by the proposed method outweigh the potential loss in important aspects such as musicality and text correlation? 4. The appendix file is difficult to locate, as there is no indication or hint for readers that it is available on the demo page. Clear guidance or references within the paper would greatly assist reviewers and readers in accessing the relevant appendix materials. 5. The paper lacks a clear explanation for certain design choices in the proposed method, such as the motivation behind utilizing an angle parameterization for the diffusion schedule and the specific design of Equation 4. Without a detailed rationale and justification for these design decisions, it becomes difficult to comprehend their significance and how they improve the overall performance and effectiveness of the method.
Questions
NA
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
NA
Summary
The authors present a text to music model, with the aim of achieveing faster than real time generation on a V100 gpu. The authors introduce a latent space diffusion model. The latent space is obtained with an adverdarial VAE (replacing the VQ VAE from MusicLM), and a first LM is trained on the semantic tokens just as in MusicLM. The LM is also conditioned on the quantized output of a Mulan model. Then a latent diffusion model is trained to predict the latent representation of the VAE, conditioned on the semantic tokens. In order to improve the speed of the model, the authors uses a dual path architecture, i.e. they alternate between modeling long range and short range dependencies. Short range dependencies are modeled with a simple RNN, while long range ones are modeled with an attention based model. The model is trained on ~260k hours of music with tag labels, enriched with ChatGPT outputs into full descriptions. The authors compare with the publicly available samples of MusicLM and Noise2Music. They show that their model achieves better quality but overall worse musicality and text fidelity. They also report the FAD on MusicCaps (5.1) which is worse than that of MusicLM (4.0) or Noise2Music (2.1).
Strengths
- large improvement in terms of speed compared with the many models and diffusion steps required by Noise2Music. - improvements on the diffusion schedule. - interesting idea of using a dual path architecture to improve the runtime. This allows to take full advantage of the non causal aspect of the diffusion process used here instead of auto-regressive modeling. - Runtimes on Table 2 are impressive. - subjective evaluations show improved quality.
Weaknesses
- still requires a large number of models to train, e.g. Mulan model, a semantic model w2v-bert + one LM for the semantic tokens, and one dual path diffusion model, without counting the VAE. - ablation studies are limited to the noise schedule. Not strong objective motivation for most of the design choices. - authors mention that the Mulan Cycle consistency score is worse for the test set MusicCaps than their model. However that could be an artefact of how the authors' Mulan model is trained, e.g. on weak labels generated from partial tags rather than full descriptions, in particular considering that their own model is trained with this Mulan model, so that it could overfit any bias it has. - when comparing the number of function calls in Table 3, the authors forget to mention that for MusicLM, each call require the evaluation of a single time step, while their own model require to process all the timesteps of the input sequence. In particular, in the context of batched generation, this can make a big difference between the two models. - objective metric FAD is worse than existing methods, and two of the subjective metrics (musicality and text correlation). Subjective evaluations are done on the samples released by the other methods which might be better than the average though. Note that the authors provide their supplementary material through a website, which might not be an acceptable way of doing so, as it doesn't enforce the deadline for the supplementary material submission.
Questions
What are "velocities" ? this term is not properly defined when introduced. Why are the authors using different noise levels for different segments? I see in the supplementary this is used for inpainting. Other than that regular sampling would always use the same noise level for all segments? Is there a benefit to this training method for regular sampling? In Section 5.1, the authors mention they filter the data to focus on non vocal music. Is the 257k hours number given before or after this filtering? The sample rate of the proposed of the VAE seems relatively large compared to the one used in MusicLM, is this something the authors experimented with?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The authors fail to discuss potential adverse effect of automation of creative jobs. The authors fail to discuss the potential breaches to intellectual property for the data used. In particular, the authors should discuss how the dataset was assembled.
Summary
The authors introduce a framework (MeLoDy) for music generation. This new framework significantly improves the inference speed of MusicLM while producing high-quality results. in terms of musicality, audio quality, and text correlation. The main reason for this speedup is that the authors replaced the coarse and fine acoustic stages in MusicLM and AudioLM from a language model to a dual-path diffusion model (DPD) in the latent space, while keeping a LM to perform the high level semantic modeling process similar to MusicLM. The authors show that the resulting model can perform similar to MusicLM while being faster at sampling.
Strengths
1. The authors address efficiency, which is a very important issue and can be easily overlooked in new research. 2. I find the idea of combining LM and diffusion models very intuitive and has a lot of potential. 3. The authors put a lot of effort into evaluating the generated samples by using human feedback. 4. The generated samples sound good (since they are not cherrypicked).
Weaknesses
1. Complex pipeline system: The authors propose a complex pipeline with many components, moving parts, and many hyperparameters. This makes the system hard to reproduce, and the contribution of individual components is not clear. The authors provide a limited ablation study in the appendix. I think in a system like this, an extensive ablation study is needed. Examples: (a) What role does the encoder play? (b) Would different encoders affect the quality? (c) The authors used different Mulan audio and text towers compared to MusicLM; what effect does that have? (d) The authors mention they used attention on the course path and an SRU on the fine path; what role do these play? (e) What is the effect of merging and repeating? I think a lot of these decisions sound arbitrary making the work read like a technical report. 2. Reproducibility, private dataset: the dataset is very vaguely described, making the proposed system even harder to reproduce and evaluate. The dataset plays a large role in the quality of the generation. 3. Clarity: Section 4 in particular is very hard to read; the description of the system is not very clear. Additionally, some details are omitted, which can be in part because of the complexity of the pipeline. I recommend a restructure of this section. The appendix significantly helps in understanding the details. 4. Evaluation: I think a comparison with the more specialized models can greatly improve the quality of the evaluation. Even if the specialized models (like genre) perform better under specific conditions. I believe it's important to illustrate the gap between these models and those that rely on free-form text generation. I think the current evaluation is limited. 5. I believe the related work section is limited. If possible, I think this section should be extended to include related work and alternatives that can be used in the proposed pipeline.
Questions
A- Would you please address the weekness above? B- In line 158, you say: "In our experiments, we find that the stability of generation can be significantly improved if we use token-based discrete conditions to control the semantics of the music and let the diffusion model learn the embedding vector for each token itself." For me, it's not clear what you mean here: are the tokens coming from the SSL Wav2Vec2-Conformer? How are you "letting the diffusion model learn the embedding vector for each token" if the embedding vectors are coming from the k-means clusters of the Wav2Vec2-Conformer? C-Line 183, The notation is not very clear for me; I assume the same MLP is performed in parallel for each token. and this is a different MLP than the one in equation 10? Is this correct? If so, I suggest stating the parameterization to make it more clear. D-Line 192, In order to reduce complexity, you set k = sqrt(L), which allows you to process long sequences with complexity that grows linearly with the sequence length L (after the self-attention over the coarse path); this is the reason you state in the abstract that "infinitely continuable generation." Is that correct? Does not this come with the limitation that as this number K grows, the coarse path becomes even more coarse (more abstract)? Is this explained somewhere? E-Line 284, why did you choose to use an outdated VGGish model? Wouldn't the results be more accurate if one used more capable models, like a transformer (in AudioGen) or models that are trained to perform musical tasks, e.g., detect musical instruments?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The authors discussed the limitations of their work. One additional thing that comes to mind is copyright in music, which is a general issue for generative models. Is there a way to determine if the model reproduces copyrighted material from its training data?
Summary
This paper proposes MeLoDy, a diffusion-based text-to-music generation system. The system comprises four parts: 1) the MuLan contrastive model, 2) semantic language modeling using Wav2Vec2-Conformer, 3) a dual-path diffusion model to generate the latent, and 4) a 1D VAE-GAN autoencoder for learning the latent. The greatest contribution of this paper is the dual-path diffusion model, while other components follow a similar paradigm to that proposed in previous models, albeit with minor upgrades. In the dual-path diffusion model, instead of applying diffusion on the full audio length, the paper suggests segmenting the target of length L into approximately sqrt(L) chunks. A RoFormer model is then used to process inter-segment, while an RNN is utilized to process the intra-segment. The dual-path diffusion model trains on a velocity objective with a novel linear angle schedule. A subjective listening test with music producers as listeners demonstrates that the proposed model outperforms previous methods in quality but falls short in musicality and text correlation.
Strengths
The paper proposes a novel dual-path diffusion model, which could be an effective alternative to the current design of the diffusion model. A new schedule for the diffusion process is also proposed. The model generates satisfactory results, as shown on the demo website, and the writing in the paper is clear.
Weaknesses
From my perspective, there are two minor weaknesses: 1. The authors should further justify the choice of the dual-path diffusion model in terms of its effectiveness. When compared to an alternative diffusion module for the proposed system, the prediction length in the dual-path diffusion is smaller. However, according to the paper, in the dual-path diffusion, at each diffusion (reverse) step, the input needs to run through two modules: RoFormer and SRU. While I can imagine that the RoFormer can be run in parallel for both inter-segment and intra-segment, I believe the SRU, as a type of RNN, still requires sequential computation. Therefore, an alternative diffusion model that works on 1D (waveform) or 2D (spectrogram) latent could leverage the benefits of parallel computing and run faster than the dual-path diffusion model on a GPU under the same sampling steps. I would appreciate the authors' insights on this point. It would be beneficial if this could be included in the paper to provide a more solid justification for the design choice. 2. The experiments show that the proposed model does not outperform in terms of musicality and text correlation, which calls the design of the proposed model into question. However, I acknowledge that the data used in each model differ, making a controlled and reproducible comparison nearly impossible.
Questions
Typos: Line 277: The sentence appears to be incomplete: "For sampling, the predicted velocity is linearly combined as ." Issues with introducing the system: In section 4.1.1, the concept of "velocity" should be introduced beforehand. Its sudden appearance made my initial reading confusing. Also, I would like to know about the justification of the dual-path diffusion design choice. Please see the first point in the weakness section.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations of the proposed system are adequately discussed in the final section.
Summary
The paper introduces MeLoDy, a music generation method combining Language Models (LMs) and Diffusion Probabilistic Models (DPMs). It addresses the challenge of generating music from diverse free-form text descriptions while mitigating high computational costs associated with current state-of-the-art models like MusicLM and Noise2Music. Key contributions include the development of MeLoDy, which significantly reduces computational requirements, the proposal of efficient Dual-Path Diffusion (DPD) models, an improved sampling scheme for DPD, and a successful implementation of audio VAE-GAN for effective continuous latent representation learning.
Strengths
The DPD model is particularly innovative as it is a variant of continuous-time diffusion probabilistic models that operates on low-dimensional latent representations. As for significance, the paper offers a substantial contribution to the field of music generation by showcasing how complex relationships across long-term contexts can be modeled and by overcoming limitations of prior works.
Weaknesses
The proposed method builds upon existing research, employing the diffusion probabilistic model for music generation. Although the combination with the dual-path architecture is intriguing, it still largely borrows from previous architectures like AudioLM, MusicLM, and MuLan.
Questions
What's the rule of thumbs for selecting the number of chunks when training?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The authors have discussed about the limitations of this work in the discussion section.
Thanks, authors, for explaining. According to the reply and the reply to other reviewers, I believe my concerns have been addressed.
I'd like to thank the authors for the reply. First, of course, I'd like to agree with the authors that the complexity of the method is subjective. I'd like to stress what I mean for clarity: I think presenting a complex system without an ablation study that shows the contributions of the components, the design choices, and the hyper-parameters is only acceptable in the context of a technical report. I think for a conference paper, the reader should be able to (1) reproduce your work and (2) understand the effect of each component you propose on the results. > which leads to a combinatorial cost. I agree that trying different combinations is, of course, not feasible, but as you explain, leave-one-out comparisons are standard in the literature. However, I didn't seem to find the leave-one-out comparison in the paper or the appendices. For example, you explain that you started with the musicLM framework and switched from w2v-BERT to Wav2Vec2-Conformer but there are no results showing the effect of this change. Another example, replacing the components of mulan. There, you can also show the role this plays in your results. Another examples include augmenting the text for training mulan etc...
Response to the reviewer
Thanks for the reviewer's insightful comments. We would like to reply point by point here: - (1) **Reproducibility**: We agree that the reproduction of MeLoDy is essential for this paper, therefore we made much effort to formally describe every details of our proposed method (mainly on the design of dual-path diffusion). We believe that our detailed presentation as well as the setting of all hyper-parameters would facilitate the reproduction of this model and our results. - (2) **Understanding the effect of each component**: We would like to express our regret that not all the design choices have been explained well for their motivations, as we tried to present all the details for better reproducibility. As discussed with other reviewers, we have accordingly modified the structure of our paper, such that more explanations in textual form has been added in the main paper and much detail has been moved to the Appendix. The author rebuttals to the reviewers also include a part of description explaining the design choice of our model. For example, reviewer **ieEr** replied that our further explanations have adequately addressed all his/her concerns. - (3) **Switching from w2v-BERT to Wav2Vec2-Conformer**: As mentioned in the rebuttal, considering the scale of data we are training and the difficulty of measuring the effects of SSL module, "we can only listen to some random samples (10-100 per system) led by different SSL modules (i.e., Wav2Vec2, Wav2Vec2-Conformer, and w2v-BERT), and subjectively vote for a better module." We will also add this justification in our revised paper. We believe objectively evaluating the SSL module in music/audio generation is still an unsolved open research problem, and is beyond the scope of this paper. If the reviewer has a better suggestion on this evaluation, please let us know. - (4) **Replacing the components of mulan**: As per mentioned in the rebuttal, we would like to emphasize that "we actually used comparable network architectures for audio and text towers". Please correct us if we misunderstood the meaning of "replacing the components of mulan" stated by the reviewer. - (5) **``Another examples include augmenting the text for training mulan''**: We agree to the reviewer on this point that the effects of text augmentation on MuLan were not well discussed in the main paper. We would like to express our apology and justify the training pipeline of our MuLan in the revised paper. In fact, after the text augmentation, some objective metrics used in music retrieval, e.g., mAP, were declined yet the music generation on long-form text prompt was obviously improved (improvements were subjectively determined by music professionals). As a result, we finally take the subjective test as the golden rule in evaluation, and ignored somewhat contradictory objective measures. - (6) **``I didn't seem to find the leave-one-out comparison in the paper or the appendices.''**: We would like to re-claim our results of leave-one-out comparisons here. In the main paper, since we highlighted two novelties of this paper, each of which has been compared to the conventional methods in our experiments: 1. The novel **dual-path architecture** for the latent diffusion model (resulting in dual-path diffusion (DPD)): the dual-path architecture has been compared to UNet-1d [1] and UNet-2d [2]: | Architecture | Velocity MSE (↓) | SI-SNR (↑) | | -------- | ------- | ------- | | UNet-1d [1] | 0.13 | 5.33 | | UNet-2d [2] | 0.15 | 4.96 | | **DPD** (Ours) | **0.12** | **6.15** | 2. The novel **angle schedule** for angular-parameterized diffusion: | Angle Schedule | Steps | FAD (↓) | MCC (↑) | | -------- | ------- | ------- |------- | | Uniform [1] | 10 | 8.52 | 0.45 | | Uniform [1] | 10 | 6.31 | 0.49 | | Ours proposed in Eq. (4) | 10 | **5.93** | **0.52** | | Ours proposed in Eq. (4) | 20 | **5.41** | **0.53** | Despite an argument of "technical report", we sincerely ask the reviewer to re-consider the significance of this work, as a novel music generation model that first allow generating music audio of high quality and competitive musicality faster than real-time on a consumer-level GPU. We believe our revised paper has been greatly improved with the proposed amendment in the rebuttal and has addressed all concerns about the justification of the design choices. [1] Flavio Schneider et al.. Moûsai: Text-to-music generation with long-context latent diffusion. 2023. [2] S Forsgren and H Martiros. Riffusion - stable diffusion for real-time music generation. 2023.
about the complexity
Regarding concern #4: this is only true if the attention is dominating the run time, which should be experimentally validated. I don't see any novel elements regarding ablation studies, although the authors state they have reorganised the paper. Having no new elements, I will not increase my score. Regarding the multiple noise levels, another possibility is to use the same noise level everywhere but to do "teacher forcing" on the already generated segments, i.e. discarding the output of the model for those segment, and using the previously obtained output, interpolated with the proper amount of noise. It would be interesting to compare the two, as the "teacher forcing" approach has a simpler training procedure. Regarding the VAE-GAN: if the authors have any results on that, it would make the paper stronger to include those.
Response to the reviewer
Thanks for the reviewer's insightful comments. We agree to the reviewer's suggestion on providing additional empirical evidence to strengthen our arguments on concern #4. We thereby would like to supplement the actual time costs measured for different LMs in comparison to DPD on a V100 GPU. | Component | Time for generating 10s music (s) | | -------- | :-: | | Semantic LM | 3.5 | | Coarse Acoustic LM | 26.4 | | Fine Acoustic LM | 67.7 | | **DPD** (5 steps) | **1.3** | | **DPD** (10 steps) | **2.3** | | **DPD** (20 steps) | **4.2** | *P.S.* The time costs of LMs were measured with a popular unofficial implementation of MusicLM [1] (using 12 RVQ quantizers -- 4 for coarse and 8 for fine acoustic LMs, respectively). In other words, adopting DPD in place of coarse and fine LMs can speed up generation roughly 22x to 63x. We will also add these results in our final paper. To address the reviewer's other concerns, we will also try to experiment on the "teaching forcing" strategy for training and adding more analysis on VAE-GAN. Yet, because of the tight schedule, please pardon us if these results could not be released during this discussion phase. [1] https://github.com/lucidrains/musiclm-pytorch.
Decision
Accept (poster)