Multi-Source Diffusion Models for Simultaneous Music Generation and Separation

Summary

This paper proposes a unified approach (called Multi-Source Diffusion Model) to solve audio generation, source inputation and source separation. In training time, the diffusion model learns the joint distribution of the audio mixture and solves the three tasks using different inference methods. Audio generation is done via directly sampling the prior. Source inputation is done using the inpainting technique in diffusion models. The source separation is a tailored method: reconstructing audio signals under the constraint that audio mixture is the summation of each audio tracks. Experiment results show the model is successful in all three tasks.

Strengths

1. Originality. The idea of using a generative way to view music source separation is so natural and experiments show diffusion model can perform pretty well. The diffusion sampling process under summation constraint is borrowed from works in CV but it is a natural fit for source separation tasks. 2. Quality: theory is presented clearly and experiments are effective. 3. Clarity: paper is well-written. High-level idea and detail are both clear. 4. Significance: the result is convincing according to the demo in the supplementary material.

Weaknesses

The paper is well-written. I have some minor comments. If it can be clarified, the significance of the work will be increased. 1. The idea of using a generative way to model audio and solve multiple audio tasks in one model is not a new idea. For example, we have a unified VAE approach to do source separation and transcription [1]. Of course the current work is novel but it would be better to include those studies and compare the approaches in related work. 2. What is the novelty compared to NCSN-BASIS? Any intuition for MSDM Dirac? Is it a marginal improvement from existing methods, or not? 3. The introduction of ISDM method is not clear. Is it a baseline method where the assumptions are obviously wrong? Is it another valid approach? Compared to MSDM, is there a trade-off in terms of generation quality and separation performance? 4. The introduction of correction step is okay. Since it is evaluated in the experiment section, could you explain more in the paper? What is the statement to be expected prior to the experiment regarding correction step? [1] Liwei Lin, Gus Xia, Qiuqiang Kong, Junyan Jiang: A unified model for zero-shot music source separation, transcription and synthesis. ISMIR 2021: 381-388

Questions

Q1: The separation result of ISDM and MSDM in Table 2 suggests a trade-off in terms of generation quality and separation performance. Is it confirmed? If so, why using a unified approach for source separation and generation? I understand on one hand, ISDM helps to show the diffusion method is comparable to the SOTA method. However, since MSDM is always lower than ISDM, I doubt the fundamental assumption in this paper is not valid that it is superior to use joint distribution to model generation and separation at the same time. Q2: Could you provide some demos for ISDM method (Now or after publication)?

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Summary

This paper proposes a Multi-Source Diffusion Model (MSDM) for performing both music source separation and generation. Modeling multiple sources jointly allows the system to perform source imputation / accompaniment generation by conditioning on a partial mixture during inference. The authors demonstrate the efficacy of MDSM in both source separation and generation tasks, achieving separation performance competitive with state-of-the-art models. Moreover, by modeling sources independently and introducing a novel score function, the authors achieve separation results surpassing the state-of-the-art for certain instrument classes. While generation and imputation abilities appear to be more limited, MSDM demonstrates the strong potential of generative separation systems that model sources jointly.

Strengths

The proposed method is novel and interesting. Given recent advances in generative source separation and the apparent benefits of iterative refinement for separation [1], a diffusion-based approach seems natural. The proposed method achieves very good separation performance using mostly off-the-shelf components, demonstrating the strength of the general approach. Additional improvements (e.g. weakly-supervised ISDM variant and Dirac score function) are well motivated in the paper. The problem of source imputation / accompaniment generation is more relevant to many music creation workflows than full mixture generation, and is comparatively under-studied. While the accompaniment generation results here don't necessarily "improve" on those demonstrated in the referenced concurrent work [2] in terms of realism, the proposed method approaches the problem from a different angle (joint source separation and generation, diffusion rather than language modeling) and allows for more fine-grained control using multiple conditioning instrument classes (as opposed to singing only). [1] Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, and Jesse Engel. Improving source separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 291–295. IEEE, 2022. [2] Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Generating musical accompaniments from singing. arXiv preprint arXiv:2301.12662, 2023.

Weaknesses

I think the paper would benefit from some additional discussion of the computational and data requirements of the proposed method. Presumably separation time is linear in the number of inference steps (plus correction); it would be nice to see this explicitly compared to Demucs with and without Gibbs sampling. Similarly, if the proposed method is more data-hungry than discriminative methods such as Demucs (e.g. if the proposed method is not competitive when trained/evaluated on MUSDB), this might be worth emphasizing further. Parameter counts for each method would also be nice to see. Based on the listener study and provided listening examples, MSDM struggles to produce coherent and high-quality generations -- even when judged in the context of its synthetic training data. To my ears, it seems like tempo sometimes degrades within the model's 12-second context for unconditional generations, and for imputation generations when strongly metric signals (drums, bass) are not given as conditioning. Overall, the separation results seem much stronger than the generation results.

Questions

The fact that the ISDM method outperforms MSDM on certain sources (Table 2) seems contrary to the intuition that jointly modeling dependencies between sources should improve separation results. Could the authors elaborate on these results, and perhaps conjecture as to why independently modeling sources with ISDM improves (or at least does not substantially deteriorate) performance versus jointly modeling with MSDM? Given the apparently high data requirements of MSDM, have the authors explored fine-tuning the Slakh2100 model on smaller datasets (e.g. MUSDB)? The authors mention potentially using the outputs of a source separation system to scale up the dataset (akin to SingSong), but fine-tuning also seems like an interesting avenue to explore -- especially given its popularity with diffusion models more generally. Did the authors conduct any separation or generation experiments with out-of-distribution data?

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Summary

The authors propose to tackle the source separation problem by seeing it as a separate-tracks music generation problem. They model this distribution using a diffusion model. This approach allows to tackle more use cases such as "source inputation" (accompaniment generation) using the same model.

Strengths

The paper is well written, well-organized, self-contained and the background and reference section are comprehensive and detailed. The proposed method is sound and the authors obtain positive results. Even if not state of the start (source separation is a highly studied task), this original method shows promising results and has the advantage of being conceptually simpler and more general. The new proposed modeling method IDSM Dirac seems relevant, clearly improves on previous methods and might be of interest in other application domains. This method is of course more costly in terms of the amount of data required or in terms of computational resources needed. But this is addressed in the limitations section. The appendix showcases interesting hyperparameter searches about the MSDM Dirac approach, like the impact on the constrained source.

Weaknesses

Adding details on the correction step could make the paper even more self-contained.

Questions

It would have been great to compare the inference time between the different methods, as the diffusion-based methods are likely to be order of magnitudes bigger.

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Summary

My review did not seem to have the right visibility, so I am re-submitting. The authors present an approach for mixture-based diffusion modeling, complete with a novel inference approach that leverages a posterior based on Dirac delta functions and a Monte Carlo approximation of the Dirac likelihood in place of the mixture density. The authors also demonstrate that this approach can be used for generation. The authors evaluate for stem separation using Slakh2100 against established baselines, and also show quantitative and qualitative human-evaluated results for a generation task, establishing a new task and baseline in the process.

Strengths

- The approach the authors proposed is novel and in fact quite general. Orthogonally, I am interested in understanding the potential of the approach in settings with other mixture-based diffusion. - The manuscript is compelling to read, well organized, and very clear. The discussion of related work is quite comprehensive. - Using a waveform as the representation of data generalizes the approach further, opening the door to applicability in other domains. - The authors provide a strong baseline for accompaniment generation with a data-rich setup (Slakh2100) which will serve to be a solid foundation. If not already planned, I would encourage the authors to release as much as they can with respect to evaluation methodology and reproducible artifacts that others can use to evaluate in a similar manner.

Weaknesses

The results of the paper would be made stronger with more discussion of the computational footprint and details for training and inference. How does the footprint compare to Demucs or other methods? How does does computation scale with the amount of data? Further, some brief qualitative analysis of the results might be warranted. Do the authors have an explanation for the relative performance on certain stems relative to Demucs? What are the high-level takeaways from the qualitative results in Table 3 beyond those that the reader could intuit or speculate about? Some feedback on the manuscript: - Section 2.1, in the last paragraph: should "the minute" read "a minute" denoting duration of the context length?

Questions

- Do the authors have a hypothesis as to why in Table 2 performance on the indicated stem categories (i.e. bass, drums) outperforms Demucs? Is there any intuition as to the variance in results across the techniques? - Is there an understanding of the approach's data efficiency? How much does Slakh2100 versus MusDB alone? What about compared to other settings in Demucs?

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Summary of changes to revised paper

We wish to thank the reviewers for their positive feedback and recognition of our method's novelty, clarity, conceptual simplicity, and generalizability to other domains. Based on the reviewers' recommendations, we made the following changes to the main paper: 1. We included a table containing inference times and the number of parameters in the revised paper (Table 4). 2. We included a table in Appendix F of the revised paper (Table 6) where we compare the performance metrics of Demucs against MSDM either tested, finetuned, or trained on MusDB. 3. Moved the section on the sampler (Section 4.2.4) to the main revised paper. There we specify and explain further the functioning of the correction step. 4. Other minor changes. These changes are highlighted in blue in the revised paper.