Music is an integral part of human culture, embodying human intelligence and creativity, of which songs compose an essential part. While various aspects of song generation have been explored by previous works, such as singing voice, vocal composition and instrumental arrangement, etc., generating songs with both vocals and accompaniment given lyrics remains a significant challenge, hindering the application of music generation models in the real world. In this light, we propose SongCreator, a song-generation system designed to tackle this challenge. The model features two novel designs: a meticulously designed dual-sequence language model (DSLM) to capture the information of vocals and accompaniment for song generation, and a series of attention mask strategies for DSLM, which allows our model to understand, generate and edit songs, making it suitable for various songrelated generation tasks by utilizing specific attention masks. Extensive experiments demonstrate the effectiveness of SongCreator by achieving state-of-the-art or competitive performances on all eight tasks. Notably, it surpasses previous works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally, it is able to independently control the acoustic conditions of the vocals and accompaniment in the generated song through different audio prompts, exhibiting its potential applicability. Our samples are available at https://thuhcsi.github.io/SongCreator/.
Paper
Similar papers
Peer review
Summary
This paper introduces SongCreator, a novel song-generation system designed to create complete songs with both vocals and accompaniment from given lyrics, addressing a significant gap in music generation. The system incorporates a dual-sequence language model (DSLM) and an innovative attention mask strategy, enabling it to understand, generate, and edit songs for various music-related tasks. Extensive testing shows that SongCreator performs exceptionally well, outperforming previous models in tasks such as lyrics-to-song and lyrics-to-vocals generation, and offers the unique ability to control the acoustic characteristics of vocals and accompaniment independently.
Strengths
1. The paper tackles an important topic within the field of song generation, a complex and evolving area of AI research. 2. Detailed and comprehensive experiments are conducted to validate the effectiveness of the proposed models and methods. 3. The work is thoroughly developed, presenting a holistic approach from theory to practical application.
Weaknesses
1. The introduction lacks clear logic and motivation, making it difficult to discern the unique contributions of this work compared to existing models like Suno and Udio. The discussion about the applicability of AI-generated content across various media types feels outdated, especially given the recent advancements in music generation technologies. 2. The presentation of related works in Table 1 is confusing; Jukebox is omitted from the table yet discussed extensively in the text. The paper primarily frames its contributions as extensions of Jukebox, which may understate the originality of the proposed methods. 3. The exclusion of prominent music generation products like Suno and Udio from a detailed discussion is a notable oversight. This could either be addressed comprehensively within the limitations section or by providing a clearer comparison within the main text to delineate how this work differentiates from those products.
Questions
1. How does the proposed model handle the arrangement task without access to traditional music sheets? 2. While SongComposer operates on MIDI files, how does the proposed model manage composition tasks directly from audio files? 3. Given that harmony is a subjective quality and a fundamental goal of music generation, how do the authors define and assess harmony in their evaluations, and why do references [15-25] lack this aspect? 4. Does the term "universal song generation" carry specific implications within this context, and is its universality considered a significant contribution of this work? 5. Is this the first model capable of generating both vocals and accompaniment from lyrics alone? What unique capabilities does your model have that distinguish it from others? 6. The related work section touches on singing voice synthesis and speech editing but lacks a detailed discussion on lyric-based music generation. What is the rationale behind this selection? 7. Figure 1 lacks a clear caption explaining its elements, which could lead to confusion. Clarifications on what the audio icons and the term "song" represent would be beneficial. 8. On line 215, how was the decision made to allocate 80% and 20% in your methodology? 9. The structure of input and output for both training and generation phases appears complex. Could you clarify whether the model supports multiple combinations of lyrics, vocals, and accompaniments during these phases? 10. How is the song editing task implemented? How does the system manage edits that only modify part of the lyrics but require corresponding changes in vocals and accompaniments? It would also be beneficial to understand the robustness of the editing performance across diverse and extensive datasets.
Rating
3
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
The most critical limitation noted is the suboptimal audio quality, which appears fragmented and affects the overall user experience with the generated music. This issue could significantly impact the practical deployment and acceptance of the proposed model.
Summary
The paper presents a novel approach for lyrics-based song generation. The method leverages language models for semantic tokens modeling and then applies latent diffusion model to generate final music. A dual-sequence language model (DSLM) is introduced to not only handle vocals and accompaniment but also integrate them together. The model is applicable to a variety of lyrics-based song generation tasks and extensive experimental results demonstrate its effectiveness.
Strengths
- SongCreator is flexible and applicable to eight different lyrics-based song generation tasks. - The components in SongCreator are mostly open-sourced and details of training and model hyper-parameters are provided. - Generated samples are in good quality.
Weaknesses
- While the proposed system looks promising, it requires multi-stage and multiple models during inference. The latency is not discussed. - Training data heavily relies on the quality of sound separation tool (Demucs). While its quality is ok for two stream (vocal & accompaniment), it becomes more problematic when more streams are separated and therefore has limitations on more instrumental-level controls. - Semantic tokens (either vocal or accompaniment) are a mixture of multiple features, and therefore it is challenging to support disentanglement control (e.g., tempo).
Questions
- How do you align lyrics and Voice Activity Detection (VAD) results? could you elaborate more on this? - Does it support languages in addition to English?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Authors discussed limitations in Section 5.
Summary
The authors present SongCreator, a music generation system capable of simultaneous generation of vocals and accompaniment tracks. SongCreator consists of a language model generating two streams of BEST-RQ ([57] in the paper) semantic tokens, one for the vocals and the other for the musical accompaniment, a non-autoregressive transformer mixing the two streams, followed by a latent diffusion model translating the semantic tokens to VAE latents that is then decoded back to audio. The model is conditioned on lyrics and optional style audio prompts for either vocals or accompaniment. The authors suggest using Bidirectional Cross Attention (BCA) for the two-stream language model, which is the main modeling contribution of the paper. A thorough ablation study is performed suggesting that BCA is crucial for coherent generation of songs - i.e. music with both vocals and instrumental accompaniment. Additionally, the zero-shot voice cloning capabilities of SongCreator is demonstrated through comparison to baselines such as VALL-E ([9] in the paper), showing superiority in terms of singer voice similarity. Finally, the benefits of multi-task training is demonstrated, and in specific, in an interesting contribution, the authors show that vocal generation benefits from dual accompaniment and vocal generation objectives seen during training.
Strengths
1. A diverse lyrics editing capabilities of SongCreator is demonstrated. Namely three variations: Direct editing of the mixture track, editing a separate vocal track or editing the vocals given an accompaniment track. This is highly practical for music production, and demonstrates the flexibility of the proposed system. 2. The design of the Bidirectional Cross Attention (BCA) between the vocal semantic decoder and the accompaniment semantic decoder is an important modeling contribution. Moreover, it is validated extensively throughout the experiments section (section 4), demonstrating clear advantage of BCA compared to both independent two stream modeling and single stream modeling alternatives. 3. The superiority in the SECS metric compared to a VALL-E like architecture is an important contribution (table 6), demonstrating effective zero-shot voice cloning capabilities of the proposed model. 4. The significant increase in vocal generation quality, when given an accompaniment track is an important contribution, demonstrating the effectiveness of the Dual Sequence Language Modeling (DSLM) in learning from temporally aligned auxiliary musical signals. 5. The authors perform an extensive experimentation in order to validate their modeling design choices. A wide range of baselines is implemented using reproduction of prior work, in addition to ablation study on the main components of SongCreator.
Weaknesses
- The quality of samples, as demonstrated in the demo page, is relatively low compared to prior work. - As stated in line 361, SongCreator cannot control and doesn't support global textual descriptions of genre, style or instrumentation. This is a major weakness compared to prior work. - line 119 - The decision to use best-RQ as the semantic tokenizer should be supported with an ablation study, comparing it to open-source alternatives such as MusicFM [1] or MERT [2]. In addition, it is unclear how did the authors validate it indeed "encapsulate sufficient semantic and acoustic details" as stated in line 121? - Though the baseline set is broad, neither baseline is an official checkpoint. All baselines are reproductions of prior work, which lowers the reliability of comparison. The comparison in table 15 reveals a significant gap in performance comparing to the SingSong official samples. - It is unclear which samples were used for subjective evaluation. In specific, a crucial factor is whether a source separated data was used or studio stemmed data. A source separated data may bias the results towards models with audio prompts due to information leakage between the artificially extracted stems. - No planned model checkpoints release, which reduces the reliability and reproducibility of the research significantly. [1] A Foundation Model for Music Informatics, Won et al. 2023 [2] MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training, Li et al. 2023
Questions
- line 215 - what is the rationale behind using the None strategy 20% of the time? - The sentence in lines 149-153 needs rephrasing. It is difficult to understand. - Table 3 - how do the authors explain the opposite trend in FAD compared to the trend in subjective quality? - Table 4 shows a significant increase in vocal quality, when given an accompaniment track. Did the author validate the diversity of such a model, in the sense that there's no information leakage from the artificially separated accompaniment track? - In the detailed description of the dataset, in line 607, it is unclear whether "separate instrumental music and vocals" refers to stemmed data, based on studio recorded tracks, or to single instrument / a cappella performances. - table 9 - music continuation study - why was the reproduced MusicGen omitted from this study? - The quality of the "original song" samples from the demo page is relatively poor. In the paper, it is reported that a sampling rate of 44.1kHz is used in the work. What is the reason for the relatively low quality of the original data samples?
Rating
8
Confidence
4
Soundness
4
Presentation
4
Contribution
3
Limitations
The authors adequately addressed the limitations of their work, as well as the broader impact of the proposed system. In specific, potential negative usage of the voice cloning capabilities of SongCreator is discussed. Moreover, the authors decide not to publish model checkpoints due to the harmful potential of such feature.
I thank the authors for the clarifications. My concerns were adequately answered, and my score would remain unchanged. The only thing that remained unclear to me is the quality of the "original song" samples in the demo page. Were this samples taken from the DISCO-10M ([68] in the paper), or from the in-house datasets? In both cases, the low-quality isn't fully explained by recording environment and equipment. In case the samples were either preprocessed, or processed using one of the encoder models of SongCreator, this should be mentioned both in the paper and in the demo page.
Response to the reviewer
We appreciate the reviewer's constructive comments. We would like to clarify that the "Original Song" samples on the demo page are reconstructed samples, not the original recordings. These samples have been reconstructed using BEST-RQ encoding and LDM decoding to eliminate the potential impact from the encoding and decoding processes during our experiments. Talking into account the reviewer's suggestion, we have updated the text on the demo page to accurately reflect this information. Specifically, we have changed "Original Song" to "Original Song (Reconstructed)" and added a note to explain this. We will also make the necessary revisions in the final paper to ensure that this information is clearly stated.
Summary
The authors introduce a novel system for lyrics-based song generation. It can handle various inputs (lyrics, vocal prompts, accompaniment prompts) and generate different outputs (full songs, vocals only, etc.). The paper proposes a dual-sequence language model (DSLM) that separately models vocals and accompaniment while capturing their interactions through a bidirectional cross-attention mechanism. An attention mask strategy is specifically designed for song generation to song generation tasks of various forms. The authors present competitive performance on eight different song-related tasks.
Strengths
- The model can handle multiple song-related tasks within a single framework, including lyrics-to-song, lyrics-to-vocals, accompaniment-to-song, vocals-to-song, music continuation, and various editing tasks. - The bidirectional cross-attention mechanism in DSLM enables the model to capture the mutual influences between vocals and accompaniment, contributing to more harmonious and coherent generation. - The attention masking strategy enables the model to perform various tasks like generation, editing, and continuation within a single architecture.
Weaknesses
- It is no clear how the data is collected, how the audios and lyrics are processed, and what is the input of lyric encoder. (Words or phonemes? If so, how do you obtain it from lyrics?) Will the dataset be open-sourced? If not, it is very challenging to reproduce the experiments in the paper to validate the proposed method. - It's not clear how to obtain the vocal prompt and accompaniment prompt and how to pass them into the model. It may be better to further explain this issue. - In Figure 2, I can get the attention mask strategies, but the figure is confusing. The authors should use a straight line with an arrow to illustrate the mask relationship between tokens in two sequences. - Although the authors describe the effects of different mask strategies, they only list the SA mask and BCA mask strategies for each task in Table 2 without clarifying why these masks are chosen for each task. The authors need to clarify this, even if just for one of the tasks.
Questions
- Have you compared semantic tokens obtained from different models? Or have you explored other forms of intermediate representations (acoustic tokens or continuous representations)? If so, have you analyzed and compared the differences between these different types of tokens? - From the reviewer’s experience, Demucs does not perform very well for source separation tasks. The vocal samples often have reverb, which significantly affects the quality of the synthesis. How do the authors address these issues? - What is the training detail of the baselines in the paper? Are the models trained on the same dataset? - What is the training strategy for the VAE? What are the components of its loss function? - The Lyrics-to-vocals samples are quite expressive. They sound natural and exhibit some singing techniques such as trill. What do the authors think contributes to this?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
No Limitations
Summary
The paper presents SongCreator, a system for full song generation including the vocals and accompaniments. The system comprises several steps: - First a quantizer is trained to tokenize audio, which is used to tokenize the song, vocals, and accompaniments. - Next, a language model (DLSM) conditioned on various conditioning signals such as lyrics, vocal prompt, accompaniment prompt is trained to predict the tokenized forms of the input conditioning signals. The DLSM consists of two transformer decoders operating on the vocal and accompaniment prompts respectively. The decoders both cross-attend to each other and the authors propose multiple masking strategies for the self and cross-attention modules. - The final component is a latent diffusion model which generates audio conditioned on the semantic tokens. The first and last steps are pretrained and based on existing literature while the DLSM is trained using a multi-task setup carefully designed to account for various tasks that the model should be able to handle such as generating songs from lyrics, generating songs from pre-determined vocal/accompaniment track, or song editing.
Strengths
- The authors have conducted a very thorough evaluation of their method using subjective and objective metrics. They have also compared against fairly strong baselines for the various tasks. - The results reported in the paper show that their proposed DLSM is superior to standard GPT-based LMs for the chosen tasks.
Weaknesses
- The paper lacks some clarity and could benefit from improvements in the illustrations and writing. More specifically, while Fig. 1 gives a good overview of the approach, it seems to be misaligned with what is being presented in the text. At first glance it felt like the text is mentioning that the model accepts text prompts for describing the vocals and accompaniments, but the figures shows those signals passing through the semantic token extractor. The figure is correct, but it took a few minutes to realize that. Similarly, it is not obvious that the attention masking strategies are different for different tasks. Some pre-conditioning in the figure/method section would be beneficial for clarifying the design for the readers. - The audio quality of the final generations are not very convincing. It is indeed better than Jukebox though. - The significance of the paper’s contributions seems low. The authors have themselves mentioned that the semantic tokenizer and the LDM are prior work. Furthermore, the overall design is very similar to other recent work. The main difference is in the specific tasks the authors have chosen and indeed I find that the design is useful for those tasks.
Questions
- One of the major contributions is the use of the BCA and the authors have ablated the utility of that component, however they seem to have skipped any ablation studies on the different masking strategies for BCA. This might be an interesting point of discussion in the paper.
Rating
4
Confidence
3
Soundness
3
Presentation
2
Contribution
2
Limitations
The authors have discussed limitations both in terms of technicality as well as societal impact of their work.
Several critical issues remain after reviewing the response: **Unconvincing Performance Demonstration:** As an audio-based music generation work, this paper does not compare with models such as Suno and Udio. It is not convincing that the lack of comparison is because Suno and Udio do not publicly disclose their methods (mentioned in the response). In most cases, a user input is sufficient for Suno and Udio to generate results for comparison. I have used this method to generate music and compared it with the music generated by this paper for tasks such as lyrics-to-song, lyrics-to-vocals, accompaniment-to-song, vocals-to-song, music continuation, song continuation, vocals continuation, and accompaniment-to-song (no lyrics). The proposed model fails to generate high-quality music in terms of vocal pronunciation, fluency, and background noise. Additionally, the proposed model only generates music at the phrase level, while Suno and Udio can generate music with a complete structure of multiple sections. **Lack of novelty:** The key contribution claimed by the paper is the encoder-decoder architecture on two audios. However, the use of encoder-decoder architecture in music generation is not novel. For example, various works from different groups have reported relevant work from 2020 to 2024, as shown below: [1] Dhariwal, Prafulla, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. "Jukebox: A generative model for music." arXiv preprint arXiv:2005.00341 (2020). [2] Donahue, Chris, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti et al. "Singsong: Generating musical accompaniments from singing." arXiv preprint arXiv:2301.12662 (2023). [3] Zhiqing, Hong, Huang Rongjie, Cheng Xize, Wang Yongqi, Li Ruiqi, You Fuming, Zhao Zhou, and Zhang Zhimeng. "Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompaniment." arXiv preprint arXiv:2404.09313 (2024).
Response to the reviewer
Thanks for the reviewer's insightful comments. We would like to reply point by point here: **Regarding Novelty** Firstly, we would like to note that our primary focus is on **improving the modelling of discrete music representations** within the encoder-decoder architecture to enhance the musicality of generated songs. As mentioned in lines 59-73 of the paper, different from the single-stream language model (LM) used in Jukebox, Singsong and Text-to-Song, our key innovation lies in **introducing a novel LM, DSLM, along with corresponding attention masking strategies.** It significantly enhances the musicality of generated songs by **improving performance for dual-sequence modelling**, and **completes song generation tasks of various forms in a single model**. 1. **Improving Performance for Dual-Sequence modelling**: For tasks involving aligned sequences such as vocals and accompaniment, DSLM demonstrates superior performance. In our experiments, we have compared our model with various LM-based models: - MusicLM (Similar to Singsong) uses LMs for semantic tokens followed by acoustic tokens. - MusicGen (Similar to Text-To-Song) uses transformer-based models to directly predict acoustic tokens. - GPT (Similar to Jukebox) uses autoregressive transformers to model discrete tokens. Our experimental results indicate that DSLM outperforms these LM-based single-stream modelling approaches in most tasks. Additionally, DSLM supports independent control over generated vocals and accompaniment, which is beyond the capabilities of the mentioned previous works. 2. **Accomplishing Various Song Generation Tasks in a Single Model**: As mentioned earlier, DSLM can perform multiple tasks within a single model, such as generation, editing and accompaniment-to-song, and multi-task learning further enhances the musicality of generated songs. These advantages are beyond the capabilities of previous literature works, which required training specialised models for each task. We believe our approach provides valuable insights for other dual-sequence modeling tasks. **Regarding Comparison with Suno** We are thankful for the reviewer's comment and would like to discuss this issue to address the reviwer's concerns. Firstly, it is challenging to make a fair comparison between our proposed method and Suno. Due to constraints related to **data collection costs and music copyright**, our dataset is relatively small (270,000 songs compared to Jukebox's 1.2 million) and of lower quality (primarily sourced from non-professional singers online). These limitations significantly affect the fluency, background noise, and vocal pronunciation in the generated songs. As a commercial product, Suno likely has access to **more extensive and higher-quality data**. Nevertheless, it is noteworthy that our generated songs **achieve musicality and quality close to the ground truth samples** in most tasks. This indicates that DSLM effectively maximizes performance despite the data limitations. We believe that increasing the quantity and quality of data will further enhance the results. Secondly, our focus is on better modelling music representations rather than improving the encoding and decoding processes of music. To ensure **fair comparisons**, we conducted all experments using **the same components (BEST-RQ, LDM, and Encodec) for audio encoding and decoding**. This approach aligns with previous works, such as MusicGen and Singsong, to prevent the influence of different encoding and decoding methods. Experiments demonstrate the strong performance and flexibility of DSLM, capable of **handling multiple song generation tasks with a single model** and **outperforming specialized baselines in most tasks**. In contrast, Suno's audio encoding and decoding methods are not disclosed, making **it difficult to rule out the influence of these modules**. Additionally, audio encoding and decoding are areas of long-standing research, significantly impacting audio quality, noise, and clarity. We plan to explore this in future work to enhance the quality of synthesized songs. Finally, our model supports several capabilities that Suno and previous works do not, such as **song editing, vocal editing, and vocal editing in songs**. As ***#Reviewer H7YT*** mentioned, these diverse editing capabilities are highly practical for music production. In **accompaniment-to-song and vocals-to-song**, our model also differs from Suno's. We follow the requirements of previous works to ensure that **the input tracks remain unchanged** in the final output, whereas Suno may alter the content and melody of the input tracks. This demonstrates that DSLM offers **a broader set of capabilities**, and is the first attempt in music generation to integrate such diverse capabilities within a single model rather than relying on multiple specialized models for music generation.
Thanks for the response. I appreciate the clarification for alignment between lyrics and VAD and the possible extension for instrument/disentanglement control. However, I am not sure why authors do not run a simple test for inference latency. I totally understand it requires multiple modules/stages to achieve good music quality just like the SOTA method but since you have already implemented these baselines for comparison, why not include the latency as well. It should give readers a high-level idea how fast it is and if it is not fast, what is the trade-off between performance v.s. speed. I strongly suggest authors adding such a table in the final version. In addition, while I don't think it is critical to have comparison with SUNO for accept/reject, it would be great to include some examples to see the gap between academic research and commercial product. Finally, given the flexibility of the proposed system and overall performance, I believe the paper is valuable to appear in NeurIPS but I will recommend authors considering the above comments. I will keep my original score.
Response to the reviewer
We appreciate the reviewer's overall positive feedback and constructive comments. Thanks very much for the suggestion to conduct test for inference latency. We would like to supplement the evaluation by comparing the real-time factor (RTF) for SongCreator and other baselines. RTF represents the time (in seconds) required for the system to synthesize one second of waveform. The evaluation was performed on a single NVIDIA V100 GPU with a batch size of 1. We randomly selected 20 generated audio samples, each longer than 20 seconds, to conduct the evaluation. These additional results will be included in the final paper. | model | RTF | | ------------------- | ------ | | MusicLM | 14.545 | | MusicGen | 2.104 | | GPT | 1.525 | | GPT (Vocals & Song) | 3.059 | | SongCreator | 2.793 | The results indicate that methods utilizing a single LM module are significantly faster than MusicLM, which employs multiple LMs in cascading manner. Taking into account the experiments corresponding to Table 3 in the paper, we observe that although GPT and MusicGen, which only model the song token sequence, are faster than GPT (Vocals & Song) and SongCreator, which predict multiple sequences, this gain in speed comes at the cost of reduced performance. In comparison to GPT (Vocals & Song), our proposed SongCreator, which leverages DSLM to simultaneously model both vocals and accompaniment, achieves not only faster speeds but also better results. Furthermore, we acknowledge the comments related to Suno and will include Suno-generated samples on the final demo page as advised.
Decision
Accept (poster)