Summary
The authors propose JEN-1, a text-to-music diffusion generation model. The model uses a denoising diffusion approach on latent continuous embeddings generated from a neural audio codec, to synthesize 10-second music.
The neural audio codec was trained by the authors to encode and decode stereo signals at 48kHz. To train the audio codec autoencoder the authors use various tricks, such as masking parts of the embedding before decoding, normalizing the latent embedding, using a combination of reconstruction losses over the time and frequency domains, as well as a patch-based adversarial objective.
The diffusion model is based on a 1D Unet, using unidirectional and bidirectional convolutional and transformer layers. The diffusion model is trained on three tasks simultaneously: text-guided music generation, music inpainting, and music continuation. The authors use the Unet with unidirectional layers to train the music continuation, the bidirectional layers to train the music inpainting and both layer types to train text-guided music generation.
The authors compare the generated music from JEN-1 quantitatively and qualitatively to other models, and demonstrate superior performance. Furthermore, the authors demonstrate the effect of their design choices in an ablation study.
Strengths
The paper is well presented and easy to read. The demo page and the provided samples are excellent. The ideas used to make the autoencoder audio codec work well for 48kHz stereo music are great. The paper compares the generated music to related work using relevant metrics. Furthermore, the combined training of different tasks makes for a compelling argument, demonstrated further by an ablation study that shows improvements across the board.
Weaknesses
Unfortunately, the paper struggles from a lot of omitted details and statements that do not hold under scientific scrutiny. While the results sound very promising, the paper lacks evidence to support its claimed contributions. I don't think we can accept a paper that almost feels like the interesting details have been removed in order to keep fellow researchers from understanding and repeating the results.
In the following we list some of the issues.
In the first contribution the authors mention “JEN-1 utilizes an extremely efficient approach by directly modeling waveforms and avoids the conversion loss associated with spectrograms”, while I understand the point the authors are trying to make, the current statement is not true. JEN-1 requires the use of an audio codec to offload the encoding/decoding of music to latent embeddings, which is very similar to what related work has been doing (MusicLM with SoundStream, MusicGen with EnCodec, Stable Audio with VAE, etc.). Similarly to Stable Audio, JEN-1 also uses continuous latent representations. Furthermore, “extremely efficient” is a strong statement, the paper has no comparisons or ablations to substantiate such claims.
The authors argue that they use the continuous embeddings from their autoencoder since it gives better results than the quantized approach used by previous work (musicLM, musicgen). While this intuition might be true, there is no evidence provided to support this (neither through experimentation, nor through references). Furthermore, the authors state “This utilization of powerful autoencoder representations enables us to achieve a nearly optimal balance between complexity reduction and high-frequency detail preservation, leading to a significant improvement in music fidelity.” - This is a bold claim, nearly optimal balance - how do the authors know this balance is almost optimal? And how can they state a significant improvement in music fidelity?
In the second contribution the authors mention “JEN-1 integrates both autoregressive and non-autoregressive diffusion modes in an end-to-end manner [...]” A diffusion model is non-autoregressive in time since we are generating the entire sequence after N denoising steps, irrespective of the type of layers used (uni/bidirectional). If there has been related work on this, please cite it. As far as I understood, the authors claim to have an autoregressive diffusion model because of the unidirectional convolutional and transformer layers. If this is the case, I would disagree with this statement.
The paper would benefit from an extensive ablation study separating the audio codec from the diffusion model, at this time it is unclear which component contributes how much to the final performance. For example, using EnCodec and Descript Audio Codec as audio encoders/decoders and training the authors’ proposed diffusion model on their latent music representations would demonstrate the effect of the proposed audio codec.
The paper could greatly benefit from an architecture diagram visualizing how all components connect to each other. Many of the questions asked below would be answered with such a figure.
There is very little information on the setting and methodology used to collect the qualitative results with human subjects. (Though I agree that the results sound good, so I don't question the results per se.)
Apart from the information given in the paper, which does not suffice to reconstruct the provided results, there seems to be little effort made regarding reproducibility. It would be highly beneficial for the research community and the reproducibility of this work if the corresponding code (and music copyrights permitting, model weights) were open-sourced.
Finally, even though the results sound very good, it is unclear whether they were selected randomly or cherry-picked. Furthermore, the authors mention JEN-1 has autoregressive capabilities, but do not demonstrate this on a prolonged generation (compare with generating for 30+ seconds as demonstrated by recent autoregressive models such as musicLM or musicgen).
Generally, the paper needs more detail. If the authors can explain these weaknesses, I will increase my score.
Questions
Is the audio codec trained separately and then frozen while the diffusion model is trained? This is what I expect. In case the authors manage to train the audio codec and diffusion model end-to-end simultaneously, I will revise my review.
Can the authors give more information on the losses used to train the autoencoder audio codec? There are many important details omitted from “which consists of an autoencoder trained by a combination of a reconstruction loss over both time and frequency domains and a patch-based adversarial objective operating at different resolutions.” - if the authors are using the same setup as EnCodec, SoundStream, or Descript Audio Codec then please clarify this, otherwise more information on the setup is needed.
Is the conditioning information passed via channel or via cross-attention? I assume only the mask is passed via channel and the Flan-T5 text embedding is passed via cross-attention? Section 4.2 states “[...] the input consists of the noisy sample denoted as $x_t$, which is stacked with additional conditional information” - what is this additional conditional information? The masks for inpainting and continuation tasks?
In Section 4.2 Task Generalization via In-context Learning, the authors mention “apart from the original latent channels, the U-Net block has 129 additional input channels (128 for the encoded masked audio and 1 for the mask itself).” - how many channels does the Unet have in total? Does the Unet have 129 channels? It is never specified what the exact dimensions of the latent representation are (variable c in Section 4.1), so it is unclear what size of tensor is being diffused by the model.
In Section 4.1 Normalizing Latent Embedding Space, the authors mention “Additionally, we incorporate a dimension reduction strategy with 5% least important channels to enhance the whitening process further and improve the overall effectiveness of our approach.“ - What is meant by dimension reduction with 5% least important channels? How is this done?
Can the authors further explain how their unidirectional layers represent an autoregressive diffusion model? Since the authors do not cite any related work on this, I assume that this is an original contribution? Nevertheless, please provide experiments or theoretical formulations demonstrating how using unidirectional layers in a diffusion model allows it to become an auto-regressive model.
Can the authors give more technical details on how the ablation study was conducted? Is the model trained from scratch for each entry in the table? What about the batch splitting described in Implementation Details?
Regarding qualitative evaluations, could the authors give more information? Did human raters follow a MUSHRA or MOS test, or something else? How many raters were involved, how many samples did each human rate, etc.?
Finally, can the authors give more details on the data used to train the model? Where does it come from? How were the captions sourced/created? What exact information is contained in the captions?
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Ethics concerns
The authors use 5’000 hours of music with corresponding textual descriptions to train JEN-1. No additional information on this data is given and whether the authors have the rights to use this data.
In Section 4.3 Music Inpainting Task, the authors mention “[...] eliminate undesired elements like noise and watermarks from musical compositions”. The reason for watermarking is to discourage users from misusing copyrighted material; so the authors are suggesting to remove watermarks without the right holder’s consent?