From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion

Summary

The paper proposes to use diffusion model to recover high-fidelity audio from compressed audio tokens. To improve the performance of diffusion model, the paper proposes multi-band diffusion with 1) Frequency Eq. Processor; 2) Scheduler Tuning; and 3) Band-Specific Training. The experiments show that the proposed method surpasses baseline method (EnCodec) by a margin.

Strengths

1. The paper proposes a diffusion-based model as decoder for audio compression model (EnCodec), which surpasses the default decoder joint trained with encoder. 2. The paper proposes multi-band diffusion model to improve the generation quality.

Weaknesses

1. The motivation of multiband is not clear. According to the description in paper, the multi-band is conducted on the hidden of encodec, which seems not have frequency information. If there is no intuition or explanation about the multi-band, it seems that using multi-band is a kind of improving the model parameters (using multiple model, one per band). 2. The experiments are only on EnCodec tokens, which are insufficient. The design in paper should be verified on vocoder tasks (recovering audio from mel-spectrum) and compared with more baselines (e.g., bigvgan, hifigan, wavegrad and multi-band MelGAN). Since mel-spectrum has frequency information, using multi-band on mel-spectrum is more convincing. 3. In ablation study, the comparison with single band seems to be unfair in terms of total model parameters.

Questions

My main concern is the proposed method should be verified on vocoder task and compared with GAN based and diffusion based vocoder baselines.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper presents Multi-Band Diffusion, a diffusion-based decoder for neural audio codecs. The proposed method demonstrates superior generation quality across various audio domains compared to a publicly available state-of-the-art neural audio codec method. Based on the analysis of time-frequency representation of audio, the model provides a novel diffusion-based model for audio synthesis.

Strengths

* The authors provide extensive experimental results, along with a detailed explanation of their methodology. * They present novel methods such as the band-specific diffusion model, frequency equalizer processor, and power noise scheduler, demonstrating their necessity and assessing their effectiveness through an ablation study. * The model not only outperforms benchmarks in terms of the generation quality of reconstructed samples across different audio domains, but also proves its applicability in tasks such as text-to-audio and text-to-speech through empirical evidence.

Weaknesses

* There is no comparison of generation quality when producing from high bit-rate latent representations. While the generation quality remains inferior compared to the ground truth audio, the authors only showcase generation quality from low bit-rate representations. * There is a lack of a comparison of computational cost or model size. Although the authors mention that the proposed method requires more computation compared to standard decoders, a comparison of generation speed or parameter size would have provided a better understanding of the trade-off between generation quality and synthesis speed of this model. Furthermore, given that the model was trained using 1000 diffusion steps and only 20 steps were employed during inference, an evaluation of quality at different diffusion steps during inference could also be provided.

Questions

From the selection of low bit-rate latent representation as the model's conditional input and the use of a small number of diffusion steps during inference, it seems that the authors have considered efficiency alongside the model's performance while empirically validating the proposed method. However, it would be beneficial to extend it and demonstrate the peak performance of the proposed model with larger bit-rate representation and high diffusion steps, as diffusion-based generative models have shown remarkable performance in various fields.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately addressed both the limitations of their research and its possible societal impacts.

Summary

This paper proposes a novel multi-band diffusion (MBD) model that generates high-fidelity audio of multiple modalities, e.g., speech, music, and environmental sounds, from low-bitrate discrete representations. The authors show that MBD outperforms EnCodec and Opus in terms of perceptual quality.

Strengths

The band-specific diffusion model is novel and appears to be a general technique for universal audio synthesis. The authors design a frequency equalizer to reduce the discrepancy between the prior Gaussian distribution and the data distribution in different frequency bands, which is sensible for improving the consistency and stability of audio generation, especially for general audios. The authors also propose a novel power noise scheduler, which empirically surpasses other commonly used schedulers, e.g., linear and cosine schedulers. I appreciate the high variety of audio generation experiments conducted for evaluating MBD. The generated samples of MBD on the demo page seem promising and are apparently better than EnCodec and Opus at the same bit rates.

Weaknesses

1. Baselines are not strong enough. The authors only compare MBD to Encodec and Opus, where Opus is an old method proposed in 2012. I am therefore skeptical of the choice of the baselines in this paper. Are these baselines strong enough? The authors mention SoundStream (Zeghidour et al., 2021) but do not take it as a baseline. Considering the wide and successful application of SoundStream in many recent audio generation works, e.g., in AudioLM, Vall-E, and Natural Speech 2, it would be more convincing if MBD can be compared to SoundStream and evidently demonstrate superiority. I consider increasing my rating if such an experiment can be supplemented. 2. MBD depends on the frozen latent representations of a pre-trained EnCodec. The dependency of MBD on EnCodec complicates the analysis of the significance of this work. It remains unclear whether the high quality comes from a well-trained EnCodec or the proposed MDB model. The training of MBD also becomes more unstable. Please explain why the encoder of EnCodec cannot be jointly trained with MBD. 3. It is unfair that the generation speed and model size of MBD are not compared to those of Encodec. While the authors only present the strength of their proposed MBD model in terms of different quality measures, they do not mention the drawback of the generative diffusion model. As an iterative sampling method, it is foreseeable that MBD could be slow than Encodec up to orders of magnitude. It is then unfair to only focus on the quality when it is more reasonable to evaluate the audio codecs based on their practical values.

Questions

Some questions have been stated above. In summary: (i) What are the considerations when choosing the baselines? Are these baselines strong enough? (ii) Is the frozen latent representations of a pre-trained EnCodec necessary for training MBD? Can MBD be end-to-end trained from scratch? (iii) What are the speed and model sizes of MBD in comparison to EnCodec or other baselines (if any)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations of this work have been stated in the weaknesses.

Summary

This is an interesting submission proposing the use of a diffusion model in a band-by-band manner to generate high-fidelity audio from (potentially) a variety of low bit rate inputs. The literature so far has only applied diffusion models for audio generation given spectrogram inputs. A specific diffusion noise schedule is proposed; and a diffusion-oriented frequency equalization method is proposed. Significant improvements in audio quality are described from a number of experimental evaluations.

Strengths

Strengths: - Clear presentation (in spite of persistent minor problems with grammar & spelling); - Original multi-band diffusion model; - Good literature review; - Original frequency equalization model; - Good empirical results, i.e. nice improvements in generated audio quality.

Weaknesses

The main weakness is a persistent pattern of small mistakes in grammar and spelling. I mention a number below but this is just a small sample. Overall the work is very readable, but the problems with the writing should be corrected. I note that there is mention of the proposed method being significantly slower than the alternatives examined, but I don't see a clear description of the compute cost of the proposed method compared to existing methods. Specific comments: > 27 artefacts [here and below] --> artifacts > have led to rich contextual representations that contains more contains --> contain > 32 They are optimized using complex combination combination --> combinations > 47 Results suggest the proposed method achieves significantly > 48 superior performance than the evaluated baselines. "significantly superior" is an odd phrasing, rewrite? > 67 Clustering using a few centroids leads to the speech content representation being > 68 mostly disentangled from the speaker and the f0 and thus controllable speech generation. End of sentence is not grammatical; missing words before/around "thus controllable speech generation"? "Eq. Processor" in Figure 1 caption vs. "EQ Processor" in the figure itself: choose one or the other, and keep consistent. > 165 As a result, training a diffusion model on full-band audio data would always > 166 provide the ground truth low frequencies when generating high frequencies I don't quite understand this statement, rephrase? > 170 Interestingly, dividing the frequency band > 171 along model channels I don't understand this statement. What are "model channels"? > 238 5.1 Multi modalities model Reading this section, i'm not sure what are the "modalities" referred to. Table 1: "MBD", though it is clear that this is the proposed method, I don't see the acronym introduced. Introduce it early on, e.g. state "... Multi Band Diffusion (MBD) ..."?

Questions

What is the computational cost of the proposed MBD compared to Opus and EnCodec?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors cite reasonable limitations to their work, including the possibility of their method being used to generate deep fakes, as well as the usual potential issue with training set bias, in spite of their efforts to avoid that.

Summary

This paper presents a novel approach to processing audio data by developing and implementing a band-specific diffusion model, a frequency equalizer processor, and a power noise scheduler. The band-specific diffusion model processes various frequency bands independently, thereby reducing the accumulation of entangled errors. The frequency equalizer (Eq.) processor helps to lessen the discrepancy between the Gaussian prior distribution and the actual data distribution across different frequency bands by balancing energy-level between gaussian noise and each frequency band. The power noise scheduler, specifically designed for audio data with high sampling rate, is another contribution. The authors conduct extensive evaluations to gauge the efficiency of their approach, using both objective metrics and human studies. The results show that the proposed approach surpasses current state-of-the-art methods, encompassing both GAN and diffusion-based methods.

Strengths

Generating high-frequency bands using diffusion model is something that has not been tackled very well. I personally have also suffered from this in various experiments. All three contributions, training of band-specific models, frequency energy balancer, and power noise scheduler, are all valid approaches and were validated by metrics in experiments.

Weaknesses

It would have been nicer if there were more GAN-based / Diffusion-based vocoder baseline models other than just encodec.

Questions

1. Regarding the power noise schedule, what were the rules of thumbs to select hyperparameters? 2. Did the authors set the power noise schedule differently for each band?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations of this work in the conclusions such as slow speed of the current model.

Response to the authors

I appreciate the authors' efforts made for the detailed response. The new strong baselines appear to be more convincing than the previous ones. Besides, following from the clarification from the authors, I am convinced that MBD is practically valuable for improving EnCodec's generation from codes. Yet, it seems unfair to me to compare a 411M MBD to a 8x smaller EnCodec. The immediate question raised here would be, if we train a 8x larger EnCodec, could it perform even better (I guess a ~400M EnCodec can still run much faster than MBD). Overall, considering the practical values of MBD, I decide to increase my previous rating to 5.

Rebuttal Response

I thank the authors for addressing the previous concerns and emphasizing the importance of the model's performance at low bitrates, and presenting new evaluation results coupled with computational trade-offs. However, considering the results indicating this model as a high-performing, yet slower and larger alternative to EnCodec, I have chosen to maintain my initial review.

Thanks for your response

The further results address most of my concerns. I have increased my score to 5.