High-Fidelity Audio Compression with Improved RVQGAN

Summary

In the present paper, the authors introduce RVQGAN, a neural audio codec that uses a convolutional encoder / decoder along with Residual Vector Quantization as a bottleneck, with a multi scale mel reconstruction loss and different adversarial losses. They show state of the art performance from 3 to 8kbps, compared with the EnCodec model [8]. The key novelties are: - in each VQ layer, the authors perform the retrieval of the nearest codebook entry into a lower dimension space, and use cosine similarity instead of L2 distance to boost the utilization of the codebooks. - the authors drop the exponential moving average rule for learning the codebooks. - the author notice that the original technique from Soundstream [45] to select a varying number of quantizers can hurt the full bandwidth performance, and thus select 50% of the time all the quantizers in RVQ. - refinement of the losses and adversaries from previous work (in particular using different weights for different frequency bands). - balancing of the dataset to sample more often fullband audio. The authors provide extensive ablation studies with objective metrics, and one subjective comparison with EnCodec with various bitrates.

Strengths

- great execution and illustration of the various issues tackled here and the proposed solutions. - quality of the final model clearly surpasses the existing state of the art. - detailed ablation study with objective metrics. - single model for fullband audio over multiple audio domains.

Weaknesses

- incremental improvement over previous work: overall method is coming from [45], adversarial losses are a combination of the one from [45] and [8]. Minor changes to the objective loss compared with [15, 45]. The authors however claim novelty: l.59, "we make impactful design changes [...]: multi scale stft discriminator, multi scale losses". - some details are unclear to me, in particular, the authors mention they do not use the EMA rule from [9]. How are the codebooks updated then? The authors also mention a low dimension projection, but do not mention when and how it is computed and updated. See questions. - no ablation with subjective evaluations: could have been interesting to clearly identify where most of the subjective gains is coming from e.g. is it from the quantizer, adversarial losses or dataset balancing? - seems like the authors do a comparison of a 24kHz baseline model with a 44.1kHz, keeping the ground truth as 44.1kHz, which can have a high impact on subjective and objective metrics independently of the design choices made by the authors. In particular the visqol for Encodec in Table 3 is much lower than reported in [8].

Questions

As mentioned before, I would need more clarification on the exact algorithm used for VQ. Is a PCA computed ? if so how often is it updated (as the encoder output distribution might change). How are the codebooks updated ? In Section 3.4, the architecture for the multi scale discriminator is missing. Is it the same one as [8] or [45]? Paragraph starting 193: this insight has been noted and motivated before in [45], [5] and [40], it doesn't seem like the authors bring any new material evidence here? It would be interesting to see the breakdown of Figure 3 by the category of audio. A last question would be over learnt snake activation parameters. Do the authors have any insight over the distribution of the learnt $\alpha$? does this vary with the layer? I'm trying to get a better sense of how exactly the model is utilizing this feature.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

authors properly address societal impact.

Summary

This paper introduces a novel high-fidelity neural audio compression algorithm that achieves impressive compression ratios while maintaining audio quality. The authors combine advancements in high-fidelity audio generation with improved vector quantization techniques from the image domain, along with enhanced adversarial and reconstruction losses. Their approach achieves a remarkable 90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. One of the notable strengths of this work is its universal applicability, as it can compress various audio domains (speech, environment, music) using a single model. The authors conduct a thorough comparison with competing audio compression algorithms and demonstrate the superior performance of their method. Furthermore, they provide detailed ablations for each design choice, allowing readers to gain insights into the effectiveness of different components. Additionally, the paper offers open-source code and trained model weights, which contribute to the reproducibility of the results.

Strengths

- **Impressive compression performance**: The proposed algorithm achieves a 90x compression ratio for 44.1 KHz audio at just 8kbps bandwidth, demonstrating its effectiveness in reducing data size while preserving audio quality. - **Novel Method**: The proposed "codebook collapse" and "quantizer dropout" effectively address the issues in lossy audio compression. - **Universal applicability**: The single model's ability to compress various audio domains makes it highly versatile and applicable to generative modeling of different audio types. - **Comprehensive evaluation**: The authors compare their method against existing audio compression algorithms, demonstrating its superiority in terms of performance. - **Thorough ablations**: The paper provides detailed insights into the impact of design choices, allowing readers to understand the effectiveness of different components and their contributions to the overall results. - **Reproducibility**: The availability of open-source code and trained model weights enhances the reproducibility of the research, enabling other researchers to build upon and validate the findings.

Weaknesses

- The novelty of the proposed model structure is a combination of existing models: - factorized codes and L2-normalized codes are from Improved VQGAN image model; - Snake activation function from BigVGAN - This paper presents a strong audio compression technique. However, since the proposed novel points are specifically tailored for a narrow domain, their impact may be limited to the machine learning community and other domains like computer vision/NLP

Questions

- Have you attempted to apply a similar architecture to the vocoder in TTS? - Which components do you believe can be applied and generalized to other domains or tasks?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

This paper introduces a RVQGAN-based neural audio codec method, demonstrating superior audio reconstruction quality, a high compression rate, and generalization across diverse audio domains. The authors substantiate the significant performance superiority of their model over alternatives through extensive and thorough qualitative and quantitative experiments. They present and validate their technique to fully utilize residual vector quantization, alongside model, discriminator, and loss design choices for enhanced performance.

Strengths

* The paper addresses some of the key challenges in the neural audio codec domain. * The authors conducted strong and extensive experiments, providing comprehensive results. * The reference list appears to be thorough and comprehensive. * The authors support their findings by sharing the developed model, which is beneficial for the research community.

Weaknesses

* The authors derived the proposed methods from existing studies and experimentally validate them in the neural audio codec domain. This approach seems to compromise the scientific novelty of the research.

Questions

* Could the model be applied to downstream applications such as training text-to-speech (TTS) models? Previous works like EnCodec and SoundStream utilized causal architectures to make them suitable for in-context learning or prompting in TTS tasks.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The authors have adequately addressed both the limitations of their research and its possible societal impacts.

Summary

The authors propose a neural audio codec model that demonstrates superior performance compared to previous works, and present experimental results.

Strengths

- The authors appropriately explain the problem they aim to address. - Their method is adequately described. - The authors provide a specific implementation, ensuring reproducibility. - The claims made are reasonable, and the experiments and results support them. - The authors' various ablation studies can be helpful for future work.

Weaknesses

- For a neural audio codec to be utilized like traditional audio codecs, it should not fail in any patterns. Data-driven neural audio codecs have not been proven to be sufficiently stable from this perspective. Although the authors divided the original dataset into a training set and evaluation set, it is necessary to validate whether the proposed audio codec works well on more diverse and completely different audio data. Additionally, finding failure cases of previous works and comparing them can serve as strong evidence supporting the superiority of the authors' proposed method.

Questions

Based on the MUSHRA scores curves, it appears that higher bitrates yield better scores, and the highest quality that this method can achieve remains unconfirmed. Is there a specific reason for this?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations have been well described.

replying to the authors

I would like to thank the authors for their detailed response. I would kindly ask them to repost the explanation on the updated RVQ rules as it seems the formatting is broken and hence hard to parse.

Latex correction for correct equation rendering

We present the equations above again with the correct latex code. --- The modified quantization operation equation used in our work: $$ z_q(x) = W_\text{out} e_k, \quad \text{where}\quad k = \text{argmin}_j ||\ell\_2(W\_\text{in}z_e(x)) - \ell\_2(e_j) ||_2 $$ $$ \text{$W_\text{in} \in R^{D \times M}$ and $W_\text{out} \in R^{M \times D}$ are projection matrices}\\ $$ $$ \text{$D$ is the output dimension of encoder} $$ $$ \text{$M$ is the codebook dimension, $M \ll D$} $$ --- The vector quantizer loss function: $$ z\_\text{proj}(x) = W\_\text{in}\ z_e(x) $$ $$ \mathcal{L}_\text{VQ} = ||\text{sg}[\ell\_2(z\_\text{proj}(x))] - \ell\_2(e_k) ||_2^2 + \beta ||\ell\_2(z\_\text{proj}(x)) - \text{sg}[\ell\_2(e_k)] ||_2^2 \\ \text{where sg is the stop gradient operator} $$

Rebuttal Response

I would like to thank the authors for addressing the raised concerns. I wish to further discuss the last query I presented, and if there are any misunderstandings on my part, I would be grateful for any clarifications. When preparing audio data for training LM-like TTS model, it's typical not to differentiate between the audio for the prompt and the subsequent audio for training. Instead, the entire audio is encoded all at once. In such a setup, a non-causal encoder might cause the prompt code to incorporate trailing audio information. For instance, assuming the length of an audio prompt is 3 seconds and the receptive field of the encoder spans 2 seconds, the prompt code derived from a 5-second ground truth audio won't be confined to just the 0 to 3-second audio data. It will also cover the subsequent 3 to 5 seconds, as they fall within the receptive field of the final code. During the inference of an LM-like TTS model trained in this manner, potential issues can emerge: a) If the entire 0 to 5 second ground truth audio is encoded and the first 3/5 of it is selected as the prompt code, it results in a cheating problem. This is because the last code of the prompt will represent not just the audio information from 0 to 3 second but also from the subsequent 3 to 5 second. b) Conversely, if only the first 3 seconds of the ground truth audio are clipped first and then encoded to be used as the prompt code, the encoded code might be interpreted that the audio from 3 to 5 seconds is silent due to the effect of zero-padding in the encoder. On the contrary, encoders with a causal architecture do not present these issues, making them seemingly more suited for providing audio prompts to LM-like TTS models. I am curious to hear the authors' perspective on this matter.

Please read rebuttals

Dear reviewers ZhX4 and d8D3, if you didn't already, please read the rebuttals ASAP and at least acknowledge them explicitly. Best, Area Chair