UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner

Summary

The paper proposes a LLM-codec module that can plug into an existing LLM, i.e., LLAMA-2, to perform few-shot in-context learning for tasks including classification (emotion & sound event), and text-to-speech synthesis. The proposed module takes the raw audio waveform as an input, and encodes it into latent space where the corresponding features are mapped into VQ codebooks in the LLM dictionary space. A multi-layer alignment designs are considered in the LLM-codec, where the shallow layer is responsible for semantic information, and deeper layers are responsible for more fine-grained information. Four losses are consider in aligning the features with the LLM pretrained embedding space, semantic loss, consistency loss, reconstruction loss and discriminator loss. For semantic loss and consistency loss, the goal is to guide the learned embeddings to have semantic meaning as well as align with the pretrained audio features for stability reason. Experiments demonstrate that the proposed module can be plugged into the pretrained LLAMA-2 for in-context few-shot learning for simple audio understanding tasks and TTS task.

Strengths

1. The paper proposes an interesting way of solving few-shot audio-related tasks using frozen LLMs. Different from previous methods, this work designs a plug-in module for LLM in-context learning for audio modality to avoid LLM training or fine-tuning. The plug-in module is designed to be efficient, with only 160M parameters. 2. The tasks can cover both audio understanding and simple text-to-speech synthesis, which is flexible enough considering the limitations of in-context learning. 3. The paper is mostly well-written and presentation is clear enough to understand the motivation and the proposed method.

Weaknesses

1. The task seems to be really simple, I am wondering how could the model performs under the more challenging scenarios as N-way goes larger? Also how the TTS performance degrades as the scripts become more complicated? 2. In table 2, for 2-way speech emotion classification, why is random guess only 40% accuracy rather than a number close to 50%. 59% acc is also not high enough for binary classification task. What is missing here in order to achieve better acc? 3. In Table 4, what is ACC for GT and FastSpeech 2, is the number not be able to compute?

Questions

See weaknesses above.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

The paper introduces LLM-Codec, which enables frozen LLMs to perform various audio tasks in a few-shot manner without fine-tuning LLMs. LLM-Codec operates in a RVQ-manner, hierarchically converting audio tokens into words or sub-words in the LLM vocabulary to compress the audio modality into the text space.

Strengths

The approach is validated through experiments on tasks such as speech emotion classification, audio classification, text-to-speech generation, and speech enhancement, demonstrating its feasibility and effectiveness.

Weaknesses

It’s hard to understand the meaning of the model setup. The authors train the system with distillation from T5 and Whisper instead of mapping them to the text-LLM like BLIP does. What is the motivation for building a system in this pipeline? Why is the decoder needed? What is the advantage of this approach compared to using external TTS or speech enhancement modules? These questions are difficult to answer with the current version of the presentation and authors should compare with BLIP-like approach to show its benefits. From a presentation perspective, there are many grammatical errors and misleading sentences throughout the manuscript. The authors should put more effort into clarifying their claims and making the manuscript easier to comprehend by providing solid details. - The captions of Figures 1 and 2 are hard to understand and contain grammatical errors. - The details of the experimental setup are also insufficient to fully understand the setting. - The purpose of Figure 4 is very unclear. There is no apparent correlation between the outputs of the semantic layer and the given audio. The purpose of this analysis is not evident, and the information provided is incomplete. The authors conducted various downstream tasks, yet, the setup of experiments are insufficient to validate the system’s capacity. I’ve pointed out questionable setups at the Questions section.

Questions

- Why is RVQ setting adopted as the system pipeline? Have the authors try comparing with just performing downstream tasks using the outputs from T5 and Whisper? - for Table 1, - What is the evaluation dataset being used? Does it only contain speech data? - How about performance comparisons with other metrics such as me reconstruction loss or SI-SDR? - How about using better configurations of the baseline models? (e.g., 44K DAC) - I didn’t fully get what 3 Vanilla RVQ means for Encodec_24k and DAC_16k. - for Table 3, why is accuracy the only metric? How about other metrics such as AUC? - for Table 4, - How is ACC being computed? How about WER? - What is the intuition of proposed method having better DNSMOS than the GT? Why not compute on subjective MOS?

Rating

5

Confidence

4

Soundness

1

Presentation

1

Contribution

3

Limitations

Authors included their potential limitations in the Appendix section.

Summary

The paper introduces LLM-Codec, a novel audio codec model that leverages Large Language Models (LLMs) to perform various audio tasks with minimal training examples. By translating audio signals into the token space of LLMs, it enables these models to understand and generate audio content. The model uses a multi-scale residual vector quantization approach to maintain audio quality while reducing token sequence length.

Strengths

- This paper shows a novel codec model for audio compression and can be generated by a frozen LLM for in-context learning. - The task setting is challenging, and the semantic codec is a challenging task, while the paper shows good exploration to it. - The ablation study is sufficient, showing the elements in the RVQ codec model's usages.

Weaknesses

Experiments. Considering the paper proposes an encodec model, the most important result is the reconstruction performance. Providing Table 1, the necessary explaination of the results is lacking. I suggest this paper adds additional demonstration to the experimental main results to improve the readability. Although the paper does a lot of experiments, showing promising results, the audio generation-related tasks still lack enough experiment results. In both introduction and related work section, the paper claims that previous codec models do not support audio generation tasks, while the text-to-audio evaluation results are not shown. Considering the strong claim and the AudioCaps training data, it is strongly recommended to show its performance comapred to other text-to-audio models.

Questions

N/A

Rating

6

Confidence

2

Soundness

3

Presentation

4

Contribution

3

Limitations

This paper provides sufficient discussion in this field.

Summary

The authors introduce a three-step audio-to-discrete codec to encode continuous acoustic information into a form suitable for large language model-based audio and speech understanding. Overall, this method is novel and represents an important step in audio modeling. The architecture targets different levels of acoustic information, from semantics to acoustic representation, using only discrete codecs. The results are empirically strong and solid. However, there are fewer theoretical connections to justify the meaning of the lexical representation of the "trainable new (pseudo) language" of speech, which can be improved in future work. For example, former works on word-level model reprogramming, learning equivalent pseudo tokens [A] from random embeddings, and theoretical bounds [B] on connecting different layers for latent alignment (e.g., when and how to align these three RVQ adapters) could help justify the uniqueness of the latent distance compared to embedding injection-based methods. This analysis could strengthen the theoretical foundations of the paper. Despite these points, the overall contributions remain high-quality. - A few grammatical and formatting issues need to be very fixed in the final draft for a much ready version. In sum, I highly recommend accepting this paper and suggest the authors address these issues in the final version.

Strengths

1. the model architecture design is overall new and effective 2. the designs on the different concept of audio representation is interesting

Weaknesses

1. there are less discussion on the representation difference of proposed codec-based method to the embedding injection based works. 2. some minor grammar and formation issues 3. there are less theoretical discussion on the representation related to how many RVQ layers needed eventually

Questions

See the weakness (to be improved) 1. Are there any streaming or token merging limitation? 2. Is there any scaling effects of the backbone LM selection? 3. what would be the semantic alignment from the codec to the lexical level representations?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

minor, in terms of performance, the gap between cascaded LM for ASR and translation tasks.