Summary
The authors present an approach to combine multiple non-ASR audio tasks into a single model. Motivated by audio language models and VLM, the authors propose using a frozen LLM to create an audio LM that can be used for open-ended (purely generative) and close-ended (classification) tasks. The main idea is to use an audio encoder and a text encoder to create fixed length prompts that can be used with a frozen LM after training. The authors train various tasks by using task specific text prompts (“generate audio caption”, “this is the sound of”, etc.). In this sense, the model is similar to a multitask learning model (more on this below). Results show that the model is competitive compared to other approaches in the literature that are task-specific, and an alternative multi-task model, CLAP.
Strengths
- A single model that can handle multiple tasks, both open-ended and close-ended.
- One of the first approaches that incorporate VLM / LLMs to create a general purpose audio LM tasks.
- Results show gains using LLM for building the model, compared to a baseline that doesn't.
Weaknesses
- While the model can cover a number of tasks with competitive results, the model still performs worse on some of the tasks considered compared to task-specific models.
- Furthermore, it is unclear if the method generalizes to new tasks, which is a main strength of non-audio LLMs / VLMs. The authors show zero-shot capabilities for new labels within an existing task. Although impressive, not entirely novel given prior works have approached the problem along similar veins (like CLAP, e.g.) but in limited settings. The results in Tab. 9 also show poor performance in zero-shot settings for a new task.
- The results in Tab. 8 also show that the model is not as best as some of the other models on zero-shot classification task.
Questions
1) The authors tokenize and map text prompts. Why is an explicit mapping needed? Can’t they re-use the same tokenizer as the LLM?
2) Why does the audio and text prompt prefix have to be of fixed length? Does the length of the input audio affect quality?
3) Line 180 and Fig. 3: Text matching is not appropriately described and is a little hard to follow. Consider providing a more detailed description. Are the embeddings created for each token or for the entire sentence? If for the sentence, how are they created (summarized)?
4) The authors claim sota in a few different close-ended tasks. Are the gains coming from the new formulation (audio LM) or the extra datasets that the authors are now using? For example, what if the authors train a single model with multiple classification heads for the N-tasks (close-ended, at least)? Does it work worse than the current model that uses the frozen LLM and the proposed architecture?
5) Line 181: When computing log likelihoods, do the authors normalize based on the number of tokens in the desired values (dog barking vs. sea, e.g.)? Does the number of tokens affect the overall score?
6) Sec. 5.5: Does the presented result show that perhaps a good portion of the performance comes from training data. For instance, wav2vec2 works better on emotion recognition, likely because the training data is relevant. Are the training set the same for the remaining non-speech models in Tab. 9?
Minor
1) The authors use close-ended and open-ended tasks a lot, but only define it Sec. 4.2. Consider moving this to an earlier section to avoid confusion.
2) The authors say the text encoder is frozen (Line 111). Where does this encoder come from?
3) Why are the results in Tab. 4 for the best setting, worse than those in Tab. 7?
5) Please provide more descriptive captions for Tab. 4 – 6.
6) Line 271: Why does Pengi outperform human performance by such a large margin? It is interesting, so perhaps an explanation is warranted.
7) It would be useful to include the best results from Mei et al to Tab. 8. Also, it’d be useful to include some representative contrastive methods to Tab. 6 for audio retrieval.
8) Why does Pengi work better on some zero-shot tasks and not the others, compared to techniques like CLAP? Is it related to the training data?
Typos:
1) Line 51: For example, -> Examples include
2) Line 70: Missing citation (?)
3) Figure 2 caption: a text a prompt -> a text prompt
4) Line 151: question: question -> question: {question}
5) Line 205: 320 secs / 1024 secs: Perhaps the authors mean milliseconds?
6) Line 270: Missing Table# (Tab. 8?).
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
1) Does not support ASR, which is, arguably, one of the most important audio task.
2) Unclear if the model can learn new tasks. The existing tasks use a fixed prefix prompt, which is tokenized and mapped. This makes it cumbersome to add new tasks to the model. This is a significant drawback compared to current LLMs that can learn new tasks by prompting.
3) How much does the LLM help? What is the quality if the LM component is trained from scratch with the current dataset?