AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

Summary

This paper proposes a zero-shot generative editing model for audio signals. The text-based user edits include adding, dropping, replacing, inpainting, and perceptually upsampling audio signals. The audio is not limited to any particular audio domain (e.g., speech or music). Comparisons to recent text-to-audio and zero-shot editing models demonstrate a clear improvement in the state-of-the-art.

Strengths

This is a notable advance in text-to-audio generation as well as audio editing. This paper outlines a framework for general purpose text-based neural audio editing that I could foresee becoming a blueprint for how this type of editing is performed. I imagine many possible subsequent works focusing on quality improvements and further editing capabilities—which there are many. One of the challenges of producing such an editing system is that each editing capability requires its own evaluation. That is well handled here.

Weaknesses

This is not “the first audio editing model based on human instructions” (line 43-44) as any editing model is technically following the user’s instructions. Consider “the first model to perform semantic audio editing from text-based prompts” or something similar. Also, you could have a machine generate the instructions, so they don’t have to be human users. The evaluation of super-resolution would benefit from a comparison with a recent task-specific model (e.g., [1, 2]). I’m not yet convinced that performing super-resolution via text prompt would be preferable over having a dedicated super-resolution model. It is not clear what each of the baseline models is in Table 7. [1] Su, Jiaqi, Zeyu Jin, and Adam Finkelstein. "HiFi-GAN-2: Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features." 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021. [2] Andreev, Pavel, et al. "Hifi++: a unified framework for bandwidth extension and speech enhancement." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.

Questions

Why regenerate the context conditioning during inpainting? Don’t you already have access to the ground truth? It seems speech is excluded throughout the paper—including in the audio samples. Why exclude speech? Have you evaluated performance formally or informally on out-of-domain data? What sort of results do you get? How fragile is the model outside the training distribution?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Listening to the audio samples, the pitch reconstruction seems noticeably off—enough that the current system would not yet be amenable to fine-grained music editing. As well, the reconstruction of high frequencies could be improved and the context conditioning degrades noticeably in inpainting. The paper could benefit from a reconstruction task (i.e., inpaint with no mask) to better understand why context conditioning is degrading when the ground truth is provided to the model. Otherwise, it's not clear whether quality/accuracy errors can be attributed to the audio generator or the editing process.

Summary

This work proposes a new editing technique to audio data.

Strengths

The authors introduce a new task and a base method to tackling it.

Weaknesses

**Conditioning:** The authors concatenate the input audio to the sampled noise, why did you pick this option on top of sampling a Gaussian around the input sample of giving it as part of the cross attention (which will probably allow for modifications in the generation length)? Can the authors elaborate on this point? **FAD** The authors do not show the FAD of audiogen and make-an-audio. Why? I noted that the classifier is different (PANN vs. VGGISH), but it is hard to understand what are the differences in performance. **Comparison to prior work:** I am having hard time understanding what is the cause for the model's performance - is it the data or the modeling. All the baselines present results when trained on different (mostly open) datasets; To do an appropriate comparison, it is expected to at least train one-two of those models on the same data the authors used. **Subjective study** I find the experimental study not reproducible (even after reading Appendix D). Did the authors also check metrics compared to ground truth data? At least for dropping and super resolution, you have a hard gt data. **Error bars** Some of the metric differences are small - can the authors report error bars? Overall: I think that the task is interesting. However, since it is a new task and the paper doesn't introduce a new modeling approach, I would expect extensive experiments, which I was unable to find.

Questions

**Technical questions:** - Which diffusion process was used? - How do you pass the condition to the model? I couldn't find answers to those questions explicitly in the paper.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

None

Summary

This paper proposes AUDIT, an instruction-guided audio editing model based on latent diffusion models. AUDIT is trained on triplet data consisting of an instruction, an input audio, and an output audio. The instruction is used to guide the diffusion model to only modify the audio segments that need to be edited. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks, such as adding, dropping, replacement, inpainting, and super-resolution.

Strengths

AUDIT is a novel audio editing model that can be trained using human text instructions. AUDIT is trained on a dataset of triplet data, which consists of an audio clip, a text instruction, and the edited audio clip. AUDIT can maximize the preservation of audio segments that do not need to be edited, and it only requires simple instructions as text guidance. AUDIT has achieved state-of-the-art results in both objective and subjective metrics for several audio editing tasks.

Weaknesses

.

Questions

1. Please add ablation analysis for each component. 2. Can you train it on one task and try to zero shot in other task? 3. The model compare to SDEdit. please add more comparison to other method (classic / deep learning based).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

.

Summary

This paper proposes AUDIT, an instruction-guided audio editing model based on latent diffusion models. The model is trained with triplet training data (instruction, input audio, output audio) for different audio editing tasks and is based on a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio. The proposed model outperforms several baselines in the editing tasks

Strengths

The AUDIT model is able to perform audio editing conditioned on text instructions, which have rarely been studied by prior work on audio generation. The data construction strategy of using paired text-audio data and refining text instructions based on templates and LLM is also useful for making instruction datasets. In terms of performance, it is better than several baselines (SDEdit and its variants).

Weaknesses

The main weakness lies in the evaluation. The model is mostly compared to SDEdit and its variants, which is zero-shot audio editing model. It is unclear how it compares to task-specific models such as NU-Wave2 [1] for audio super-resolution. Some of the editing tasks considered in the paper (e.g., Adding) can also be addressed via text-to-audio generation, while proposed method has not been compared to the methods of this category. There also lacks analysis on the gain of the proposed method over SDEdit and its variants. For example, how robust the model is to the category of the dropped sound in Dropping task? As both the training and evaluation data are synthesized, it is unclear how the model will perform on real data. [1]. Han et al., NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates

Questions

N/A

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes