Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Summary

This paper proposes a new method for performing the task of "Foley", which entails the creation of sounds that match a video, for example, making an audio clip in a studio that matches a character's footsteps in a film, as a post-production task. The authors proposed to do this automatically via deep learning, which has been proposed in other works, notably Im2Wav from last year. They propose 4 main contributions: audio-visual contrastive pre-training, latent diffusion modelling, an augmentation technique based on cropping different clips together, and the combination of classifier guidance and classifier-free guidance. Through a set of ablations, the authors show the importance of these four contributions. They also show superior results compared to 2 previous works on 2/4 metrics, and on inference time. Many qualitative results are shown and the appendix includes a large amount of discussion around many of the central topics.

Strengths

Fundamentally, the paper has enough contributions, in my opinion. Out of the 4, the first is very substantial, and the other 3 are reasonably novel when applied to this field. The writing is quite consistent with little to no mistakes, and in general the clarity of the discussions is remarkable. The paper does well in introducing readers to the task of video foley, which is not particularly well known amongst most readers, and also does a good job at presenting a lot of the modeling techniques used. The ablations satisfy the vast majority of questions the reader may have, and the more fine-grained ones are a nice addition. Quantitative results are explicit and well-presented. The lengthy appendix is welcome with some fruitful discussions. The demos are really impressive when compared to other works.

Weaknesses

From the demos and quantitative results, the previous approaches seem to feature quite weak reproductions of the original sound. The authors only compare with two methods, which leaves me wondering if there could be comparisons with more works. I have found, for example, that FoleyGAN (2021) seems to perform the same task, but no comparison (or citation) is present. I believe the results would be a lot more convincing with more comparisons. Also, Diff-Foley is outperformed on 2/4 metrics, which is also not very reassuring, although they are FID and KL which are known to be somewhat unreliable. While the authors clearly show that adding Classifier guidance helps compare to only CFG, I would like to see an ablation where only Classifier guidance is used as well, as a matter of completeness, to show that both components of the double guidance are truly important. Figure 2 is good for the most part, but the latent diffusion section is, in my opinion, too vague - instead of just having a big box labelled "latent diffusion", it would be nice to have more detail to explain exactly what is happening here. Particularly for readers who are not already familiar with LDMs. The third contribution "Temporal Split & Merge Augmentation" is similar to augmentations that have long been used in other domains. Namely cropping parts of different data points together to form a new one. For example, "MixUp" does exactly this by mixing two different images and labels together. Some discussion around previous approaches similar to this contribution with adequate referencing is necessary. In table 1, if inference time is mentioned, then I believe number of parameters and training time, if possible, should also be presented for a more fair comparison. In the related work, there is a section on "Video-to-Audio Generation", but existing works on a specific subset of video-to-audio which has received a lot of attention, video-to-speech, are not mentioned at all.

Questions

As mentioned above, are there any other methods that this work could be compared with in the results section? Will the code be released upon acceptance? If so, will this include training code, pre-trained models, etc.? These contributions to the open-source community would be very welcome.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Although I realize there are space limitations, the broader impact is really insufficient. Please put a bit more substance into this discussion. For example, you can talk about how this technology could be used to create fake videos more accurately, which could help spread false information. Or perhaps you can talk about the consequences of using producing an inaccurate reproduction of the audio, which may change the meaning of the video and mislead viewers. This may also be used to improve video surveillance, etc.

Summary

The paper presents an approach to generate and align audio with an existing video track, using a diffusion process. This is useful in video post-processing, where frequently sound effects have to be aligned with existing video footage (and not just match semantically). The authors demonstrate the quality of their approach, which relies on a "double guidance" technique to improve the quality of the reverse process in LDM, and improve the alignment. Authors perform fine-tuning and ablation studies, to further understand the properties of the proposed approach.

Strengths

Originality The paper presents a novel approach to learning audio-visual alignment, using a CLIP-style loss, taking within-clip and across-clip contrastive pairs from audio and video. This CVAP is a novel contribution, and at the core of the proposed work. LDMs are still a non-standard use in audio, although they are finding applications. Quality, Significance Given the paper's improvements (mostly given through the CVAP formulation and its application to the LDM), I think the paper makes a significant contribution, although the topic is somewhat "niche" Clarity The paper is mostly well written and covering a reasonable set of experiments, it describes the main argument.

Weaknesses

I don't understand the effect of temporal split & merge, section 3.3 (also see the question below) -- IIUC, the model could be trained on any audio-video pair, so if AudioSet+VGGSound is not enough, why not simply crawl more data from the Internet? It is clear that temporal augmentation improves results, specifically around the alignment metric, but Table 5 does not seem to indicate that more data always improves results?

Questions

- In Section 3.3, you say "We validate the effectiveness of this augmentation method in Sec 3.3." - please explain? - In the abstract (and in other places): "we demonstrate DIFF-FOLEY practical applicability and generalization capabilities via downstream finetuning." -- how does fine-tuning, which is kind of a specialization of the model, demonstrate generalization capability? - Figure 5 - I am not sure I understood what is presented. The model makes mistakes for the first example - are these being resolved through fine-tuning the diffusion model, or do they still present?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

n/a

Summary

This paper present DIFF-FOLEY, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates audio with improved synchronization and audio-visual relevance. The method adopts contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. During inference, the method adopts a combination of classifier-free guidance and classifier guidance based on a synchronization classifier. The proposed model outperforms existing methods in audio-visual synchronicity and inception score.

Strengths

The proposed model achieves greater audio-visual synchronicity compared to several baselines. Through analysis, the audio-visual pre-training module indeed brings significant gains to the synchronicity. The paper is overall well-written and the authors have conducted a thorough analysis on the effectiveness of different modules, including the effect of two guidances, the used features, effect of pre-training.

Weaknesses

Despite its advantages in audio-visual synchronicity and inception score, the proposed method falls below the baselines in other metrics such as FID and KL divergence. There is no human evaluation metrics in the comparison among the methods. Thus it is unclear overall how realistic the generated audios are compared to other methods. Regarding the synchronicity classifier, the authors have not thoroughly analyzed the synchronicity classifier, which is the key model to measure the main metric used in the paper. The paper mentioned it achieves 90% accuracy on the test set. However, it is unclear how the test set is constructed. For example, how is the negative-sample set constructed? What is the precision and recall respectively? How well does the accuracy match the human perception of audio-visual synchronicity? In terms of modeling, the generative model is a typical LDM model used in audio generation and lacks general novelty.

Questions

* What is the audio-visual synchronicity classifier used in the model? Is it the same with the classifier for accuracy measurement?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The focus of the paper is audio synthesis. More specifically, it focuses on video to audio synthesis and in particular on synchronized synthesis of audio. It relies on Latent Diffusion models for the synthesis and proposes an aligned audio-visual representation learning approach to improve synchronization of synthesized audio and input video. CAVP - Contrastive Audio-Visual Pre-training – tries to learn these features through contrastive learning. Experiments are conducted on the VGGSound dataset and both quantitative and qualitative results are shown in the dataset. In quantitative terms, the proposed DIff-foley method outperforms prior works by a considerable margin with respect to some of the metrics.

Strengths

-- The paper addresses a key problem in audio/video synthesis. Generating synchronized audio for videos is challenging and most of the current struggle at. -- Learning good audio-visual representation is essential to solving this problem. The approach taken by this paper makes sense. -- The gain in IS metric and inference time using the proposed method is large. Moreover, the demo and the qualitative results does show superior and more synchronized

Weaknesses

-- In Eq 1, aren’t the two terms same ? Why have they been separated ? Same is true for Eq 2. -- Analysis and discussion on duration as a factor is completely missing. It appears that all empirical results are given for fixed 8 seconds audio. Some results for variable length video inputs would be helpful to understand the duration factor. More importantly, producing synchronized video on longer form audio/video would be more challenging than relatively short audio/video. How does the method work out in those cases ? -- WHile the Diff-foley method does well (compared to others in Table 1) in terms of IS and synchro. Acc. it does not do well in terms of FID and KL. Some discussion on why that might be happening is desirable. Diff-foley is significantly inferior to other methods on these metrics. In Table 2, on impact of visual features - CAVP ends up doing better than others in terms of FID but not IS and KL. Overall, these results and lack of discussion in what might be happening leads to unclear understanding of the performances. -- Audio/speech synthesis works in my opinion should include some form of subjective tests. Objective metrics (especially when some of them are basically imperfect pre-trained models) do not paint a clear picture. This is also necessary because the generated sounds themselves are often not corresponding to the visual objects. The frequency content of the generated sound in Fig 3 and Fig 4 is clearly far from the ground truth. From the demo also, in many cases the generated sound does not correspond to the visual object. --- updated score after rebuttal ---

Questions

Please respond to the points outlined in the weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss limitations of the method in terms of scalability. A bit more discussion on how generative AI for audio can have societal impact might be good.

Response to authors

Thank you for answering my questions. I appreciate the new ablation and human evaluation. Also happy to hear you'll be releasing code, acknowledging previous works in video-to-speech, adjusting Figure 2, and have added num. params and training time. Happy to raise my score slightly.

Awaiting Your Valuable Feedback

We appreciate the time and effort the reviewer has dedicated to providing us with thorough and constructive feedback. Please inform us if our response addresses all concerns and let us know if more information is needed. We are committed to providing any necessary clarifications.

Awaiting Your Valuable Feedback

We appreciate the time and effort the reviewer has dedicated to providing us with thorough and constructive feedback. Please inform us if our response addresses all concerns and let us know if more information is needed. We are committed to providing any necessary clarifications.