Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching

Summary

They propose FRIEREN, an efficient video-to-audio generation model based on rectified flow matching that obtains state-of-the-art results.

Strengths

- They successfully propose rectified flow matching for video-to-audio, that is a problem that is important in current generative AI setup where most video generative models are generating video without audio. - They run a perceptual study. - The paper (specially section 3) is very well written and clear. - I appreciate the examples in the demo webpage. Also the selection of the examples, that does not feel cherry picked.

Weaknesses

The introduction lacks scientific rigor. - line 36: "leave room for further advancement". This is a general statement, can you be more specific? - line 36: "autoregressive models lack the ability to align the generated audio with the video explicitly". This is not true, because AudioLM and MusicLM are autoregressive models that use explicit semantic tokens to capture structure similar to the conditioning in video-to-audio. I could not find how you compute alignment accuracy. Minor comment related to scientific writing: - line 18: "revolutionary enhancements". It feels like marketing and this is a scientific paper.

Questions

- Do you plan to release the code? - How do you compute alignment accuracy? - Why not using CLIP for visual representation?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- 16kHz and short-form videos. - No code/weights provided.

Summary

This paper presents a new model for video-to-audio generation. The proposed model is based on rectified flow formulation and adopts Transformer-based architecture. A conditional video is fed into the model via channel-level concatenation to the audio tokens after processed by a length regulator. After training of the model, the model is further fine-tuned with the reflow and distillation. They are conducted with synthetic data generated by the firstly trained model with classifier-free guidance. The experimental results demonstrate that the proposed model outperforms the existing models by a large margin both quantitatively and qualitatively.

Strengths

- The design of the proposed model is simple and reasonable. The proposed model is based on Transformer, and the video condition is fed into the model via channel-level concatenation to the audio tokens after adjusting the number of tokens. This design would be beneficial for boosting the temporal alignment, as it explicitly utilizes the temporal correspondance between the conditional video and the generated audio. - In the experiments, the proposed method outperforms the other existing methods by a large margin. I have checked the generated examples on the website, and they are really amazing. - The proposed model is quite light-weight, and it is great to be able to train the model with only two GPUs. In addition, the inference speed is substantially fast thanks to the reflow and distillation as well as the light-weight design. - The manuscript is well-written and easy to follow.

Weaknesses

- The experiments have only been conducted with one dataset, which is VGGSound. Training or zero-shot evaluation with other datasets (such as Landscape dataset) would be beneficial to validate the generalization capability of the proposed method. - The empirical analysis on why the proposed method performs well seems insufficient. According to the results shown in Table 2, DDPM with the proposed model architecture already achieves substantially better performance than the existing methods. Thus, it appears that the model architecture, rather than the usage of the rectified flow, is the key for the impressive performance. As far as I understand, its major difference from the standard Transformer is two-fold: channel-level concatenation for the conditional inputs instead of sequence-level one (or cross-attention mechanism as in [23]) and the usage of 1D-conv instead of 2D-conv. It would be great if this paper could provide an empirical analysis on which component actually boosts the performance for video-to-audio generation. The current manuscript places significant emphasis on the rectified flow aspect, which is not particularly novel as the proposed model largely follows to the settings of previous works.

Questions

- Is there any particular challenge (and its solution) when applying rectified flows for audio generation? - Minor questions: - Is CFG also applied for the reflowed models? I understand that it is applied when generating the training data for the reflow process but cannot find how it is set during the inference phase. --- <After the rebuttal> I updated my rating from 5 to 7.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations have been discussed in the appendix.

Summary

The following work proposes a video to audio generation model. The model architecture closely follows that of prior work diff-foley, which operates on 4 frames per second, fits a temporally aligned latent space between audio and video content, and then a latent diffusion model to map from this latent space to audio. This work proposes to replace the latent diffusion model architecture with a transformer-based rectified flow model, and opts to use the MAViL audio-video joint latent representation instead of the one from diff-foley (CAVP). Results are qualitatively much better than that of diff-foley, and also faster to sample from due to the rectified flow matching formulation.

Strengths

- Qualitative results significantly improve over prior work - Decent ablation studies over critical architectural design choices, such as CAVP vs MaVIL, loss-reweighting for training flow matching models.

Weaknesses

- Despite improvements over prior works such as diff-foley, the contributions of this work remain limited. The time-aligned audio generation appears to stem from architectural choices made in diff-foley. - Furthermore conditional-optimal-transport flow-matching generative models have been applied audio models with similar conclusions. The specific application to the video-to-audio task, in my opinion, is not sufficiently different from prior applications in audio for the findings in this work to be particularly new. Specifically, it should be considered to be very closely related to other temporally-aligned conditional generation tasks such as text-to-speech. - It's also worth noting that Diff-Foley uses a very simple griffin-lim to map predicted spectrograms to audio waveforms, whereas this work makes use of the much more effective BigVGAN model. This makes it very difficult to pinpoint the qualitative improvements of the proposed work compared to prior methods.

Questions

- I'm curious how the authors were able to try MAViL given that the code for this project does not appear to be publicly available?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper proposes a diffusion model based on rectified flow matching. Besides, to generate better audio quality, the authors propose re-weighting objective. The method achieves the state-of-the-art results on V2A benchmark.

Strengths

* The proposed method is the first to leverage rectified flow matching on video-to-audio generation tasks. * The quantitative and qualitative results demonstrate the superiority comparing with existing baselines.

Weaknesses

* Although FRIEREN shows impressive results, the competing methods ( i.e., Diff-foley) based on U-Net style diffusion model are relatively weak. The performance gain seems mostly come from transformer architecture. * Following previous point, DDPM shows pretty similar results when increasing steps. It makes the proposed method less stronger. Thus, it would be great to show the results with more steps. * The proposed method, reflow, cannot consistently benefit FAD on different number of step. It seems not reasonable. Overall, the results are good. If the authors can address some questions and more insight (comparing to simple adapting reflowing in V2A like speech model), that would make the paper more convincing.

Questions

* Do the authors use any pretrained initialization for transformer? * The design of Fig2b is very similar to standard ViT block. Any intuitions or differences between these two? * In Fig2b, are the c latents, video features, performed any pooling layer?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

See weakness.

Thanks for the rebuttal

Thanks for the response and additional experimental results. I have read them as well as the other reviews. The additional experimental results clarify the advantage of the proposed method as well as which component contributes the performance gain. As my concerns have been properly addressed in the rebuttal, I would like to update my rating from 5 to 7.