Summary
The paper proposed a Video-to-Audio generation framework. The model consists of a pretrained audio VAE and vocoder, a latent-diffusion-model, and CLIP-based vision encoder. The paper studied different design choices, such as using different vision encoders, input different conditioning features, and different data augmentation methods during training.
Weaknesses
1. Lack of evaluation dataset and synchronization metrics
VGGSound is in fact a noisy dataset to measure both: on one hand, it contains overwhelming audio noises, and many video audio pairs are not semantically aligned; on the other hand, the video-audio synchronization is even poor due to camera shaking, out of scene sound sources, etc. Previous papers on VTA has evaluated on some other datasets, which can be simpler but much cleaner, e.g., evaluate semantics alignment on Landscapes dataset (Seeing&Hearing), or evaluate both semantics alignment and synchronization on AVSync15 (FoleyCrafter). Without evaluation on these datasets, the single-dataset (and also noisy dataset) evaluation results is pretty hard to convince the community.
The author explained in Sec. 5.2 to "omit alignment metrics, as they are tailored to produce shorter audio segments, which does not align with competitive standards". This reason is weak. Suppose the other datasets/metrics do have shorter audio-video segments, the author can simply extend the input video with black/white/the last frame, then after getting the audio, only retrieve the first several seconds' content. In fact, there are lots of videos in VGGSound that have frame sudden changes (some frames in one video are even not related to each other), therefore, the method I mentioned to deal with shorter video-audio segments is absolutely feasible.
2. Inconsistent / strange evaluation.
On VGGSound, the author mentioned in Sec. 4.1 that, on VGGSound, they are using 200k video for training and 3k for testing. Even without considering broken youtube links, VGGSound's original annotation file only contains 183,971 training videos and 15496 testing videos, summing up to no more than 200k. How come the author can use 203k VGGSound videos? Do the training and testing splits overlapping with each other?
Also, the evaluation metrics are inconsistent. For the same model and CLIP4CLIP encoder, the AV-Align metrics is reported as 0.243 in Table 1 but 0.225 in Table 2, while other metrics being the same. If this was a typo, I would consider it as very serious mistake.
3. Weaker performance compared to SOTA.
When comparing with other methods in Table 1, the author used CLIP4CLIP as the encoder. However, the proposed model, once equipped with ImageBind encoder (Note that ImageBind vision encoder uses frozen CLIP image encoder, which makes it the same as CLIP encoder in Table 1), the performance becomes much worse than some other methods. E.g., AV-Align is 0.189 compared to FoleyCrafter's 0.239, CLAP is 0.186 compared to IM2WAV's 0.306, PAM is 0.166, the worst among all compared models. That means, under fair comparison, the paper cannot achieve SOTA performance, as opposed to what is claimed.
Besides, even the proposed model is not stronger than previous methods, there should always be a more detailed comparison between the proposed framework and previous works. E.g., DiffFoley and FoleyCrafter also used LDM for generation, what is the difference between the proposed LDM and theirs?
4. Some missing but important details.
(1) In Sec. 3.3, when UNet attending the vision features, does each audio feature only see vision features within its corresponding time window? Or, each audio features is attended to all vision features, regardless its time? If it is the former, how exactly did you split the vision tokens to achieve time-aware attention? If it is the latter (every audio feature in UNet sees exactly the same vision encoded feature, regardless of time) and the model can still achieve good synchronization results, then the only explanation should be that, the vision features have already be implicitly encoded with time information (e.g., CLIP4CLIP might already have positional encodings on temporal dimension), which might be the reason why using PE does not help in Table 3? Any more discussions on this?
(2) How did you sample the 3k validation data? Did you first apply data alignment filter to get a clean subset first, then choose from it? Or did you simply random sample from original noisy VGGSound?
(3) There should be formal explanation on each metric you are using, at least in supplementary material. The current rough explanation in Sec. 4.2 is vague. For example, the author mentioned to use CAVP as semantics and synchronization evaluation metric. However, CAVP is only a contrastively trained model in DiffFoley. The original DiffFoley paper used a trained classifier to tell synchronization and semantics alignment instead, and called the metric "Align Acc". Did you use CAVP features just as in CLIPSim for evaluation, or did you use Align Acc? The detail is missing though.
Questions
Following my weakness mentioned above, I would like to see answers to the following questions:
1. The proposed method's performance, and comparison, on Landscapes (Compare to Seeing&Hearing) and AVSync15(Compare to FoleyCrafter).
2. The author's explanation on strange training/testing splits of VGGSound, and inconsistent data in Table 1 & Table 2.
3. A conclusion on, under fair comparison, whether the proposed model is truly SOTA or not, and provide more concrete comparison between the proposed model and others, as well as some analysis on why (or why not) the proposed model is SOTA.
4. The 3 questions I've asked in Last point in weakness.