MAViL: Masked Audio-Video Learners

Summary

This paper proposes a self-supervised audiovisual representation learning framework. It combines MAE with contrastive learning in both intra and inter modality settings. The authors also suggest a self-training schema where the student model, which only sees masked input, needs to align with the latent representations of the unmasked input produced by the teacher. This shows to improve the quality of learned representations in just a few iterations without relying on external pre-trained teacher models.

Strengths

1. The work is well-motivated and for the most part easy to follow. 2. Experiments and ablation studies are comprehensive with some reservations (ref to Questions section) 3. Components of the proposed approach (Fig 1) are not novel, however putting them together and studying how they interact with one another is valuable. 4. While there are other works which have combined MAE with contrastive learning, this paper has done a good job distinguishing itself from those concurrent/earlier efforts. 5. Self-training results are promising (Table 4.b) and the ablation for reconstruction target (Table 4.a) provides valuable insight.

Weaknesses

1. The effectiveness of the fusion layer is not clear: Table 1 does not show any meaningful gain added by the fusion layer over unimodal MAE. It is worth noting that “None” has fewer parameters compared to other columns in Table 1 due to not having a fusion layer. Similarly in Table 4.a when comparing M-Fusion against M-Uni.
 2. Table 3: Intra is lower than Inter, and their combination provides marginal gain over the inter (0.6 on audio and 1.2 on video). It is worth remembering that the addition of intra doubles the computation cost since we need to process two views from each input-modality. Intra loss is one of the main points that this work is using to distinguish itself from CAV-MAE.
 3. Table 5.e summarizes the effectiveness of the contributions claimed by MAViL. We can see that the addition of joint AV-MAE adds nothing meaningful to the baseline. Similarly is the marginal gain due to the addition of intra loss over the inter one. The student teacher learning is the only novel contribution that shows to be effective.
 4. Minor note (not a weakness): arxiv paper https://arxiv.org/abs/2303.12001 also combines MAE and contrastive losses though the setup and modalities are different from the current paper.

Questions

1. What is difference between $\tilde{a}$ (similarly $\tilde{v}$) in line 183 and $\hat{a}$ (similarly $\hat{v}$) in line 135? 2. Eq 5 is confusing: $\tilde{a}$ is in the raw space (ref Eq 1) but here it is being compared against contextualized latent embedding of the audio (similarly for video). It made sense if $\tilde{a}$ did not have the decoder (i.e $f^{-1}$). The current form also contradicts with default setting in Table 4.a. Authors should clarify. 3. Authors should provide intuition on $\tau$ values for inter and intra losses. Why do we need to be more tolerant for intra? How are the actual values been chosen? 4. Linear probe is more favorable for evaluation than fine-tuning. As also noted in Sec 4.3, there are so many bells and whistles (e.g all sorts of data augmentations and learning rate annealing strategies etc) when it comes to fine-tuning. Linear probe on the other hand relies on minimal parameters and evaluates the quality of learned representations not the pre-trained model. It would be more helpful if we were able to see linear probe results. 5. Table 5.b: Different visual backbone initializations impact the quality of the corresponding modality in a meaningful way (as expected) however the performance is flat on the audio side. Considering fusion layer is supposed to mix the representations between modalities, one would expect to see a lift for the audio backbone too. It would help if authors provide some insights. 6. Table 6: Is it fair to compare against audio-only models when MAViL was pre-trained with video and audio?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes, they have.

Summary

The authors present Masked Audio-Video Learners (MAViL), pretrained on 3 pseudo tasks and consisting of a 2-stage approach. i) masked reconstruction (adoption of MAE, AudioMAE, VideoMAE) ii) intra-modal and inter-modal contrastive learning (similar contrastive loss like SIMCLR, MOCO), and iii) student-teacher learning (performed in stage 2, by minimizing the error between earlier pretrained encoders referred to as teachers and the current encoders referred to as students). The proposed framework is pretrained and evaluated with AudioSet and VGGSound. Additionally, audio-to-video retrieval experiments are done on MSR-VTT and YouCook.

Strengths

- The proposed method is a nice extension of masked reconstruction in a multimodal setup. - Fair amount of experiments are conducted as part of ablation to support the design choices. - Good presentation and easy to follow. - Thanks to the authors for making the code to be public and sharing the details of the implementations which are helpful for reproducibility.

Weaknesses

- This work does not provide results on linear evaluation. The whole point of SSL is to learn rich abstract representations that can be used through minimal training with labelled data, i.e., linear SVM or FC-tuning. - The comparison setup with prior works is insufficient. For example, the SOTA evaluation on visual tasks is extremely weak and most of the entries in Tab. 6 for the video are blank (-), while there are plenty of methods that can be compared. This work does not compare the results of ESC50 with several AV-SSL prior works. - The proposed method lacks technical novelty as all the pieces (masked reconstruction, contrastive learning, student-teacher) of the proposed method are already been studied quite heavily in the literature, be it in a uni-modal or multimodal/cross-modal setups.

Questions

1. The method uses 10 seconds of audio and 4 seconds of video, which means the network sees asynchronous audio-video segments. Why is such a setup used? Is there a reason for using different durations of audio-video sequences? How would the results be affected if the same 4 seconds of audio and video are used? Could you please provide such results? 2. What would happen if you drop the fusion encoder and directly reconstruct the encoded representation? Could you please provide the results by removing the fusion encoder while keeping the rest of the setup intact? The loss function would remain the same. To clarify further, replace $\hat{v}$ and $\hat{a}$ in Eq 1 with the following values: $\hat{v} =f^{-1}_v(f_v(v'))$ and $\hat{a} =f^{-1}_a(f_a(a')) $. 3. What would happen if you initialize the encoders with VideoMAE and AudioMAE checkpoints and only perform stage 2, which involves training with Eq. 6? Could you please provide the results for such a scenario? You can utilize the pretraining weights released by the respective papers. This approach would simplify the process, as currently, the intra-modal and inter-modal contrastive losses are optimized twice (in both Eq. 4 and 6), which appears to be somewhat redundant. 4. The authors have maintained a fixed teacher in their approach. However, I'm curious to know how it would perform if the teachers are updated slowly using Exponential Moving Average (EMA). Additionally, the current loss between the teachers and students is Mean Squared Error (MSE). I am curious to understand why MSE is used and what would happen if the loss function is changed to minimize cross-entropy or cosine distance instead. Could you please provide the findings for such experiments? 5. The authors are requested to include the following information in the SOTA comparison tables (Tab. 6) to ensure a fair and informed comparison with other methods: - model names (e.g., ViT, ResNet, etc.) - number of parameters during pretraining (backbone+auxiliary network parameters such as heads, decoders, etc.) - number of params during evaluation, - pretraining input size - evaluation input size. 6. The authors are strongly recommended to compare MAViL with prior works in linear evaluation. As a reference for such evaluation, you may see Brave (Tab. 6). 7. The authors are suggested to evaluate their method on the following setups. - linear and finetune comparison on (i) large-scale datasets like Kinetics400 and Kinetics600; (ii) small-scale datasets like UCF101 and HMDB51; and (iii) temporal-focused datasets like SSv2. Similar to: VideoMOCO, CVRL, p-BYOL, ELO, VideoMAE, Brave, AVID, Robust-xID, Selavi, GDT, CrissCross, STiCA, and CM-ACC. - action retrieval on UCF101 and HMDB51 to report R@1, R@5, R@20. Similar to: Robust-xID, Selavi, GDT, and CrissCross. - You may check Brave Tab 6 as a reference for a comprehensive comparison. 8. The authors are also recommended to compare the results of ESC50 with prior AV-SSL methods, please use similar references for comparison. Refs.: XDC, AVID, Brave, CrissCross, CM-ACC. 9. Typo: in Figure 2: iter 2, V-Enc -> V-Dec for the reconstruction part. 10. Minor suggestions: - There are a few works that are very similar to MAViL and have some overlapping ideas such as joint-masked reconstruction, student-teacher etc. They are available on arXiv for 5-6 months, I would suggest discussing them as well, e.g., AVMAE and XKD. - I recommend adding a pseudocode, it gives more clarity. ========= References: VideoMAE: https://arxiv.org/pdf/2203.12602.pdf; Brave: https://arxiv.org/pdf/2103.16559.pdf; AVID: https://arxiv.org/pdf/2004.12943.pdf; Robust-xID: https://arxiv.org/pdf/2103.15916.pdf; Selavi: https://arxiv.org/pdf/2006.13662.pdf; GDT: https://arxiv.org/pdf/2003.04298.pdf; CrissCross: https://arxiv.org/pdf/2111.05329.pdf; CM-ACC: https://arxiv.org/pdf/2009.09805.pdf; VideoMOCO: https://arxiv.org/pdf/2103.05905.pdf; CVRL: https://arxiv.org/pdf/2008.03800.pdf; p-BYOL: https://arxiv.org/pdf/2104.14558.pdf; ELO: https://arxiv.org/pdf/2002.12177.pdf; STiCA: https://arxiv.org/pdf/2103.10211.pdf; AVMAE: https://arxiv.org/pdf/2212.05922.pdf; XKD: https://arxiv.org/pdf/2211.13929.pdf;

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

In summary, this work lacks novelty, a comprehensive evaluation on different downstream benchmarks, and a thorough comparison with several prior works. These weaknesses outweigh the reasons to accept. The proposed method has three objectives and all of them are heavily explored in the literature, and I do not see solid technical novelty. First, the masked reconstruction technique has already been implemented in several published works such as MAE, MAE_ST, VideoMAE, AudioMAE, CAV-MAE, and many more. Secondly, contrastive learning in multimodal/cross-modal setups has been widely studied. Lastly, the use of a frozen pretrained network to supervise an online encoder is not novel, whether in uni-modal or multi-modal setups.

Summary

The authors propose to learn audio-visual representations with two stage process with three self-supervision losses including 1. masked reconstruction 2. intra and inter-modal contrastive learning with masking 3. self-training via teacher-student setup to contrast on contextualized representations in 2nd stage. The authors show that proposed approach achieve state-of-the-art audio-visual classification performance on both AudioSet and VGGSound, audio-only tasks, and audio-to-video retrieval.

Strengths

- The authors provide thorough ablation studies for each loss function introduced, masked ratio, choices of dataset size, backbone init, model size. This helps to understand the proposed system with a more complete view. - The proposed approach to combine two popular loss function masked reconstruction and contrastive learning, and further introduce the self-training on contextualized representation is original. - This paper is well written with clear narrative on the motivation and it provides sequences of experiments which help the reader to understand their thoughts step by step.

Weaknesses

- This work lacks evaluation on the visual modality, especially video, as this work claims to be audio-video learner. Both AudioSet and VGGSound are mainly used in audio research, it would be beneficial to provide performance on some video benchmarks. For example, MBT also include EpicKitchens-100. - Minor suggestion: consider add AS-20K in the caption of table 1, as it is less clear which dataset/task is used. Relevant to this, the use of AS-20K for ablation study is mentioned in the last paragraph of section 4.3, consider move it to the first paragraph of section 4.4 to better highlight this choice.

Questions

- In section 4.3 it is mentioned that for downstream evaluation fine-tuning is applied. What is the performance on frozen encoders with linear probe if matching the output number of classes is necessary? Both fine-tuning and frozen can provide different perspectives of proposed system and can be used in different applications. It can provide a more complete view if provided. - In table 4(b), what is the performance beyond three iterations, does it keep improving or the performance drops? It helps to include iteration 4 in either case.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

- This work provides thorough study on proposed approach applied to audio related tasks including classification and crossmodal retrieval. It lacks some aspects on the video modality. It would be beneficial to include more evaluation with video downstream tasks.

Summary

This paper presents a new self-supervised learning framework for audio-visual modalities. It combines three types of SSL techniques, namely MAE, contrastive learning, and student-teacher distillation. The proposed method is a two-stage process by first training a teacher network to reconstruct the raw signal and then using this network to guide a student network to learn in the second stage. Authors show that this method improves the SOTA on AudioSet, VGG-Sounds as well as several other classification and retrieval benchmarks.

Strengths

Although the three SSL losses in this paper are not new, this paper did a good job of putting them together and making them work. The performance improvement over the SOTA method is also pretty decent, validating the effectiveness of the approach. The authors conducted a very sufficient ablation study of the model architecture, showing that multi-modal fusion, contrastive learning, reconstruction, and distillation all contribute to the model's performance. The writing is very clear-written and easy to follow.

Weaknesses

In the implementation details, authors mentioned many data augmentations are used in training. How much do they matter? I do not see an ablation on this component. The limitation of this work is not discussed in this paper.

Questions

I'd appreciate the authors' response on how much data augmentation matters and the potential limitation of this work. ====post rebuttal=== The authors' response addressed my concerns and I am keeping the original score.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitations and societal impact are not discussed in this work.

Additional discussion about choice of objective (Q4)

**Q4 Why MSE and what if using cross-entropy or cosine distance?** Regarding your question concerning the choice of loss function, we adhered to methodologies used in previous works such as MAE, VideoMAE, Aud-MAE to choose MSE as the reconstruction objective. MSE has demonstrated its efficacy in facilitating auto-encoding across continuous signals such as images, videos, and audio. With regards to your insightful suggestion of using other objectives such as binary-cross-entropy (BCE) and cosine distance, we carried out a comparison listed below (due to rebuttal time limit, we only compare MAViL-stage1 results): ||MSE|L1|BCE|cos-dist| |---|---|---|---|---| |A|39.0|38.5|38.3|36.9| |V|22.2|21.5|21.5|19.0| As illustrated above, BCE also works but it is not as good as the widely used MSE in the context of masked auto-encoding. This disparity can be attributed to BCE's asymmetry, which introduces a bias towards a probability of 0.5, while MSE symmetrically regresses towards the exact input intensities, thereby enhancing performance. Furthermore, the attempt to minimize the cosine distance does not yield comparable outcomes in this context. This is likely due to the increased difficulty of reconstructing the precise input, which, in turn, compels the encoder to grasp intricate contextual elements like intensity, which is important for downstream classification tasks. Consequently, MSE results in a more robust self-supervisory process. In contrast, minimizing the cosine distance may inadvertently disregard intensity information. For instance, scenarios like a daylight scene and its corresponding darker or night-view might exhibit a small cosine distance between normalized patches. Failing to differentiate visual and spectrogram intensity may ultimately result in inferior SSL representations.