Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

Summary

This paper proposes a method called VALOR to assign segment level audio-visual (AV) event/object labels given weak labels at the video-level. It makes use of pseudo-labels obtained from unimodal pretrained models CLIP and CLAP to derive additional guidance for the AV model. The final model is trained using a combination of AV, audio-only and video-only losses, and the AV component is based on the hybrid attention network (HAN) structure. Experiments are performed on two tasks, AV video parsing and AV event localization. In both cases, the VALOR approach outperformed previous studies on the respective tasks by a clear margin. Some additional ablation experiments further validate the design choices that have been made in the system. * After rebuttal Given the authors' response, and them clarifying my concerns, I am increasing my score, especially for presentation.

Strengths

The main strength is that the experimental results clearly outperforms older approaches on the two tasks described in the paper. * Originality: Even though the individual sub-components such as CLIp, CLAP, HAN are not novel, the paper brings them together in a certain way (unimodal guidance with dense labeling of the video segments) to solve the AV video parsing problem on a weakly labeled dataset. * Quality: Experimental evaluations successfully support the use of the newly proposed VALOR technique. * Clarity: Language is mostly clear. * Significance: The paper probably establishes the new state-of-the-art the for the AV video parsing task on the LLP dataset.

Weaknesses

* The main weakness of the paper is probably the flow of the paper. 1. There is a section called Preliminaries and then in Section 3.1, CLIP and CLAP are introduced. They can go to preliminaries. 2. The motivation behind adding Section 4.2 is not clear. Similar applications have been already included in the Introduction. * Another weakness is lacking clarity about some of the experimental settings: Instead of Section 4.2, it could be better to save some space and describe the dataset a little clearer. For example, 1. how many labels are there, are they coming from a closed vocab? (We later get the answers to these questions by looking at Figs. 3 and 4) 2. Even though the timestamps are not given, do we know the order of the labels, or is it more like a bag of labels at the video level? b. What does CLIP+CLAP without modality labels work in Table 2? Do we take the union of CLIP and CLAP labels and then use this as the ground truth for both audio and video losses? * The improvements in F-scores are mostly stated between HAN and the VALOR approach, but as mentioned at times, there were other stronger baselines to compare (JoMoLD and CMBS in Tables 1 and 4, respectively). It could be better to mention improvements w.r.t. those ones although the gains will look slightly smaller in that case.

Questions

1. Do we know the order of the weak labels? 2. In AV event localization evaluations, did the authors allowed a margin to evaluate the F-scores? 3. The paper describes Type@AV as the Average of A-only, V-only and AV F-scores. It might make it clearer if the name reflects this fact. 4. Event@ AV score is also not very self-explanatory. How does it differ from the AV F-score? 5. Do the authors consider how the number of occurrences of an event in the training dataset correlate with the final performance? Is there any bias do to class imbalance? 6. Paper mentions class-dependent threshold to binarize the CLIP/CLAP outputs. Are they manually or automatically tuned? If automatic, which subsets are used to determine these numbers? 7. Instead of binarizing the labels from CLIP/CLAP, have the authors tried to use the soft labels directly in the loss computation? Soft-labels have sometimes been useful in other semi/self-supervised applications. 8. What is the chosen segment length (duration)? Have the authors tested various granularity levels? 9. The notation is Section 2 can be improved. Even though we can understand their meaning from the context, there are many f's in various forms: $f$, $F$, $\mathrm{F}$ 10. The level of detail in the case of MMIL looks a little long given that the rest of the paper does not mention it that frequently in the rest of the paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Some limitations have been mentioned in the paper. For example, whether the model will be effective in the case of large vocabulary labels. Also how the inherent limitations of CLIP can harm the proposed VALOR approach. 1. It is not clear whether there are any issues due to class imbalance in the dataset. Or if the LLP data has a similar label distribution in train/dev/test splits?

Summary

The paper aims to improve performance of audio-visual video parsing (AVVP) on the LLP dataset via a novel pseudo-labelling technique. This is because LLP contains only coarse video-level labels, and expects the model to match fine-grained, temporally dense labels during testing. This mismatch (due to the intense labelling effort that would be required to have temporally dense labels in the training set) is obviously an issue, and has been approached by prev. works using the Multi-modal Multiple Instance Learning (MMIL) loss for soft-selection and label smoothing, without much success. This paper proposes to use CLIP and CLAP, two pre-trained image/audio encoders, to extract pseudo-labels for each video/audio in an attempt to create temporally dense labels for the training set. This work also uses CLIP and CLAP as feature encoders for better results. This method considerably outperforms previous works, and its design choices are motivated by an ablation study. The method also performs well on audio-visual event localization.

Strengths

The paper is well written and does a good job at introducing the reader to the LLP task, the challenges it entails and why previous works have struggled with it. The figures are great, in particular figure 2, and the tables are all clear and have a clear purpose. The narrative of the paper is consistent and has good momentum, and the initial questions and issues it poses are adequately solved/answered by the end of the paper, with good justifications and conclusions. The method is rather simple but quite elegant, and seems to be the a clear low-hanging fruit to improve the LLP task. The preliminaries are well-explained and the choice not to perform distillation (which, again, would seem like a low-hanging fruit) is well-justified. Related work seems to be adequately, comprehensively, and fairly referenced and discussed. When compared with other works, the results are very convincing. Ablations are also welcome and have clear motivations and conclusions. Table 3 is especially good in my opinion - it answers a lot of questions.

Weaknesses

I don't think the method has any particularly noticeable weaknesses, apart from the limitations that are brought up after the conclusion, which are understandable and are largely outside of the scope of this work. However, I think the paper would benefit heavily from more experiments. CLIP and CLAP are great, and they are compared with HAN (which is effectively a completely different model), but some comparisons with other audio/image encoders would help justify the choice of using CLIP and CLAP specifically. Also it would be great if the authors could experiment with different datasets or even new tasks, if this is possible, as it would give us a wider breadth of results to draw conclusions from. Would be good to have the best numbers in bold for Table 2. Would be good to highlight second and third best to highlight valor and valor+ rather than just valor++. Some extra comments on the poor performing classes in Figure 3 would be nice. I would suggest that the authors focus more on the term "pseudo-labelling" rather than "teacher" since student-teacher modelling is a broad topic with multiple techniques, but clearly here the authors are performing pseudo-labelling, which is a much more specific technqiue.

Questions

When do you plan to make the code available? To the best of the authors' knowledge, are there any other works that use CLIP or CLAP for pseudo-labelling, perhaps in audio-only or image-only literature? If so, it would be very relevant to discuss such approaches in the related work.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Some limitations are mentioned, which is good. However, there should also be a broader impact discussion (can be brief) about the potential misuse of this technology (e.g., aiding automated audio-visual surveillance, what happens if this model is deployed and gives a wrong output, etc.).

Summary

The paper proposes visual-audio label elaboration (VALOR) for weakly supervised audio-visual video parsing. It generates fine-grained temporal labels in audio and visual modalities by harnessing large-scale pretrained contrastive models CLIP and CLAP and providing explicit supervision to guide the learning of AVVP models. The paper shows that utilizing modality-independent pretrained models and generating modality-aware labels are essential for AVVP.

Strengths

1 The paper proposes a simple and effective AVVP framework, VALOR, to harvest modality and temporal labels directly from video-label annotations, with an absolute improvement of +8.0 F-score. 2 The paper is the first to point out that modality independence could be crucial for audio-visual learning in the unaligned and weakly-supervised setup. 3 VALOR achieves new state-of-the-art results with significant improvements on AVVP (+5.4 F-score) with generalization to AVE (+4.4 accuracy) jointly verified.

Weaknesses

1 The paper is really well-written and clear. It would be great if the paper can have a wider use scenerio. The current paper and contribution, experiments are limited to LLP dataset. It would be great if the paper could extend the method to other datasets. Adopting other widely used datasets to validate the proposed method can be a big plus for the paper. 2 The paper achieves big improvements than previous methods. It would be great if the paper could add FLOPs, throughput and number of parameters comparisons with previous methods. Model complexity comparison is a common practice for video related methods.

Questions

Please check [*Weaknesses]

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper tackles the task of audio-visual event parsing, where the goal is to independently recognize the localize the events occurring in the visual and audio modality. The paper argues that modality independent processing can be crucial for this task compared to joint modeling of the two modalities. Consequently, the paper makes use of pre-trained vision and audio models like CLIP and CLAP to harvest temporal labels independently for each modality. The proposed method achieves clear boost in scores across the board. The experiments are clear and sufficient. Rebuttal: I have read the rebuttal. The rebuttal answers all my questions. So I have raised my rating.

Strengths

I think the paper is clearly written (in most places), has a clear motivation and proposes simple intuitive methods to solve the discussed issues. The improvement in performance is significant and is useful for future works in this space.

Weaknesses

The proposed automatic label harvesting using CLIP and CLAP have not been evaluated directly. Since a test set is available with segment labels, is it possible to evaluate the automatic annotation technique using the ground-truth labels on these test sets to directly evaluate how well the automatic annotation procedure works? The automatic training signal extraction technique could have been explored in more detail, as it is the crux of the paper. To start with line 156-157 are quite unclear. What does “contrastive models understanding logits” mean? The intersection operation with y is not described in the text at all. Also, how are the threshold values decided for each class? Is it helpful to keep a slightly lower confidence label (z_t < theta) that is present in “y”? If both CLIP and CLAP models predict the same label with low probability (because something else is dominating the scene), can we keep that? Also some captions that are used to query CLIP or CLAP could be too fine-grained. E.g. the exact musical instrument or vehicle (car or motorcycle) might be hard to identify, so during the automatic training label harvesting, multiple variations of the captions could have been tried, such as using synonyms for example.

Questions

See weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have discussed the limitations.

Summary

This paper proposes methods for weakly-supervised audio-visual event learning. The method relies primarily on pre-trained audio-language (CLIP) and visual-language (CLAP) models to guide the learning process. These pre-trained models serve as teachers and provide pseudo-labels which are then used to compute loss which guides the overall learning process. Experiments are done on an LLP dataset and the proposed method improves the state of the art by a good margin.

Strengths

– The proposed approach is simple – it relies on pre-trained CLIP and CLAP models and achieves good performance in both segment level and event level. – One key claim the authors make is that modality independent learning can be crucial for audio-visual learning in some conditions. While most multimodal learning works focus on using multiple modalities to improve performance on a given task, the significance of learning independently from each modality is a good point to highlight. – The paper is mostly clear and easy to follow except for some mathematical descriptions which I believe can be improved at a few places.

Weaknesses

— The proposed approach relies heavily on the pre-trained models audio-language and image-language. Considering that it might be important to understand the impact of these models themselves. I believe that the paper is missing some crucial analysis in those areas. Some which might help pain a better picture. --- What kind of information (pseudo-labels) these pre-trained models are providing for the audio and video modalities ? Can we get some insight into the distribution of \hat{y}^m_t and how that is related (if any) to the actual video level ? Do the audio and visual model provide complimentary or similar information ? This can be analyzed before any training. --- It’s not fully clear (Table 3) what happens if the ground truth event labels are not used at all and just the pre-trained models are used to train. --- What is the impact of using different pre-trained models -- different CLIP/CLAP type models? — I think experiments on some other weakly supervised data might be helpful. — While the “non alignment” of the modalities in the videos does sound like a valid problem to address – it is not clear how much of that is present in the LLP dataset. How often does the video and the audio do align with the event label ? There is no analysis of to what extent that is present in the current dataset and to what extent it gets addressed. Some other comments/questions -- What is KD in in Table 3 ? Is it the one where ground truth labels are not used ? Just knowledge distillation from teachers ? -- Why is label smoothing applied to modality training targets ( line 106 ) but it’s not applied anywhere else ? -- For event localization, I am not sure accuracy is the right metric. Some metric as well as visualization which factors in the beginning and of the event will be more informative. -- The notations (line 145 - 147) q^P and q^m are confusing. Simplifying and clearly explaining what precisely they represent will improve clarity. -- How are the threshold parameters \theta^P obtained for each class ? -- In the current VALOR formulation only CLIP/CLAP outputs for only classes marked to be present are used (the logical and with y). What if you --- Updated review after rebuttal----

Questions

Please follow-up on the questions/concerns in the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors describe limitations of their work in terms of whether the approach will generalize to larger settings. Societal impact and other such limitations are not discussed. Given the nature and scope of this paper that might be okay.

Summary

In this work, the authors propose a unified weakly supervised audio-visual scene understanding framework for audio-visual video parsing and audio-visual event localization. Different from previous works, the proposed Visual-Audio Label Elaboration (VALOR) method is simple and effective. It leverages large-scale audio-text and visual-text contrastively pre-trained models as the modality teachers to predict individual labels for audio and visual modalities to tackle the modality and temporal uncertainty issue and boost event parsing performance. Extensive experiments and ablation studies on the LLP and AVE datasets can validate the effectiveness of the proposed approach.

Strengths

+ The proposed method is new and technically sound. To alleviate drawbacks in past approaches, the proposed VALOR leverages large-scale audio-text and visual-text contrastively pre-trained models as the modality teachers to predict individual labels for audio and visual modalities for weakly-supervised audio-visual event parsing tasks. + Extensive experiments and ablation studies on two datasets are provided, and the strong results can demonstrate the effectiveness of the proposed method. + The core code is provided in the supplementary material, and the authors promised to release the source code. + The paper is easy to follow.

Weaknesses

+ The process for selecting class-dependent thresholds in VALOR is not clearly outlined in the paper. These thresholds are crucial hyperparameters for generating audio and visual labels for training, and it appears that different classes may require different thresholds. Further explanation is needed on how these thresholds are chosen and how changes in these thresholds could potentially impact the performance of event parsing. + The authors did not provide a clear explanation as to why directly using Knowledge Distillation (KD) with audio and visual semantic class embeddings results in even worse parsing performance (especially for audio event parsing) on the LLP dataset, as shown in Table 3. While the proposed VALOR method does improve performance, a more detailed analysis on the KD models would be beneficial for further understanding. + The CLIP model uses a text prompt that is created by adding a "A photo of" prefix to the event's natural language form. However, the visual data involved in audio-visual event parsing are videos, not single images. This raises the question of whether there is a better prompt that could be used to improve model performance by taking into account the video nature of the data.

Questions

Please address questions in Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discussed method limitations in the paper.

Response to authors

Thank you for answering my questions. You have clarified all my doubts. Very happy to hear that the code will be made available. Happy to raise my score slightly based on the convincing responses to all reviewers. I am not completely familiarized with this specific field in depth, which is why my confidence score is not 5. But overall, I think the paper is quite good.

Response to Authors

The rebuttal can address my questions. I will keep my positive rating. Thanks!

Thank you for the rebuttal!

Thank you for these results! I do not have further concern of the paper. It seems the paper and responses are good to all reviewers. Thus, I increase the score a little bit. Thanks for the contribution!