Weakly-Supervised Audio-Visual Segmentation

Summary

This paper presents a new framework for weakly supervised audio-visual segmentation which does not need pixel level annotations. This is achieved via the proposed multi-scale multiple-instance contrastive learning approach which can capture audio-visual alignment in multiple scales. Comparison with existing methods show that the proposed methodology leads to state-of-the-art results.

Strengths

The paper proposes an interesting approach ti audio-visual segmentation which does not rely on pixel-level annotations. This is an important contribution since obtaining such annotations is time consuming. The paper is easy to follow. Convincing results are presented and a detailed ablation study is presented.

Weaknesses

The main weakness of the paper is that several important details are not described. - In table 1, results for multiple sources are presented without providing details, e.g., how many sources are used, how were these examples obtained? - It is not clear how the multi-scale features are generated. - How are the negative examples selected in multi-scale multiple instance contrastive learning? Would be very helpful if the authors elaborate on this. - Equal weights are used for the two loss terms in Eq. 8. Have the authors considered using different weights? This might improve the performance. - It is not clear why the face and body are predicted as the sound producing regions in case a person speaks. Ideally, just the face or just the mouth should be identified as the sound source. - The audio which corresponds to the images in Fig.2 is not presented. Most of the examples are included in the supplementary material so adding such a comment in the caption would be helpful. - Finally, it would be helpful to add additional failure cases. Only one example is included in the supplementary material. Another weakness, is that only one dataset is used for evaluation.

Questions

Please see the questions above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

As mentioned above, it would be desirable if the authors include additional failure cases in the supplementary material. At the moment there is only one.

Summary

This work introduces a new setting for audio-visual segmentation, Weakly-Supervised Audio-Visual Segmentation (WS-AVS). The authors address the challenge of the costly and not always available pixel-level masks by employing weakly-supervised audio-visual segmentation. This framework uses multi-scale multiple-instance contrastive learning for capturing multi-scale audio-visual alignment. Tested on the AVS Bench, WS-AVS demonstrates superior performance in both single-source and multi-source scenarios when compared to previous methods.

Strengths

1. To eliminate the requirement of pixel-level annotations, this work propose to solve the audio-visual segmentation problem in a weakly-supervised way. Correspondingly, it uses adopts Audio-Visual Fusion with Multi-scale Multiple-Instance Contrastive Learning and Pseudo Mask Refinement by Contrastive Class-agnostic Maps to solve the challenges in the weakly supervised setting. 2. The authors provide comprehensive evaluation under various setting to illustrate the performance of the proposed method.

Weaknesses

1. Clarifying Novelty The paper's novelty looks somewhat unclear. This reviewer recommends clearly defining the unique technical contributions proposed by this work, especially those set apart from the developments in weakly-supervised segmentation. For instance, it would be advantageous to underscore the importance of Multi-scale Multiple-Instance Contrastive Learning in the given task and its distinctiveness from similar concepts previously employed in weakly-supervised segmentation techniques. 2. Further analysis The authors offer sufficient ablation studies to ascertain the performance improvement attributed to AVF and PMR. However, the reviewer anticipates a more in-depth analysis of the outcomes. Specifically, what distinguishes the audio-visual relationship as learned via supervised methods from those learned in weakly-supervised ways? Considering the final classification outcome, it's intuitive to hypothesize that weakly-supervised methods might result in a sub-optimal or noisy audio-visual mapping. But, without human bias, these weakly-supervised maps might demonstrate superior generalization or aid in identifying human annotation-induced preconceptions. The reviewer encourages exploring these insights rather than viewing weakly-supervised audio-visual segmentation merely as a workaround for data limitations.

Questions

The reviewer believes the task of weakly-supervised audio-visual segmentation is worth exploring, and hence currently vote for borderline accept. However, the authors should adequately discuss the key novelty of this paper for the final acceptance.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes a new contrastive learning approach with pseudo-mask generation/refinement process for weakly supervised video segmentation with audio guidance. The authors show that their method outperforms several other state-of-the-art methods for the visual segmentation task when training and testing on the AVSBench dataset.

Strengths

- The paper shows strong results compared to other methods in the literature. - The paper tries to tackle a very important problem in the vicinity of audio-visual perception which is how to find the pixel regions that correspond to sounding sources when limited annotated data are available. - The paper is well written and the ideas are clearly presented.

Weaknesses

Unfortunately the paper also contains several weaknesses that narrow down the scope and the potential impact of the paper that could be addressed using experiments and rewriting some parts of the manuscript when needed. - The authors only include experiments with a very small scale in terms of multiple aspects (video frames, classes of sounds only 23 objects, number of videos < 5k) to compare their method with other established methods. Although having smaller scale experiments help with finetuning of the hyperparameters and early experimentation, it becomes difficult to extrapolate the findings of this study to large scale solutions and real-world data with only showing results on the chosen dataset. Some dataset that come on top of my head and could be used by the authors to enrich the results and show the true potential of their method would be some more limited datasets like Flickr-SoundNet [A], a medium-sized mostly single-source dataset like VGG Sound Source [B] or a real-world video dataset like AudioSet [C] or YFCC100M [D]. - Building upon my previous argument, using more realistic and large scale datasets would not only put away potential skepticism revolving around the validity of training neural networks with only 5k examples and limited video resolutions (e.g. capturing motion features when the average video has 2 frames is almost impossible for the dataset used in this paper) but would also show whether the proposed approach is robust when trained with videos that contain no on-screen objects which is very difficult to solve for related problems like on-screen sound separation [E]. I don’t want to extend the problem to cases where more than one sounds appear on-screen / off-screen and thus some audio separation front-end might be needed before obtaining the audio features but I would still expect some more convincing large-scale experiments. - In the current version of the paper, the authors only consider evaluation datasets where the sounding object appears on-screen at some point in the video but they completely omit to report the performance for off-screen-only videos. To that end, the authors should also evaluate the performance of their model when there is some visual activity on-screen but the sounding object/action cannot be seen in camera and test how robust their model is. According to my experience, there will be a trade-off in the sensitivity of how accurate the model will predict the masks for on-screen objects which correspond to the on-screen audio and how well the model will predict zeros for on-screen objects and actions that do not make any sound (e.g. a human talking in the background while a non-talking human is displayed on-screen). Overall, I am willing to increase my score if most of the most important above concerns are addressed by the authors since I truly believe that the paper has a great potential to become a good benchmark method for weakly supervised audio-guided video segmentation. [A] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018. [B] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020. [C] Gemmeke, J.F., Ellis, D.P., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M. and Ritter, M., 2017, March. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 776-780). IEEE. [D] Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D. and Li, L.J., 2016. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2), pp.64-73. [E] Tzinis E, Wisdom S, Jansen A, Hershey S, Remez T, Ellis D, Hershey JR. Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds. In International Conference on Learning Representations 2021.

Questions

Have you considered an analysis of the proposed approach for the choice of the model for ensuring robustness towards the “reliable pseudo mask extraction with multi-scale visual features”?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors discuss some limitations of their work and I think they should include the issues that I raised in the Section above and that they will not be able to address through experiments and/or rebuttal.

Summary

The authors propose a weakly-supervised audio-visual segmentation framework called WS-AVS that predicts sounding source masks from audio and images without pixel-level ground truth masks. It leverages multi-scale contrastive learning in audio-visual fusion to capture multi-scale alignment, addressing modality uncertainty in weakly supervised segmentation. The refined pseudo masks guide training, enabling generating accurate segmentation masks. Experiments show it outperforms weakly-supervised baselines.

Strengths

The manuscript presents a solution to the challenging problem of performing segmentation tasks without pixel-level annotations by effectively integrating multi-scale contrastive learning and pseudo mask refinement. It is well written and easy to read through.

Weaknesses

1. Although these methods have been proven effective independently, their combination in this manuscript does not elevate them to a novel plane but rather presents a blend of established tactics, the novelty aspÅect therefore seems insufficient. 2. While the max-pooled cosine similarity function works well in [9], is there any other similarity function that can be compared with, as AVS has multi-source scenarios? 3. As the WS-AVS framework relies heavily, to some extent, on the generation of pseudo-labels, so is there any other ablations on pseudo-label generation? 4. Noticed that there is no schematic for multi-source in the manuscript, wondering what the pseudo-label for multi-source looks like. 5. In Line 188-190, the authors wrote: the recent audio-visual segmentation baseline [1] does not give any cross-modal constraint on the audio and visual representations in the audio-visual fusion, which causes a significant performance drop when removing the ground-truth masks during training. I notice that [1] proposed an audio-visual matching loss that is used in the multiple sound source case as [1] claimed this loss does not bring further improvement in single-source case. I guess the pixel-level ground truth provides enough supervision in single-source AVS. However, in the studied weakly-supervised AVS setting, will this loss help in the audio-visual fusion? 5. Some minor issues: Typo. Line 54, “WS-VAS, that” should be “WS-AVS, which” Typo. Line 176, “V^s” should be “V^s_i” Typo. Line 194 “WSSL” should be “WSSS” The result in F-score of Multiple Source in Table 1 should be bold.

Questions

As above

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

N/A

Additional response to the reviewer

Dear Reviewer yiGS, Thank you for your detailed review and the valuable feedback. We have carefully addressed each of your concerns and provided clarifications in our previous response. We would like to kindly request your response to the provided explanations and revisions. We appreciate your thorough evaluation of our work, and your feedback will greatly contribute to the improvement of our manuscript. Thank you for your continued engagement and support.

Additional response to the reviewer

Dear Reviewer CQaV, Thank you for your thorough review and valuable feedback. We have carefully addressed each of your concerns and provided clarifications in our previous response. We would like to kindly request your response to the provided explanations and revisions. We appreciate your engagement and the time you have dedicated to reviewing our work. Your feedback will greatly contribute to the improvement of our manuscript. Thank you for your continued support.