Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective

Summary

The goal of this paper is to create a model that can solve the task of audio-visual video parsing in a weakly-supervised way. For this, the authors propose to use the language modality which proves to provide benefits and improve the performance of the proposed model over the previous baseline. The language modality is used to provide additional supervision for the system. The paper provides an extensive comparison with previous baselines. Also, the authors provide extensive ablations showing the importance of each component and some qualitative examples which provide another perspective of how the model behaves.

Strengths

- The proposed system performs better than the previous baselines. - Multiple baselines were used to compare against the proposed method. - Extensive ablation showing the importance of each component - Presence of qualitative results.

Weaknesses

- Line 2: typo “andvisual”->”and visual” - Figure 1: last line of the description: the cross product looks different than in the figure. Make them consistent. - Line 127: Do the authors also train the Resnnet and VGGish? Or are they pre-trained and then frozen? - Equation 1. I have some questions here. What is the shape of w^a_t and w^v_t? Are these scalars or vectors? Moreover, p_t^a/p_t^v is a vector. What will p^a and p^v? Also vectors? If w is a scalar then p^a/p^v will be vectors. If w is a vector, then I would assume there is a dot product between w and p which may be wrong depending on the dimension of w. Is there a dot product in the p^a and p^v? Moreover, if there is a dot product then p^a and p^v will be a single scalar, and not a tensor. However, I think the authors want to obtain p^a which will have the same shape as p^a_t, but it will aggregate the scores from the whole video. I also observed that p^{av} uses a Hadamard product. The first thing would be to clarify the shape of the weights and that would also clarify a lot of things in the equations. Right now, for me, it is very hard to understand what is going on there. - Line 154: How are the features per segment obtained? CLIP gives features per frame. Do the authors average the per-frame features from CLIP to obtain per-segment features? - Line 179: How is the overlap region chosen? What are the thresholds? It is said before that “...some segments in orange are labeled as without car, their similarity is still high…”. How high and in comparison with what? Thus, provide some details on how it is judged if the similarity is still too high or too low. - Line 182: is this t-th segment one of the unreliable segments, or it is just the t-th segment of the video? Make this explicit, as right now it seems that it is just the t-th segment of a video, but I understood that the re-weighting is only applied to the unreliable segments, as mentioned in Line 170. - Line 184: What are the values of alpha and beta? - Line 191-194. I am a bit confused by these lines. First, I will refer to the figure. In the upper part of Figure 1 (d), it is seen that the method is learning in a supervised way to label segments correctly, based on the labels that are provided by the denoising and the re-weighting mechanism. However, for the lower part of Figure 1 (d), only the VGGish features are used for the MMIL. Is this correct? Shouldn’t both the visual and the audio features be used by MMIL, as in equation (1)? As far as I understand the only thing that changes is the loss of the video modality, which now can provide segment-level supervision. But everything else is staying the same. - Line 193: Here it is said that CLIP is used to extract the visual features during training. Does this mean that the authors train HAN but they replace the Resnet with CLIP? If yes, is CLIP trained, or it is frozen? - Line 230: typo “leave the label” -> “leave out the label”

Questions

Most of the above questions can be easily fixed and clarified. For now, I think this paper looks good, so my score is weak accept. However, I would encourage the authors to clarify the mathematical equations, as this will make the proposed method easier to be understood.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss the limitations of their work and how the work can be extended.

Summary

This paper tackles the audio-visual parsing task by dividing the video-level label into segment-level labels with the help of language based on CLIP. Training with the fine-grained segment-level labels, instead of the coarse video-level labels, makes the model perform better. In the process, the processing for noisy and unreliable labels bring obvious improvement on the task.

Strengths

The presentation is clear and it’s easy to follow. The manner to introduce CLIP to generate segment-level labels to replace video-level labels for training is smart, simple and effective.

Weaknesses

I think the main contribution in this paper is to introduce large models such as CLIP to help the audio-visual video parsing task, and the exploration could be more thorough. The main task is the “audio-visual” video parsing task, then how would the relationship between the two modalities influence the task? In default case, would the audio modality perform better than visual modality? In Table 1,2 and so on, the results show that “A” always perform better than “V” when V has not been helped by CLIP; and Even when introducing CLIP, the “HAN +CLIP” also shows that “A” better than “V”. Does this mean that “A” always perform better than “V” by default? And also compared the results of “A-V” and others with “A”, the improvements are small. If so, enhancing the audio modality may be more effective for this task, while the audio modality in this work is weaken in the process. This is because that there is no proper audio-based large models like CLIP?

Questions

How to generate the prompts? how to obtain A and B? Randomly choose a negative class from 25 categories together with the original class to compose A and B respectively? And the paper says that “the language prompts of the video [could] be xx”. Are there any other formats besides the “other/none/notclass” format in experiment?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

One main motivation of this paper it to tackle the problem of unmatching between the audio and visual modality. Then what’s the actual situation in the evaluated data? And it would be better to provide a statistics of this unmatching in this data to show that the data is proper to evaluate this point, together with quantitative comparison of the performance on the data with or without this unmatching.

Summary

This paper proposes LSLD, which leverages CLIP model to denoise unreliable segment-level video labels for audio-visual video parsing.

Strengths

$+$ The proposed method can achieve state-of-the-art results on LLP datasets in several metrics. $+$ The improvement of video and audio-visual events is significant.

Weaknesses

$-$ The proposed method can only denoise labels for visual domains. It will limit the overall accuracy of audio-visual video parsing. Although the authors mentioned CLAP model is not applicable to the proposed pipeline in L.59-L.60, the audio waveform can be simply blocked out to get segment-level audio information as well. $-$ Introducing the language model might not be necessary. Since LLP dataset is collected from AudioSet, pre-trained models from AudioSet can find the corresponding labels for LLP in both audio and visual labels. These models may contribute to stronger label-denoising. $-$ The effectiveness of LSLD is not clear. * In L.231, it mentioned that LSLD could also benefit audio accuracy. However, LSLD causes an accuracy drop from 62.4 to 62.3 (Segment-Leve) and from 53.9 to 53.0 (Evet-Level). Since LSLD is based on HAN, which leverages audio-visual feature aggregation, LSLD would be expected to improve audio results as well. * Does LSLD also benefit other approaches (e.g., MGN)? The proposed approach should also benefit other baselines including CLIP encoder or Resnet encoder. * Is LSLD complementary to other denoising approaches (i.e., MA and JoMoLD)? These approaches also denoise audio labels. They should be complementary to LSLD.

Questions

$-$ Do the baselines with labels denoising (e.g., MA) also leverage CLIP to do denoise? For example, for MA+CLIP baseline, the refined labels are from CLIP or ResNet?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors mentioned this in the paper. The proposed method can only work on visual segment-level labels. However, this claim might not be entirely correct. The possible way to do this is mentioned at weakness.

Summary

The authors have observed that weakly-supervised labels in audio-visual tasks are noisy at the segment-level label. To solve the problem, the authors introduce language as an additional source of information to assign soft labels to each of the 1-second video segments. More specifically, a similarity score is computed between each segment and possible prompts, and segment-level labels are softly assigned by the proposed dynamic re-weighting method. The experimental results show the performance boost with the re-weighing.

Strengths

1. The paper is mostly well-written, with details and figures. 2. The idea of dynamic re-weighting for segment-level label noise is interesting. 3. The authors show various experimental results, including the ablation study of different prompts.

Weaknesses

1. Some manuscripts need to be clarified. * Section 3.4, especially Equation (4), needs to be clarified. According to Equation (4), $\tilde{\mathbf{y}}_t^v[c]$ is already assigned with 1 or 0. Where does the value come from? * Moreover, there is a discrepancy between the descriptions in Lines 171-179 and the equation. For example, how is *relatively high similarity* applied to the equation? * Figure 1 (c): Does```w/o Cello``` stand for a a prompt "w/o Cello"?   * In Table 1's caption, it is written, "The last 3 lines are all label denoising methods ..." Do the last 3 lines indicate ```MA+Clip```, ```JoMoLD+Clip```, and ```LSLD (ours)```? 2. Although Section 4 shows multiple experimental results, all of the results are on a single dataset (LLP dataset). Ideally, the proposed approach should be demonstrated on various datasets, especially those with no segment-level labels or longer video clips.

Questions

1. Does ```other``` include the case where the segment is similar to some other labels than class A or class B?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discussed the limitations of the work (e.g. segment-level labels for audio) in Section 5.

Welcome to discuss!

Dear reviewer, We are following up on our paper rebuttal. In summary, we elaborated on deriving the value of $\mathbf{\widetilde{y}}_t^v[c]$ and modifying Eq. 4 to incorporate *relatively high similarity*. We explained the meanings of 'w/o Cello' and 'other', and emphasized the reason for experimenting solely on the LLP dataset. Your response and feedback are highly valued. Thank you for your time and consideration.