SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation

Summary

This paper proposes a novel framework that can effectively exploit video-level visual-linguistic alignment to improve the performance of RVOS, filling a gap in existing methods. The extensive experiments on three datasets demonstrate the effectiveness of SOC.

Strengths

The proposed method outperforms existing state-of-the-art methods on all datasets. The paper provides a clear and concise description of the SOC framework and its experimental results.

Weaknesses

None

Questions

How to handle objects that have been occluded for a short time (> training frames number)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations and potential negative societal impact of their work.

Summary

This work studies the referring video object segmentation problem and proposes the semantic-assisted object cluster approach. It aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment.

Strengths

1) It performs object aggregation and promotes visual-linguistic alignment at the video level, which considers the temporal variations and alignment across different modalities of frames. 2) It introduces the visual-linguistic contrastive learning to provide semantic supervision. 3) Empirical studies on several benchmarks demonstrate the advantage of the proposed method.

Weaknesses

1) The novelty seems a bit limited. The multi-modal fusion model just uses the common multi-head cross-attention to model the relations of language and visual information. 2) Is there other clustering methods for object grouping? Some discussions are requried. 3) Ablations studies on the hyper-parameters $\lambda$ are missing.

Questions

What about considering the object-level temporal relations across frames compared to the video-level ones?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See the above.

Summary

This research paper focuses on referring video object segmentation. It combines temporal frame features and language features using a cross attention design. Additionally, it extracts object features from each frame and applies contrastive learning with language representation. The paper presents various designs to enhance the referring performance. Results from experiments on multiple benchmarks demonstrate that this method achieves state-of-the-art performance on this task.

Strengths

1. The RVOS framework demonstrates remarkable results on various benchmarks. 2. Through experimentation, the proposed fusion of features significantly enhances the baseline. 3. The concept of utilizing temporal and language features is effectively communicated.

Weaknesses

1. The current method is quite complex, and it is not clear what the main contributions of the RVOS in figure are. It appears that most of the modules shown in Figure 1 are not mentioned in the manuscript. This makes it difficult to understand which proposed components are essential for the given task. 2. The Multi-Modal Fusion (MMF) and Semantic Integration Module (SIM) are presented in Section 3.2 and 3.3. However, there are no experimental tests conducted to validate these modules. The authors also claim that SIM contributes to the task in the introduction. 3. The results of visualizing the impact of the proposed modules are missing. It is important to provide results that demonstrate how each module enhances the referring performance.

Questions

1. Figure 6 shows a video with only one person. Why does the compared method fail in these scenes? Are there any complex scenes that demonstrate the advantages of the present method? 2. The figures lack clarity and are difficult to understand. It is important to highlight the main contributions of the present modules. 3. The Semantic Integration Module appears to be resource-intensive, and it would be helpful to present the model parameters compared to previous methods. 4. The use of off-the-shelf designs in all the modules limits the novelty of the approach.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are only briefly mentioned in the current manuscript.

Summary

This article introduces the Semantic-assisted Object Cluster model, which is used to solve the problem of visual language alignment between videos and texts. The model uses a semantic integration module and visual language contrastive learning to improve the understanding of video content and has achieved significant improvements in different scenes and text descriptions.

Strengths

1．The Semantic-assisted Object Cluster (SOC) framework has been proposed, which effectively aligns video and text, realizes object aggregation and visual-language alignment, and improves the performance of the RefVOS model. 2．The Semantic Integration Module (SIM) has been designed to efficiently aggregate intra-frame and inter-frame information, understand video content from a global perspective, and help understand temporal changes and alignment of different modalities and granularities. 3．Visual-linguistic contrastive learning has been introduced to provide semantic supervision, guide the establishment of video-level multimodal joint space, and improve the performance of the RefVOS model. 4、It can effectively handle text descriptions that express temporal changes and significantly improve general scene processing. 4.On popular RVOS benchmarks such as Ref-YouTube-VOS, Ref-DAVIS, A2D-Sentences, and JHMDB-Sentences, the performance is significantly better than existing methods, and inference speed is faster.

Weaknesses

1．The effectiveness of Visual-Linguistic Contrastive Learning module heavily depends on the quality of the text descriptions used for supervision. If the text descriptions are inaccurate or incomplete, the performance of this module could be limited. 2．The efficiency of SIM is not clearly explained, and it is unclear how well it would scale to larger and more complex datasets. 3. The caption of many figures (such as Fig.3) are too simple, and more details should be provided. 4.Ablation experiments were conducted only within each module, without verifying the effectiveness of each module on the whole network framework.

Questions

1．For the Video-Level Object Cluster module, the design of the object clustering module does not fully consider the similarities and differences between instances, which may affect the clustering effect. The paper should provide some considerations on this issue. 2．For the Visual-Linguistic Contrastive Learning module, the issue of potentially non-target responses in video-level embeddings due to the simple use of text priors should not be ignored. 3．The model uses Video-Swin-B and Video-Swin-T as backbones, so it is unfair to compare it with other models that use resnet as backbone.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

1．The model may rely on the accuracy of semantic information, as it uses the Semantic Integration Module (SIM) to aggregate information in the video. 2．The model may not be able to handle some complex video scenes because it only performs inference based on known semantic information.

Summary

This paper mainly studies referring video object segmentation (RVOS) which is a fundermental problem in vision-and-language applications. Compared with previous works which pay little attention to temporally modeling relationships between different frames in the entire video clip, the authors proposed a framework called SOC for RVOS which conducts temporal modeling and cross-modal alignment in a unified framework. Specifically, SOC contains several modules including textual and vision encoding and multi-modal fusion, semantic integration module and vision-language contrastive learning. The experimental results show SOC outperforms existing SOTA method, e.g. ReferFormer published in CVPR 2022.

Strengths

(1) Overall, the motivation for poposing the temporal modeling intra-frame and inter-frame information for RVOS is quite clear and straightforward. Besides, the paper is well-writen and the idear as well as each module is easy-to-follow. (2) The most valuable contribution is the video-level object cluster module in the semantic integration module (SIM). Instead of using traditional clustering method such as k-means, the authors proposed an object grouping decoder architecure to perform the clustering process. Specifically, N_v video-level object queries are introduced to learn the overall object centroids in the entire video. After repeating such operation T times, the semantic representation enhancement is achieved by enriching the video-level object information. Figure 1 in the supplementary material demonstrates that most of these learnable queries focus on the region-of-interests (ROIs) of he referred objects. (3) The experimental results are promising. The proposed SOC framework outperforms other existing SOTA works under different settings. The provided video demo shows the referred objects have been segmented perfectly.

Weaknesses

(1) According to my understanding of this work, the main contributions is the video-level object cluster module. Hence, the albation studies or comparison between different methods should focus on such module. Specifically, is there any other clustering-based method that could obtain video-level object embeddings? Why is the proposed object grouping decoder-style model is the best solution among these optional methods? Besides such high level comparisons, some details (or parameters) of abaltion studies should be further discussed. For example, the number of object queries and number of temporal frames are tuned independently in Table 5 and 6. However, jointly tuning these two parameters should be investigated. Another example is that the final representation is the element-wis sum of frame-level and video-level semantic embeddings. Are there any other better learning-based fusion implementations? (2) Generally, the proposed framework follows a two-stream fashion which includes a textual encoder and a vision encoder. Note that the vision encoder could be a temporal convolutional network (3D CNN) or video transformer such as in this work. No matter which implementation, it undergoes the temporal processing. The question is: could the video-level object clustering module be embedded inside such temporal encoding procedure? I think the whole inferece speed could be further decreased if it is possible. (3) I think the main contribution and the motivation is clear. Some trivial parts such as the visual-linguistic contrastive learning could be removed in order to make the key points stand out, which will not reduce the whole work's contribution or value.

Questions

(1) Why the SIM module is not designed inside the video encoder? Please refer to the second part in the above weaknesses. (2) It is reported that the SOC runs at 32.3 FPS on a single 3090 GPU. I think the overhead of each module should be further decomposed, e.g. the encoding backbone, multi-modal fusion module, and the SIM. (3) As illustrated in Table 4, L2V outperforms V2L significantly. Deeper insights are expected to be uncovered. Although using both of them achieves the best performance, some missing details should be claimed. For example, what is each branch's weight? (4) Could the proposed SOC be extended to solve multiple referred objects given a textual query?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have mentioned that malicious use of such model could lead to negative social impacts such as unauthorized video surveillance. However, I believe the positive impacts of these vision-and-language applications would improve the embodied AI research and generally outweigh the risks.

Summary

In this paper, the author presents a framework called SOC, and a visual-linguistic contrastive loss for RVOS to unify temporal modeling and cross-modal alignment. The performance is good according to the leaderboard (https://paperswithcode.com/sota/referring-video-object-segmentation-on-refer) The extra module seems to have many parameters, and somehow incremental.

Strengths

In this paper, the author presents a framework called SOC, and a visual-linguistic contrastive loss for RVOS to unify temporal modeling and cross-modal alignment. The performance is good.

Weaknesses

1. Parameters The extra module seems to have many parameters, and somehow incremental. - Unfair Comparison. It is unfair to compare with other methods. - Design of MMF is not new. Many multi-modality fusion has used similar structure such as [a]. The different part is that authors adopt transformer structure. [a] Comprehensive Linguistic-Visual Composition Network for Image Retrieval 2. Motivation in Figure 1 is not clear to me. I think it can be easily solved by adopting the video-based backbone such as TSM or I3DNet, 3. The visual-linguistic contrastive loss I do not see significant difference from the basic contrastive loss in Clip or other works. The input is changing from image to video features. Could you further explain it?

Questions

Please see Weakness. I am not working in this field. If the author could address my concerns, I will upgrade my rate.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please see Weakness. I am not working in this field. If the author could address my concerns, I will upgrade my rate.

More explaination on Semantic Integration Module

Thank you. 1. Could you provide more details? - How about the text backbone? Are they the same for the three works? - Why the proposed SIM is good for fine-grained details? How do you define the fine-grained and coarse-grained for VOS? 2. Yes. Could you try TSM? I think TSM can do good efficiency in term of Figure 1. Why TSM is coarse-grained? 3. Could you explain more on "the contrast between different object queries and the same text descriptions" ? I think it is still basic contrastive learning idea.