SyncVIS: Synchronized Video Instance Segmentation

Summary

The paper argues that existing methods for video instance segmentation use asynchronous designs, leading to difficulties in handling complex video scenarios. To address this problem, the paper proposes a SyncVIS method for synchronized modeling, achieving state-of-the-art results on several benchmarks.

Strengths

1. The paper is generally clear to understand. 2. The paper achieves good performance by synchronized modeling. 3. Visualizations are provided to illustrate the effectiveness of the method.

Weaknesses

1. The novelty of synchronized embedding optimization is limited. 2. The paper did not compare with DVIS++. For example, using ResNet-50 and offline mode, DVIS++ outperforms SyncVIS by 2.5% on YouTube-VIS 2019. DVIS++: Improved Decoupled Framework for Universal Video Segmentation, arXiv, 2023. 3. More implementation details like training steps should be reported. 4. Compute resources, such as the type of GPU, are not reported in the paper, yet the authors answered YES to question 8 in the checklist.

Questions

In addition to the Weaknesses, 1. How does the method ensure that the video queries learn motion information? 2. Ablation study on different values of $N_{k}$ (line 173) should be conducted. 3. How is the efficiency of the method?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes.

Summary

This paper focuses on improving synchronization between frame and video queries in video instance segmentation for better long-range video analysis. The authors propose encoding frame and video queries separately, then using confidence scores to select Nk queries. These queries are updated through mutual information exchange with momentum updates. To train each query, they introduce a synchronized optimization method using a divide-and-conquer approach. They also identify that video-level bipartite matching complexity increases with the number of frames, and address this by suggesting sub-clip level matching. Their technique, when applied to CTVIS and VITA, demonstrates enhanced performance in both online and offline settings compared to existing methods.

Strengths

- The paper is comprehensively written, with clear explanations of its contributions and a detailed analysis of prior research. The proposed methods are innovative and seem well-founded. - Extensive experiments validate the effectiveness of the proposed methods, significantly bolstering the paper's credibility.

Weaknesses

- The analysis primarily focuses on early work (Mask2Former-VIS), while the baselines used are CTIVS and VITA. Despite this, there is a lack of thorough analysis on these baselines. - The GT assignment method of Synchronized Embedding Optimization is not compared with existing methods such as TCOVIS. Additionally, although the authors aim for long-range video modeling, their performance on the long video dataset YouTube-VIS 2022 is lower than that of TCOVIS. - In the Checklist under Experiments Compute Resources, the authors answered "yes" but did not specify the equipment used. Reference Li, Junlong, et al. "Tcovis: Temporally consistent online video instance segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Questions

- In Section 3.1, the author highlights the performance drop in Mask2Former-VIS when increasing the number of frames. CTVIS, on the other hand, shows improved performance with more frames and seems to handle temporal modeling well through its memory bank. If CTVIS were used as the baseline, why does the proposed method show improved performance and what specific aspects are enhanced? - TCOVIS aggregates the cost for each frame and matches GT with predictions at the video level globally. The proposed method matches at the sub-clip level, which is a contrasting approach. What are the respective strengths and weaknesses of each method? Additionally, why does the proposed method perform worse on long videos like YouTubeVIS-2022 compared to TCOVIS? - In L219, it is mentioned that inference with the Swin-L backbone was done at 480p, but the code indicates 448p. Which is correct? Furthermore, in L218, the learning rate is stated as 1e-4, but the code shows 5e-5. Also, the iteration and learning rate decay settings for YouTubeVIS-2019 and 2021 datasets seem inconsistent. How were the optimization settings determined?

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors address the limitations of the proposed method in the appendix.

Summary

This paper concentrates on the video instance segmentation task. To address the problem of motion information loss when existing methods use frame queries, and the optimization issue of bipartite matching across multiple frames, the authors propose a synchronized video-frame modeling paradigm. This paradigm allows for interaction between frame queries and video queries and introduces a synchronized embedding optimization strategy in a video-level buffer, making multi-frame tracking optimization feasible. The effectiveness of the proposed method is verified on four VIS benchmarks, and detailed ablation studies are conducted.

Strengths

1. The proposed SyncVIS shows impressive performance across multiple VIS benchmarks. 2. The method introduced can be adapted to enhance multiple existing VIS frameworks, demonstrating a high level of versatility.

Weaknesses

1. Line 45 mentions that 'image encoding stage (rather than video encoding)' could lead to motion information loss; however, the proposed method still employs image encoding. Frame queries take video query for feature aggregation, the aggregation is merely an accumulation of multi-frame information, similar to the video query in SeqFormer, which can help achieve robustness, but how does it model object motion? This statement might be inappropriate.

Questions

1. The proposed Synchronized Embedding Optimization for bipartite matching is it aimed at the bipartite matching between prediction and Ground Truth, or object query bipartite matchings during the tracking process (Like MinVIS)? This aspect is quite confusing for me. From lines L186-L193, it seems that the author aims to enhance bipartite matching between prediction and GTs. Yet, lines L194-L199 appear to discuss associating the tracking results of two frames. 2. The conclusion in Sec4.4 seems counterintuitive. Longer Sub-clips could provide the model with more temporal information to help it model complex scenes and trajectories. This should be especially evident on OVIS, which includes many objects and complex inter-object motions, including occlusions and disappearances and reappearances. According to the paper's motivation, longer Sub-clips should perform better on OVIS, but the experimental conclusion states the optimal length on OVIS to be T_s = 2, which corresponds to a pair of frames, providing very limited temporal information. Doesn't this contradict the motivation or the conclusion?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

This paper proposes SyncVIS, an approach for Video Instance Segmentation (VIS), which tries to jointly model frame-level and video-level embeddings thus can capture both semantics and movement of instances. The new architecture design is intuitive and generic enough to be applied to various VIS models. Experiments are done on Youtube-VIS 2019, 2021, 2022, and OVIS which show that SyncVIS achieves state-of-the-art results on these benchmarks. Ablations are thorough and enough to understand the proposed approach. Written presentation is clear and easy to read.

Strengths

- The newly proposed architecture design is intuitive and effective. - The new approach can be applied to most of existing VIS architectures and gives consistent improvements. - SyncVIS achieves state-of-the-art performance on current benchmarks. - Ablation experiments are solid and provide good insights about the newly proposed method.

Weaknesses

- The paper may be benefit from having from qualitative results to highly what SyncVIS can improve from the base models which were trained on asynchronous manner in the main text if space allows.

Questions

- Why the results of Mask2Former-VIS on OVIS dataset in Table 9 are too low?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The author(s) have some statements about their method's limitation and further provide more details in supplementary.