Summary
This paper presents a novel spatial-semantic block that effectively integrates semantic information with spatial features, resulting in a more comprehensive representation of target objects, especially those with complex or distinct parts. By utilizing a pre-trained Vision Transformer (ViT) backbone without the need to retrain all parameters, the proposed method significantly enhances the efficiency of video object segmentation (VOS).
Additionally, the development of a discriminative query mechanism marks a substantial advancement in the field. This mechanism prioritizes the most representative regions of target objects, thereby improving the reliability of target representation and query updates. This is particularly advantageous in long-term video scenarios, where appearance changes and occlusions can lead to noise accumulation during query propagation.
The authors also highlight the importance of learning comprehensive target features that encompass semantic, spatial, and discriminative information. This holistic approach effectively addresses challenges related to appearance variations and identity confusion among similar-looking objects in long-term videos, making it a valuable contribution to the VOS community.
Finally, extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance across multiple benchmark datasets, including DAVIS 2017, YouTube VOS 2019, MOSE, and LVOS.
Strengths
This paper presents a spatial-semantic modeling method and a discriminative query mechanism that significantly enhance the model's performance. Extensive experiments have been conducted to demonstrate the effectiveness of the model, and several visual examples are provided to clearly illustrate the results at different processing stages. Additionally, the final results showcase the model's considerable potential.
Weaknesses
Writing Style:
1. The writing language is not concise enough, with many long sentences that significantly reduce readability. This is particularly evident in the introduction, such as on the second page: "We construct a Spatial-Semantic Block comprising a semantic embedding module and a spatial dependencies modeling module to efficiently leverage the semantic information and local details of the pre-trained ViTs for VOS without training all the parameters of the ViT backbone."
Image Details:
1. In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.
Method:
1. In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?
2. If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.
3. The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?
4. In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.
5. Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?
6. The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.
Questions
1.In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.
2.In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?
3.If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.
4.The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?
5.In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.
6.Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?
7.The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.