Dear Reviewer vGHj,
Thank you very much for your positive feedback and valuable suggestions! We are glad to hear that our rebuttal has addressed some of your concerns.
Regarding the two points you mentioned:
- We would like to clarify that "One-Token-Seg-All" emphasizes the capacity of using one single token to segment **multiple video frames**, rather than **multiple objects**. We also do not claim the ability to handle multiple objects in our paper.
- As we discussed in the rebuttal, single-object referring/reasoning segmentation is a standard and widely adapted setting. Since our main focus is video data, rather than single versus multi-object, we believe this standard setting is suitable and effective to assess our method.
- We greatly thank the reviewer for pointing out this potential misleading expression in the name of the approach. We will consider revise the name and add more clarification in the revised paper.
- In the comparison between VideoLISA and PixelLM on video object segmentation benchmarks, VideoLISA shows better performance than PixelLM. The performance gap mainly comes from **the model's design on processing videos**. We provide the analysis below and will add this in our revised paper.
- PixelLM focuses on multi-object in **images** but has little design for videos. When applying such image-based models on videos, they present two challenges: 1) cannot understand video temporal dynamics, making it unable to handle temporal-related text queries, such as actions, state changes, etc. 2) cannot output temporal-consistent masks. The two issues make image-based models, including LISA and PixelLM, struggle to properly process video data.
- Assume that even we equip the image-based model with a "temporal association module", it is still hindered by the first challenge on video understanding. **This argument has been validated by experiments in our paper**. In Table 6 of the paper and Table 9 of the appendix, we tried to equip LISA with a "temporal association module" XMem. It still shows significant worse performance than VideoLISA.
- In contrast, our VideoLISA is dedicatedly designed for videos. Rather than a "temporal association module", our VideoLISA actually has **two novel designs for videos**, including a Sparse Dense Sampling strategy and a One-Token-Seg-All approach. The design motivation and rationale have been discussed in the rebuttal. The effectiveness has been validated by extensive experiments in our paper. Corresponding to the two challenges stated above, both of the two modules are essential. The Sparse Dense Sampling makes the model be aware of both spatial and temporal information in videos while the One-Token-Seg-All enables temporal consistent segmentation.
- From the functionality perspective, PixelLM addresses image understanding, language reasoning; previous video segmentation models may mainly address image (frame) understanding and video temporal association. **Our VideoLISA is the first model that systematically integrates all these features**, and this is non-trivial to achieve (not simply stacking these existing models together). As mentioned above, the experiment in Table 9 of the appendix also demonstrates that simply stacking these existing models only yields sub-optimal results.
We greatly thank the reviewer for the feedback and constructive questions, which are very helpful in improving our work. We sincerely hope this response answers your questions. Any questions/comments are warmly welcomed!